UPDATE: My hosted sites down for 30 minutes tonight for RAM update.

[MRL17]

I would guess this is all BELCO hosted sites, but I am only in the BFL and USFL.

The sites are down and citing database errors.

I wasn't sure where else to post this, but I have been trying to access the BFL site since 10pm EST to upload my export. I have had no luck for two hours.

[PhinsFan]

I would guess this is all BELCO hosted sites, but I am only in the BFL and USFL.

The sites are down and citing database errors.

I wasn't sure where else to post this, but I have been trying to access the BFL site since 10pm EST to upload my export. I have had no luck for two hours.

Same here for the BFL and the CCFL, guess we just wait.

[Bako]

Yup....unable to upload to the CCFL site also.

[Ben E Lou]

Awake here, and everything seems ok now. Looking at the logs, I'm seeing something very strange, and what almost *has* to be the culprit. I get an automated email every time there's a database error. I started getting them at 11:34am, and the last one came at 1:03am. What's unusual is that I can't see the site access logs back beyond 11:55am, because the same IP Address hit IHOF's Conscriptor over 900 times in the 1 hour and 8 minutes from 11:55-1:03. When I see something like this, I normally assume "bot." But I check before I ban an IP, and this time, it doesn't appear to be a bot. It's a legit IP address of someone I trust who is a member of IHOF. The real kicker here is that the most recent IHOF draft ended two weeks ago. It's almost as if he went in to take a look at something from the draft and fell asleep at his keyboard with his head on the refresh key. :p All I can think of is either that or some virus/malware from his machine, or someone/something spoofing his IP. Point being, that IP stopped hitting Conscriptor every 4ish seconds at 1:03, and the last db error was also at 1:03. The two seem like they have to be related. I'll check in with my hosting company, but seeing this usage log, I really think it was caused by the Conscriptor stuff from that one IP. *shurg*

As best as I can tell, everything looks fine now, and I didn't do anything...

<table border="0" cellpadding="0" cellspacing="0" width="100%"><tbody><tr><td valign="top" width="50%"><table class="summaryTitle" border="0" cellpadding="0" cellspacing="0" width="100%"><tbody><tr><td class="qE">CPU</td></tr></tbody></table> <table border="0" cellpadding="0" cellspacing="0" width="100%"> <tbody><tr> <td class="q7 label" nowrap="nowrap" width="180">CPU Usage</td> <td class="q7_" valign="top" width="16">https://108.59.255.76:4643/vz/skins/winxp.silver/icons/green_16.gif</td> <td class="q7_" nowrap="nowrap"><table border="0" cellpadding="0" cellspacing="0" width="170"><tbody><tr> <td class="qD">https://108.59.255.76:4643/vz/skins/winxp.silver/images/m1.gif</td> <td nowrap="nowrap" width="100%" align="left">8.1%</td> </tr></tbody></table></td> </tr> <tr> <td class="q8 label" nowrap="nowrap" width="180">CPU Load Average</td> <td class="q8_" valign="top" width="16">https://108.59.255.76:4643/vz/skins/winxp.silver/icons/green_16.gif</td> <td class="q8_" nowrap="nowrap">0.49, 0.7, 1.09</td> </tr> </tbody></table>

<table class="summaryTitle" border="0" cellpadding="0" cellspacing="0" width="100%"><tbody><tr><td class="qE">Memory Usage</td></tr></tbody></table> <table class="sTable" border="0" cellpadding="0" cellspacing="0" width="100%"> <tbody><tr> <th class="qA label" width="150" align="left">Resource</th> <th class="qB label" width="105" align="right">
</th> <th class="qB label" width="41" align="right">Used</th> <th class="qA label" width="104" align="left">
</th> <th class="qA label" width="100" align="right">Free</th> <th class="qB label" width="100" align="right">Total</th> </tr> <tr class="sEvenRow"> <td class="qA" nowrap="nowrap" align="left"> https://108.59.255.76:4643/vz/skins/winxp.silver/icons/green_16.gifMemory</td> <td class="qB" nowrap="nowrap" align="right">470.1 MB</td> <td class="qB" nowrap="nowrap" align="right">https://108.59.255.76:4643/vz/skins/winxp.silver/images/m1.gif</td> <td class="qA" nowrap="nowrap" align="left">7.7%</td> <td class="qA" nowrap="nowrap" align="right">5.5 GB</td> <td class="qB" nowrap="nowrap" align="right">6 GB</td> </tr> </tbody></table>

<table class="summaryTitle" border="0" cellpadding="0" cellspacing="0" width="100%"><tbody><tr><td class="qE">Disk Usage</td></tr></tbody></table> <table class="sTable" border="0" cellpadding="0" cellspacing="0" width="100%"> <tbody><tr> <th class="qA label" width="150" align="left">Resource</th> <th class="qB label" width="105" align="right">
</th> <th class="qB label" width="41" align="right">Used</th> <th class="qA label" width="104" align="left">
</th> <th class="qA label" width="100" align="right">Free</th> <th class="qB label" width="100" align="right">Total</th> </tr> <tr class="sEvenRow"> <td class="qA" nowrap="nowrap" align="left"> https://108.59.255.76:4643/vz/skins/winxp.silver/icons/green_16.gifDisk Space</td> <td class="qB" nowrap="nowrap" align="right">42.89 GB</td> <td class="qB" nowrap="nowrap" align="right">https://108.59.255.76:4643/vz/skins/winxp.silver/images/m4.gif</td> <td class="qA" nowrap="nowrap" align="left">35.7%</td> <td class="qA" nowrap="nowrap" align="right">77.11 GB</td> <td class="qB" nowrap="nowrap" align="right">120.00 GB</td> </tr> <tr class="sOddRow"> <td class="qA" nowrap="nowrap" align="left"> https://108.59.255.76:4643/vz/skins/winxp.silver/icons/green_16.gifDisk Inodes</td> <td class="qB" nowrap="nowrap" align="right">258,242</td> <td class="qB" nowrap="nowrap" align="right">https://108.59.255.76:4643/vz/skins/winxp.silver/images/m1.gif</td> <td class="qA" nowrap="nowrap" align="left">12.9%</td> <td class="qA" nowrap="nowrap" align="right">1,741,758</td> <td class="qB" nowrap="nowrap" align="right">2,000,000</td> </tr> </tbody></table>

</td> <td>https://108.59.255.76:4643/vz/skins/winxp.silver/images/1x1.gif</td> <td valign="top" width="50%">
</td></tr></tbody></table>

[Ben E Lou]

Hmmmm..and now the site has gone down...wtih no db errors. Checking...

[Ben E Lou]

Server is rebooting, but I don't know why. On phone with tech support to see if this is just normal, or related to whatever happened last night.

[Ben E Lou]

Sounds like a big server outage in their Rochester datacenter, affecting all VPS hosting. They say they've got their entire operations team working on it, however many people that is. *shurg*

[Yoda]

http://4.bp.blogspot.com/-1u41FHusvw8/ToVpcujNEQI/AAAAAAAAMjo/tTyAjZvjW6E/s1600/monkeys_with_computers%255B1%255D.jpg

[BreizhManu]

Sounds like a big server outage in their Rochester datacenter, affecting all VPS hosting. They say they've got their entire operations team working on it, however many people that is. *shurg*

Usually 5-6 people ;)

[Kozure]

I would guess this is all BELCO hosted sites, but I am only in the BFL and USFL.

The sites are down and citing database errors.

I wasn't sure where else to post this, but I have been trying to access the BFL site since 10pm EST to upload my export. I have had no luck for two hours.

Hmmmm. This post was made at midnight and mike failed to show up for work this morning. Seems like we have a premeditated call off. Awwww, I'm telling!!

[Steel]

I'm still having trouble accessing all the sites. They will work for a bit, then go back down.

[Ben E Lou]

I'm still having trouble accessing all the sites. They will work for a bit, then go back down.I'm not "officially" back up yet. The hosting company is making some changes and needing to reboot from time to time, and I initiated one server reboot myself this afternoon because I did a php upgrade. Based on the last email I got from them, I'd expect more stability by this evening.

[aston217]

Moral of the story: don't fall asleep on that refresh key? :p

[Ben E Lou]

Moral of the story: don't fall asleep on that refresh key? :pHeh.

I suspect that what the IHOF member experienced was a symptom of a bigger problem, although we all know that Conscriptor can be a massive resource hog, so who knows?

I haven't received any emails lately from the company, but my server logs show that things have been running smoothly for a while now. I am expecting an "all clear" email or status blog update from them at some point, though.

[Ben E Lou]

Please be aware that our Operations team will be performing maintenance on one of our Virtuozzo Linux servers, specifically RVZLIN27, which is located at our Rochester data center. The maintenance will replace RAM on this particular server to optimize its stability and speed.


This maintenance will occur tonight – Thursday, January 26, 2012 - at 11:00 p.m. EST. You will be unable to access your server during the time the maintenance is being performed; the maintenance is expected to last, at maximum, approximately 30 minutes.


Thank you for your attention to this notice!.