Re: wikipedia down, slashdot covering, mysql mentioned

[Greg Whalin]

Daniel Kasak wrote:
> Greg Whalin wrote:
> 
>> Many data centers do not allow customers to install their own UPS 
>> inside the rack.  I am not sure if this is the case with Wikipedia, 
>> but it is definitely the case at the data center we are hosted in.  I 
>> would love to shove one in after reading the horror stories at 
>> Livejournal and now Wikipedia.
> 
> 
> I see.
> Do the data centers offer their own UPS ( as opposed to the customer 
> setting up one for themselves ), or do they simple ban the use of UPSs 
> outright?
> 
> Either way, if you *need* a UPS, you can't hide behind the excuse "oh 
> yeah but XYZ wouldn't let me". You find a way to make it happen ... or 
> if you can't you at least don't start blaming your software for what is 
> obviously a hardware issue.
> 

Our data center spans two power grids, has a massive battery backup, and 
is backed by two massive generators, making power loss pretty difficult.

However, we once lost power to our entire cage by hooking into too many 
servers into a single circuit and tripping a breaker.  We now have two 
power feeds coming into each rack, and redundant power supplies on each 
db server split between the two circuits.  We also carefully watch our 
power utilization to make sure we don't come close to hitting this limit.

As I understand it, both LJ and Wiki suffered from someone hitting EPO 
switch (emergancy power off used to kill all power for when fire 
fighters have to come through ... a safety to make sure the firemen are 
not electrocuted when putting out fires).  This EPO switch is precisly 
why UPSs are sometimes not allowed in each individual rack.  For safety 
reasons, they need to ensure that all power is off in event of 
fire/rescue crews having to come through.

I am not sure that LJ or Wiki blamed mysql.  In fact, the only mysql 
complaint I saw from LJ was that they had to spend time rebuilding index 
files for their few remaining myisam tables.  This I can easily believe 
as I see myisam indexes get corrupted when I sneeze in their general 
direction.  However, LJ mentioned that all of their innodb tables 
recovered with no problems.  In fact, both Wiki and LJ mentioned drive 
write caching to be the major cause of their problems.

Greg
--
greg@stripped
Meetup.com