Re: wikipedia down, slashdot covering, mysql mentioned

[Greg Whalin]

Rich Lafferty wrote:
> On Thu, Feb 24, 2005 at 09:10:32AM -0500, Greg Whalin <gwhalin@stripped>
> wrote:
> 
>>Exactly.  No ACID database can ensure integerity in such a situation. 
>>Postgres, Oracle, or any other transactional DB would have suffered the 
>>same fate in these two cases (LiveJournal, Wiki).
> 
> 
> FWIW, my understanding of LiveJournal's integrity problem after the
> power outage involved tables they weren't yet able to migrate off of
> MyISAM, and getting replication content in all directions. The
> stuff migrated to InnoDB already came up fine. Their replication
> setup is a bit complex -- see
> http://www.danga.com/words/2004_mysqlcon/mysql-slides.pdf for an
> overview.
> 
> (My kit at the same facility was hit too, and recovered correctly, for
> what it's worth. :-)
> 
> At least this mailing list has progressed beyond "Why didn't they have
> a UPS?", I suppose. :-)
> 
>   -Rich
> 

They had most of their dbs transitioned to innodb, but even some of 
those came up corrupted due to write caching being enabled on individual 
drives.  Their myisam tables simply needed indexes rebuilt (which is a 
pain in the butt and takes forever).  Their complete story: 
http://www.livejournal.com/community/lj_dev/670215.html

Needless to say, between LJ and Wiki, I am fairly paranoid about db 
corruption now (though our writes are considerably less than either of 
those two places as we are 99% reads ... our reads are considerably less 
than either of those sites as well :) )

Greg