List:General Discussion« Previous MessageNext Message »
From:Maurice Volaski Date:September 17 2007 7:32pm
Subject:Re: Could drbd randomly flip bits? Was:
Database page corruption on disk occurring during mysqldump on a fresh
View as plain text  
>Hi Maurice,
>
>If you're running into corruption both in ext3 metadata and in MySQL 
>data, it is certainly not he fault of MySQL as you're likely aware.

I am hoping they are not related. The problems with MySQL surfaced 
almost immediately after upgrading to 5.0.x.


>[details deleted]
>
>You can see that there are in fact many bits flipped in each.  I 
>would suspect higher-level corruption than

I initially thought this as well, but the explanation on the ext3 
mailing list is that it really is just a lone flipped bit in both 
instances. The other differences are due to fsck padding out the 
block when it guesses what the correct size is.

>Do note that data on e.g. the PCI bus is not protected by any sort 
>of checksum.  I've seen this cause corruption problems with PCI 
>risers and RAID cards.  Are you using a PCI riser card?  Note that 
>LSI does *not* certify their cards to be used on risers if you are 
>custom building a machine.
>

Yes, there is a riser card. Wouldn't this imply that LSI is saying 
you can't use a 1U or a 2U box?

It's kind of scary there is no end-to-end parity implemented 
somewhere along the whole data path to prevent this. It sort of 
defeats the point of RAID 6 and ECC.

How did you determine this was the cause?


>Do you mean a Serially-Attached SCSI aka SAS controller, I assume?

No, it's SATA to SCSI.

>Is this a custom build machine or a vendor integrated one?

It is custom-built.


>
>Maurice Volaski wrote:
>>In using drbd 8.0.5 recently, I have come across at least two 
>>instances where a bit on disk apparently flipped spontaneously in 
>>the ext3 metadata on volumes running on top of drbd.
>>
>>Also, I have been seeing regular corruption of a mysql database, 
>>which runs on top of drbd, and when I reported this as a bug since 
>>I also recently upgraded mysql versions, they question whether drbd 
>>could be responsible!
>>
>>All the volumes have been fscked recently and there were no 
>>reported errors. And, of course, there have been no errors reported 
>>from the underlying hardware.
>>
>>I have since upgraded to 8.0.6, but it's too early to say whether 
>>there is a change.
>>
>>I'm also seeing the backup server complain of not being files not 
>>comparing, though this may be a separate problem on the backup 
>>server.
>>
>>
>>
>>The ext-3  bit flipping:
>>At 12:00 PM -0400 9/11/07, ext3-users-request@stripped wrote:
>>>I have come across two files, essentially untouched in years, on two
>>>different ext3 filesystems on the same server, Gentoo AMD 64-bit with
>>>kernel 2.6.22 and fsck version 1.40.2 currently, spontaneously
>>>becoming supremely large:
>>>
>>>Filesystem one
>>>Inode 16257874, i_size is 18014398562775391, should be 53297152
>>>
>>>Filesystem two
>>>Inode 2121855, i_size is 35184386120704, should be 14032896.
>>>
>>>Both were discovered during an ordinary backup operation (via EMC
>>>Insiginia's Retrospect Linux client).
>>>
>>>The backup runs daily and so one day, one file must have grew
>>>spontaneously to this size and then on another day, it happened to
>>>the second file, which is on a second filesystem. The backup attempt
>>>generated repeated errors:
>>>
>>>EXT3-fs warning (device dm-2): ext3_block_to_path: block > big
>>>
>>>Both filesystems are running on different logical volumes, but
>>>underlying that is are drbd network raid devices and underlying that
>>>is a RAID 6-based SATA disk array.
>>
>>
>>
>>The answer to the bug report regarding mysql data corruption, who 
>>is blaming drbd!
>>>http://bugs.mysql.com/?id=31038
>>>
>>>  Updated by:  Heikki Tuuri
>>>  Reported by: Maurice Volaski
>>>  Category:    Server: InnoDB
>>>  Severity:    S2 (Serious)
>>>  Status:      Open
>>>  Version:     5.0.48
>>>  OS:          Linux
>>>  OS Details:  Gentoo
>>>  Tags:        database page corruption locking up corrupt doublewrite
>>>
>>>[17 Sep 18:49] Heikki Tuuri
>>>
>>>Maurice, my first guess is to suspect the RAID-1 driver.
>>
>>
>>My initial report of mysql data corruption:
>>>>A 64-bit Gentoo Linux box had just been upgraded from MySQL 4.1 
>>>>to5.0.44 fresh (by dumping in 4.1 and restoring in 5.0.44) and 
>>>>almostimmediately after that, during which time the database was 
>>>>not used,a crash occurred during a scripted mysqldump. So I 
>>>>restored and dayslater, it happened again. The crash details seem 
>>>>to be trying tosuggest some other aspect of the operating system, 
>>>>even the memoryor disk is flipping a bit. Or could I be running 
>>>>into a bug in thisversion of MySQL?
>>>>
>>>>Here's the output of the crash
>>>>-----------------------------------
>>>>InnoDB: Database page corruption on disk or a failed
>>>>InnoDB: file read of page 533.
>>>>InnoDB: You may have to recover from a backup.
>>>>070827  3:10:04  InnoDB: Page dump in ascii and hex (16384 bytes):
>>>>  len 16384; hex
>>>>
>>>>[dump itself deleted 
>>>>forbrevity]                                                               
>                                            
>>>>
>>>>
>>>> 
>>>>                                                                          
>                                                                                           
>                            
>>>>  ;InnoDB: End of page dump
>>>>070827  3:10:04  InnoDB: Page checksum 
>>>>646563254,prior-to-4.0.14-form checksum 2415947328
>>>>InnoDB: stored checksum 4187530870, prior-to-4.0.14-form 
>>>>storedchecksum 2415947328
>>>>InnoDB: Page lsn 0 4409041, low 4 bytes of lsn at page end 4409041
>>>>InnoDB: Page number (if stored to page already) 533,
>>>>InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
>>>>InnoDB: Page may be an index page where index id is 0 35
>>>>InnoDB: (index PRIMARY of table elegance/image)
>>>>InnoDB: Database page corruption on disk or a failed
>>>>InnoDB: file read of page 533.
>>>>InnoDB: You may have to recover from a backup.
>>>>InnoDB: It is also possible that your operating
>>>>InnoDB: system has corrupted its own file cache
>>>>InnoDB: and rebooting your computer removes the
>>>>InnoDB: error.
>>>>InnoDB: If the corrupt page is an index page
>>>>InnoDB: you can also try to fix the corruption
>>>>InnoDB: by dumping, dropping, and reimporting
>>>>InnoDB: the corrupt table. You can use CHECK
>>>>InnoDB: TABLE to scan your table for corruption.
>>>>InnoDB: See also 
>>>>InnoDB:http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
>>>>InnoDB: about forcing recovery.
>>>InnoDB: Ending processing because of a corrupt database page.
>>
>
>--
>high performance mysql consulting
>www.provenscaling.com


-- 

Maurice Volaski, mvolaski@stripped
Computing Support, Rose F. Kennedy Center
Albert Einstein College of Medicine of Yeshiva University
Thread
Could drbd randomly flip bits? Was: Databasepage corruption on disk occurring during mysqldump on a fresh databaseand Was: Spontan...Maurice Volaski17 Sep
  • Re: Could drbd randomly flip bits? Was: Databasepage corruption on disk occurring during mysqldump on a fresh database andWas: Spo...Jeremy Cole17 Sep
    • Re: Could drbd randomly flip bits? Was:Database page corruption on disk occurring during mysqldump on a freshdatabase and Was: S...Maurice Volaski17 Sep
      • Re: Could drbd randomly flip bits? Was: Databasepage corruption on disk occurring during mysqldump on a fresh database andWas: Spo...Jeremy Cole17 Sep
        • Re: Could drbd randomly flip bits? Was:Database page corruption on disk occurring during mysqldump on a freshdatabase and Was:...Maurice Volaski18 Sep
Re: Could drbd randomly flip bits? Was:Database page corruption on disk occurring during mysqldump on a freshdatabase and Was: Spo...Maurice Volaski17 Sep
  • Re: Could drbd randomly flip bits? Was: Databasepage corruption on disk occurring during mysqldump on a fresh database andWas: Spo...Jeremy Cole17 Sep
Re: Could drbd randomly flip bits? Was:Database page corruption on disk occurring during mysqldump on a freshdatabase and Was:...Maurice Volaski18 Sep
Re: Could drbd randomly flip bits? Was: Database page corruption on disk occurring during mysqldump on a fresh database and Was: Spontaneous developme...Stephen Samuel18 Sep