List:General Discussion« Previous MessageNext Message »
From:Andrew Miklas Date:October 4 2012 10:21pm
Subject:Re: InnoDB corrupt after power failure
View as plain text  
Hi Rick,

On Oct 4, 2012, at 2:40 PM, Rick James wrote:

> I hope you turned OFF caching on the drives, themselves.  The BBU should be the
> single place that caches and is trusted to survive a power outage.

The DB server in question is running in a virtualized environment, so the array shows up
as a SCSI device inside our VM.  I can't use hdparm to directly check whether the disks
are doing write caching, but our hosting provider assures us that once data is sent to
the virtual SCSI device from inside the VM, it will be persisted to disk even if there's
a power failure.

I'm a bit suspicious of a recent change we did to switch our ext3 journals from
data=ordered to data=writeback.  The ext3 docs say "a crash+recovery can cause incorrect
data to appear in files which were written shortly before the crash" [1].  As a result,
if a tablespace were extended just before the power failure, it might be possible that
when MySQL restarts, it will see random data at the end of the tablespace.  It seems like
this could happen even if the disks are BBU / not write caching, because the increase of
the ibd's file size in the inode and the zeroing out of the new blocks assigned to the
file are not atomic with respect to one another.

Is the InnoDB recovery process OK with this scenario?  Has anyone else seen corruption
problems with data=writeback?


-- Andrew


[1] http://lxr.linux.no/linux+v3.5.2/Documentation/filesystems/ext3.txt
Thread
InnoDB corrupt after power failureAndrew Miklas4 Oct
  • Re: InnoDB corrupt after power failureManuel Arostegui4 Oct
    • Re: InnoDB corrupt after power failureAndrew Miklas4 Oct
      • RE: InnoDB corrupt after power failureRick James4 Oct
        • Re: InnoDB corrupt after power failureAndrew Miklas4 Oct
          • RE: InnoDB corrupt after power failureRick James4 Oct