From: Andrew Miklas
Date: October 4 2012 10:21pm
Subject: Re: InnoDB corrupt after power failure
List-Archive: http://lists.mysql.com/mysql/228331
Message-Id: <0B7E0830-14AC-4C43-BF54-074B112EFF75@pagerduty.com>
MIME-Version: 1.0 (Apple Message framework v1278)
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable

Hi Rick,

On Oct 4, 2012, at 2:40 PM, Rick James wrote:

> I hope you turned OFF caching on the drives, themselves.  The BBU =
should be the single place that caches and is trusted to survive a power =
outage.

The DB server in question is running in a virtualized environment, so =
the array shows up as a SCSI device inside our VM.  I can't use hdparm =
to directly check whether the disks are doing write caching, but our =
hosting provider assures us that once data is sent to the virtual SCSI =
device from inside the VM, it will be persisted to disk even if there's =
a power failure.

I'm a bit suspicious of a recent change we did to switch our ext3 =
journals from data=3Dordered to data=3Dwriteback.  The ext3 docs say "a =
crash+recovery can cause incorrect data to appear in files which were =
written shortly before the crash" [1].  As a result, if a tablespace =
were extended just before the power failure, it might be possible that =
when MySQL restarts, it will see random data at the end of the =
tablespace.  It seems like this could happen even if the disks are BBU / =
not write caching, because the increase of the ibd's file size in the =
inode and the zeroing out of the new blocks assigned to the file are not =
atomic with respect to one another.

Is the InnoDB recovery process OK with this scenario?  Has anyone else =
seen corruption problems with data=3Dwriteback?


-- Andrew


[1] http://lxr.linux.no/linux+v3.5.2/Documentation/filesystems/ext3.txt=