From: Rick James
Date: October 4 2012 11:19pm
Subject: RE: InnoDB corrupt after power failure
List-Archive: http://lists.mysql.com/mysql/228333
Message-Id: <2E7DD7ADE53B044C8C8BCD9C5829E1EB148CF92369@SP2-EX07VS01.ds.corp.yahoo.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

On Linux, XFS is preferred.  Noop or Deadline, not CFQ is preferred.
I don't know if any of this has any impact on the crash you describe.

I am quite suspicious of VMs.

> -----Original Message-----
> From: Andrew Miklas [mailto:andrew@stripped]
> Sent: Thursday, October 04, 2012 3:21 PM
> To: Rick James
> Cc: Manuel Arostegui; mysql@stripped
> Subject: Re: InnoDB corrupt after power failure
>=20
> Hi Rick,
>=20
> On Oct 4, 2012, at 2:40 PM, Rick James wrote:
>=20
> > I hope you turned OFF caching on the drives, themselves.  The BBU
> should be the single place that caches and is trusted to survive a
> power outage.
>=20
> The DB server in question is running in a virtualized environment, so
> the array shows up as a SCSI device inside our VM.  I can't use hdparm
> to directly check whether the disks are doing write caching, but our
> hosting provider assures us that once data is sent to the virtual SCSI
> device from inside the VM, it will be persisted to disk even if there's
> a power failure.
>=20
> I'm a bit suspicious of a recent change we did to switch our ext3
> journals from data=3Dordered to data=3Dwriteback.  The ext3 docs say "a
> crash+recovery can cause incorrect data to appear in files which were
> written shortly before the crash" [1].  As a result, if a tablespace
> were extended just before the power failure, it might be possible that
> when MySQL restarts, it will see random data at the end of the
> tablespace.  It seems like this could happen even if the disks are BBU
> / not write caching, because the increase of the ibd's file size in the
> inode and the zeroing out of the new blocks assigned to the file are
> not atomic with respect to one another.
>=20
> Is the InnoDB recovery process OK with this scenario?  Has anyone else
> seen corruption problems with data=3Dwriteback?
>=20
>=20
> -- Andrew
>=20
>=20
> [1] http://lxr.linux.no/linux+v3.5.2/Documentation/filesystems/ext3.txt