List:General Discussion« Previous MessageNext Message »
From:Alec.Cawley Date:August 21 2003 9:05am
Subject:Re: Grown defects in a RAID array
View as plain text  

> I've notice that when grown defects (bad blocks on the disk caused by
> usage over time) that my dedicated mysql server is adversely effected.

> Since the action of Grown defects does not flush the table with a write
> lock-that the block marked as bad will effect prior to that block
> becoming a grown defect-what is another way for mysql to recover from
> this situation?

> I'm currently using EXT3 does this same effect happen with Reiser-Fs /
> UFS / Veritas?

I assume you are talking about Scsi disks, which I know about. If not,
ignore the following.

Defect handling on Scsi disks is done by the drive itself, at a level below
both the RAID controller, the file system, or MySQL caching. The drive,
once it has noticed a defect, flags it as faulty and substitutes a spare
sector from a stock of hidden sectors it keeps. Usually, when it flags a
defect, the drive has managed to retrieve your data, but with an excessive
level of retries or after having to use its second level, software
implemented ECC instead of the hardware based first level ECC. Therefore,
by the time the drive manages to report a grown defect, the data has been
salvaged and the drive is presenting the appearance of an undamaged disk.
This is independent of filesystem, since the drive is managing to simulate
a perfect unit even though it is defective.

Perfect, that is, in data storage, but not in performance. Defects are
replaced in units of sectors - blocks of 512 bytes. However, most systems
tend to page files in and out in units of several sectors, for (normal)
performance reasons. I don't know what page size MySQL uses, but if it more
than 512 bytes, accessing pages with grown defects will be inefficient. For
example, imagine a page occupies sectors 100-103 - a 2048 byte page. If
sector 102 goes faulty, the drive has to read sectors 100 and 101, seek to
wherever it has put the replacement for 102, then seek back to read 103,.
This effectively trebles access time. (Actually, the drive is cleverer, and
will probably only double access time).

Drives vary dramatically in their efficiencies in handling grow defects. It
is not a parameter that is usually tested in performance tests (nor, I
think, would it be easy to test). Two algorithms that I know had wildly
different results. On family was very efficient for the first defect on any
cylinder, then dropped off dramatically. Another was less efficient for the
first, but could handle several adjacent defects before performance fell
off a cliff. All of them, AFAIK, emphasize data retention rather than
performance - which I think most users would want.

However, I would say that if a disk shows (a) any grown defects in its
first three months of life, or (b) more than three or four grown defects
*ever*, that disk is on its way out, and should be replaced at the first
opportunity. Drive warranties specify a maximum number of grown defects
during the warranty period, and it is usually quite small (ten springs from
memory, but read your warranty), so if you are getting enough problems to
get a significant performance drop, you should be able to exchange your
drive.

      Alec



Thread
Grown defects in a RAID arrayDathan Vance Pattishall21 Aug
  • Re: Grown defects in a RAID arrayDan Nelson21 Aug
Re: Grown defects in a RAID arrayAlec.Cawley21 Aug