From: Date: October 12 2005 2:08am
Subject: RE: Non-linear degradation in bulk loads?
List-Archive: http://lists.mysql.com/mysql/190274
Message-Id: <D927B4E9BC644E4BBEE04C52F1B64898028BA594@smvcmail01.yfdirect.net>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

> Two solutions: 1) sort the rows to be inserted on the key=20
> 'email' before inserting.
>=20
> 2) Or:
>=20
> http://dev.mysql.com/doc/mysql/en/innodb-tuning.html
> "
> If you have UNIQUE constraints on secondary keys, starting from MySQL
> 3.23.52 and 4.0.3, you can speed up table imports by=20
> temporarily turning off the uniqueness checks during the=20
> import session:
> SET UNIQUE_CHECKS=3D0;
>=20
> For big tables, this saves a lot of disk I/O because InnoDB=20
> can use its insert buffer to write secondary index records in a batch.
> "
>=20
> But make sure you do not have any duplicates in the rows!

After sending my mail, I discovered SET UNIQUE_CHECKS=3D0, and =
subsequent
to that it also occurred to me to try putting the data in in sorted
order.  Unfortunately, doing UNIQUE_CHECKS=3D0 did not work, and even =
the
combination of both did not work.  First chunk (3.4m rows) was ~1.5
minutes, second was ~5 minutes...

At this point I'm inclined to believe that there is something very wrong
with the disk subsystem because of this and other problems (doing a
large cp from the datapool filesystem to another filesystem brought the
database to a near-halt, among other things).

As a stop-gap solution, I created the table with no indexes, and loaded
all the data (loaded in linear time), and plan on doing a CREATE UNIQUE
INDEX on the table.  Will this happen in linear time, or near-linear
time?

*sigh*

-JF