List:Replication« Previous MessageNext Message »
From:Joe Hammerman Date:June 29 2009 6:19pm
Subject:Corruption in Master - Master server setup
View as plain text  
Hello MySQL users list,

                I have an issue, and I can't seem to turn up any information relating to
its resolution; I was hoping someone on the replication list would be kind of enough to
shed some light on the possible causes of the problem I am encountering.  We have two
servers performing Master - Master replication (theoretically) on a pair of CentOS 5.2
servers. The my.cnf files define different server-id's and auto-increment-offset values.
They fell out of sync, and I was asked to restore them. A brief overview of the steps I
took follows:

1)Obtain read lock on Master 1, and get the log pos and log file.
2)Snapshot the datadir on Master 1 with LVM snapshot.
3)tar cvzf  - ./* | scp xxx@yyy:/path/to/tarball.
4)On Master 2 unpack tarball, start mysql with -skip-slave-start.
5)Run CHANGE MASTER TO with the log pos and log file obtained in 1), above.
6)START SLAVE on Master 2
7)After Master 2 has caught up, STOP SLAVE on MASTER 2, and get MASTER LOG POS and LOG
FILE from Master 2. I use this data to START SLAVE on Master 1.

Everything seems great for a while, but after some time, we always see unrecoverable data
corruption on Master 2, specifically on the largest table (we have seen corruption on
other tables as well on Master 2). There is only on user created database, and all it's
tables are utilizing the MyISAM engine.

                I have run REPAIR TABLE EXTENDED on this table Master 1; repeating the
above process following this action has the same result.

                Any ideas or troubleshooting steps are highly appreciated! It's possible
that I'm doing something terribly wrong without realizing it.

Regards,
                Joseph Hammerman

Thread
Corruption in Master - Master server setupJoe Hammerman29 Jun