List:General Discussion« Previous MessageNext Message »
From:Tim Gustafson Date:October 16 2012 6:18pm
Subject:Odd Behavior During Replication Start-Up
View as plain text  
Hi,

I have had a MySQL replication pair going for a while now, and we
recently had some hardware issues on the slave.  We've fixed the
hardware issues on the slave, and now I went to re-start replication.
I've done this probably 100 times, but for some reason I just cannot
get this one to go, and I suspect that something may be wrong with my
InnoDB databases and tables.  Here's the steps I've taken:

On the functioning master server, I did:

slave stop;
reset master;
flush tables with read lock;
show master status;

And I copied and pasted the master status to a text file for later.
Then, in a separate window, I created a ZFS snapshot, like this:

zfs snapshot tank/root/var@2012-10-16

And then once that was complete, back on the master server I did:

unlock tables

Then I rsync'd the data from the snapshot to the slave server:

rsync --progress --recursive --delete --times --perms --owner --group
--links --hard-links
root@db-01:/.zfs/snapshot/2012-10-16/var/db/mysql/ /var/db/mysql/

But, when I started up MySQL, I get the following in my error log:

121016 10:40:20 mysqld_safe Starting mysqld daemon with databases from
/var/db/mysql
121016 10:40:20 InnoDB: The InnoDB memory heap is disabled
121016 10:40:20 InnoDB: Mutexes and rw_locks use GCC atomic builtins
121016 10:40:20 InnoDB: Compressed tables use zlib 1.2.5
121016 10:40:20 InnoDB: Initializing buffer pool, size = 256.0M
121016 10:40:20 InnoDB: Completed initialization of buffer pool
121016 10:40:20 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
121016 10:40:20  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Error: trying to add tablespace 1182 of name './foo/TECH.ibd'
InnoDB: to the tablespace memory cache, but tablespace
InnoDB: 1182 of name './bar/patches.ibd' already exists in the tablespace
InnoDB: memory cache!
121016 10:40:38 mysqld_safe mysqld from pid file
/var/db/mysql/db-07.soe.ucsc.edu.pid ended

So, I went back to the master server, backed up the "foo" database and
dropped and re-created it, and then restored the data, and repeated
the whole process, but then I just get the same error for another pair
of database names.  I did this three times before giving up.  No data
appears to be corrupted at all on the master server.

The master is running 5.5.25 on FreeBSD 8.2p4, and the slave is
running 5.5.27 on FreeBSD 9.0p4.

So, am I doing something wrong here?  It seems like something is
broken with InnoDB, but I can't be sure.  I'm doing an "optimize
table" on all my tables across all my databases now to make sure
there's no weird corruption going on, but I'm not holding my breath
with regard to if that will fix anything.

-- 

Tim Gustafson
tjg@stripped
831-459-5354
Baskin Engineering, Room 313A
Thread
Odd Behavior During Replication Start-UpTim Gustafson16 Oct
  • Re: Odd Behavior During Replication Start-UpReindl Harald16 Oct
    • Re: Odd Behavior During Replication Start-UpMichael Dykman16 Oct
Re: Odd Behavior During Replication Start-UpTim Gustafson16 Oct
  • Re: Odd Behavior During Replication Start-UpMichael Dykman16 Oct
    • Re: Odd Behavior During Replication Start-UpTim Gustafson16 Oct
      • Re: Odd Behavior During Replication Start-Upspameden16 Oct
    • Re: Odd Behavior During Replication Start-UpReindl Harald16 Oct
      • Re: Odd Behavior During Replication Start-UpJohan De Meersman17 Oct
        • Re: Odd Behavior During Replication Start-UpReindl Harald17 Oct
          • Re: Odd Behavior During Replication Start-UpJohan De Meersman17 Oct
            • Re: Odd Behavior During Replication Start-UpReindl Harald17 Oct
              • Re: Odd Behavior During Replication Start-UpJohan De Meersman17 Oct
              • RE: Odd Behavior During Replication Start-UpRick James17 Oct
                • Re: Odd Behavior During Replication Start-UpReindl Harald17 Oct
                  • Re: Odd Behavior During Replication Start-UpTim Gustafson17 Oct
  • Re: Odd Behavior During Replication Start-Upspameden16 Oct
    • Re: Odd Behavior During Replication Start-UpShawn Green16 Oct