List:Replication« Previous MessageNext Message »
From:Johan De Meersman Date:July 17 2013 8:29am
Subject:Re: 2 slaves - one Master; continues to fail
View as plain text  
If pt-table-checksum tells you everything is in sync, and you don't see any
non-deterministic updates in the master's error log; and you still get replication
errors, then I would very much assume someone or something is connecting to your slaves
and changing things there. 

As mentioned before, even when the slave is read_only, any user with super privileges can
still do updates. Try turning on the general log on the slave - keeping in mind that that
takes both a lot of diskspace, and quite a bit of overhead. You should be able to analyse
it and sort by user@host using pt-query-digest - or simply grep for "Connect" :-p 

I can't quite remember if you mentioned this, but do both slaves die on the exact same
statement/master log position? If so, that probably means the error does come through
replication; if only one dies (or they die at diferent points) that is a fair indicator
of slave updates (not 100%, though, but nondeterministic queries should show up in the
master's log). 

----- Original Message -----

> From: "Jamen McGranahan" <jamen.mcgranahan@stripped>
> To: "Johan De Meersman" <vegivamp@stripped>
> Sent: Tuesday, 16 July, 2013 8:44:13 PM
> Subject: RE: 2 slaves - one Master; continues to fail

> Well, I thought Percona was going to be the answer. Alas, no such
> thing – I tried to do a backup of our running Master MySQL server
> and it brought down all of our other servers because they were
> trying to connect to it and the MySQL server was locked. Ugh… I’m so
> frustrated by this.

> Jamen McGranahan
> Systems Services Librarian
> Vanderbilt University Library

> From: Johan De Meersman [mailto:vegivamp@stripped]
> Sent: Monday, July 15, 2013 9:36 AM
> To: McGranahan, Jamen
> Cc: Replication
> Subject: Re: 2 slaves - one Master; continues to fail

> ----- Original Message -----

> Oh, how I loathe Drupal, but that's not something I have room to get
> into here :-)

> I'm familiar with such problems, but I've found that it's all too
> often caused by developers connecting to the wrong server. If you've
> already set the slaves read-only, though, that shouldn't be the
> issue - unless there's already more inconsistencies than you think.
> The Percona Toolkit is your friend, pt-table-checksum will verify
> data consistency and pt-table-sync will use the black arts to resync
> without having to dump/import. Read the manpages carefully, though
> :-)

> --

> Unhappiness is discouraged and will be corrected with kitten
> pictures.
-- 

Unhappiness is discouraged and will be corrected with kitten pictures. 

Thread
2 slaves - one Master; continues to failJamen McGranahan14 Jul
  • Re: 2 slaves - one Master; continues to failSuresh Kuna14 Jul
    • RE: 2 slaves - one Master; continues to failJamen McGranahan14 Jul
      • Re: 2 slaves - one Master; continues to failSuresh Kuna14 Jul
  • RE: 2 slaves - one Master; continues to failJamen McGranahan15 Jul
    • RE: 2 slaves - one Master; continues to failJohan De Meersman15 Jul
    • Re: 2 slaves - one Master; continues to failTruls Bergskaug15 Jul
  • Re: 2 slaves - one Master; continues to failJesper Wisborg Krogh15 Jul
    • Re: 2 slaves - one Master; continues to failDivesh Kamra16 Jul
Re: 2 slaves - one Master; continues to failJohan De Meersman15 Jul
  • RE: 2 slaves - one Master; continues to failJamen McGranahan15 Jul
    • Re: 2 slaves - one Master; continues to failJohan De Meersman15 Jul
      • Re: 2 slaves - one Master; continues to failSuresh Kuna15 Jul
        • RE: 2 slaves - one Master; continues to failJamen McGranahan15 Jul
          • Re: 2 slaves - one Master; continues to failSuresh Kuna15 Jul
            • RE: 2 slaves - one Master; continues to failJamen McGranahan15 Jul
              • Re: 2 slaves - one Master; continues to failSuresh Kuna15 Jul
Re: 2 slaves - one Master; continues to failJohan De Meersman17 Jul
Re: 2 slaves - one Master; continues to failJohan De Meersman18 Jul
  • RE: 2 slaves - one Master; continues to failJamen McGranahan18 Jul
    • Re: 2 slaves - one Master; continues to failMarcus Bointon18 Jul
      • RE: 2 slaves - one Master; continues to failJamen McGranahan18 Jul
        • Re: 2 slaves - one Master; continues to failSuresh Kuna18 Jul
          • RE: 2 slaves - one Master; continues to failJamen McGranahan18 Jul
        • Re: 2 slaves - one Master; continues to failMarcus Bointon18 Jul
          • RE: 2 slaves - one Master; continues to failJamen McGranahan18 Jul
            • Re: 2 slaves - one Master; continues to failMarcus Bointon18 Jul
              • RE: 2 slaves - one Master; continues to failJamen McGranahan18 Jul
              • RE: 2 slaves - one Master; continues to failJamen McGranahan22 Jul
                • Re: 2 slaves - one Master; continues to failSuresh Kuna22 Jul