List:General Discussion« Previous MessageNext Message »
From:Daevid Vincent Date:December 23 2009 8:51pm
Subject:RE: HELP! "RESET MASTER" hosed replication
View as plain text  
Thank you guys for the tips. 

I've also updated the documentation so others don't run into this:
http://dev.mysql.com/doc/refman/5.0/en/reset.html


> -----Original Message-----
> From: prabhat kumar [mailto:aim.prabhat@stripped] 
> Sent: Wednesday, December 23, 2009 12:46 PM
> To: Carlos Proal
> Cc: mysql@stripped
> Subject: Re: HELP! "RESET MASTER" hosed replication
> 
> *Next time look for 'purge binary logs'!*
> 
> Purge may also cause problem , before doing purge make sure 
> slave is in sync
> with master or at least cross check slave currently which bin 
> file getting
> sync.
> 
> On Thu, Dec 24, 2009 at 2:05 AM, Carlos Proal 
> <carlos.proal@stripped>wrote:
> 
> >
> > The issue is that replication relies on  this logs !!!, so 
> when you deleted
> > them .....
> > Generally speaking you have to:
> > > stop the slave
> > > sync the master with the slave (there are several ways to 
> do this and
> > depending how busy is your master)
> > >grab the master status (position)
> > >change the slave to point to the master new position
> > >start the slave
> >
> > Hope this helps.
> >
> > Carlos
> >
> >
> > On 12/23/2009 2:20 PM, Daevid Vincent wrote:
> >
> >> I got an alert that one of the drives was filling up (3% 
> free). So I
> >> figured out that a large chunk was from /var/log/mysql
> >>
> >> root@pse10:~# find / -type d -print0 | xargs -0 -n1 du -sk 
> | sort -rn |
> >> head -n20>  ~/dir-sizes.txt
> >>
> >> root@pse10:~# cat ~/dir-sizes.txt
> >> 159121012 /
> >> 70442396  /var
> >> 70127764  /var/log
> >> 69991160  /var/log/mysql<-------- big offender
> >> 56307436  /data
> >> 31479936  /home
> >> 29386076  /data/mysql
> >> 26899784  /data/archive
> >>
> >> It looked like the /var/log/mysql was pretty full of these
> >> Mysql-bin.00XXXX
> >> log files,
> >>
> >> ...
> >> -rw-rw---- 1 mysql adm 105019928 2009-12-23 05:07 mysql-bin.001196
> >> -rw-rw---- 1 mysql adm 105004751 2009-12-23 05:08 mysql-bin.001197
> >> -rw-rw---- 1 mysql adm 104978518 2009-12-23 05:10 mysql-bin.001198
> >> -rw-rw---- 1 mysql adm 104949073 2009-12-23 05:11 mysql-bin.001199
> >> -rw-rw---- 1 mysql adm 104925795 2009-12-23 05:13 mysql-bin.001200
> >> -rw-rw---- 1 mysql adm 104974354 2009-12-23 05:14 mysql-bin.001201
> >> -rw-rw---- 1 mysql adm 105089249 2009-12-23 05:16 mysql-bin.001202
> >> -rw-rw---- 1 mysql adm 105165487 2009-12-23 05:17 mysql-bin.001203
> >> -rw-rw---- 1 mysql adm 104926853 2009-12-23 05:19 mysql-bin.001204
> >> -rw-rw---- 1 mysql adm 105139076 2009-12-23 05:20 mysql-bin.001205
> >> -rw-rw---- 1 mysql adm 104891552 2009-12-23 05:22 mysql-bin.001206
> >> -rw-rw---- 1 mysql adm 104959626 2009-12-23 05:25 mysql-bin.001207
> >> -rw-rw---- 1 mysql adm 104883048 2009-12-23 05:27 mysql-bin.001208
> >> -rw-rw---- 1 mysql adm 104993511 2009-12-23 05:28 mysql-bin.001209
> >> -rw-rw---- 1 mysql adm 104945974 2009-12-23 05:30 mysql-bin.001210
> >> -rw-rw---- 1 mysql adm  35468892 2009-12-23 05:30 mysql-bin.001211
> >> -rw-rw---- 1 mysql adm     21728 2009-12-23 05:30 mysql-bin.index
> >> -rw-r----- 1 mysql adm     12836 2009-12-23 00:12 mysql-slow.log
> >> ...
> >>
> >> so I took the liberty of resetting them...
> >> http://dev.mysql.com/doc/refman/5.0/en/reset.html
> >>
> >> vincentd@pse10 /var/log/mysql $ dbroot
> >> (root@localhost) [(none)]>  RESET MASTER;
> >>
> >> vincentd@pse10 /var/log/mysql $ ll
> >> total 2792
> >> -rw-rw---- 1 mysql adm 2801618 2009-12-23 05:35 mysql-bin.000001
> >> -rw-rw---- 1 mysql adm      32 2009-12-23 05:35 mysql-bin.index
> >> -rw-r----- 1 mysql adm   14987 2009-12-23 05:35 mysql-slow.log
> >> -rw-r----- 1 mysql adm    1102 2009-12-22 00:13 mysql-slow.log.1.gz
> >> -rw-r----- 1 mysql adm     891 2009-12-21 00:02 mysql-slow.log.2.gz
> >> -rw-r----- 1 mysql adm    1318 2009-12-20 00:02 mysql-slow.log.3.gz
> >> -rw-r----- 1 mysql adm     687 2009-12-19 00:02 mysql-slow.log.4.gz
> >> -rw-r----- 1 mysql adm    5246 2009-12-17 20:38 mysql-slow.log.5.gz
> >> -rw-r----- 1 mysql adm     156 2009-12-16 06:25 mysql-slow.log.6.gz
> >> -rw-r----- 1 mysql adm    1114 2009-12-15 16:26 mysql-slow.log.7.gz
> >>
> >> Which freed up a tremendous amount of space again...
> >>
> >> However, a co-worker informed me that now our slaves are broken and
> >> replication is hosed!
> >> What did I do wrong or forget to do?
> >> I see no mention of something I was supposed to do for replication
> >> scenarios...
> >> Was I supposed to "RESET SLAVE" too?
> >>
> >> As of right now, /var/log/mysql has grown to 
> mysql-bin.000028 since last
> >> night when I reset it.
> >> How do I recover from this?
> >>
> >>
> >>
> >>
> >
> >
> > --
> > MySQL General Mailing List
> > For list archives: http://lists.mysql.com/mysql
> > To unsubscribe:
> > http://lists.mysql.com/mysql?unsub=1
> >
> >
> 
> 
> -- 
> Best Regards,
> 
> Prabhat Kumar
> MySQL DBA
> Datavail-India Mumbai
> Mobile     : 91-9987681929
> www.datavail.com
> 
> My Blog: http://adminlinux.blogspot.com
> My LinkedIn: http://www.linkedin.com/in/profileprabhat
> 

Thread
HELP! "RESET MASTER" hosed replicationDaevid Vincent23 Dec
  • Re: HELP! "RESET MASTER" hosed replicationClaudio Nanni23 Dec
  • Re: HELP! "RESET MASTER" hosed replicationCarlos Proal23 Dec
    • Re: HELP! "RESET MASTER" hosed replicationClaudio Nanni23 Dec
      • Re: HELP! "RESET MASTER" hosed replicationCarlos Proal23 Dec
    • Re: HELP! "RESET MASTER" hosed replicationprabhat kumar23 Dec
      • RE: HELP! "RESET MASTER" hosed replicationDaevid Vincent23 Dec