List:General Discussion« Previous MessageNext Message »
From:Daevid Vincent Date:December 23 2009 8:20pm
Subject:HELP! "RESET MASTER" hosed replication
View as plain text  
I got an alert that one of the drives was filling up (3% free). So I
figured out that a large chunk was from /var/log/mysql

root@pse10:~# find / -type d -print0 | xargs -0 -n1 du -sk | sort -rn |
head -n20 > ~/dir-sizes.txt

root@pse10:~# cat ~/dir-sizes.txt
159121012 /
70442396  /var
70127764  /var/log
69991160  /var/log/mysql    <-------- big offender
56307436  /data
31479936  /home
29386076  /data/mysql
26899784  /data/archive

It looked like the /var/log/mysql was pretty full of these Mysql-bin.00XXXX
log files, 

...
-rw-rw---- 1 mysql adm 105019928 2009-12-23 05:07 mysql-bin.001196
-rw-rw---- 1 mysql adm 105004751 2009-12-23 05:08 mysql-bin.001197
-rw-rw---- 1 mysql adm 104978518 2009-12-23 05:10 mysql-bin.001198
-rw-rw---- 1 mysql adm 104949073 2009-12-23 05:11 mysql-bin.001199
-rw-rw---- 1 mysql adm 104925795 2009-12-23 05:13 mysql-bin.001200
-rw-rw---- 1 mysql adm 104974354 2009-12-23 05:14 mysql-bin.001201
-rw-rw---- 1 mysql adm 105089249 2009-12-23 05:16 mysql-bin.001202
-rw-rw---- 1 mysql adm 105165487 2009-12-23 05:17 mysql-bin.001203
-rw-rw---- 1 mysql adm 104926853 2009-12-23 05:19 mysql-bin.001204
-rw-rw---- 1 mysql adm 105139076 2009-12-23 05:20 mysql-bin.001205
-rw-rw---- 1 mysql adm 104891552 2009-12-23 05:22 mysql-bin.001206
-rw-rw---- 1 mysql adm 104959626 2009-12-23 05:25 mysql-bin.001207
-rw-rw---- 1 mysql adm 104883048 2009-12-23 05:27 mysql-bin.001208
-rw-rw---- 1 mysql adm 104993511 2009-12-23 05:28 mysql-bin.001209
-rw-rw---- 1 mysql adm 104945974 2009-12-23 05:30 mysql-bin.001210
-rw-rw---- 1 mysql adm  35468892 2009-12-23 05:30 mysql-bin.001211
-rw-rw---- 1 mysql adm     21728 2009-12-23 05:30 mysql-bin.index
-rw-r----- 1 mysql adm     12836 2009-12-23 00:12 mysql-slow.log
...

so I took the liberty of resetting them...
http://dev.mysql.com/doc/refman/5.0/en/reset.html

vincentd@pse10 /var/log/mysql $ dbroot
(root@localhost) [(none)]> RESET MASTER;

vincentd@pse10 /var/log/mysql $ ll
total 2792
-rw-rw---- 1 mysql adm 2801618 2009-12-23 05:35 mysql-bin.000001
-rw-rw---- 1 mysql adm      32 2009-12-23 05:35 mysql-bin.index
-rw-r----- 1 mysql adm   14987 2009-12-23 05:35 mysql-slow.log
-rw-r----- 1 mysql adm    1102 2009-12-22 00:13 mysql-slow.log.1.gz
-rw-r----- 1 mysql adm     891 2009-12-21 00:02 mysql-slow.log.2.gz
-rw-r----- 1 mysql adm    1318 2009-12-20 00:02 mysql-slow.log.3.gz
-rw-r----- 1 mysql adm     687 2009-12-19 00:02 mysql-slow.log.4.gz
-rw-r----- 1 mysql adm    5246 2009-12-17 20:38 mysql-slow.log.5.gz
-rw-r----- 1 mysql adm     156 2009-12-16 06:25 mysql-slow.log.6.gz
-rw-r----- 1 mysql adm    1114 2009-12-15 16:26 mysql-slow.log.7.gz

Which freed up a tremendous amount of space again...

However, a co-worker informed me that now our slaves are broken and
replication is hosed! 
What did I do wrong or forget to do? 
I see no mention of something I was supposed to do for replication
scenarios... 
Was I supposed to "RESET SLAVE" too?

As of right now, /var/log/mysql has grown to mysql-bin.000028 since last
night when I reset it.
How do I recover from this?

Thread
HELP! "RESET MASTER" hosed replicationDaevid Vincent23 Dec
  • Re: HELP! "RESET MASTER" hosed replicationClaudio Nanni23 Dec
  • Re: HELP! "RESET MASTER" hosed replicationCarlos Proal23 Dec
    • Re: HELP! "RESET MASTER" hosed replicationClaudio Nanni23 Dec
      • Re: HELP! "RESET MASTER" hosed replicationCarlos Proal23 Dec
    • Re: HELP! "RESET MASTER" hosed replicationprabhat kumar23 Dec
      • RE: HELP! "RESET MASTER" hosed replicationDaevid Vincent23 Dec