List:General Discussion« Previous MessageNext Message »
From:Brent Clark Date:March 27 2012 10:47am
Subject:mysqld got signal 6 (problem why bigger than I initially realised)
View as plain text  
Hey Guys

Yesterday I sent an email, about '1 client is using or hasn't closed the table properly',
but the problem is actually bigger than I realised.

We run Mysql replication, and on the second node, Mysql is crashing with 'mysqld got
signal 6' every so often.

Other than the drives, we have replaced the hardware, and still the problem persists.

Googling, I thought it was a memory exhaustion issue, so I started playing with the
following variables and values.

key_buffer_size=8384512
max_connections=41

But it still crashes

Mar 27 12:36:53 xyz-web02 mysqld: key_buffer_size=8384512
Mar 27 12:36:53 xyz-web02 mysqld: read_buffer_size=131072
Mar 27 12:36:53 xyz-web02 mysqld: max_used_connections=3
Mar 27 12:36:53 xyz-web02 mysqld: max_threads=41
Mar 27 12:36:53 xyz-web02 mysqld: thread_count=1
Mar 27 12:36:53 xyz-web02 mysqld: connection_count=1
Mar 27 12:36:53 xyz-web02 mysqld: It is possible that mysqld could use up to
Mar 27 12:36:53 xyz-web02 mysqld: key_buffer_size + (read_buffer_size +
sort_buffer_size)*max_threads = 97828 K  bytes of memory

xyz-web02:# free -m
              total       used       free     shared    buffers     cached
Mem:         12038      11966         71          0        117      11378
-/+ buffers/cache:        469      11568
Swap:         2047          0       2047

We running Debian Squeeze 64bit.

I have attached the full crash message.

If anyone can help, I would be *most* grateful (If you are in South Africa, I will buy you
a meal, let alone a beer :)

Kindest Regards
Brent Clark

*** glibc detected *** /usr/sbin/mysqld: double free or corruption (!prev):
0x00007f5e38003b60 ***
======= Backtrace: =========
/lib/libc.so.6(+0x71bd6)[0x7f5e4575dbd6]
/lib/libc.so.6(cfree+0x6c)[0x7f5e4576294c]
/usr/sbin/mysqld(_ZN15Query_log_eventD0Ev+0x20)[0x7f5e474f2ed0]
/usr/sbin/mysqld(handle_slave_sql+0x835)[0x7f5e47567255]
/lib/libpthread.so.0(+0x68ca)[0x7f5e46c738ca]
/lib/libc.so.6(clone+0x6d)[0x7f5e457bb86d]
======= Memory map: ========
7f5e30000000-7f5e30021000 rw-p 00000000 00:00 0 
7f5e30021000-7f5e34000000 ---p 00000000 00:00 0 
7f5e38000000-7f5e38158000 rw-p 00000000 00:00 0 
7f5e38158000-7f5e3c000000 ---p 00000000 00:00 0 
7f5e3e019000-7f5e3e01a000 ---p 00000000 00:00 0 
7f5e3e01a000-7f5e3e81a000 rw-p 00000000 00:00 0 
7f5e3e81a000-7f5e3e81b000 ---p 00000000 00:00 0 
7f5e3e81b000-7f5e3f01b000 rw-p 00000000 00:00 0 
7f5e3f01b000-7f5e3f01c000 ---p 00000000 00:00 0 
7f5e3f01c000-7f5e3f81c000 rw-p 00000000 00:00 0 
7f5e3f81c000-7f5e3f81d000 ---p 00000000 00:00 0 
7f5e3f81d000-7f5e4001d000 rw-p 00000000 00:00 0 
7f5e4001d000-7f5e4001e000 ---p 00000000 00:00 0 
7f5e4001e000-7f5e40a24000 rw-p 00000000 00:00 0 
7f5e40c47000-7f5e40c48000 ---p 00000000 00:00 0 
7f5e40c48000-7f5e41448000 rw-p 00000000 00:00 0 
7f5e41448000-7f5e41449000 ---p 00000000 00:00 0 
7f5e41449000-7f5e41c49000 rw-p 00000000 00:00 0 
7f5e41c49000-7f5e41c4a000 ---p 00000000 00:00 0 
7f5e41c4a000-7f5e448c4000 rw-p 00000000 00:00 0 
7f5e448c4000-7f5e448d0000 r-xp 00000000 fd:00 328894                    
/lib/libnss_files-2.11.3.so
7f5e448d0000-7f5e44acf000 ---p 0000c000 fd:00 328894                    
/lib/libnss_files-2.11.3.so
7f5e44acf000-7f5e44ad0000 r--p 0000b000 fd:00 328894                    
/lib/libnss_files-2.11.3.so
7f5e44ad0000-7f5e44ad1000 rw-p 0000c000 fd:00 328894                    
/lib/libnss_files-2.11.3.so
7f5e44ad1000-7f5e44adb000 r-xp 00000000 fd:00 328896                    
/lib/libnss_nis-2.11.3.so
7f5e44adb000-7f5e44cda000 ---p 0000a000 fd:00 328896                    
/lib/libnss_nis-2.11.3.so
7f5e44cda000-7f5e44cdb000 r--p 00009000 fd:00 328896                    
/lib/libnss_nis-2.11.3.so
7f5e44cdb000-7f5e44cdc000 rw-p 0000a000 fd:00 328896                    
/lib/libnss_nis-2.11.3.so
7f5e44cdc000-7f5e44ce3000 r-xp 00000000 fd:00 328892                    
/lib/libnss_compat-2.11.3.so
7f5e44ce3000-7f5e44ee2000 ---p 00007000 fd:00 328892                    
/lib/libnss_compat-2.11.3.so
7f5e44ee2000-7f5e44ee3000 r--p 00006000 fd:00 328892                    
/lib/libnss_compat-2.11.3.so
7f5e44ee3000-7f5e44ee4000 rw-p 00007000 fd:00 328892                    
/lib/libnss_compat-2.11.3.so
7f5e44eeb000-7f5e44eec000 ---p 00000000 00:00 0 
7f5e44eec000-7f5e456ec000 rw-p 00000000 00:00 0 
7f5e456ec000-7f5e45845000 r-xp 00000000 fd:00 328847                    
/lib/libc-2.11.3.so
7f5e45845000-7f5e45a44000 ---p 00159000 fd:00 328847                    
/lib/libc-2.11.3.so
7f5e45a44000-7f5e45a48000 r--p 00158000 fd:00 328847                    
/lib/libc-2.11.3.so
7f5e45a48000-7f5e45a49000 rw-p 0015c000 fd:00 328847                    
/lib/libc-2.11.3.so
7f5e45a49000-7f5e45a4e000 rw-p 00000000 00:00 0 
7f5e45a4e000-7f5e45a64000 r-xp 00000000 fd:00 328801                    
/lib/libgcc_s.so.1
7f5e45a64000-7f5e45c63000 ---p 00016000 fd:00 328801                    
/lib/libgcc_s.so.1
7f5e45c63000-7f5e45c64000 rw-p 00015000 fd:00 328801                    
/lib/libgcc_s.so.1
7f5e45c64000-7f5e45ce4000 r-xp 00000000 fd:00 328889                    
/lib/libm-2.11.3.so
7f5e45ce4000-7f5e45ee4000 ---p 00080000 fd:00 328889                    
/lib/libm-2.11.3.so
7f5e45ee4000-7f5e45ee5000 r--p 00080000 fd:00 328889                    
/lib/libm-2.11.3.so
7f5e45ee5000-7f5e45ee6000 rw-p 00081000 fd:00 328889                    
/lib/libm-2.11.3.so
7f5e45ee6000-7f5e45fdc000 r-xp 00000000 fd:02 173092                    
/usr/lib/libstdc++.so.6.0.13
7f5e45fdc000-7f5e461dc000 ---p 000f6000 fd:02 173092                    
/usr/lib/libstdc++.so.6.0.13
7f5e461dc000-7f5e461e3000 r--p 000f6000 fd:02 173092                    
/usr/lib/libstdc++.so.6.0.13
7f5e461e3000-7f5e461e5000 rw-p 000fd000 fd:02 173092                    
/usr/lib/libstdc++.so.6.0.13
7f5e461e5000-7f5e461fa000 rw-p 00000000 00:00 0 
7f5e461fa000-7f5e4620f000 r-xp 00000000 fd:00 328891                    
/lib/libnsl-2.11.3.so
7f5e4620f000-7f5e4640e000 ---p 00015000 fd:00 328891                    
/lib/libnsl-2.11.3.so
7f5e4640e000-7f5e4640f000 r--p 00014000 fd:00 328891                    
/lib/libnsl-2.11.3.so
7f5e4640f000-7f5e46410000 rw-p 00015000 fd:00 328891                    
/lib/libnsl-2.11.3.so
7f5e46410000-7f5e46412000 rw-p 00000000 00:00 0 
7f5e46412000-7f5e4641a000 r-xp 00000000 fd:00 328885                    
/lib/libcrypt-2.11.3.so
7f5e4641a000-7f5e46619000 ---p 00008000 fd:00 328885                    
/lib/libcrypt-2.11.3.so
7f5e46619000-7f5e4661a000 r--p 00007000 fd:00 328885                    
/lib/libcrypt-2.11.3.so
7f5e4661a000-7f5e4661b000 rw-p 00008000 fd:00 328885                    
/lib/libcrypt-2.11.3.so
7f5e4661b000-7f5e46649000 rw-p 00000000 00:00 0 
7f5e46649000-7f5e4664b000 r-xp 00000000 fd:00 328887                    
/lib/libdl-2.11.3.so
7f5e4664b000-7f5e4684b000 ---p 00002000 fd:00 328887                    
/lib/libdl-2.11.3.so
7f5e4684b000-7f5e4684c000 r--p 00002000 fd:00 328887                    
/lib/libdl-2.11.3.so
7f5e4684c000-7f5e4684d000 rw-p 00003000 fd:00 328887                    
/lib/libdl-2.11.3.so
7f5e4684d000-7f5e46855000 r-xp 00000000 fd:00 328825                    
/lib/libwrap.so.0.7.6
7f5e46855000-7f5e46a54000 ---p 00008000 fd:00 328825                    
/lib/libwrap.so.0.7.6
7f5e46a54000-7f5e46a56000 rw-p 00007000 fd:00 328825                    
/lib/libwrap.so.0.7.6
7f5e46a56000-7f5e46a6d000 r-xp 00000000 fd:02 173161                    
/usr/lib/libz.so.1.2.3.4
7f5e46a6d000-7f5e46c6c000 ---p 00017000 fd:02 173161                    
/usr/lib/libz.so.1.2.3.4
7f5e46c6c000-7f5e46c6d000 rw-p 00016000 fd:02 173161                    
/usr/lib/libz.so.1.2.3.4
7f5e46c6d000-7f5e46c84000 r-xp 00000000 fd:00 328787                    
/lib/libpthread-2.11.3.so
7f5e46c84000-7f5e46e83000 ---p 00017000 fd:00 328787                    
/lib/libpthread-2.11.3.so
7f5e46e83000-7f5e46e84000 r--p 00016000 fd:00 328787                    
/lib/libpthread-2.11.3.so
7f5e46e84000-7f5e46e85000 rw-p 00017000 fd:00 328787                    
/lib/libpthread-2.11.3.so
Mar 27 12:30:39 xyz-web02 mysqld: 10:30:39 UTC - mysqld got signal 6 ;
Mar 27 12:30:39 xyz-web02 mysqld: This could be because you hit a bug. It is also possible
that this binary
Mar 27 12:30:39 xyz-web02 mysqld: or one of the libraries it was linked against is
corrupt, improperly built,
Mar 27 12:30:39 xyz-web02 mysqld: or misconfigured. This error can also be caused by
malfunctioning hardware.
Mar 27 12:30:39 xyz-web02 mysqld: We will try our best to scrape up some info that will
hopefully help
Mar 27 12:30:39 xyz-web02 mysqld: diagnose the problem, but since we have already crashed,

Mar 27 12:30:39 xyz-web02 mysqld: something is definitely wrong and this may fail.
Mar 27 12:30:39 xyz-web02 mysqld: 
Mar 27 12:30:39 xyz-web02 mysqld: key_buffer_size=8384512
Mar 27 12:30:39 xyz-web02 mysqld: read_buffer_size=131072
Mar 27 12:30:39 xyz-web02 mysqld: max_used_connections=2
Mar 27 12:30:39 xyz-web02 mysqld: max_threads=41
Mar 27 12:30:39 xyz-web02 mysqld: thread_count=1
Mar 27 12:30:39 xyz-web02 mysqld: connection_count=1
Mar 27 12:30:39 xyz-web02 mysqld: It is possible that mysqld could use up to 
Mar 27 12:30:39 xyz-web02 mysqld: key_buffer_size + (read_buffer_size +
sort_buffer_size)*max_threads = 97828 K  bytes of memory
Mar 27 12:30:39 xyz-web02 mysqld: Hope that's ok; if not, decrease some variables in the
equation.
Mar 27 12:30:39 xyz-web02 mysqld: 
Mar 27 12:30:39 xyz-web02 mysqld: Thread pointer: 0x7f5e38000970
Mar 27 12:30:39 xyz-web02 mysqld: Attempting backtrace. You can use the following
information to find out
Mar 27 12:30:39 xyz-web02 mysqld: where mysqld died. If you see no messages after this,
something went
Mar 27 12:30:39 xyz-web02 mysqld: terribly wrong...
Mar 27 12:30:39 xyz-web02 mysqld: stack_bottom = 7f5e47033848 thread_stack 0x30000
Mar 27 12:30:39 xyz-web02 mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x29)
[0x7f5e47714b59]
Mar 27 12:30:39 xyz-web02 mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x483)
[0x7f5e475294a3]
Mar 27 12:30:39 xyz-web02 mysqld: /lib/libpthread.so.0(+0xeff0) [0x7f5e46c7bff0]
Mar 27 12:30:39 xyz-web02 mysqld: /lib/libc.so.6(gsignal+0x35) [0x7f5e4571e1b5]
Mar 27 12:30:39 xyz-web02 mysqld: /lib/libc.so.6(abort+0x180) [0x7f5e45720fc0]
Mar 27 12:30:39 xyz-web02 mysqld: /lib/libc.so.6(+0x6837b) [0x7f5e4575437b]
Mar 27 12:30:39 xyz-web02 mysqld: /lib/libc.so.6(+0x71bd6) [0x7f5e4575dbd6]
Mar 27 12:30:39 xyz-web02 mysqld: /lib/libc.so.6(cfree+0x6c) [0x7f5e4576294c]
Mar 27 12:30:39 xyz-web02 mysqld:
/usr/sbin/mysqld(Query_log_event::~Query_log_event()+0x20) [0x7f5e474f2ed0]
Mar 27 12:30:39 xyz-web02 mysqld: /usr/sbin/mysqld(handle_slave_sql+0x835)
[0x7f5e47567255]
Mar 27 12:30:39 xyz-web02 mysqld: /lib/libpthread.so.0(+0x68ca) [0x7f5e46c738ca]
Mar 27 12:30:39 xyz-web02 mysqld: /lib/libc.so.6(clone+0x6d) [0x7f5e457bb86d]
Mar 27 12:30:39 xyz-web02 mysqld: 
Mar 27 12:30:39 xyz-web02 mysqld: Trying to get some variables.
Mar 27 12:30:39 xyz-web02 mysqld: Some pointers may be invalid and cause the dump to
abort.
Mar 27 12:30:39 xyz-web02 mysqld: Query (0): is an invalid pointer
Mar 27 12:30:39 xyz-web02 mysqld: Connection ID (thread ID): 2
Mar 27 12:30:39 xyz-web02 mysqld: Status: NOT_KILLED
Mar 27 12:30:39 xyz-web02 mysqld: 
Mar 27 12:30:39 xyz-web02 mysqld: The manual page at
http://dev.mysql.com/doc/mysql/en/crashing.html contains
Mar 27 12:30:39 xyz-web02 mysqld: information that should help you find out what is
causing the crash.
Mar 27 12:30:39 xyz-web02 mysqld_safe: Number of processes running now: 0
Mar 27 12:30:39 xyz-web02 mysqld_safe: mysqld restarted
Mar 27 12:30:39 xyz-web02 mysqld: 120327 12:30:39 [Warning] 'for replication startup
options' is deprecated and will be removed in a future release. Please use ''CHANGE
MASTER'' instead.
Mar 27 12:30:39 xyz-web02 mysqld: 120327 12:30:39 [Note] Plugin 'FEDERATED' is disabled.
Mar 27 12:30:39 xyz-web02 mysqld: 120327 12:30:39  InnoDB: Initializing buffer pool, size
= 8.0M
Mar 27 12:30:39 xyz-web02 mysqld: 120327 12:30:39  InnoDB: Completed initialization of
buffer pool
Mar 27 12:30:39 xyz-web02 mysqld: InnoDB: Log scan progressed past the checkpoint lsn 21
2468511949
Mar 27 12:30:39 xyz-web02 mysqld: 120327 12:30:39  InnoDB: Database was not shut down
normally!
Mar 27 12:30:39 xyz-web02 mysqld: InnoDB: Starting crash recovery.
Mar 27 12:30:39 xyz-web02 mysqld: InnoDB: Reading tablespace information from the .ibd
files...
Mar 27 12:30:39 xyz-web02 mysqld: InnoDB: Restoring possible half-written data pages from
the doublewrite
Mar 27 12:30:39 xyz-web02 mysqld: InnoDB: buffer...
Mar 27 12:30:39 xyz-web02 mysqld: InnoDB: Doing recovery: scanned up to log sequence
number 21 2468634155
Mar 27 12:30:40 xyz-web02 mysqld: 120327 12:30:40  InnoDB: Starting an apply batch of log
records to the database...
Mar 27 12:30:41 xyz-web02 mysqld: InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11
12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41
42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71
72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 
Mar 27 12:30:41 xyz-web02 mysqld: InnoDB: Apply batch completed
Mar 27 12:30:41 xyz-web02 mysqld: InnoDB: Last MySQL binlog file position 0 106, file name
/var/log/mysql/mysql-bin.002274
Mar 27 12:30:41 xyz-web02 mysqld: 120327 12:30:41  InnoDB: Started; log sequence number 21
2468634155
Mar 27 12:30:41 xyz-web02 mysqld: 120327 12:30:41 [Note] Recovering after a crash using
/var/log/mysql/mysql-bin
Mar 27 12:30:41 xyz-web02 mysqld: 120327 12:30:41 [Note] Starting crash recovery...
Mar 27 12:30:41 xyz-web02 mysqld: 120327 12:30:41 [Note] Crash recovery finished.
Mar 27 12:30:41 xyz-web02 mysqld: 120327 12:30:41 [Note] Slave SQL thread initialized,
starting replication in log 'mysql-bin.000845' at position 244888243, relay log
'/var/lib/mysql/slave-relay.000262' position: 4602980
Mar 27 12:30:41 xyz-web02 mysqld: 120327 12:30:41 [Note] Event Scheduler: Loaded 0 events

Thread
mysqld got signal 6 (problem why bigger than I initially realised)Brent Clark27 Mar
  • Re: mysqld got signal 6 (problem why bigger than I initially realised)P.R.Karthik27 Mar
    • Re: mysqld got signal 6 (problem why bigger than I initially realised)Brent Clark27 Mar
      • Re: mysqld got signal 6 (problem why bigger than I initially realised)Rick James27 Mar
        • Re: mysqld got signal 6 (problem why bigger than I initially realised)Brent Clark27 Mar
          • Re: mysqld got signal 6 (problem why bigger than I initially realised)Rick James27 Mar
            • Re: mysqld got signal 6 (problem why bigger than I initially realised)Reindl Harald28 Mar
              • Re: mysqld got signal 6 (problem why bigger than I initially realised)Rick James29 Mar
                • Re: mysqld got signal 6 (problem why bigger than I initially realised)Reindl Harald29 Mar