List:General Discussion« Previous MessageNext Message »
From:Simon Loewenthal Date:August 28 2013 10:12am
Subject:mysql 5.5 crashed on Debian 6 after server crash
View as plain text  
Hi there,

  I just had my mysql dB crash really badly on a small server I'm running
and wonder if someone could point me in the right direction.  I'm pretty
basic with mysql... I set up master/slave replication to another site, so
this could be a backup, so long as the errors were not replicated over
there.

Mysql boots with these types of errors recorded in syslog. Also is the
my.cnf


If you know of ideas that could be of help, then I woul dbe really
grateful.  I've been trying to get this to run for 4 hours solid.

Best regards, Simon

My.cnf
( My server only has 512Mb of RAM so mysql has to use not a lot).


# grep -v ^$ /etc/mysql/my.cnf|grep -v ^#
[client]
port            = 3306
socket          = /var/run/mysqld/mysqld.sock
[mysqld_safe]
socket          = /var/run/mysqld/mysqld.sock
nice            = 0
[mysqld]
user            = mysql
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
port            = 3306
basedir         = /usr
datadir         = /var/lib/mysql
tmpdir          = /tmp
skip-external-locking
bind-address            = 127.0.0.1
max_allowed_packet = 1M
key_buffer              = 16K
sort_buffer_size = 64K
read_buffer_size = 256K
thread_stack = 64K
thread_cache_size       = 8
myisam-recover          = BACKUP
max_connections         = 34
log_slow_queries        = /var/log/mysql/mysql-slow.log
query_cache_limit       = 64K
query_cache_size        = 2M
expire_logs_days        = 10
max_binlog_size         = 100M
table_open_cache        = 4
table_cache             = 48
read_rnd_buffer_size = 256K
net_buffer_length = 128K
server-id       = 1
log-bin=/var/log/mysql/mysql-bin.log
innodb_flush_log_at_trx_commit  = 2
max_binlog_size         = 15M
expire_logs_days        = 2
[mysqldump]
quick
quote-names
max_allowed_packet = 1M
[mysql]
[isamchk]
key_buffer              = 16M
!includedir /etc/mysql/conf.d/
[myisamchk]
key_buffer_size = 386K
sort_buffer_size = 64K
read_buffer_size = 256K

Syslog records 36000 lines of errors from mysql and below please find a
snip it:

Aug 28 11:50:14 emailserver mysqld_safe: Number of processes running now: 0
Aug 28 11:50:14 emailserver mysqld_safe: mysqld restarted
Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 [Warning] option
'thread_stack': unsigned value 65536 adjusted to 131072
Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 [Warning] The syntax
'--log-slow-queries' is deprecated and will be removed in a future release.
Please use '--slow-query-log'/'--slow-query-log-file' instead.
Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 [Note] Plugin
'FEDERATED' is disabled.
Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: The InnoDB
memory heap is disabled
Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: Mutexes and
rw_locks use GCC atomic builtins
Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: Compressed
tables use zlib 1.2.3.4
Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: Using Linux
native AIO
Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: Initializing
buffer pool, size = 128.0M
Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: Completed
initialization of buffer pool
Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: highest
supported file format is Barracuda.
Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14  InnoDB: Waiting for
the background threads to start
Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 InnoDB: 5.5.31 started;
log sequence number 1211163767
Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note] Recovering after
a crash using /var/log/mysql/mysql-bin
Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note] Starting crash
recovery...
Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note] Crash recovery
finished.
Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note] Server hostname
(bind-address): '127.0.0.1'; port: 3306
Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note]   - '127.0.0.1'
resolves to '127.0.0.1';
Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note] Server socket
created on IP: '127.0.0.1'.
Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note] Event Scheduler:
Loaded 0 events
Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note]
/usr/sbin/mysqld: ready for connections.
Aug 28 11:50:15 emailserver mysqld: Version: '5.5.31-1~dotdeb.0-log'
socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Debian)
Aug 28 11:50:15 emailserver mysqld: InnoDB: Dump of the tablespace extent
descriptor:  len 40; hex
00000000000000000000000001de00000000016600000002aaaaaaaaaaaaaaaaeaaaeafaabaabbaa;
asc                    f                    ;
Aug 28 11:50:15 emailserver mysqld: InnoDB: Serious error! InnoDB is trying
to free page 815
Aug 28 11:50:15 emailserver mysqld: InnoDB: though it is already marked as
free in the tablespace!
Aug 28 11:50:15 emailserver mysqld: InnoDB: The tablespace free space info
is corrupt.
Aug 28 11:50:15 emailserver mysqld: InnoDB: You may need to dump your
InnoDB tables and recreate the whole
Aug 28 11:50:15 emailserver mysqld: InnoDB: database!
Aug 28 11:50:15 emailserver mysqld: InnoDB: Please refer to
Aug 28 11:50:15 emailserver mysqld: InnoDB:
http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
Aug 28 11:50:15 emailserver mysqld: InnoDB: about forcing recovery.
Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15  InnoDB: Assertion
failure in thread 139642230933248 in file fsp0fsp.c line 3309
Aug 28 11:50:15 emailserver mysqld: InnoDB: We intentionally generate a
memory trap.
Aug 28 11:50:15 emailserver mysqld: InnoDB: Submit a detailed bug report to
http://bugs.mysql.com.
Aug 28 11:50:15 emailserver mysqld: InnoDB: If you get repeated assertion
failures or crashes, even
Aug 28 11:50:15 emailserver mysqld: InnoDB: immediately after the mysqld
startup, there may be
Aug 28 11:50:15 emailserver mysqld: InnoDB: corruption in the InnoDB
tablespace. Please refer to
Aug 28 11:50:15 emailserver mysqld: InnoDB:
http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
Aug 28 11:50:15 emailserver mysqld: InnoDB: about forcing recovery.
Aug 28 11:50:15 emailserver mysqld: 09:50:15 UTC - mysqld got signal 6 ;
Aug 28 11:50:15 emailserver mysqld: This could be because you hit a bug. It
is also possible that this binary
Aug 28 11:50:15 emailserver mysqld: or one of the libraries it was linked
against is corrupt, improperly built,
Aug 28 11:50:15 emailserver mysqld: or misconfigured. This error can also
be caused by malfunctioning hardware.
Aug 28 11:50:15 emailserver mysqld: We will try our best to scrape up some
info that will hopefully help
Aug 28 11:50:15 emailserver mysqld: diagnose the problem, but since we have
already crashed,
Aug 28 11:50:15 emailserver mysqld: something is definitely wrong and this
may fail.
Aug 28 11:50:15 emailserver mysqld:
Aug 28 11:50:15 emailserver mysqld: key_buffer_size=16384
Aug 28 11:50:15 emailserver mysqld: read_buffer_size=262144
Aug 28 11:50:15 emailserver mysqld: max_used_connections=0
Aug 28 11:50:15 emailserver mysqld: max_threads=34
Aug 28 11:50:15 emailserver mysqld: thread_count=0
Aug 28 11:50:15 emailserver mysqld: connection_count=0
Aug 28 11:50:15 emailserver mysqld: It is possible that mysqld could use up
to
Aug 28 11:50:15 emailserver mysqld: key_buffer_size + (read_buffer_size +
sort_buffer_size)*max_threads = 11283 K  bytes of memory
Aug 28 11:50:15 emailserver mysqld: Hope that's ok; if not, decrease some
variables in the equation.
Aug 28 11:50:15 emailserver mysqld:
Aug 28 11:50:15 emailserver mysqld: Thread pointer: 0x0
Aug 28 11:50:15 emailserver mysqld: Attempting backtrace. You can use the
following information to find out
Aug 28 11:50:15 emailserver mysqld: where mysqld died. If you see no
messages after this, something went
Aug 28 11:50:15 emailserver mysqld: terribly wrong...
Aug 28 11:50:15 emailserver mysqld: stack_bottom = 0 thread_stack 0x20000
Aug 28 11:50:15 emailserver mysqld:
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7f011094b14e]
Aug 28 11:50:15 emailserver mysqld:
/usr/sbin/mysqld(handle_fatal_signal+0x493)[0x7f0110802843]
Aug 28 11:50:15 emailserver mysqld:
/lib/libpthread.so.0(+0xeff0)[0x7f010ffbbff0]
Aug 28 11:50:15 emailserver mysqld:
/lib/libc.so.6(gsignal+0x35)[0x7f010e86d1b5]
Aug 28 11:50:15 emailserver mysqld:
/lib/libc.so.6(abort+0x180)[0x7f010e86ffc0]
Aug 28 11:50:15 emailserver mysqld:
/usr/sbin/mysqld(+0x682405)[0x7f0110a6b405]
Aug 28 11:50:15 emailserver mysqld:
/usr/sbin/mysqld(+0x682735)[0x7f0110a6b735]
Aug 28 11:50:15 emailserver mysqld:
/usr/sbin/mysqld(+0x6394b9)[0x7f0110a224b9]
Aug 28 11:50:15 emailserver mysqld:
/usr/sbin/mysqld(+0x6d1b0f)[0x7f0110abab0f]
Aug 28 11:50:15 emailserver mysqld:
/usr/sbin/mysqld(+0x6d2438)[0x7f0110abb438]
Aug 28 11:50:15 emailserver mysqld:
/usr/sbin/mysqld(+0x6c8255)[0x7f0110ab1255]
Aug 28 11:50:15 emailserver mysqld:
/usr/sbin/mysqld(+0x613435)[0x7f01109fc435]
Aug 28 11:50:15 emailserver mysqld:
/usr/sbin/mysqld(+0x60656c)[0x7f01109ef56c]
Aug 28 11:50:15 emailserver mysqld:
/usr/sbin/mysqld(+0x6092a3)[0x7f01109f22a3]
Aug 28 11:50:15 emailserver mysqld:
/lib/libpthread.so.0(+0x68ca)[0x7f010ffb38ca]
Aug 28 11:50:15 emailserver mysqld:
/lib/libc.so.6(clone+0x6d)[0x7f010e90ab6d]
Aug 28 11:50:15 emailserver mysqld: The manual page at
http://dev.mysql.com/doc/mysql/en/crashing.html contains
Aug 28 11:50:15 emailserver mysqld: information that should help you find
out what is causing the crash.
Aug 28 11:50:15 emailserver mysqld_safe: Number of processes running now: 0
Aug 28 11:50:15 emailserver mysqld_safe: mysqld restarted
Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 [Warning] option
'thread_stack': unsigned value 65536 adjusted to 131072
Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 [Warning] The syntax
'--log-slow-queries' is deprecated and will be removed in a future release.
Please use '--slow-query-log'/'--slow-query-log-file' instead.
Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 [Note] Plugin
'FEDERATED' is disabled.
Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: The InnoDB
memory heap is disabled
Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: Mutexes and
rw_locks use GCC atomic builtins
Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: Compressed
tables use zlib 1.2.3.4
Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: Using Linux
native AIO
Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: Initializing
buffer pool, size = 128.0M
Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: Completed
initialization of buffer pool
Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: highest
supported file format is Barracuda.
Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16  InnoDB: Waiting for
the background threads to start
Aug 28 11:50:16 emailserver mysqld_safe: mysqld from pid file
/var/run/mysqld/mysqld.pid ended
Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 InnoDB: 5.5.31 started;
log sequence number 1211163767
Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note] Recovering after
a crash using /var/log/mysql/mysql-bin
Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note] Starting crash
recovery...
Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note] Crash recovery
finished.
Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note] Server hostname
(bind-address): '127.0.0.1'; port: 3306
Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note]   - '127.0.0.1'
resolves to '127.0.0.1';
Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note] Server socket
created on IP: '127.0.0.1'.
Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note] Event Scheduler:
Loaded 0 events
Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note]
/usr/sbin/mysqld: ready for connections.
Aug 28 11:50:17 emailserver mysqld: Version: '5.5.31-1~dotdeb.0-log'
socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Debian)
Aug 28 11:50:17 emailserver mysqld: InnoDB: Dump of the tablespace extent
descriptor:  len 40; hex
00000000000000000000000001de00000000016600000002aaaaaaaaaaaaaaaaeaaaeafaabaabbaa;
asc                    f                    ;
Aug 28 11:50:17 emailserver mysqld: InnoDB: Serious error! InnoDB is trying
to free page 815
Aug 28 11:50:17 emailserver mysqld: InnoDB: though it is already marked as
free in the tablespace!
Aug 28 11:50:17 emailserver mysqld: InnoDB: The tablespace free space info
is corrupt.
Aug 28 11:50:17 emailserver mysqld: InnoDB: You may need to dump your
InnoDB tables and recreate the whole
Aug 28 11:50:17 emailserver mysqld: InnoDB: database!
Aug 28 11:50:17 emailserver mysqld: InnoDB: Please refer to
Aug 28 11:50:17 emailserver mysqld: InnoDB:
http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
Aug 28 11:50:17 emailserver mysqld: InnoDB: about forcing recovery.
Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17  InnoDB: Assertion
failure in thread 140076906338048 in file fsp0fsp.c line 3309
Aug 28 11:50:17 emailserver mysqld: InnoDB: We intentionally generate a
memory trap.
Aug 28 11:50:17 emailserver mysqld: InnoDB: Submit a detailed bug report to
http://bugs.mysql.com.
Aug 28 11:50:17 emailserver mysqld: InnoDB: If you get repeated assertion
failures or crashes, even
Aug 28 11:50:17 emailserver mysqld: InnoDB: immediately after the mysqld
startup, there may be
Aug 28 11:50:17 emailserver mysqld: InnoDB: corruption in the InnoDB
tablespace. Please refer to
Aug 28 11:50:17 emailserver mysqld: InnoDB:
http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
Aug 28 11:50:17 emailserver mysqld: InnoDB: about forcing recovery.
Aug 28 11:50:17 emailserver mysqld: 09:50:17 UTC - mysqld got signal 6 ;
Aug 28 11:50:17 emailserver mysqld: This could be because you hit a bug. It
is also possible that this binary
Aug 28 11:50:17 emailserver mysqld: or one of the libraries it was linked
against is corrupt, improperly built,
Aug 28 11:50:17 emailserver mysqld: or misconfigured. This error can also
be caused by malfunctioning hardware.
Aug 28 11:50:17 emailserver mysqld: We will try our best to scrape up some
info that will hopefully help
Aug 28 11:50:17 emailserver mysqld: diagnose the problem, but since we have
already crashed,
Aug 28 11:50:17 emailserver mysqld: something is definitely wrong and this
may fail.
Aug 28 11:50:17 emailserver mysqld:
Aug 28 11:50:17 emailserver mysqld: key_buffer_size=16384
Aug 28 11:50:17 emailserver mysqld: read_buffer_size=262144
Aug 28 11:50:17 emailserver mysqld: max_used_connections=0
Aug 28 11:50:17 emailserver mysqld: max_threads=34
Aug 28 11:50:17 emailserver mysqld: thread_count=0
Aug 28 11:50:17 emailserver mysqld: connection_count=0
Aug 28 11:50:17 emailserver mysqld: It is possible that mysqld could use up
to
Aug 28 11:50:17 emailserver mysqld: key_buffer_size + (read_buffer_size +
sort_buffer_size)*max_threads = 11283 K  bytes of memory
Aug 28 11:50:17 emailserver mysqld: Hope that's ok; if not, decrease some
variables in the equation.
Aug 28 11:50:17 emailserver mysqld:
Aug 28 11:50:17 emailserver mysqld: Thread pointer: 0x0
Aug 28 11:50:17 emailserver mysqld: Attempting backtrace. You can use the
following information to find out
Aug 28 11:50:17 emailserver mysqld: where mysqld died. If you see no
messages after this, something went
Aug 28 11:50:17 emailserver mysqld: terribly wrong...
Aug 28 11:50:17 emailserver mysqld: stack_bottom = 0 thread_stack 0x20000
Aug 28 11:50:17 emailserver mysqld:
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7f664541014e]
Aug 28 11:50:17 emailserver mysqld:
/usr/sbin/mysqld(handle_fatal_signal+0x493)[0x7f66452c7843]
Aug 28 11:50:17 emailserver mysqld:
/lib/libpthread.so.0(+0xeff0)[0x7f6644a80ff0]
Aug 28 11:50:17 emailserver mysqld:
/lib/libc.so.6(gsignal+0x35)[0x7f66433321b5]
Aug 28 11:50:17 emailserver mysqld:
/lib/libc.so.6(abort+0x180)[0x7f6643334fc0]
Aug 28 11:50:17 emailserver mysqld:
/usr/sbin/mysqld(+0x682405)[0x7f6645530405]
Aug 28 11:50:17 emailserver mysqld:
/usr/sbin/mysqld(+0x682735)[0x7f6645530735]
Aug 28 11:50:17 emailserver mysqld:
/usr/sbin/mysqld(+0x6394b9)[0x7f66454e74b9]
Aug 28 11:50:17 emailserver mysqld:
/usr/sbin/mysqld(+0x6d1b0f)[0x7f664557fb0f]
Aug 28 11:50:17 emailserver mysqld:
/usr/sbin/mysqld(+0x6d2438)[0x7f6645580438]
Aug 28 11:50:17 emailserver mysqld:
/usr/sbin/mysqld(+0x6c8255)[0x7f6645576255]
Aug 28 11:50:17 emailserver mysqld:
/usr/sbin/mysqld(+0x613435)[0x7f66454c1435]
Aug 28 11:50:17 emailserver mysqld:
/usr/sbin/mysqld(+0x60656c)[0x7f66454b456c]
Aug 28 11:50:17 emailserver mysqld:
/usr/sbin/mysqld(+0x6092a3)[0x7f66454b72a3]
Aug 28 11:50:17 emailserver mysqld:
/lib/libpthread.so.0(+0x68ca)[0x7f6644a788ca]
Aug 28 11:50:17 emailserver mysqld:
/lib/libc.so.6(clone+0x6d)[0x7f66433cfb6d]
Aug 28 11:50:17 emailserver mysqld: The manual page at
http://dev.mysql.com/doc/mysql/en/crashing.html contains
Aug 28 11:50:17 emailserver mysqld: information that should help you find
out what is causing the crash.

Thread
mysql 5.5 crashed on Debian 6 after server crashSimon Loewenthal28 Aug
  • Re: mysql 5.5 crashed on Debian 6 after server crashSimon Loewenthal28 Aug