List:General Discussion« Previous MessageNext Message »
From:Simon Loewenthal Date:August 28 2013 10:17am
Subject:Re: mysql 5.5 crashed on Debian 6 after server crash
View as plain text  
Hi,
 I have managed to get the dB open with
 [mysqld]
innodb_force_recovery = 2

I did a mysqldump for all tables, and restarted with innodb_force_recordy
but still have the same old error messages.

Regards, S


On Wed, Aug 28, 2013 at 12:12 PM, Simon Loewenthal <
simon.loewenthal@stripped> wrote:

> Hi there,
>
>   I just had my mysql dB crash really badly on a small server I'm running
> and wonder if someone could point me in the right direction.  I'm pretty
> basic with mysql... I set up master/slave replication to another site, so
> this could be a backup, so long as the errors were not replicated over
> there.
>
> Mysql boots with these types of errors recorded in syslog. Also is the
> my.cnf
>
>
> If you know of ideas that could be of help, then I woul dbe really
> grateful.  I've been trying to get this to run for 4 hours solid.
>
> Best regards, Simon
>
> My.cnf
> ( My server only has 512Mb of RAM so mysql has to use not a lot).
>
>
> # grep -v ^$ /etc/mysql/my.cnf|grep -v ^#
> [client]
> port            = 3306
> socket          = /var/run/mysqld/mysqld.sock
> [mysqld_safe]
> socket          = /var/run/mysqld/mysqld.sock
> nice            = 0
> [mysqld]
> user            = mysql
> pid-file        = /var/run/mysqld/mysqld.pid
> socket          = /var/run/mysqld/mysqld.sock
> port            = 3306
> basedir         = /usr
> datadir         = /var/lib/mysql
> tmpdir          = /tmp
> skip-external-locking
> bind-address            = 127.0.0.1
> max_allowed_packet = 1M
> key_buffer              = 16K
> sort_buffer_size = 64K
> read_buffer_size = 256K
> thread_stack = 64K
> thread_cache_size       = 8
> myisam-recover          = BACKUP
> max_connections         = 34
> log_slow_queries        = /var/log/mysql/mysql-slow.log
> query_cache_limit       = 64K
> query_cache_size        = 2M
> expire_logs_days        = 10
> max_binlog_size         = 100M
> table_open_cache        = 4
> table_cache             = 48
> read_rnd_buffer_size = 256K
> net_buffer_length = 128K
> server-id       = 1
> log-bin=/var/log/mysql/mysql-bin.log
> innodb_flush_log_at_trx_commit  = 2
> max_binlog_size         = 15M
> expire_logs_days        = 2
> [mysqldump]
> quick
> quote-names
> max_allowed_packet = 1M
> [mysql]
> [isamchk]
> key_buffer              = 16M
> !includedir /etc/mysql/conf.d/
> [myisamchk]
> key_buffer_size = 386K
> sort_buffer_size = 64K
> read_buffer_size = 256K
>
> Syslog records 36000 lines of errors from mysql and below please find a
> snip it:
>
> Aug 28 11:50:14 emailserver mysqld_safe: Number of processes running now: 0
> Aug 28 11:50:14 emailserver mysqld_safe: mysqld restarted
> Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 [Warning] option
> 'thread_stack': unsigned value 65536 adjusted to 131072
> Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 [Warning] The syntax
> '--log-slow-queries' is deprecated and will be removed in a future release.
> Please use '--slow-query-log'/'--slow-query-log-file' instead.
> Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 [Note] Plugin
> 'FEDERATED' is disabled.
> Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: The InnoDB
> memory heap is disabled
> Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: Mutexes and
> rw_locks use GCC atomic builtins
> Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: Compressed
> tables use zlib 1.2.3.4
> Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: Using Linux
> native AIO
> Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: Initializing
> buffer pool, size = 128.0M
> Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: Completed
> initialization of buffer pool
> Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14 InnoDB: highest
> supported file format is Barracuda.
> Aug 28 11:50:14 emailserver mysqld: 130828 11:50:14  InnoDB: Waiting for
> the background threads to start
> Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 InnoDB: 5.5.31
> started; log sequence number 1211163767
> Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note] Recovering
> after a crash using /var/log/mysql/mysql-bin
> Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note] Starting crash
> recovery...
> Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note] Crash recovery
> finished.
> Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note] Server hostname
> (bind-address): '127.0.0.1'; port: 3306
> Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note]   - '127.0.0.1'
> resolves to '127.0.0.1';
> Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note] Server socket
> created on IP: '127.0.0.1'.
> Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note] Event
> Scheduler: Loaded 0 events
> Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15 [Note]
> /usr/sbin/mysqld: ready for connections.
> Aug 28 11:50:15 emailserver mysqld: Version: '5.5.31-1~dotdeb.0-log'
> socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Debian)
> Aug 28 11:50:15 emailserver mysqld: InnoDB: Dump of the tablespace extent
> descriptor:  len 40; hex
> 00000000000000000000000001de00000000016600000002aaaaaaaaaaaaaaaaeaaaeafaabaabbaa;
> asc                    f                    ;
> Aug 28 11:50:15 emailserver mysqld: InnoDB: Serious error! InnoDB is
> trying to free page 815
> Aug 28 11:50:15 emailserver mysqld: InnoDB: though it is already marked as
> free in the tablespace!
> Aug 28 11:50:15 emailserver mysqld: InnoDB: The tablespace free space info
> is corrupt.
> Aug 28 11:50:15 emailserver mysqld: InnoDB: You may need to dump your
> InnoDB tables and recreate the whole
> Aug 28 11:50:15 emailserver mysqld: InnoDB: database!
> Aug 28 11:50:15 emailserver mysqld: InnoDB: Please refer to
> Aug 28 11:50:15 emailserver mysqld: InnoDB:
> http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
> Aug 28 11:50:15 emailserver mysqld: InnoDB: about forcing recovery.
> Aug 28 11:50:15 emailserver mysqld: 130828 11:50:15  InnoDB: Assertion
> failure in thread 139642230933248 in file fsp0fsp.c line 3309
> Aug 28 11:50:15 emailserver mysqld: InnoDB: We intentionally generate a
> memory trap.
> Aug 28 11:50:15 emailserver mysqld: InnoDB: Submit a detailed bug report
> to http://bugs.mysql.com.
> Aug 28 11:50:15 emailserver mysqld: InnoDB: If you get repeated assertion
> failures or crashes, even
> Aug 28 11:50:15 emailserver mysqld: InnoDB: immediately after the mysqld
> startup, there may be
> Aug 28 11:50:15 emailserver mysqld: InnoDB: corruption in the InnoDB
> tablespace. Please refer to
> Aug 28 11:50:15 emailserver mysqld: InnoDB:
> http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
> Aug 28 11:50:15 emailserver mysqld: InnoDB: about forcing recovery.
> Aug 28 11:50:15 emailserver mysqld: 09:50:15 UTC - mysqld got signal 6 ;
> Aug 28 11:50:15 emailserver mysqld: This could be because you hit a bug.
> It is also possible that this binary
> Aug 28 11:50:15 emailserver mysqld: or one of the libraries it was linked
> against is corrupt, improperly built,
> Aug 28 11:50:15 emailserver mysqld: or misconfigured. This error can also
> be caused by malfunctioning hardware.
> Aug 28 11:50:15 emailserver mysqld: We will try our best to scrape up some
> info that will hopefully help
> Aug 28 11:50:15 emailserver mysqld: diagnose the problem, but since we
> have already crashed,
> Aug 28 11:50:15 emailserver mysqld: something is definitely wrong and this
> may fail.
> Aug 28 11:50:15 emailserver mysqld:
> Aug 28 11:50:15 emailserver mysqld: key_buffer_size=16384
> Aug 28 11:50:15 emailserver mysqld: read_buffer_size=262144
> Aug 28 11:50:15 emailserver mysqld: max_used_connections=0
> Aug 28 11:50:15 emailserver mysqld: max_threads=34
> Aug 28 11:50:15 emailserver mysqld: thread_count=0
> Aug 28 11:50:15 emailserver mysqld: connection_count=0
> Aug 28 11:50:15 emailserver mysqld: It is possible that mysqld could use
> up to
> Aug 28 11:50:15 emailserver mysqld: key_buffer_size + (read_buffer_size +
> sort_buffer_size)*max_threads = 11283 K  bytes of memory
> Aug 28 11:50:15 emailserver mysqld: Hope that's ok; if not, decrease some
> variables in the equation.
> Aug 28 11:50:15 emailserver mysqld:
> Aug 28 11:50:15 emailserver mysqld: Thread pointer: 0x0
> Aug 28 11:50:15 emailserver mysqld: Attempting backtrace. You can use the
> following information to find out
> Aug 28 11:50:15 emailserver mysqld: where mysqld died. If you see no
> messages after this, something went
> Aug 28 11:50:15 emailserver mysqld: terribly wrong...
> Aug 28 11:50:15 emailserver mysqld: stack_bottom = 0 thread_stack 0x20000
> Aug 28 11:50:15 emailserver mysqld:
> /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7f011094b14e]
> Aug 28 11:50:15 emailserver mysqld:
> /usr/sbin/mysqld(handle_fatal_signal+0x493)[0x7f0110802843]
> Aug 28 11:50:15 emailserver mysqld:
> /lib/libpthread.so.0(+0xeff0)[0x7f010ffbbff0]
> Aug 28 11:50:15 emailserver mysqld:
> /lib/libc.so.6(gsignal+0x35)[0x7f010e86d1b5]
> Aug 28 11:50:15 emailserver mysqld:
> /lib/libc.so.6(abort+0x180)[0x7f010e86ffc0]
> Aug 28 11:50:15 emailserver mysqld:
> /usr/sbin/mysqld(+0x682405)[0x7f0110a6b405]
> Aug 28 11:50:15 emailserver mysqld:
> /usr/sbin/mysqld(+0x682735)[0x7f0110a6b735]
> Aug 28 11:50:15 emailserver mysqld:
> /usr/sbin/mysqld(+0x6394b9)[0x7f0110a224b9]
> Aug 28 11:50:15 emailserver mysqld:
> /usr/sbin/mysqld(+0x6d1b0f)[0x7f0110abab0f]
> Aug 28 11:50:15 emailserver mysqld:
> /usr/sbin/mysqld(+0x6d2438)[0x7f0110abb438]
> Aug 28 11:50:15 emailserver mysqld:
> /usr/sbin/mysqld(+0x6c8255)[0x7f0110ab1255]
> Aug 28 11:50:15 emailserver mysqld:
> /usr/sbin/mysqld(+0x613435)[0x7f01109fc435]
> Aug 28 11:50:15 emailserver mysqld:
> /usr/sbin/mysqld(+0x60656c)[0x7f01109ef56c]
> Aug 28 11:50:15 emailserver mysqld:
> /usr/sbin/mysqld(+0x6092a3)[0x7f01109f22a3]
> Aug 28 11:50:15 emailserver mysqld:
> /lib/libpthread.so.0(+0x68ca)[0x7f010ffb38ca]
> Aug 28 11:50:15 emailserver mysqld:
> /lib/libc.so.6(clone+0x6d)[0x7f010e90ab6d]
> Aug 28 11:50:15 emailserver mysqld: The manual page at
> http://dev.mysql.com/doc/mysql/en/crashing.html contains
> Aug 28 11:50:15 emailserver mysqld: information that should help you find
> out what is causing the crash.
> Aug 28 11:50:15 emailserver mysqld_safe: Number of processes running now: 0
> Aug 28 11:50:15 emailserver mysqld_safe: mysqld restarted
> Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 [Warning] option
> 'thread_stack': unsigned value 65536 adjusted to 131072
> Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 [Warning] The syntax
> '--log-slow-queries' is deprecated and will be removed in a future release.
> Please use '--slow-query-log'/'--slow-query-log-file' instead.
> Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 [Note] Plugin
> 'FEDERATED' is disabled.
> Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: The InnoDB
> memory heap is disabled
> Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: Mutexes and
> rw_locks use GCC atomic builtins
> Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: Compressed
> tables use zlib 1.2.3.4
> Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: Using Linux
> native AIO
> Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: Initializing
> buffer pool, size = 128.0M
> Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: Completed
> initialization of buffer pool
> Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16 InnoDB: highest
> supported file format is Barracuda.
> Aug 28 11:50:16 emailserver mysqld: 130828 11:50:16  InnoDB: Waiting for
> the background threads to start
> Aug 28 11:50:16 emailserver mysqld_safe: mysqld from pid file
> /var/run/mysqld/mysqld.pid ended
> Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 InnoDB: 5.5.31
> started; log sequence number 1211163767
> Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note] Recovering
> after a crash using /var/log/mysql/mysql-bin
> Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note] Starting crash
> recovery...
> Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note] Crash recovery
> finished.
> Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note] Server hostname
> (bind-address): '127.0.0.1'; port: 3306
> Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note]   - '127.0.0.1'
> resolves to '127.0.0.1';
> Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note] Server socket
> created on IP: '127.0.0.1'.
> Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note] Event
> Scheduler: Loaded 0 events
> Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17 [Note]
> /usr/sbin/mysqld: ready for connections.
> Aug 28 11:50:17 emailserver mysqld: Version: '5.5.31-1~dotdeb.0-log'
> socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Debian)
> Aug 28 11:50:17 emailserver mysqld: InnoDB: Dump of the tablespace extent
> descriptor:  len 40; hex
> 00000000000000000000000001de00000000016600000002aaaaaaaaaaaaaaaaeaaaeafaabaabbaa;
> asc                    f                    ;
> Aug 28 11:50:17 emailserver mysqld: InnoDB: Serious error! InnoDB is
> trying to free page 815
> Aug 28 11:50:17 emailserver mysqld: InnoDB: though it is already marked as
> free in the tablespace!
> Aug 28 11:50:17 emailserver mysqld: InnoDB: The tablespace free space info
> is corrupt.
> Aug 28 11:50:17 emailserver mysqld: InnoDB: You may need to dump your
> InnoDB tables and recreate the whole
> Aug 28 11:50:17 emailserver mysqld: InnoDB: database!
> Aug 28 11:50:17 emailserver mysqld: InnoDB: Please refer to
> Aug 28 11:50:17 emailserver mysqld: InnoDB:
> http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
> Aug 28 11:50:17 emailserver mysqld: InnoDB: about forcing recovery.
> Aug 28 11:50:17 emailserver mysqld: 130828 11:50:17  InnoDB: Assertion
> failure in thread 140076906338048 in file fsp0fsp.c line 3309
> Aug 28 11:50:17 emailserver mysqld: InnoDB: We intentionally generate a
> memory trap.
> Aug 28 11:50:17 emailserver mysqld: InnoDB: Submit a detailed bug report
> to http://bugs.mysql.com.
> Aug 28 11:50:17 emailserver mysqld: InnoDB: If you get repeated assertion
> failures or crashes, even
> Aug 28 11:50:17 emailserver mysqld: InnoDB: immediately after the mysqld
> startup, there may be
> Aug 28 11:50:17 emailserver mysqld: InnoDB: corruption in the InnoDB
> tablespace. Please refer to
> Aug 28 11:50:17 emailserver mysqld: InnoDB:
> http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
> Aug 28 11:50:17 emailserver mysqld: InnoDB: about forcing recovery.
> Aug 28 11:50:17 emailserver mysqld: 09:50:17 UTC - mysqld got signal 6 ;
> Aug 28 11:50:17 emailserver mysqld: This could be because you hit a bug.
> It is also possible that this binary
> Aug 28 11:50:17 emailserver mysqld: or one of the libraries it was linked
> against is corrupt, improperly built,
> Aug 28 11:50:17 emailserver mysqld: or misconfigured. This error can also
> be caused by malfunctioning hardware.
> Aug 28 11:50:17 emailserver mysqld: We will try our best to scrape up some
> info that will hopefully help
> Aug 28 11:50:17 emailserver mysqld: diagnose the problem, but since we
> have already crashed,
> Aug 28 11:50:17 emailserver mysqld: something is definitely wrong and this
> may fail.
> Aug 28 11:50:17 emailserver mysqld:
> Aug 28 11:50:17 emailserver mysqld: key_buffer_size=16384
> Aug 28 11:50:17 emailserver mysqld: read_buffer_size=262144
> Aug 28 11:50:17 emailserver mysqld: max_used_connections=0
> Aug 28 11:50:17 emailserver mysqld: max_threads=34
> Aug 28 11:50:17 emailserver mysqld: thread_count=0
> Aug 28 11:50:17 emailserver mysqld: connection_count=0
> Aug 28 11:50:17 emailserver mysqld: It is possible that mysqld could use
> up to
> Aug 28 11:50:17 emailserver mysqld: key_buffer_size + (read_buffer_size +
> sort_buffer_size)*max_threads = 11283 K  bytes of memory
> Aug 28 11:50:17 emailserver mysqld: Hope that's ok; if not, decrease some
> variables in the equation.
> Aug 28 11:50:17 emailserver mysqld:
> Aug 28 11:50:17 emailserver mysqld: Thread pointer: 0x0
> Aug 28 11:50:17 emailserver mysqld: Attempting backtrace. You can use the
> following information to find out
> Aug 28 11:50:17 emailserver mysqld: where mysqld died. If you see no
> messages after this, something went
> Aug 28 11:50:17 emailserver mysqld: terribly wrong...
> Aug 28 11:50:17 emailserver mysqld: stack_bottom = 0 thread_stack 0x20000
> Aug 28 11:50:17 emailserver mysqld:
> /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7f664541014e]
> Aug 28 11:50:17 emailserver mysqld:
> /usr/sbin/mysqld(handle_fatal_signal+0x493)[0x7f66452c7843]
> Aug 28 11:50:17 emailserver mysqld:
> /lib/libpthread.so.0(+0xeff0)[0x7f6644a80ff0]
> Aug 28 11:50:17 emailserver mysqld:
> /lib/libc.so.6(gsignal+0x35)[0x7f66433321b5]
> Aug 28 11:50:17 emailserver mysqld:
> /lib/libc.so.6(abort+0x180)[0x7f6643334fc0]
> Aug 28 11:50:17 emailserver mysqld:
> /usr/sbin/mysqld(+0x682405)[0x7f6645530405]
> Aug 28 11:50:17 emailserver mysqld:
> /usr/sbin/mysqld(+0x682735)[0x7f6645530735]
> Aug 28 11:50:17 emailserver mysqld:
> /usr/sbin/mysqld(+0x6394b9)[0x7f66454e74b9]
> Aug 28 11:50:17 emailserver mysqld:
> /usr/sbin/mysqld(+0x6d1b0f)[0x7f664557fb0f]
> Aug 28 11:50:17 emailserver mysqld:
> /usr/sbin/mysqld(+0x6d2438)[0x7f6645580438]
> Aug 28 11:50:17 emailserver mysqld:
> /usr/sbin/mysqld(+0x6c8255)[0x7f6645576255]
> Aug 28 11:50:17 emailserver mysqld:
> /usr/sbin/mysqld(+0x613435)[0x7f66454c1435]
> Aug 28 11:50:17 emailserver mysqld:
> /usr/sbin/mysqld(+0x60656c)[0x7f66454b456c]
> Aug 28 11:50:17 emailserver mysqld:
> /usr/sbin/mysqld(+0x6092a3)[0x7f66454b72a3]
> Aug 28 11:50:17 emailserver mysqld:
> /lib/libpthread.so.0(+0x68ca)[0x7f6644a788ca]
> Aug 28 11:50:17 emailserver mysqld:
> /lib/libc.so.6(clone+0x6d)[0x7f66433cfb6d]
> Aug 28 11:50:17 emailserver mysqld: The manual page at
> http://dev.mysql.com/doc/mysql/en/crashing.html contains
> Aug 28 11:50:17 emailserver mysqld: information that should help you find
> out what is causing the crash.
>
>
>
>
>


-- 
--
GSM: +31.6.2000.5427

Thread
mysql 5.5 crashed on Debian 6 after server crashSimon Loewenthal28 Aug
  • Re: mysql 5.5 crashed on Debian 6 after server crashSimon Loewenthal28 Aug