I posted this problem in MySQL forum InnoDB but no one replied (
http://forums.mysql.com/read.php?22,102901,102901#msg-102901). So I want to
find help from this mail list. Thank you for reading it. Any suggestions
will be appreciated!
I will try to describe the problem again:
There are 3 mysql servers in my distributed system. My application will use
mysql's distributed transaction functionality (XA support). When the
application running, it will generate some distributed transactions in
different threads (all the transactions will operate in this mode: select
from a table, update or delete it, insert into another table).
Unfortunately, after I kill the application when it running, at least one of
mysql servers will crash when I run the application next time.
The output of resolve_stack_dump is like following:
======= stack dump begin =======
thd=0xb018848
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x6c42e988, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x81917ac handle_segfault + 668
0xffffe420 _end + -140066672
0x1 (?)
0x8214618 _Z18mysql_stmt_executeP3THDPcj + 136
0x81b64e9 _Z16dispatch_command19enum_server_commandP3THDPcj + 3433
0x81b79fc handle_one_connection + 2380
0xb7f3fced _end + -1348806307
0xb7d7cdde _end + -1350653362
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read
[dev.mysql.com<http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html>]
and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at (nil) is invalid pointer
thd->thread_id=469
The manual page at [www.mysql.com<http://www.mysql.com/doc/en/Crashing.html>]
contains
information that should help you find out what is causing the crash.
======= stack dump end =======
The version of mysql is 5.1.11. I compiled it by myself. The OS is GNU/Linux
2.6.15-1-686-smp. The compiler and libraries are gcc (GCC) 4.0.3 (Debian
4.0.3-1), libc6(2.3.6-3), libg++2.8.1.3-glibc2.2. The table engine I used is
InnoDB and one of the tables is partitioned by range of a column.
I found that following steps will cause mysql server(s) crash:
1. start mysql servers;
2. run my application (multi-threaded, distributed transactions);
3. kill the application;
4. run the application again;
5. ... (some or all of the mysql servers will crash)
But it won't crash if I operate in this way:
1. start mysql servers;
2. run my application (multi-threaded, distributed transactions);
3. kill the application;
4. restart mysql servers manually (shutdown them and start again);
5. run the application again;
6. ... (the application will run smoothly)
It seems that the data and log files are ok. However, the server runs into
an inconsistent status in memory after some connections (with distributed
transactions) aborted.
Another time, all of the three servers crashed. Maybe you can get more hints
from following output of resolve_stack_dump:
======= stack dump begin =======
=== mysql 1 ===
thd=0x6f0ec920
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x6c8de918, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x81917ac handle_segfault + 668
0xffffe420 _end + -140066672
0xf6e6ec62 _end + -292697902
0x83d1709 trx_commit_off_kernel + 633
0x83c8d78 trx_finish_rollback_off_kernel + 88
0x838d986 que_run_threads + 1750
0x83c9f9b trx_general_rollback_for_mysql + 251
0x83ca420 trx_rollback_for_mysql + 48
0x8277317 _Z17innobase_rollbackP3THDb + 119
0x8263ec8 _Z17ha_rollback_transP3THDb + 280
0x818300b _ZN3THD7cleanupEv + 27
0x8191256 _Z10end_threadP3THDb + 22
0x81b73a8 handle_one_connection + 760
0xb7f25ced _end + -1348912803
0xb7d62dde _end + -1350759858
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read
[dev.mysql.com<http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html>]
and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at (nil) is invalid pointer
thd->thread_id=418
The manual page at [www.mysql.com<http://www.mysql.com/doc/en/Crashing.html>]
contains
information that should help you find out what is causing the crash.
=== mysql 2 ===
thd=0x6f9a1458
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x6fd7d018, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x81917ac handle_segfault + 668
0xffffe420 _end + -140066672
0x701083e0 _end + 1740070992
0x818c72c _Z17mysql_lock_tablesP3THDPP8st_tablejjPb + 76
0x81e02e0 _Z11lock_tablesP3THDP13st_table_listjPb + 592
0x81e1e31 _Z20open_and_lock_tablesP3THDP13st_table_list + 49
0x8213bd7 _ZN18Prepared_statement7prepareEPKcj + 3319
0x82144ed _Z18mysql_stmt_prepareP3THDPKcj + 205
0x81b64c9 _Z16dispatch_command19enum_server_commandP3THDPcj + 3401
0x81b79fc handle_one_connection + 2380
0xb7f8eced _end + -1348482723
0xb7dcbdee _end + -1350329762
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read
[dev.mysql.com<http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html>]
and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x70104498 is invalid pointer
thd->thread_id=287
The manual page at [www.mysql.com<http://www.mysql.com/doc/en/Crashing.html>]
contains
information that should help you find out what is causing the crash.
=== mysql 3 ===
thd=0x8c0a5228
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x87e6197c, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x818711d handle_segfault + 685
0xffffe420 _end + -139825680
(nil)
0x81f9e5c _Z18mysql_stmt_executeP3THDPcj + 332
0x819d999 _Z16dispatch_command19enum_server_commandP3THDPcj + 5129
0x819c53d _Z10do_commandP3THD + 141
0x819b8b2 handle_one_connection + 482
0x40036b63 _end + 934147379
0x4024018a _end + 936282970
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read
[dev.mysql.com<http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html>]
and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x9e3b348 = SELECT col_a, ifnull(col_b,0) FROM tab_a WHERE
(col_c = 'some_value') AND (col_d = 11) FOR UPDATE
thd->thread_id=329
The manual page at [www.mysql.com<http://www.mysql.com/doc/en/Crashing.html>]
contains
information that should help you find out what is causing the crash.
======= stack dump end =======
After above crashing (all of 3 mysql servers crashed), I restarted mysql
servers manually, ran the application again and it works ok. Next, I killed
the application and ran it once more. This time, it caused one of mysql
servers crashed.
Output of resolve_stack_dump of that crashed mysql:
======= stack dump begin =======
thd=0x681a0bb0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x6f062ac8, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x81917ac handle_segfault + 668
0xffffe420 _end + -140066672
0xfbe9befe _end + -208626834
0x83d1709 trx_commit_off_kernel + 633
0x83d2b8c trx_commit_for_mysql + 108
0x827a679 _Z15innobase_commitP3THDb + 217
0x82661e2 _Z19ha_commit_one_phaseP3THDb + 114
0x81b4992 _Z21mysql_execute_commandP3THD + 42354
0x81b5641 _Z11mysql_parseP3THDPcj + 593
0x81b5c64 _Z16dispatch_command19enum_server_commandP3THDPcj + 1252
0x81b79fc handle_one_connection + 2380
0xb7f02ced _end + -1349056163
0xb7d3fdee _end + -1350903202
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read
[dev.mysql.com<http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html>]
and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0xb301e90 = XA COMMIT
0x3131313830383131383639303130363136,0x31,0x9913
thd->thread_id=405
The manual page at [www.mysql.com<http://www.mysql.com/doc/en/Crashing.html>]
contains
information that should help you find out what is causing the crash.
pure virtual method called
pure virtual method called
pure virtual method called
pure virtual method called
pure virtual method called
pure virtual method called
pure virtual method called
pure virtual method called
pure virtual method called
======= stack dump end =======
I'm expecting for any reply. Thanks a lot!
