List:Falcon Storage Engine« Previous MessageNext Message »
From:Hakan Kuecuekyilmaz Date:January 26 2009 2:48pm
Subject:Up merge is delayed due to Pushbuild failures
View as plain text  
Team,

we have following situation in PB1, which stops us from merging up to
main.

PB1 failures

sapsrv1
  ** falcon.index_merge_falcon
  Looks like disk is full
  090123 19:39:02 [ERROR] Error writing file
'/dev/shm/var-falcon-110/log/master.log' (errno: 28)
  090123 19:39:02 [ERROR] Failed to write to mysql.general_log: Error
writing file './mysql/general_log.CSV' (Errcode: 28)

pb-valgrind
  ** falcon.falcon_bug_23818_II Invalid read of size 8
  ** falcon.falcon_bug_25537 Invalid read of size 8
  Also
  ** '32,000 bytes in 2 blocks are indirectly lost in loss record 5 of
5'
  ** '32,176 (176 direct, 32,000 indirect) bytes in 1 blocks are
definitely lost in loss record 4 of 5'


The crashes in the Valgrind runs look the same as in
http://bugs.mysql.com/bug.php?id=42342

Stacktrace from Valgrind crash

==29948== Thread 24:
==29948== Invalid read of size 8
==29948==    at 0x990BED: Index::duplicateKey(IndexKey*, Record*)
(Index.cpp:688)
==29948==    by 0x990EF2: Index::update(Record*, Record*, Transaction*)
(Index.cpp:608)
==29948==    by 0x94A0D6: Table::updateIndexes(Transaction*,
RecordVersion*, Record*) (Table.cpp:1246)
==29948==    by 0x94A26B: Table::update(Transaction*, Record*, Stream*)
(Table.cpp:3179)
==29948==    by 0x938045: StorageDatabase::updateRow(StorageConnection*,
Table*, Record*, Stream*) (StorageDatabase.cpp:674)
==29948==    by 0x93D25E: StorageTable::updateRow(int)
(StorageTable.cpp:132)
==29948==    by 0x932A59: StorageInterface::update_row(unsigned char
const*, unsigned char*) (ha_falcon.cpp:1214)
==29948==    by 0x7D92D1: handler::ha_update_row(unsigned char const*,
unsigned char*) (handler.cc:5463)
==29948==    by 0x765C09: mysql_update(THD*, TABLE_LIST*, List<Item>&,
List<Item>&, Item*, unsigned, st_order*, unsigned long long,
enum_duplicates, bool) (sql_update.cc:649)
==29948==    by 0x6C13FD: mysql_execute_command(THD*)
(sql_parse.cc:3005)
==29948==    by 0x758D9B: Prepared_statement::execute(String*, bool)
(sql_prepare.cc:3756)
==29948==    by 0x75D938: Prepared_statement::execute_loop(String*,
bool, unsigned char*, unsigned char*) (sql_prepare.cc:3386)
==29948==    by 0x75DF48: mysql_sql_stmt_execute(THD*)
(sql_prepare.cc:2591)
==29948==    by 0x6BCF01: mysql_execute_command(THD*)
(sql_parse.cc:2072)
==29948==    by 0x85791B: sp_instr_stmt::exec_core(THD*, unsigned*)
(sp_head.cc:2920)
==29948==    by 0x85AFF5: sp_lex_keeper::reset_lex_and_exec_core(THD*,
unsigned*, bool, sp_instr*) (sp_head.cc:2744)
==29948==  Address 0x70 is not stack'd, malloc'd or (recently) free'd
{
   <insert a suppression name here>
   Memcheck:Addr8
   fun:Index::duplicateKey(IndexKey*, Record*)
   fun:Index::update(Record*, Record*, Transaction*)
   fun:Table::updateIndexes(Transaction*, RecordVersion*, Record*)
   fun:Table::update(Transaction*, Record*, Stream*)
   fun:StorageDatabase::updateRow(StorageConnection*, Table*, Record*,
Stream*)
   fun:StorageTable::updateRow(int)
   fun:StorageInterface::update_row(unsigned char const*, unsigned
char*)
   fun:handler::ha_update_row(unsigned char const*, unsigned char*)
   fun:mysql_update(THD*, TABLE_LIST*, List<Item>&, List<Item>&,
Item*,
unsigned int, st_order*, unsigned long long, enum_duplicates, bool)
   fun:mysql_execute_command(THD*)
   fun:Prepared_statement::execute(String*, bool)
   fun:Prepared_statement::execute_loop(String*, bool, unsigned char*,
unsigned char*)
   fun:mysql_sql_stmt_execute(THD*)
   fun:mysql_execute_command(THD*)
   fun:sp_instr_stmt::exec_core(THD*, unsigned int*)
   fun:sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool,
sp_instr*)
}
/data0/pushbuild/pb1/pb/bzr_mysql-6.0-falcon/131/mysql-6.0.10-alpha-pb131/sql/mysqld(my_print_stacktrace+0x33)[0xc1ce33]
/data0/pushbuild/pb1/pb/bzr_mysql-6.0-falcon/131/mysql-6.0.10-alpha-pb131/sql/mysqld(handle_segfault+0x33b)[0x6ac1eb]
/lib64/libpthread.so.0[0x4c3a140]
/data0/pushbuild/pb1/pb/bzr_mysql-6.0-falcon/131/mysql-6.0.10-alpha-pb131/sql/mysqld(Index::duplicateKey(IndexKey*,
Record*)+0x5d)[0x990bed]
/data0/pushbuild/pb1/pb/bzr_mysql-6.0-falcon/131/mysql-6.0.10-alpha-pb131/sql/mysqld(Index::update(Record*,
Record*, Transaction*)+0x53)[0x990ef3]
/data0/pushbuild/pb1/pb/bzr_mysql-6.0-falcon/131/mysql-6.0.10-alpha-pb131/sql/mysqld(Table::updateIndexes(Transaction*,
RecordVersion*, Record*)+0x67)[0x94a0d7]
/data0/pushbuild/pb1/pb/bzr_mysql-6.0-falcon/131/mysql-6.0.10-alpha-pb131/sql/mysqld(Table::update(Transaction*,
Record*, Stream*)+0x11c)[0x94a26c]
/data0/pushbuild/pb1/pb/bzr_mysql-6.0-falcon/131/mysql-6.0.10-alpha-pb131/sql/mysqld(StorageDatabase::updateRow(StorageConnection*,
Table*, Record*, Stream*)+0x36)[0x938046]
/data0/pushbuild/pb1/pb/bzr_mysql-6.0-falcon/131/mysql-6.0.10-alpha-pb131/sql/mysqld(StorageTable::updateRow(int)+0x4f)[0x93d25f]
/data0/pushbuild/pb1/pb/bzr_mysql-6.0-falcon/131/mysql-6.0.10-alpha-pb131/sql/mysqld(StorageInterface::update_row(unsigned
char const*, unsigned char*)+0xba)[0x932a5a]
/data0/pushbuild/pb1/pb/bzr_mysql-6.0-falcon/131/mysql-6.0.10-alpha-pb131/sql/mysqld(handler::ha_update_row(unsigned
char const*, unsigned char*)+0xc2)[0x7d92d2]
/data0/pushbuild/pb1/pb/bzr_mysql-6.0-falcon/131/mysql-6.0.10-alpha-pb131/sql/mysqld(mysql_update(THD*,
TABLE_LIST*, List<Item>&, List<Item>&, Item*, unsigned int, st_order*,
unsigned long long, enum_duplicates, bool)+0xf1a)[0x765c0a]
/data0/pushbuild/pb1/pb/bzr_mysql-6.0-falcon/131/mysql-6.0.10-alpha-pb131/sql/mysqld(mysql_execute_command(THD*)+0x511e)[0x6c13fe]
/data0/pushbuild/pb1/pb/bzr_mysql-6.0-falcon/131/mysql-6.0.10-alpha-pb131/sql/mysqld(Prepared_statement::execute(String*,
bool)+0x42c)[0x758d9c]
/data0/pushbuild/pb1/pb/bzr_mysql-6.0-falcon/131/mysql-6.0.10-alpha-pb131/sql/mysqld(Prepared_statement::execute_loop(String*,
bool, unsigned char*, unsigned char*)+0x89)[0x75d939]
/data0/pushbuild/pb1/pb/bzr_mysql-6.0-falcon/131/mysql-6.0.10-alpha-pb131/sql/mysqld(mysql_sql_stmt_execute(THD*)+0x149)[0x75df49]
/data0/pushbuild/pb1/pb/bzr_mysql-6.0-falcon/131/mysql-6.0.10-alpha-pb131/sql/mysqld(mysql_execute_command(THD*)+0xc22)[0x6bcf02]
/data0/pushbuild/pb1/pb/bzr_mysql-6.0-falcon/131/mysql-6.0.10-alpha-pb131/sql/mysqld(sp_instr_stmt::exec_core(THD*,
unsigned int*)+0x1c)[0x85791c]
/data0/pushbuild/pb1/pb/bzr_mysql-6.0-falcon/131/mysql-6.0.10-alpha-pb131/sql/mysqld(sp_lex_keeper::reset_lex_and_exec_core(THD*,
unsigned int*, bool, sp_instr*)+0xf6)[0x85aff6]
/data0/pushbuild/pb1/pb/bzr_mysql-6.0-falcon/131/mysql-6.0.10-alpha-pb131/sql/mysqld(sp_instr_stmt::execute(THD*,
unsigned int*)+0x6a9)[0x85e109]
/data0/pushbuild/pb1/pb/bzr_mysql-6.0-falcon/131/mysql-6.0.10-alpha-pb131/sql/mysqld(sp_head::execute(THD*)+0x5ec)[0x85c1ec]
/data0/pushbuild/pb1/pb/bzr_mysql-6.0-falcon/131/mysql-6.0.10-alpha-pb131/sql/mysqld(sp_head::execute_procedure(THD*,
List<Item>*)+0x473)[0x85eb53]
/data0/pushbuild/pb1/pb/bzr_mysql-6.0-falcon/131/mysql-6.0.10-alpha-pb131/sql/mysqld(mysql_execute_command(THD*)+0x9a8)[0x6bcc88]
/data0/pushbuild/pb1/pb/bzr_mysql-6.0-falcon/131/mysql-6.0.10-alpha-pb131/sql/mysqld(mysql_parse(THD*,
char const*, unsigned int, char const**)+0x1a3)[0x6c2c53]
/data0/pushbuild/pb1/pb/bzr_mysql-6.0-falcon/131/mysql-6.0.10-alpha-pb131/sql/mysqld(dispatch_command(enum_server_command,
THD*, char*, unsigned int)+0x9f8)[0x6c3788]
/data0/pushbuild/pb1/pb/bzr_mysql-6.0-falcon/131/mysql-6.0.10-alpha-pb131/sql/mysqld(do_command(THD*)+0x163)[0x6c4343]
/data0/pushbuild/pb1/pb/bzr_mysql-6.0-falcon/131/mysql-6.0.10-alpha-pb131/sql/mysqld(handle_one_connection+0xf4)[0x6b5094]
/lib64/libpthread.so.0[0x4c33193]
/lib64/libc.so.6(__clone+0x6d)[0x526547d]
090124 23:16:35 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this
binary
or one of the libraries it was linked against is corrupt, improperly
built,
or misconfigured. This error can also be caused by malfunctioning
hardware.
We will try our best to scrape up some info that will hopefully help
diagnose
the problem, but since we have already crashed, something is definitely
wrong
and this may fail.

key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=3
max_threads=151
thread_count=1
connection_count=1
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads =
60676 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x14c09b30
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x16a3a198 thread_stack 0x40000
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x11b39180 is an invalid pointer
thd->thread_id=41
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html
contains
information that should help you find out what is causing the crash.
Writing a core file

-- 
Hakan Küçükyılmaz, Senior Software Engineer DBTG/MySQL +49 160
98953296
Sun Microsystems GmbH     Sonnenallee 1, DE-85551 Kirchheim-Heimstetten
Geschaeftsfuehrer:  Thomas Schroeder, Wolfang Engels, Dr. Roland Boemer
Vorsitz d. Aufs.rat.: Martin Haering   HRB MUC 161028     49.011, 8.376

Thread
Up merge is delayed due to Pushbuild failuresHakan Kuecuekyilmaz26 Jan