Hi Daogang,
Great work,
STATUS
------
Conditionally approved.
REQUEST
------
1 - We are not writing directly to the index_file so I think
we can replace the init_io_cache by a simple seek.
2 - Rename rpl_crashed_master.test to rpl_crash_safe_master.test
3 - Rename rpl_*_crash_safe.test to rpl_*_crash_safe_slave.test
4 - Create a version with checksum and without checksum, maybe
rpl_crash_safe_master_checksum.test and rpl_crash_safe_master.test
respectively.
5. I still don't understand what fdle->flags&=
~LOG_EVENT_BINLOG_IN_USE_F is for. Please, add comments to the code
int MYSQL_BIN_LOG::recover(IO_CACHE *log, Format_description_log_event
*fdle,
my_off_t *valid_pos)
{
fdle->flags&= ~LOG_EVENT_BINLOG_IN_USE_F; // abort on the first error
6 - See more comments in-line
Cheers.
On 12/02/2010 02:56 AM, Dao-Gang.Qu@stripped wrote:
> #At file:///home/daogang/bzrwork/wl5493/mysql-trunk/ based on
> revid:alexander.nozdrin@stripped
>
> 3208 Dao-Gang.Qu@stripped 2010-12-02
> WL#5493 Binlog crash-safe when master crashed
>
> Trim the crashed binlog file to last valid transaction or event
> (non-transaction) base on binlog size(valid_pos) when MYSQL server
> crashed in the middle of binlog. And add a temp file to guarantee
> the binlog index file to be crash safe.
>
> WL#5440 Test binlog and replication when master crashed
>
> Test whether the transaction is binlogged partly and the replication
> works fine or not when the master crashed.
> @ mysql-test/r/crash_commit_before.result
> Removed the result file. Because its test file is removed.
> @ mysql-test/suite/binlog/r/binlog_index.result
> Test result for WL#5493
> @ mysql-test/suite/binlog/t/binlog_delete_and_flush_index.test
> Updated for WL#5493.
> @ mysql-test/suite/binlog/t/binlog_index.test
> Added Test cases to verify that the index file will be crash safe.
> @ mysql-test/suite/rpl/r/rpl_crashed_master.result
> Test result for WL#5493 and WL#5440
> @ mysql-test/suite/rpl/t/rpl_crashed_master.test
> Added test to verify if the crashed binlog file is trimmed to
> last valid transaction or event(non-transaction) correctly.
> And if the binlog index is crash safe.
> @ mysql-test/t/crash_commit_before-master.opt
> Removed the opt file. Because its test file is removed.
> @ mysql-test/t/crash_commit_before.test
> Removed the test file. Because the test case is tested
> in rpl_crashed_master.test file.
> @ mysys/my_rename.c
> Cleared obsolete code. Because all non-windows platforms will support
> the "rename()" POSIX interface.
> @ sql/binlog.cc
> Added code to trim the crashed binlog file to last valid transaction
> or event(non-transaction) base on valid_pos, make binlog index to be
> crash safe.
> @ sql/binlog.h
> Added code to define these functions for binlog index crash safe.
> @ sql/log_event.cc
> Updated code to remove sql_print_error() line, because its
> direct caller will invoke sql_print_error() to print error.
>
> removed:
> mysql-test/r/crash_commit_before.result
> mysql-test/t/crash_commit_before-master.opt
> mysql-test/t/crash_commit_before.test
> added:
> mysql-test/suite/rpl/r/rpl_crashed_master.result
> mysql-test/suite/rpl/t/rpl_crashed_master-master.opt
> mysql-test/suite/rpl/t/rpl_crashed_master.test
> modified:
> mysql-test/suite/binlog/r/binlog_index.result
> mysql-test/suite/binlog/t/binlog_delete_and_flush_index.test
> mysql-test/suite/binlog/t/binlog_index.test
> mysys/my_rename.c
> sql/binlog.cc
> sql/binlog.h
> sql/log_event.cc
> === removed file 'mysql-test/r/crash_commit_before.result'
> --- a/mysql-test/r/crash_commit_before.result 2007-04-03 09:36:33 +0000
> +++ b/mysql-test/r/crash_commit_before.result 1970-01-01 00:00:00 +0000
> @@ -1,14 +0,0 @@
> -CREATE TABLE t1(a int) engine=innodb;
> -START TRANSACTION;
> -insert into t1 values(9);
> -SET SESSION debug="d,crash_commit_before";
> -COMMIT;
> -ERROR HY000: Lost connection to MySQL server during query
> -SHOW CREATE TABLE t1;
> -Table Create Table
> -t1 CREATE TABLE `t1` (
> - `a` int(11) DEFAULT NULL
> -) ENGINE=InnoDB DEFAULT CHARSET=latin1
> -SELECT * FROM t1;
> -a
> -DROP TABLE t1;
ok.
>
> === modified file 'mysql-test/suite/binlog/r/binlog_index.result'
> --- a/mysql-test/suite/binlog/r/binlog_index.result 2009-12-17 17:10:18 +0000
> +++ b/mysql-test/suite/binlog/r/binlog_index.result 2010-12-02 02:56:34 +0000
> @@ -147,5 +147,180 @@ master-bin.000011
> master-bin.000012
> master-bin.000013
>
> +# Test case6: Set DEBUG POINT before rename index file to make the master
> +# crash when appending a binlog file name to index file.
> +show binary logs;
> +Log_name File_size
> +master-bin.000006 #
> +master-bin.000007 #
> +master-bin.000008 #
> +master-bin.000009 #
> +master-bin.000010 #
> +master-bin.000011 #
> +master-bin.000012 #
> +master-bin.000013 #
> +SET SESSION debug="+d,crash_create_before_rename_index_file";
> +flush logs;
> +ERROR HY000: Lost connection to MySQL server during query
> +# Restart the master server
> +# Test the index file is complete and binlog file name is added.
> +show binary logs;
> +Log_name File_size
> +master-bin.000006 #
> +master-bin.000007 #
> +master-bin.000008 #
> +master-bin.000009 #
> +master-bin.000010 #
> +master-bin.000011 #
> +master-bin.000012 #
> +master-bin.000013 #
> +master-bin.000014 #
> +master-bin.000015 #
> +# Test case7: Set DEBUG POINT after rename index file to make the master
> +# crash when appending a binlog file name to index file.
> +SET SESSION debug="+d,crash_create_after_rename_index_file";
> +flush logs;
> +ERROR HY000: Lost connection to MySQL server during query
> +# Restart the master server
> +# Test the index file is complete and binlog file name is added.
> +show binary logs;
> +Log_name File_size
> +master-bin.000006 #
> +master-bin.000007 #
> +master-bin.000008 #
> +master-bin.000009 #
> +master-bin.000010 #
> +master-bin.000011 #
> +master-bin.000012 #
> +master-bin.000013 #
> +master-bin.000014 #
> +master-bin.000015 #
> +master-bin.000016 #
> +master-bin.000017 #
> +show binary logs;
> +Log_name File_size
> +master-bin.000006 #
> +master-bin.000007 #
> +master-bin.000008 #
> +master-bin.000009 #
> +master-bin.000010 #
> +master-bin.000011 #
> +master-bin.000012 #
> +master-bin.000013 #
> +master-bin.000014 #
> +master-bin.000015 #
> +master-bin.000016 #
> +master-bin.000017 #
> +SET SESSION debug="+d,crash_create_after_rename_index_file";
> +purge binary logs TO 'master-bin.000008';
> +ERROR HY000: Lost connection to MySQL server during query
> +# Restart the master server
> +# Test the index file is complete and is purged successfully
> +show binary logs;
> +Log_name File_size
> +master-bin.000008 #
> +master-bin.000009 #
> +master-bin.000010 #
> +master-bin.000011 #
> +master-bin.000012 #
> +master-bin.000013 #
> +master-bin.000014 #
> +master-bin.000015 #
> +master-bin.000016 #
> +master-bin.000017 #
> +master-bin.000018 #
> +# Test case8: Set DEBUG POINT after rename index file to make the master
> +# crash when purging the index file.
> +show binary logs;
> +Log_name File_size
> +master-bin.000008 #
> +master-bin.000009 #
> +master-bin.000010 #
> +master-bin.000011 #
> +master-bin.000012 #
> +master-bin.000013 #
> +master-bin.000014 #
> +master-bin.000015 #
> +master-bin.000016 #
> +master-bin.000017 #
> +master-bin.000018 #
> +SET SESSION debug="+d,crash_create_after_rename_index_file";
> +purge binary logs TO 'master-bin.000010';
> +ERROR HY000: Lost connection to MySQL server during query
> +# Restart the master server
> +# Test the index file is complete, and is purged successfully.
> +show binary logs;
> +Log_name File_size
> +master-bin.000010 #
> +master-bin.000011 #
> +master-bin.000012 #
> +master-bin.000013 #
> +master-bin.000014 #
> +master-bin.000015 #
> +master-bin.000016 #
> +master-bin.000017 #
> +master-bin.000018 #
> +master-bin.000019 #
> +# Test case9: Set DEBUG POINT befor rename index file to make the master
> +# crash when purging the index file.
> +show binary logs;
> +Log_name File_size
> +master-bin.000010 #
> +master-bin.000011 #
> +master-bin.000012 #
> +master-bin.000013 #
> +master-bin.000014 #
> +master-bin.000015 #
> +master-bin.000016 #
> +master-bin.000017 #
> +master-bin.000018 #
> +master-bin.000019 #
> +SET SESSION debug="+d,crash_create_before_rename_index_file";
> +purge binary logs TO 'master-bin.000012';
> +ERROR HY000: Lost connection to MySQL server during query
> +# Restart the master server
> +# Test the index file is complete, and is purged successfully.
> +show binary logs;
> +Log_name File_size
> +master-bin.000012 #
> +master-bin.000013 #
> +master-bin.000014 #
> +master-bin.000015 #
> +master-bin.000016 #
> +master-bin.000017 #
> +master-bin.000018 #
> +master-bin.000019 #
> +master-bin.000020 #
> +# Test case10: Inject a fault to copy part content to the temp file
> +# and abort the server after the temp file is closed
> +# when purging the index file.
> +show binary logs;
> +Log_name File_size
> +master-bin.000012 #
> +master-bin.000013 #
> +master-bin.000014 #
> +master-bin.000015 #
> +master-bin.000016 #
> +master-bin.000017 #
> +master-bin.000018 #
> +master-bin.000019 #
> +master-bin.000020 #
> +SET SESSION debug="+d,fault_injection_copy_part_file";
> +purge binary logs TO 'master-bin.000014';
> +ERROR HY000: Lost connection to MySQL server during query
> +# Restart the master server
> +# Test the index file is complete, although is not purged successfully.
> +show binary logs;
> +Log_name File_size
> +master-bin.000012 #
> +master-bin.000013 #
> +master-bin.000014 #
> +master-bin.000015 #
> +master-bin.000016 #
> +master-bin.000017 #
> +master-bin.000018 #
> +master-bin.000019 #
> +master-bin.000020 #
> +master-bin.000021 #
> SET SESSION debug="";
> End of tests
ok.
>
> === modified file 'mysql-test/suite/binlog/t/binlog_delete_and_flush_index.test'
> --- a/mysql-test/suite/binlog/t/binlog_delete_and_flush_index.test 2009-10-20
> 08:39:40 +0000
> +++ b/mysql-test/suite/binlog/t/binlog_delete_and_flush_index.test 2010-12-02
> 02:56:34 +0000
> @@ -77,6 +77,7 @@ FLUSH LOGS;
> -- remove_file $datadir/master-bin.000001
>
> -- echo ### assertion: index file contains renamed binlog and the new one
> +-- chmod 0644 $index
> -- replace_result $MYSQLTEST_VARDIR MYSQLTEST_VARDIR
> -- eval SET @index=LOAD_FILE('$index')
> -- replace_regex /\.[\\\/]master/master/
> @@ -106,6 +107,7 @@ DROP TABLE t1;
> -- file_exists $datadir/$current_binlog
>
> -- echo ### assertion: show index file contents and these should match show binary
> logs issued above
> +-- chmod 0644 $index
> -- replace_result $MYSQLTEST_VARDIR MYSQLTEST_VARDIR
> -- eval SET @index=LOAD_FILE('$index')
> -- replace_regex /\.[\\\/]master/master/
>
Can you explain in the commit message why you need this chmod?
> === modified file 'mysql-test/suite/binlog/t/binlog_index.test'
> --- a/mysql-test/suite/binlog/t/binlog_index.test 2009-12-08 16:03:19 +0000
> +++ b/mysql-test/suite/binlog/t/binlog_index.test 2010-12-02 02:56:34 +0000
> @@ -230,6 +230,185 @@ flush logs;
> -- replace_regex /\.[\\\/]master/master/
> SELECT @index;
>
> +#
> +# WL#5493
> +# Test case6 verifies if the index file is complete and the binlog
> +# file name is added after the master restarts when setting
> +# DEBUG POINT before rename index file and appending binlog file name
> +# to index file to cause the master crash.
I would change this as follows
Test case6 verifies if the index file has the correct data, i.e. if
binlog file name is added after the master restarts when setting
DEBUG POINT before renaming index file.
> +#
> +# Test case7 verifies if the index file is complete and the binlog
> +# file name is added after the master restarts when setting
> +# DEBUG POINT after rename index file and appending binlog file name
> +# to index file to cause the master crash.
See above.
> +#
> +# Test case8 verifies if the index file is complete and is purged
> +# successfully after the master restarts when setting DEBUG POINT
> +# after rename index file and purging the index file to cause the
> +# master crash.
See above.
> +#
> +# Test case9 verifies if the index file is complete and is purged
> +# successfully after the master restarts when setting DEBUG POINT
> +# before rename index file and purging the index file to cause the
> +# master crash.
See above.
> +#
> +# Test case10 verifies if the index file is complete, although is
> +# not purged successfully after the master restarts when injecting
> +# a fault to the temp file and aborting the server after the temp
> +# file is closed in the process of purging the index file
> +#
See above.
> +
> +-- echo # Test case6: Set DEBUG POINT before rename index file to make the master
> +-- echo # crash when appending a binlog file name to index file.
Set DEBUG POINT before renaming index file to when appending a binlog
file name to index file.
> +-- source include/show_binary_logs.inc
> +file_exists $MYSQLD_DATADIR/master-bin.000013;
> +--error 1
> +file_exists $MYSQLD_DATADIR/master-bin.000014;
> +
> +# Write file to make mysql-test-run.pl expect crash and restart
> +SET SESSION debug="+d,crash_create_before_rename_index_file";
> +-- exec echo "restart"> $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
> +
> +--error 2013
> +flush logs;
> +
> +-- source include/wait_until_disconnected.inc
> +-- enable_reconnect
> +-- echo # Restart the master server
> +-- exec echo "restart"> $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
> +-- source include/wait_until_connected_again.inc
> +-- disable_reconnect
> +
> +-- echo # Test the index file is complete and binlog file name is added.
Test if the index file has the correct data, i.e. binlog file name is
added.
> +-- source include/show_binary_logs.inc
> +file_exists $MYSQLD_DATADIR/master-bin.000014;
> +file_exists $MYSQLD_DATADIR/master-bin.000015;
> +--error 1
> +file_exists $MYSQLD_DATADIR/master-bin.000016;
> +
> +-- echo # Test case7: Set DEBUG POINT after rename index file to make the master
> +-- echo # crash when appending a binlog file name to index file.
See above.
> +# Write file to make mysql-test-run.pl expect crash and restart
> +SET SESSION debug="+d,crash_create_after_rename_index_file";
> +-- exec echo "restart"> $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
> +
> +-- error 2013
> +flush logs;
> +
> +-- source include/wait_until_disconnected.inc
> +-- enable_reconnect
> +-- echo # Restart the master server
> +-- exec echo "restart"> $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
> +-- source include/wait_until_connected_again.inc
> +-- disable_reconnect
> +
> +-- echo # Test the index file is complete and binlog file name is added.
> +-- source include/show_binary_logs.inc
See above... There are other case in what follows.
> +file_exists $MYSQLD_DATADIR/master-bin.000016;
> +file_exists $MYSQLD_DATADIR/master-bin.000017;
> +--error 1
> +file_exists $MYSQLD_DATADIR/master-bin.000018;
> +
> +-- source include/show_binary_logs.inc
> +# Write file to make mysql-test-run.pl expect crash and restart
> +SET SESSION debug="+d,crash_create_after_rename_index_file";
> +-- exec echo "restart"> $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
> +
> +-- error 2013
> +purge binary logs TO 'master-bin.000008';
> +
> +-- source include/wait_until_disconnected.inc
> +-- enable_reconnect
> +-- echo # Restart the master server
> +-- exec echo "restart"> $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
> +-- source include/wait_until_connected_again.inc
> +-- disable_reconnect
> +
> +-- echo # Test the index file is complete and is purged successfully
> +-- source include/show_binary_logs.inc
> +--error 1
> +file_exists $MYSQLD_DATADIR/master-bin.000006;
> +--error 1
> +file_exists $MYSQLD_DATADIR/master-bin.000007;
> +file_exists $MYSQLD_DATADIR/master-bin.000008;
> +
> +-- echo # Test case8: Set DEBUG POINT after rename index file to make the master
> +-- echo # crash when purging the index file.
> +
> +-- source include/show_binary_logs.inc
> +# Write file to make mysql-test-run.pl expect crash and restart
> +SET SESSION debug="+d,crash_create_after_rename_index_file";
> +-- exec echo "restart"> $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
> +
> +-- error 2013
> +purge binary logs TO 'master-bin.000010';
> +
> +-- source include/wait_until_disconnected.inc
> +-- enable_reconnect
> +-- echo # Restart the master server
> +-- exec echo "restart"> $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
> +-- source include/wait_until_connected_again.inc
> +-- disable_reconnect
> +
> +-- echo # Test the index file is complete, and is purged successfully.
> +-- source include/show_binary_logs.inc
> +-- error 1
> +file_exists $MYSQLD_DATADIR/master-bin.000008;
> +-- error 1
> +file_exists $MYSQLD_DATADIR/master-bin.000009;
> +file_exists $MYSQLD_DATADIR/master-bin.000010;
> +
> +-- echo # Test case9: Set DEBUG POINT befor rename index file to make the master
> +-- echo # crash when purging the index file.
> +
> +-- source include/show_binary_logs.inc
> +# Write file to make mysql-test-run.pl expect crash and restart
> +SET SESSION debug="+d,crash_create_before_rename_index_file";
> +-- exec echo "restart"> $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
> +
> +-- error 2013
> +purge binary logs TO 'master-bin.000012';
> +
> +-- source include/wait_until_disconnected.inc
> +-- enable_reconnect
> +-- echo # Restart the master server
> +-- exec echo "restart"> $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
> +-- source include/wait_until_connected_again.inc
> +-- disable_reconnect
> +
> +-- echo # Test the index file is complete, and is purged successfully.
> +-- source include/show_binary_logs.inc
> +-- error 1
> +file_exists $MYSQLD_DATADIR/master-bin.000010;
> +-- error 1
> +file_exists $MYSQLD_DATADIR/master-bin.000011;
> +file_exists $MYSQLD_DATADIR/master-bin.000012;
> +
> +-- echo # Test case10: Inject a fault to copy part content to the temp file
> +-- echo # and abort the server after the temp file is closed
> +-- echo # when purging the index file.
> +
> +-- source include/show_binary_logs.inc
> +# Write file to make mysql-test-run.pl expect crash and restart
> +SET SESSION debug="+d,fault_injection_copy_part_file";
> +-- exec echo "restart"> $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
> +
> +-- error 2013
> +purge binary logs TO 'master-bin.000014';
> +
> +-- source include/wait_until_disconnected.inc
> +-- enable_reconnect
> +-- echo # Restart the master server
> +-- exec echo "restart"> $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
> +-- source include/wait_until_connected_again.inc
> +-- disable_reconnect
> +
> +-- echo # Test the index file is complete, although is not purged successfully.
> +-- source include/show_binary_logs.inc
> +file_exists $MYSQLD_DATADIR/master-bin.000012;
> +file_exists $MYSQLD_DATADIR/master-bin.000013;
> +file_exists $MYSQLD_DATADIR/master-bin.000014;
> +
> eval SET SESSION debug="$old";
>
> --echo End of tests
>
> === added file 'mysql-test/suite/rpl/r/rpl_crashed_master.result'
> --- a/mysql-test/suite/rpl/r/rpl_crashed_master.result 1970-01-01 00:00:00 +0000
> +++ b/mysql-test/suite/rpl/r/rpl_crashed_master.result 2010-12-02 02:56:34 +0000
> @@ -0,0 +1,150 @@
> +stop slave;
> +drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t9;
> +reset master;
> +reset slave;
> +drop table if exists t1,t2,t3,t4,t5,t6,t7,t8,t9;
> +start slave;
> +STOP SLAVE;
> +RESET MASTER;
> +START SLAVE;
> +call mtr.add_suppression("Attempting backtrace");
> +call mtr.add_suppression("allocated tablespace *., old maximum was 0");
> +call mtr.add_suppression("Error in Log_event::read_log_event()");
> +call mtr.add_suppression("Buffered warning: Performance schema disabled");
> +CREATE TABLE t1(a LONGBLOB) ENGINE=INNODB;
> +# Test case1: Set DEBUG POINT before binlog to make
> +# the master crash for transaction
> +BEGIN;
> +INSERT INTO t1 (a) VALUES (REPEAT('a',2));
> +INSERT INTO t1 (a) VALUES (REPEAT('a',2));
> +INSERT INTO t1 (a) VALUES (REPEAT('a',2));
> +SET SESSION debug="d,crash_commit_after_prepare";
> +COMMIT;
> +ERROR HY000: Lost connection to MySQL server during query
> +# Restart the master server
> +# Test the transaction statements will not be binlogged
> +show binlog events in 'master-bin.000001' from<binlog_start>;
> +Log_name Pos Event_type Server_id End_log_pos Info
> +master-bin.000001 # Query # # use `test`; CREATE TABLE t1(a LONGBLOB) ENGINE=INNODB
> +# On master, test the data will be rolled back after restart.
> +SELECT COUNT(*) FROM t1;
> +COUNT(*)
> +0
> +# On slave, test replication will work fine, and the data
> +# is not replicated
> +SELECT COUNT(*) FROM t1;
> +COUNT(*)
> +0
> +# Test case2: Set DEBUG POINT after binlog, and before the date
> +# is committed to make crash for transaction
> +BEGIN;
> +INSERT INTO t1 (a) VALUES (REPEAT('a',2));
> +INSERT INTO t1 (a) VALUES (REPEAT('a',2));
> +INSERT INTO t1 (a) VALUES (REPEAT('a',2));
> +SET SESSION debug="d,crash_commit_after_log";
> +COMMIT;
> +ERROR HY000: Lost connection to MySQL server during query
> +# Restart the master server
> +# Test the transaction statements will be binlogged
> +show binlog events in 'master-bin.000002' from<binlog_start>;
> +Log_name Pos Event_type Server_id End_log_pos Info
> +master-bin.000002 # Query # # BEGIN
> +master-bin.000002 # Table_map # # table_id: # (test.t1)
> +master-bin.000002 # Write_rows # # table_id: # flags: STMT_END_F
> +master-bin.000002 # Table_map # # table_id: # (test.t1)
> +master-bin.000002 # Write_rows # # table_id: # flags: STMT_END_F
> +master-bin.000002 # Table_map # # table_id: # (test.t1)
> +master-bin.000002 # Write_rows # # table_id: # flags: STMT_END_F
> +master-bin.000002 # Xid # # COMMIT /* XID */
> +# On master, test the data will be recovered after the master restart
> +SELECT COUNT(*) FROM t1;
> +COUNT(*)
> +3
> +# On slave, test replication will work fine, and the data is replicated
> +SELECT COUNT(*) FROM t1;
> +COUNT(*)
> +3
> +DROP TABLE t1;
> +include/stop_slave.inc
> +CREATE TABLE t1(a LONGBLOB) ENGINE=INNODB;
> +# Test case3: Set DEBUG POINT in the middle of binlog to
> +# make the master crash for transaction.
> +SET SESSION debug="d,half_binlogged_transaction";
> +BEGIN;
> +INSERT INTO t1 (a) VALUES (REPEAT('a',6144));
> +INSERT INTO t1 (a) VALUES (REPEAT('a',6144));
> +INSERT INTO t1 (a) VALUES (REPEAT('a',6144));
> +INSERT INTO t1 (a) VALUES (REPEAT('a',6144));
> +INSERT INTO t1 (a) VALUES (REPEAT('a',6144));
> +INSERT INTO t1 (a) VALUES (REPEAT('a',6144));
> +INSERT INTO t1 (a) VALUES (REPEAT('a',6144));
> +INSERT INTO t1 (a) VALUES (REPEAT('a',6144));
> +INSERT INTO t1 (a) VALUES (REPEAT('a',6144));
> +INSERT INTO t1 (a) VALUES (REPEAT('a',6144));
> +INSERT INTO t1 (a) VALUES (REPEAT('a',6144));
> +INSERT INTO t1 (a) VALUES (REPEAT('a',6144));
> +INSERT INTO t1 (a) VALUES (REPEAT('a',6144));
> +INSERT INTO t1 (a) VALUES (REPEAT('a',6144));
> +INSERT INTO t1 (a) VALUES (REPEAT('a',6144));
> +INSERT INTO t1 (a) VALUES (REPEAT('a',6144));
> +INSERT INTO t1 (a) VALUES (REPEAT('a',6144));
> +INSERT INTO t1 (a) VALUES (REPEAT('a',6144));
> +INSERT INTO t1 (a) VALUES (REPEAT('a',6144));
> +INSERT INTO t1 (a) VALUES (REPEAT('a',6144));
> +INSERT INTO t1 (a) VALUES (REPEAT('a',6144));
> +INSERT INTO t1 (a) VALUES (REPEAT('a',6144));
> +INSERT INTO t1 (a) VALUES (REPEAT('a',6144));
> +INSERT INTO t1 (a) VALUES (REPEAT('a',6144));
> +COMMIT;
> +ERROR HY000: Lost connection to MySQL server during query
> +# Restart the master server
> +# Test the halfly binlogged transaction will be trimmed
> +# from the crashed binlog file
> +show binlog events in 'master-bin.000003' from<binlog_start>;
> +Log_name Pos Event_type Server_id End_log_pos Info
> +master-bin.000003 # Query # # use `test`; CREATE TABLE t1(a LONGBLOB) ENGINE=INNODB
> +# Test the data will not be recovered successfully
> +# after the master restart.
> +SELECT COUNT(*) FROM t1;
> +COUNT(*)
> +0
> +# Test case4: Set DEBUG POINT in the middle of binlog to
> +# make the master crash for non-transaction.
> +SET SESSION debug="d,half_binlogged_transaction";
> +CREATE TABLE t2(a LONGBLOB) ENGINE=MYISAM;
> +INSERT INTO t2 (a) VALUES (REPEAT('a',16384));
> +ERROR HY000: Lost connection to MySQL server during query
> +# Restart the master server
> +# Test the halfly binlogged non-transaction statement will be trimmed
> +# from the crashed binlog file
> +show binlog events in 'master-bin.000004' from<binlog_start>;
> +Log_name Pos Event_type Server_id End_log_pos Info
> +master-bin.000004 # Query # # use `test`; CREATE TABLE t2(a LONGBLOB) ENGINE=MYISAM
> +# Test the data will not be recovered successfully
> +# after the master restart.
> +SELECT COUNT(*) FROM t2;
> +COUNT(*)
> +0
> +# Test case5: Inject wrong value of crc for a log event, and
> +# then set DBUG POINT to casue the master crash.
> +INSERT INTO t2 (a) VALUES (REPEAT('a',1));
> +BEGIN;
> +INSERT INTO t1 (a) VALUES (REPEAT('a',1));
> +SET SESSION debug="d,fault_injection_crc_value";
> +INSERT INTO t1 (a) VALUES (REPEAT('a',2));
> +COMMIT;
> +BEGIN;
> +INSERT INTO t1 (a) VALUES (REPEAT('a',3));
> +SET SESSION debug="d,crash_commit_after_prepare";
> +COMMIT;
> +ERROR HY000: Lost connection to MySQL server during query
> +# Restart the master server
> +# Test the transaction with a log event injected a wrong crc value
> +# will be trimmed from the crashed binlog file
> +show binlog events in 'master-bin.000005' from<binlog_start>;
> +Log_name Pos Event_type Server_id End_log_pos Info
> +master-bin.000005 # Query # # BEGIN
> +master-bin.000005 # Table_map # # table_id: # (test.t2)
> +master-bin.000005 # Write_rows # # table_id: # flags: STMT_END_F
> +master-bin.000005 # Query # # COMMIT
> +DROP TABLE t1, t2;
>
> === added file 'mysql-test/suite/rpl/t/rpl_crashed_master-master.opt'
> --- a/mysql-test/suite/rpl/t/rpl_crashed_master-master.opt 1970-01-01 00:00:00 +0000
> +++ b/mysql-test/suite/rpl/t/rpl_crashed_master-master.opt 2010-12-02 02:56:34 +0000
> @@ -0,0 +1 @@
> +--binlog-checksum=CRC32 --master-verify-checksum=1
>
> === added file 'mysql-test/suite/rpl/t/rpl_crashed_master.test'
> --- a/mysql-test/suite/rpl/t/rpl_crashed_master.test 1970-01-01 00:00:00 +0000
> +++ b/mysql-test/suite/rpl/t/rpl_crashed_master.test 2010-12-02 02:56:34 +0000
> @@ -0,0 +1,237 @@
> +#
> +# WL#5493& WL#5440
> +# Test case1 verifies if the transaction statements will not be
> +# binlogged and replication will work fine, but the data will
> +# be rolled back on master after the master restarts when setting
> +# DEBUG POINT before binlog to make the master crash.
> +#
> +# Test case2 verifies if the transaction statements will be
> +# binlogged and replication will work fine, and the data will
> +# be recovered after the master restarts when setting DEBUG
> +# POINT after binlog, and before the date is committed to make
> +# the master crash.
> +#
> +# Test case3 verifies if the halfly binlogged transaction
> +# statements will be trimmed from the crashed binlog file
> +# and the data will not be recovered successfully after
> +# the master restarts when setting DEBUG POINT in the
> +# middle of binlog to make the master crash
> +#
> +# Test case4 verifies if the halfly binlogged non-transaction
> +# statement will be trimmed from the crashed binlog file
> +# and the data will not be recovered successfully after
> +# the master restarts when setting DEBUG POINT in the
> +# middle of binlog to make the master crash.
> +#
> +# Test case5 verifies if a transaction with a wrong crc value
> +# will be trimmed from the crashed binlog file after
> +# master restarts when injecting a wrong crc value
> +# for a statment of the transaction and then setting
> +# DEBUG POINT to cause master crash.
> +#
> +
> +# Don't test this under valgrind, memory leaks will occur
> +-- source include/not_valgrind.inc
> +-- source include/not_embedded.inc
> +-- source include/master-slave.inc
> +-- source include/have_debug.inc
> +-- source include/have_innodb.inc
> +-- source include/have_binlog_format_row.inc
> +
> +# Reset master
> +connection slave;
> +STOP SLAVE;
> +--source include/wait_for_slave_to_stop.inc
please, use --source include/stop_slave.inc
> +
> +connection master;
> +RESET MASTER;
> +
> +connection slave;
> +START SLAVE;
> +--source include/wait_for_slave_to_start.inc
please, use --source include/start_slave.inc
> +
> +connection master;
> +call mtr.add_suppression("Attempting backtrace");
> +call mtr.add_suppression("allocated tablespace *., old maximum was 0");
> +call mtr.add_suppression("Error in Log_event::read_log_event()");
> +call mtr.add_suppression("Buffered warning: Performance schema disabled");
Please, move to this part of the code to beginning and call
syn_slave_with_master.
> +-- let $binlog_start= query_get_value(SHOW MASTER STATUS, Position, 1)
> +-- let $binlog_file= query_get_value(SHOW MASTER STATUS, File, 1)
> +CREATE TABLE t1(a LONGBLOB) ENGINE=INNODB;
> +
> +-- echo # Test case1: Set DEBUG POINT before binlog to make
> +-- echo # the master crash for transaction
> +#SET SESSION debug="d,crash_trans_commit_before_binlog";
> +BEGIN;
> +let $rows= 3;
> +WHILE($rows)
> +{
> + INSERT INTO t1 (a) VALUES (REPEAT('a',2));
> + dec $rows;
> +}
> +# Write file to make mysql-test-run.pl expect crash and restart
> +-- exec echo "restart"> $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
> +SET SESSION debug="d,crash_commit_after_prepare";
> +# Run the crashing query
> +-- error 2013
> +COMMIT;
Use the following pattern
--exec echo "wait" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
# CRASH OR SHUTDOWN
--source include/wait_until_disconnected.inc
--enable_reconnect
--exec echo "restart" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
--source include/wait_until_connected_again.inc
> +
> +-- source include/wait_until_disconnected.inc
> +-- enable_reconnect
> +-- echo # Restart the master server
> +-- exec echo "restart"> $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
> +-- source include/wait_until_connected_again.inc
> +-- disable_reconnect
> +
> +-- echo # Test the transaction statements will not be binlogged
> +-- source include/show_binlog_events.inc
> +
> +-- echo # On master, test the data will be rolled back after restart.
> +SELECT COUNT(*) FROM t1;
> +
> +sync_slave_with_master;
> +-- echo # On slave, test replication will work fine, and the data
> +-- echo # is not replicated
> +SELECT COUNT(*) FROM t1;
> +
> +connection master;
> +-- let $binlog_start= query_get_value(SHOW MASTER STATUS, Position, 1)
> +-- let $binlog_file= query_get_value(SHOW MASTER STATUS, File, 1)
> +-- echo # Test case2: Set DEBUG POINT after binlog, and before the date
> +-- echo # is committed to make crash for transaction
> +#SET SESSION debug="d,crash_trans_commit_after_binlog";
> +BEGIN;
> +let $rows= 3;
> +WHILE($rows)
> +{
> + INSERT INTO t1 (a) VALUES (REPEAT('a',2));
> + dec $rows;
> +}
> +# Write file to make mysql-test-run.pl expect crash and restart
> +-- exec echo "restart"> $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
> +SET SESSION debug="d,crash_commit_after_log";
> +# Run the crashing query
> +-- error 2013
> +COMMIT;
> +
> +-- source include/wait_until_disconnected.inc
> +-- enable_reconnect
> +-- echo # Restart the master server
> +-- exec echo "restart"> $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
> +-- source include/wait_until_connected_again.inc
> +-- disable_reconnect
> +
> +-- echo # Test the transaction statements will be binlogged
> +-- source include/show_binlog_events.inc
> +
> +-- echo # On master, test the data will be recovered after the master restart
> +SELECT COUNT(*) FROM t1;
> +
> +sync_slave_with_master;
> +-- echo # On slave, test replication will work fine, and the data is replicated
> +SELECT COUNT(*) FROM t1;
> +
> +connection master;
> +DROP TABLE t1;
> +sync_slave_with_master;
> +
> +-- source include/stop_slave.inc
> +
> +connection master;
> +-- let $binlog_start= query_get_value(SHOW MASTER STATUS, Position, 1)
> +-- let $binlog_file= query_get_value(SHOW MASTER STATUS, File, 1)
> +# Test transaction with large data inserted
> +CREATE TABLE t1(a LONGBLOB) ENGINE=INNODB;
> +
> +-- echo # Test case3: Set DEBUG POINT in the middle of binlog to
> +-- echo # make the master crash for transaction.
> +SET SESSION debug="d,half_binlogged_transaction";
> +BEGIN;
> +let $rows= 24;
> +WHILE($rows)
> +{
> + INSERT INTO t1 (a) VALUES (REPEAT('a',6144));
> + dec $rows;
> +}
> +# Write file to make mysql-test-run.pl expect crash and restart
> +-- exec echo "restart"> $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
> +# Run the crashing query
> +-- error 2013
> +COMMIT;
> +
> +-- source include/wait_until_disconnected.inc
> +-- enable_reconnect
> +-- echo # Restart the master server
> +-- exec echo "restart"> $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
> +-- source include/wait_until_connected_again.inc
> +-- disable_reconnect
> +
> +-- echo # Test the halfly binlogged transaction will be trimmed
> +-- echo # from the crashed binlog file
> +-- source include/show_binlog_events.inc
> +
> +-- echo # Test the data will not be recovered successfully
> +-- echo # after the master restart.
> +SELECT COUNT(*) FROM t1;
> +
> +-- echo # Test case4: Set DEBUG POINT in the middle of binlog to
> +-- echo # make the master crash for non-transaction.
> +SET SESSION debug="d,half_binlogged_transaction";
> +# Write file to make mysql-test-run.pl expect crash and restart
> +-- exec echo "restart"> $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
> +-- let $binlog_start= query_get_value(SHOW MASTER STATUS, Position, 1)
> +-- let $binlog_file= query_get_value(SHOW MASTER STATUS, File, 1)
> +CREATE TABLE t2(a LONGBLOB) ENGINE=MYISAM;
> +-- error 2013
> +INSERT INTO t2 (a) VALUES (REPEAT('a',16384));
> +
> +-- source include/wait_until_disconnected.inc
> +-- enable_reconnect
> +-- echo # Restart the master server
> +-- exec echo "restart"> $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
> +-- source include/wait_until_connected_again.inc
> +-- disable_reconnect
> +
> +-- echo # Test the halfly binlogged non-transaction statement will be trimmed
> +-- echo # from the crashed binlog file
> +-- source include/show_binlog_events.inc
> +
> +-- echo # Test the data will not be recovered successfully
> +-- echo # after the master restart.
> +SELECT COUNT(*) FROM t2;
> +
> +-- echo # Test case5: Inject wrong value of crc for a log event, and
> +-- echo # then set DBUG POINT to casue the master crash.
> +# Write file to make mysql-test-run.pl expect crash and restart
> +-- exec echo "restart"> $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
> +-- let $binlog_start= query_get_value(SHOW MASTER STATUS, Position, 1)
> +-- let $binlog_file= query_get_value(SHOW MASTER STATUS, File, 1)
> +INSERT INTO t2 (a) VALUES (REPEAT('a',1));
> +
> +BEGIN;
> +INSERT INTO t1 (a) VALUES (REPEAT('a',1));
> +SET SESSION debug="d,fault_injection_crc_value";
> +INSERT INTO t1 (a) VALUES (REPEAT('a',2));
> +COMMIT;
> +
> +BEGIN;
> +INSERT INTO t1 (a) VALUES (REPEAT('a',3));
> +SET SESSION debug="d,crash_commit_after_prepare";
> +# Run the crashing query
> +-- error 2013
> +COMMIT;
> +
> +-- source include/wait_until_disconnected.inc
> +-- enable_reconnect
> +-- echo # Restart the master server
> +-- exec echo "restart"> $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
> +-- source include/wait_until_connected_again.inc
> +-- disable_reconnect
> +
> +-- echo # Test the transaction with a log event injected a wrong crc value
> +-- echo # will be trimmed from the crashed binlog file
> +-- source include/show_binlog_events.inc
> +
> +DROP TABLE t1, t2;
> +
>
> === removed file 'mysql-test/t/crash_commit_before-master.opt'
> --- a/mysql-test/t/crash_commit_before-master.opt 2010-06-21 07:58:54 +0000
> +++ b/mysql-test/t/crash_commit_before-master.opt 1970-01-01 00:00:00 +0000
> @@ -1,3 +0,0 @@
> ---skip-stack-trace --skip-core-file
> ---default-storage-engine=MyISAM
> ---innodb-file-per-table=0
ok.
>
> === removed file 'mysql-test/t/crash_commit_before.test'
> --- a/mysql-test/t/crash_commit_before.test 2007-12-12 17:19:24 +0000
> +++ b/mysql-test/t/crash_commit_before.test 1970-01-01 00:00:00 +0000
> @@ -1,35 +0,0 @@
> --- source include/not_embedded.inc
> -# Don't test this under valgrind, memory leaks will occur
> ---source include/not_valgrind.inc
> -
> -# Binary must be compiled with debug for crash to occur
> ---source include/have_debug.inc
> -
> ---source include/have_innodb.inc
> -
> -CREATE TABLE t1(a int) engine=innodb;
> -START TRANSACTION;
> -insert into t1 values(9);
> -
> -# Setup the mysqld to crash at certain point
> -SET SESSION debug="d,crash_commit_before";
> -
> -# Write file to make mysql-test-run.pl expect crash and restart
> ---exec echo "restart"> $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
> -
> -# Run the crashing query
> ---error 2013
> -COMMIT;
> -
> -# Turn on reconnect
> ---enable_reconnect
> -
> -# Call script that will poll the server waiting for it to be back online again
> ---source include/wait_until_connected_again.inc
> -
> -SHOW CREATE TABLE t1;
> -
> -SELECT * FROM t1;
> -
> -
> -DROP TABLE t1;
>
ok.
> === modified file 'mysys/my_rename.c'
> --- a/mysys/my_rename.c 2010-07-15 11:41:37 +0000
> +++ b/mysys/my_rename.c 2010-12-02 02:56:34 +0000
> @@ -52,8 +52,6 @@ int my_rename(const char *from, const ch
> #else
> #if defined(HAVE_RENAME)
> if (rename(from,to))
> -#else
> - if (link(from, to) || unlink(from))
> #endif
> {
> my_errno=errno;
>
Please, remove the #ifdef.
Have you checked this with runtime?
> === modified file 'sql/binlog.cc'
> --- a/sql/binlog.cc 2010-11-16 12:14:06 +0000
> +++ b/sql/binlog.cc 2010-12-02 02:56:34 +0000
> @@ -1138,58 +1138,51 @@ int query_error_code(THD *thd, bool not_
>
>
> /**
> - Move all data up in a file in an filename index file.
> + Copy content of 'from' file from offset to 'to' file.
>
> - We do the copy outside of the IO_CACHE as the cache buffers would just
> - make things slower and more complicated.
> - In most cases the copy loop should only do one read.
> + - We do the copy outside of the IO_CACHE as the cache
> + buffers would just make things slower and more complicated.
> + In most cases the copy loop should only do one read.
>
> - @param index_file File to move
> - @param offset Move everything from here to beginning
> + @param from File to copy.
> + @param to File to copy to.
> + @param offset Offset in 'from' file.
>
> - @note
> - File will be truncated to be 'offset' shorter or filled up with newlines
>
> @retval
> - 0 ok
> + 0 ok
> + @retval
> + -1 error
> */
> -
> -#ifdef HAVE_REPLICATION
> -
> -static bool copy_up_file_and_fill(IO_CACHE *index_file, my_off_t offset)
> +static bool copy_file(IO_CACHE *from, IO_CACHE *to, my_off_t offset)
> {
> int bytes_read;
> - my_off_t init_offset= offset;
> - File file= index_file->file;
> uchar io_buf[IO_SIZE*2];
> - DBUG_ENTER("copy_up_file_and_fill");
> + DBUG_ENTER("copy_file");
>
> - for (;; offset+= bytes_read)
> + mysql_file_seek(from->file, offset, MY_SEEK_SET, MYF(0));
> + while(TRUE)
> {
> - mysql_file_seek(file, offset, MY_SEEK_SET, MYF(0));
> - if ((bytes_read= (int) mysql_file_read(file, io_buf, sizeof(io_buf),
> + if ((bytes_read= (int) mysql_file_read(from->file, io_buf, sizeof(io_buf),
> MYF(MY_WME)))
> - < 0)
> +< 0)
> goto err;
> + if (DBUG_EVALUATE_IF("fault_injection_copy_part_file", 1, 0))
> + bytes_read= bytes_read/2;
> if (!bytes_read)
> - break; // end of file
> - mysql_file_seek(file, offset-init_offset, MY_SEEK_SET, MYF(0));
> - if (mysql_file_write(file, io_buf, bytes_read, MYF(MY_WME | MY_NABP)))
> + break; // end of file
> + if (mysql_file_write(to->file, io_buf, bytes_read, MYF(MY_WME | MY_NABP)))
> goto err;
> }
> - /* The following will either truncate the file or fill the end with \n' */
> - if (mysql_file_chsize(file, offset - init_offset, '\n', MYF(MY_WME)) ||
> - mysql_file_sync(file, MYF(MY_WME)))
> - goto err;
>
> - /* Reset data in old index cache */
> - reinit_io_cache(index_file, READ_CACHE, (my_off_t) 0, 0, 1);
> DBUG_RETURN(0);
>
> err:
> DBUG_RETURN(1);
> }
>
> +
> +#ifdef HAVE_REPLICATION
> /**
> Load data's io cache specific hook to be executed
> before a chunk of data is being read into the cache's buffer
> @@ -1443,6 +1436,7 @@ MYSQL_BIN_LOG::MYSQL_BIN_LOG(uint *sync_
> index_file_name[0] = 0;
> bzero((char*)&index_file, sizeof(index_file));
> bzero((char*)&purge_index_file, sizeof(purge_index_file));
> + bzero((char*)&crash_safe_index_file, sizeof(crash_safe_index_file));
> }
>
> /* this is called only once */
> @@ -1489,6 +1483,7 @@ bool MYSQL_BIN_LOG::open_index_file(cons
> const char *log_name, bool need_mutex)
> {
> File index_file_nr= -1;
> + bool need_rename= FALSE;
> DBUG_ASSERT(!my_b_inited(&index_file));
>
> /*
> @@ -1504,6 +1499,36 @@ bool MYSQL_BIN_LOG::open_index_file(cons
> }
> fn_format(index_file_name, index_file_name_arg, mysql_data_home,
> ".index", opt);
> +
> + if (set_crash_safe_index_file_name(index_file_name_arg))
> + {
> + sql_print_error("MYSQL_BIN_LOG::set_crash_safe_index_file_name failed.");
> + return TRUE;
> + }
> +
> + /*
> + We need move crash_safe_index_file to index_file if the index_file
> + does not exist and crash_safe_index_file exists when mysqld server
> + restarts.
> + */
> +#ifdef __WIN__
> + if (my_access(index_file_name, F_OK)&&
> + !my_access(crash_safe_index_file_name, F_OK))
> + need_rename= TRUE;
> +#else
> + if (access(index_file_name, F_OK)&&
> + !access(crash_safe_index_file_name, F_OK))
> + need_rename= TRUE;
> +#endif
We don't need this here. Notice that you have the following
definition in include/my_sys.h:
#ifdef _WIN32
extern int my_access(const char *path, int amode);
#else
#define my_access access
#endif
> +
> + if (need_rename&&
> + my_rename(crash_safe_index_file_name, index_file_name, MYF(MY_WME)))
> + {
> + sql_print_error("MYSQL_BIN_LOG::open_index_file failed to "
> + "move crash_safe_index_file to index file.");
> + return TRUE;
> + }
> +
> if ((index_file_nr= mysql_file_open(key_file_binlog_index,
> index_file_name,
> O_RDWR | O_CREAT | O_BINARY,
See comment above on init_io_cache.
> @@ -1711,18 +1736,15 @@ bool MYSQL_BIN_LOG::open(const char *log
> #endif
>
> DBUG_ASSERT(my_b_inited(&index_file) != 0);
> - reinit_io_cache(&index_file, WRITE_CACHE,
> - my_b_filelength(&index_file), 0, 0);
> +
> /*
> - As this is a new log file, we write the file name to the index
> - file. As every time we write to the index file, we sync it.
> + The new log file name is appended into crash safe index file after
> + all the content of index file is copyed into the crash safe index
> + file. Then move the crash safe index file to index file.
> */
> if (DBUG_EVALUATE_IF("fault_injection_updating_index", 1, 0) ||
> - my_b_write(&index_file, (uchar*) log_file_name,
> - strlen(log_file_name)) ||
> - my_b_write(&index_file, (uchar*) "\n", 1) ||
> - flush_io_cache(&index_file) ||
> - mysql_file_sync(index_file.file, MYF(MY_WME)))
> + add_log_to_index((uchar*) log_file_name, strlen(log_file_name),
> + need_mutex))
> goto err;
>
> #ifdef HAVE_REPLICATION
ok.
> @@ -1759,6 +1781,135 @@ shutdown the MySQL server and restart it
> }
>
>
> +/**
> + Move crash safe index file to index file.
> +
> + @param need_mutex Set it to FALSE if its caller already has a
> + lock on LOCK_index
> +
> + @retval
> + 0 ok
> + @retval
> + -1 error
> +*/
> +int MYSQL_BIN_LOG::move_crash_safe_index_file_to_index_file(bool need_mutex)
> +{
> + int error= 0;
> + File fd= -1;
> + DBUG_ENTER("MYSQL_BIN_LOG::move_crash_safe_index_file_to_index_file");
> +
> + if (need_mutex)
> + mysql_mutex_lock(&LOCK_index);
> + mysql_mutex_assert_owner(&LOCK_index);
> +
> + if (my_b_inited(&index_file))
> + {
> + end_io_cache(&index_file);
> + if (mysql_file_close(index_file.file, MYF(0))< 0)
> + {
> + error= -1;
> + sql_print_error("MYSQL_BIN_LOG::move_crash_safe_index_file_to_index_file "
> + "failed to close the index file.");
> + goto err;
> + }
> + mysql_file_delete(key_file_binlog_index, index_file_name, MYF(MY_WME));
> + }
> +
> + DBUG_EXECUTE_IF("crash_create_before_rename_index_file", DBUG_SUICIDE(););
> + if (my_rename(crash_safe_index_file_name, index_file_name, MYF(MY_WME)))
> + {
> + error= -1;
> + sql_print_error("MYSQL_BIN_LOG::move_crash_safe_index_file_to_index_file "
> + "failed to move crash_safe_index_file to index file.");
> + goto err;
> + }
> + DBUG_EXECUTE_IF("crash_create_after_rename_index_file", DBUG_SUICIDE(););
> +
> + if ((fd= mysql_file_open(key_file_binlog_index,
> + index_file_name,
> + O_RDWR | O_CREAT | O_BINARY,
> + MYF(MY_WME)))< 0 ||
> + mysql_file_sync(fd, MYF(MY_WME)) ||
> + init_io_cache(&index_file, fd, IO_SIZE, READ_CACHE,
> + mysql_file_seek(fd, 0L, MY_SEEK_END, MYF(0)),
> + 0, MYF(MY_WME | MY_WAIT_IF_FULL)))
See comment above on init_io_cache.
> + {
> + error= -1;
> + sql_print_error("MYSQL_BIN_LOG::move_crash_safe_index_file_to_index_file "
> + "failed to open the index file.");
> + goto err;
> + }
> +
> +err:
> + if (need_mutex)
> + mysql_mutex_unlock(&LOCK_index);
> + DBUG_RETURN(error);
> +}
> +
> +
> +/**
> + Append log file name to index file.
> +
> + - To make crash safe, we copy all the content of index file
> + to crash safe index file firstly and then append the log
> + file name to the crash safe index file. Finally move the
> + crash safe index file to index file.
> +
> + @retval
> + 0 ok
> + @retval
> + -1 error
> +*/
> +int MYSQL_BIN_LOG::add_log_to_index(uchar* log_file_name,
> + int name_len, bool need_mutex)
> +{
> + DBUG_ENTER("MYSQL_BIN_LOG::add_log_to_index");
> +
> + if (open_crash_safe_index_file())
> + {
> + sql_print_error("MYSQL_BIN_LOG::add_log_to_index failed to "
> + "open the crash safe index file.");
> + goto err;
> + }
> +
> + if (copy_file(&index_file,&crash_safe_index_file, 0))
> + {
> + sql_print_error("MYSQL_BIN_LOG::add_log_to_index failed to "
> + "copy index file to crash safe index file.");
> + goto err;
> + }
> +
> + if (my_b_write(&crash_safe_index_file, log_file_name, name_len) ||
> + my_b_write(&crash_safe_index_file, (uchar*) "\n", 1) ||
> + flush_io_cache(&crash_safe_index_file) ||
> + mysql_file_sync(crash_safe_index_file.file, MYF(MY_WME)))
> + {
> + sql_print_error("MYSQL_BIN_LOG::add_log_to_index failed to "
> + "append log file name: %s, to crash "
> + "safe index file.", log_file_name);
> + goto err;
> + }
> +
> + if (close_crash_safe_index_file())
> + {
> + sql_print_error("MYSQL_BIN_LOG::add_log_to_index failed to "
> + "close the crash safe index file.");
> + goto err;
> + }
> +
> + if (move_crash_safe_index_file_to_index_file(need_mutex))
> + {
> + sql_print_error("MYSQL_BIN_LOG::add_log_to_index failed to "
> + "move crash safe index file to index file.");
> + goto err;
> + }
> +
> + DBUG_RETURN(0);
> +
> +err:
> + DBUG_RETURN(-1);
> +}
> +
> int MYSQL_BIN_LOG::get_current_log(LOG_INFO* linfo)
> {
> mysql_mutex_lock(&LOCK_log);
ok.
> @@ -2082,6 +2233,92 @@ err:
>
>
> /**
> + Set the name of crash safe index file.
> +
> + @retval
> + 0 ok
> + @retval
> + 1 error
> +*/
> +int MYSQL_BIN_LOG::set_crash_safe_index_file_name(const char *base_file_name)
> +{
> + int error= 0;
> + DBUG_ENTER("MYSQL_BIN_LOG::set_crash_safe_index_file_name");
> + if (fn_format(crash_safe_index_file_name, base_file_name, mysql_data_home,
> + ".index_crash_safe", MYF(MY_UNPACK_FILENAME | MY_SAFE_PATH |
> + MY_REPLACE_EXT)) == NULL)
> + {
> + error= 1;
> + sql_print_error("MYSQL_BIN_LOG::set_crash_safe_index_file_name failed "
> + "to set file name.");
> + }
> + DBUG_RETURN(error);
> +}
> +
> +
> +/**
> + Open a (new) crash safe index file.
> +
> + @note
> + The crash safe index file is a special file
> + used for guaranteeing index file crash safe.
> + @retval
> + 0 ok
> + @retval
> + 1 error
> +*/
> +int MYSQL_BIN_LOG::open_crash_safe_index_file()
> +{
> + int error= 0;
> + File file= -1;
> +
> + DBUG_ENTER("MYSQL_BIN_LOG::open_crash_safe_index_file");
> +
> + if (!my_b_inited(&crash_safe_index_file))
> + {
> + if ((file= my_open(crash_safe_index_file_name, O_RDWR | O_CREAT | O_BINARY,
> + MYF(MY_WME | ME_WAITTANG)))< 0 ||
> + init_io_cache(&crash_safe_index_file, file, IO_SIZE, WRITE_CACHE,
> + 0, 0, MYF(MY_WME | MY_NABP | MY_WAIT_IF_FULL)))
> + {
> + error= 1;
> + sql_print_error("MYSQL_BIN_LOG::open_crash_safe_index_file failed "
> + "to open temporary index file.");
> + }
> + }
> + DBUG_RETURN(error);
> +}
> +
> +
> +/**
> + Close the crash safe index file.
> +
> + @note
> + The crash safe file is just closed, is not deleted.
> + Because it is moved to index file later on.
> + @retval
> + 0 ok
> + @retval
> + 1 error
> +*/
> +int MYSQL_BIN_LOG::close_crash_safe_index_file()
> +{
> + int error= 0;
> +
> + DBUG_ENTER("MYSQL_BIN_LOG::close_crash_safe_index_file");
> +
> + if (my_b_inited(&crash_safe_index_file))
> + {
> + end_io_cache(&crash_safe_index_file);
> + error= my_close(crash_safe_index_file.file, MYF(0));
> + }
> + bzero((char*)&crash_safe_index_file, sizeof(crash_safe_index_file));
> +
> + DBUG_RETURN(error);
> +}
> +
> +
> +/**
> Delete relay log files prior to rli->group_relay_log_name
> (i.e. all logs which are not involved in a non-finished group
> (transaction)), remove them from the index file and start on next
ok.
> @@ -2216,19 +2453,67 @@ err:
> DBUG_RETURN(error);
> }
>
> +
> /**
> - Update log index_file.
> -*/
> + Remove logs from index file.
> +
> + - To make crash safe, we copy the content of index file
> + from index_file_start_offset recored in log_info to
> + crash safe index file firstly and then move the crash
> + safe index file to index file.
> +
> + @param linfo Store here the found log file name and
> + position to the NEXT log file name in
> + the index file.
>
> -int MYSQL_BIN_LOG::update_log_index(LOG_INFO* log_info, bool need_update_threads)
> + @param need_update_threads If we want to update the log coordinates
> + of all threads. False for relay logs,
> + true otherwise.
> +
> + @retval
> + 0 ok
> + @retval
> + LOG_INFO_IO Got IO error while reading/writing file
> +*/
ok.
> +int MYSQL_BIN_LOG::remove_logs_from_index(LOG_INFO* log_info, bool
> need_update_threads)
> {
> - if (copy_up_file_and_fill(&index_file, log_info->index_file_start_offset))
> - return LOG_INFO_IO;
> + if (open_crash_safe_index_file())
> + {
> + sql_print_error("MYSQL_BIN_LOG::remove_logs_from_index failed to "
> + "open the crash safe index file.");
> + goto err;
> + }
> +
> + if (copy_file(&index_file,&crash_safe_index_file,
> + log_info->index_file_start_offset))
> + {
> + sql_print_error("MYSQL_BIN_LOG::remove_logs_from_index failed to "
> + "copy index file to crash safe index file.");
> + goto err;
> + }
> +
> + if (close_crash_safe_index_file())
> + {
> + sql_print_error("MYSQL_BIN_LOG::remove_logs_from_index failed to "
> + "close the crash safe index file.");
> + goto err;
> + }
> + DBUG_EXECUTE_IF("fault_injection_copy_part_file", DBUG_SUICIDE(););
> +
> + if (move_crash_safe_index_file_to_index_file(FALSE))
> + {
> + sql_print_error("MYSQL_BIN_LOG::remove_logs_from_index failed to "
> + "move crash safe index file to index file.");
> + goto err;
> + }
>
> // now update offsets in index file for running threads
> if (need_update_threads)
> adjust_linfo_offsets(log_info->index_file_start_offset);
> return 0;
> +
> +err:
> + return LOG_INFO_IO;
> }
ok.
>
> /**
> @@ -2313,7 +2598,7 @@ int MYSQL_BIN_LOG::purge_logs(const char
> }
>
> /* We know how many files to delete. Update index file. */
> - if ((error=update_log_index(&log_info, need_update_threads)))
> + if ((error=remove_logs_from_index(&log_info, need_update_threads)))
> {
> sql_print_error("MSYQL_BIN_LOG::purge_logs failed to update the index file");
> goto err;
> @@ -3386,6 +3671,9 @@ int MYSQL_BIN_LOG::write_cache(IO_CACHE
> if (do_checksum)
> crc= crc_0= my_checksum(0L, NULL, 0);
>
> + if (DBUG_EVALUATE_IF("fault_injection_crc_value", 1, 0))
> + crc= crc - 1;
> +
> do
> {
> /*
ok.
> @@ -3655,9 +3943,9 @@ bool MYSQL_BIN_LOG::write(THD *thd, IO_C
> goto err;
>
> bool synced= 0;
> + DBUG_EXECUTE_IF("half_binlogged_transaction", DBUG_SUICIDE(););
> if (flush_and_sync(&synced))
> goto err;
> - DBUG_EXECUTE_IF("half_binlogged_transaction", DBUG_SUICIDE(););
> if (cache->error) // Error on read
> {
> sql_print_error(ER(ER_ERROR_ON_READ), cache->file_name, errno);
ok.
> @@ -3920,6 +4208,9 @@ int MYSQL_BIN_LOG::open(const char *opt_
> Log_event *ev=0;
> Format_description_log_event fdle(BINLOG_VERSION);
> char log_name[FN_REFLEN];
> + my_off_t valid_pos= 0;
> + my_off_t binlog_size;
> + MY_STAT s;
>
> if (! fdle.is_valid())
> goto err;
ok.
> @@ -3941,13 +4232,17 @@ int MYSQL_BIN_LOG::open(const char *opt_
> goto err;
> }
>
> + my_stat(log_name,&s, MYF(0));
> + binlog_size= s.st_size;
> +
> if ((ev= Log_event::read_log_event(&log, 0,&fdle,
> opt_master_verify_checksum))&&
> ev->get_type_code() == FORMAT_DESCRIPTION_EVENT&&
> ev->flags& LOG_EVENT_BINLOG_IN_USE_F)
> {
> sql_print_information("Recovering after a crash using %s", opt_name);
> - error= recover(&log, (Format_description_log_event *)ev);
> + valid_pos= my_b_tell(&log);
> + error= recover(&log, (Format_description_log_event *)ev,&valid_pos);
> }
> else
> error=0;
ok.
> @@ -3958,6 +4253,51 @@ int MYSQL_BIN_LOG::open(const char *opt_
>
> if (error)
> goto err;
> +
> + /* Trim the crashed binlog file to last valid transaction
> + or event (non-transaction) base on valid_pos. */
> + if (valid_pos> 0)
> + {
> + if ((file= mysql_file_open(key_file_binlog, log_name,
> + O_RDWR | O_BINARY, MYF(MY_WME)))< 0)
> + {
> + sql_print_error("Failed to open the crashed binlog file "
> + "when master server is recovering it.");
> + return -1;
> + }
> +
> + /* Change binlog file size to valid_pos */
> + if (valid_pos< binlog_size)
> + {
> + if (my_chsize(file, valid_pos, 0, MYF(MY_WME)))
> + {
> + sql_print_error("Failed to trim the crashed binlog file "
> + "when master server is recovering it.");
> + mysql_file_close(file, MYF(MY_WME));
> + return -1;
> + }
> + else
> + {
> + sql_print_information("Crashed binlog file %s size is %llu, "
> + "but recovered up to %llu. Binlog trimmed to %llu
> bytes.",
> + log_name, binlog_size, valid_pos, valid_pos);
> + }
> + }
> +
> + /* Clear LOG_EVENT_BINLOG_IN_USE_F */
> + my_off_t offset= BIN_LOG_HEADER_SIZE + FLAGS_OFFSET;
> + uchar flags= 0;
> + if (mysql_file_pwrite(file,&flags, 1, offset, MYF(0)) != 1)
> + {
> + sql_print_error("Failed to clear LOG_EVENT_BINLOG_IN_USE_F "
> + "for the crashed binlog file when master "
> + "server is recovering it.");
> + mysql_file_close(file, MYF(MY_WME));
> + return -1;
> + }
> +
> + mysql_file_close(file, MYF(MY_WME));
> + } //end if
> }
>
> err:
ok.
> @@ -4007,11 +4347,32 @@ void MYSQL_BIN_LOG::unlog(ulong cookie,
> rotate_and_purge(0); // as ::write() did not rotate
> }
>
> -int MYSQL_BIN_LOG::recover(IO_CACHE *log, Format_description_log_event *fdle)
> +
> +/**
> + MYSQLD server recovers from last crashed binlog.
> +
> + @param log IO_CACHE of the crashed binlog.
> + @param fdle Format_description_log_event of the crashed binlog.
> + @param valid_pos The position of the last valid transaction or
> + event(non-transaction) of the crashed binlog.
> +
> + @retval
> + 0 ok
> + @retval
> + 1 error
> +*/
> +int MYSQL_BIN_LOG::recover(IO_CACHE *log, Format_description_log_event *fdle,
> + my_off_t *valid_pos)
> {
> Log_event *ev;
> HASH xids;
> MEM_ROOT mem_root;
> + my_off_t last_valid_pos= *valid_pos;
> + /*
> + The flag is used for handling the case that a transaction
> + is partially written to the binlog.
> + */
> + bool in_transaction= 0;
>
> if (! fdle->is_valid() ||
> my_hash_init(&xids,&my_charset_bin, TC_LOG_PAGE_SIZE/3, 0,
ok.
> @@ -4026,8 +4387,27 @@ int MYSQL_BIN_LOG::recover(IO_CACHE *log
> opt_master_verify_checksum))
> && ev->is_valid())
> {
> + /*
> + Recorded valid position for the crashed binlog file
> + which contains incorrect events.
> + */
> + if (ev->get_type_code() == QUERY_EVENT&&
> + !strcmp(((Query_log_event*)ev)->query, "BEGIN"))
> + {
> + in_transaction= TRUE;
> + *valid_pos= last_valid_pos;
> + }
> + last_valid_pos= my_b_tell(log);
> +
> + sql_print_information("ev->get_type_code()== %d, last_valid_pos== %llu .",
> ev->get_type_code(), last_valid_pos);
> +
> + if (ev->get_type_code() == QUERY_EVENT&&
> + !strcmp(((Query_log_event*)ev)->query, "COMMIT"))
> + in_transaction= FALSE;
> +
> if (ev->get_type_code() == XID_EVENT)
> {
> + in_transaction= FALSE;
> Xid_log_event *xev=(Xid_log_event *)ev;
> uchar *x= (uchar *) memdup_root(&mem_root, (uchar*)&xev->xid,
> sizeof(xev->xid));
ok.
> @@ -4037,6 +4417,13 @@ int MYSQL_BIN_LOG::recover(IO_CACHE *log
> delete ev;
> }
>
> + /*
> + Recorded valid position for the crashed binlog file
> + which did not contain incorrect events.
> + */
> + if (!log->error&& !in_transaction)
> + *valid_pos= last_valid_pos;
> +
> if (ha_recover(&xids))
> goto err2;
>
ok.
>
> === modified file 'sql/binlog.h'
> --- a/sql/binlog.h 2010-10-25 19:02:24 +0000
> +++ b/sql/binlog.h 2010-12-02 02:56:34 +0000
> @@ -35,6 +35,12 @@ class MYSQL_BIN_LOG: public TC_LOG, priv
> IO_CACHE index_file;
> char index_file_name[FN_REFLEN];
> /*
> + crash_safe_index_file is temp file used for guaranteeing
> + index file crash safe when master server restarts.
> + */
> + IO_CACHE crash_safe_index_file;
> + char crash_safe_index_file_name[FN_REFLEN];
> + /*
ok.
> purge_file is a temp file used in purge_logs so that the index file
> can be updated before deleting files from disk, yielding better crash
> recovery. It is created on demand the first time purge_logs is called
> @@ -153,7 +159,8 @@ public:
> void close();
> int log_xid(THD *thd, my_xid xid);
> void unlog(ulong cookie, my_xid xid);
> - int recover(IO_CACHE *log, Format_description_log_event *fdle);
> + int recover(IO_CACHE *log, Format_description_log_event *fdle,
> + my_off_t *valid_pos);
> #if !defined(MYSQL_CLIENT)
ok.
>
> int flush_and_set_pending_rows_event(THD *thd, Rows_log_event* event,
> @@ -218,7 +225,7 @@ public:
>
> void make_log_name(char* buf, const char* log_ident);
> bool is_active(const char* log_file_name);
> - int update_log_index(LOG_INFO* linfo, bool need_update_threads);
> + int remove_logs_from_index(LOG_INFO* linfo, bool need_update_threads);
> void rotate_and_purge(uint flags);
> /**
> Flush binlog cache and synchronize to disk.
ok.
> @@ -240,6 +247,11 @@ public:
> ulonglong *decrease_log_space);
> int purge_logs_before_date(time_t purge_time);
> int purge_first_log(Relay_log_info* rli, bool included);
> + int set_crash_safe_index_file_name(const char *base_file_name);
> + int open_crash_safe_index_file();
> + int close_crash_safe_index_file();
> + int add_log_to_index(uchar* log_file_name, int name_len, bool need_mutex);
> + int move_crash_safe_index_file_to_index_file(bool need_mutex);
> int set_purge_index_file_name(const char *base_file_name);
> int open_purge_index_file(bool destroy);
> bool is_inited_purge_index_file();
>
ok.
> === modified file 'sql/log_event.cc'
> --- a/sql/log_event.cc 2010-11-23 23:10:22 +0000
> +++ b/sql/log_event.cc 2010-12-02 02:56:34 +0000
> @@ -1306,6 +1306,8 @@ err:
> enough to stop the SQL thread now ; as we are skipping the current event,
> going on with reading and successfully executing other events can
> only corrupt the slave's databases. So stop.
> + The file->error is also checked to record the position of
> + the last valid event when master server recovers.
> */
> file->error= -1;
> }
> @@ -1378,20 +1380,15 @@ Log_event* Log_event::read_log_event(con
> if (crc_check&&
> event_checksum_test((uchar *) buf, event_len, alg))
> {
> -#ifdef MYSQL_CLIENT
> *error= "Event crc check failed! Most likely there is event corruption.";
> +#ifdef MYSQL_CLIENT
> if (force_opt)
> {
> ev= new Unknown_log_event(buf, description_event);
> DBUG_RETURN(ev);
> }
> - else
> - DBUG_RETURN(NULL);
> -#else
> - *error= ER(ER_BINLOG_READ_EVENT_CHECKSUM_FAILURE);
> - sql_print_error("%s", ER(ER_BINLOG_READ_EVENT_CHECKSUM_FAILURE));
> - DBUG_RETURN(NULL);
> #endif
> + DBUG_RETURN(NULL);
> }
>
> if (event_type> description_event->number_of_event_types&&
>
>
Can you improve your comments here?
I don't understan why these changes are necessary.
>
>
