| List: | Commits | « Previous MessageNext Message » | |
| From: | rsomla | Date: | November 30 2007 9:23am |
| Subject: | bk commit into 6.0 tree (rafal:1.2669) WL#4060 | ||
| View as plain text | |||
Below is the list of changes that have just been committed into a local 6.0 repository of rafal. When rafal does a push these changes will be propagated to the main repository and, within 24 hours after the push, to the public repository. For information on how to access the public repository see http://dev.mysql.com/doc/mysql/en/installing-source-tree.html ChangeSet@stripped, 2007-11-30 09:23:34+01:00, rafal@quant.(none) +26 -0 WL#4060 (kernel updates for beta release): This patch modifies backup kernel code so that it uses the backup stream library for writing and reading backup images. Now backup kernel writes backup images in the format described in WL#4063. Limitations: 1. Errors are detected but not always reported in the error log - only a general error message will be shown. 2. Object names using non-standard charsets are still not supported. The code doesn't address this issue yet. 3. The default server charset is stored in the image, but it is not set upon restore. This should not be a big problem however, since I believe that CREATE TABLE statements we use always set charset and collaction explicitly. 4. I don't obtain CREATE DATABASE statements from the server and thus don't save things like default charset or collation. It should be easy to add but I simply didn't have time to do it now. There is a bug entry for this. 5. The summary produced by BACKUP/RESTORE commands is now very minimalistic. Due to complex buffering in the backup stream library it is difficult to get exact sizes of various parts of the backup image and thus produce similar summary as before. Since format of the summary is going to change soon, I didn't spend time on improving it. 6. Eventually, the endianess of the server which created backup image should be stored in image flags. This code doesn't store this information (as I'm not sure how to get it). Note however that the image format *is* endianess agnostic (at least the part written by the kernel). The purpose of the flag is to provide more information for the user and possibly for restore drivers (not supported in the current driver API). BitKeeper/deleted/.del-map.h~3bf2f96a8cb7ffe7@stripped, 2007-11-30 09:21:40+01:00, rafal@quant.(none) +0 -0 Rename: sql/backup/map.h -> BitKeeper/deleted/.del-map.h~3bf2f96a8cb7ffe7 BitKeeper/deleted/.del-string_pool.cc~52288457d3ba0e49@stripped, 2007-11-30 09:21:40+01:00, rafal@quant.(none) +0 -0 Rename: sql/backup/string_pool.cc -> BitKeeper/deleted/.del-string_pool.cc~52288457d3ba0e49 BitKeeper/deleted/.del-string_pool.h~c9ae95ef8972eaf9@stripped, 2007-11-30 09:21:41+01:00, rafal@quant.(none) +0 -0 Rename: sql/backup/string_pool.h -> BitKeeper/deleted/.del-string_pool.h~c9ae95ef8972eaf9 mysql-test/r/backup.result@stripped, 2007-11-30 09:23:28+01:00, rafal@quant.(none) +12 -20 Result changes due to different BACKUP/RESTORE summary. mysql-test/r/backup_commit_blocker.result@stripped, 2007-11-30 09:23:28+01:00, rafal@quant.(none) +18 -30 Result changes due to different BACKUP/RESTORE summary. mysql-test/r/backup_no_data.result@stripped, 2007-11-30 09:23:28+01:00, rafal@quant.(none) +21 -35 Result changes due to different BACKUP/RESTORE summary. mysql-test/r/backup_no_engine.result@stripped, 2007-11-30 09:23:28+01:00, rafal@quant.(none) +6 -10 Result changes due to different BACKUP/RESTORE summary. mysql-test/r/backup_security.result@stripped, 2007-11-30 09:23:28+01:00, rafal@quant.(none) +12 -20 Result changes due to different BACKUP/RESTORE summary. mysql-test/r/backup_snapshot.result@stripped, 2007-11-30 09:23:29+01:00, rafal@quant.(none) +12 -20 Result changes due to different BACKUP/RESTORE summary. sql/backup/CMakeLists.txt@stripped, 2007-11-30 09:23:29+01:00, rafal@quant.(none) +3 -2 - remove string_pool.cc which is no longer used; - add backupstream library to the backup library. sql/backup/Makefile.am@stripped, 2007-11-30 09:23:29+01:00, rafal@quant.(none) +6 -4 - removed sources and headers which are not used now - added be_native.h header - added libbackupstream to libbackup sql/backup/backup_aux.h@stripped, 2007-11-30 09:23:29+01:00, rafal@quant.(none) +133 -0 - added new constructor for backup::LEX_STRING - added backup::String convenience class - added backup::Dynamic_array class template sql/backup/backup_kernel.h@stripped, 2007-11-30 09:23:29+01:00, rafal@quant.(none) +58 -67 Updated declarations of the Backup_info and Restore_info classes. sql/backup/backup_stream.h@stripped, 2007-11-30 09:23:30+01:00, rafal@quant.(none) +15 -0 New header which defines the backup stream format used. sql/backup/backup_stream.h@stripped, 2007-11-30 09:23:30+01:00, rafal@quant.(none) +0 -0 sql/backup/be_default.h@stripped, 2007-11-30 09:23:29+01:00, rafal@quant.(none) +10 -18 Updated definition of Default_image class, now called Default_snapshot. sql/backup/be_native.h@stripped, 2007-11-30 09:23:30+01:00, rafal@quant.(none) +74 -0 Definition of the Native_snapshot class using when backing-up/restoring table data using native backup engines. sql/backup/be_native.h@stripped, 2007-11-30 09:23:30+01:00, rafal@quant.(none) +0 -0 sql/backup/be_snapshot.h@stripped, 2007-11-30 09:23:29+01:00, rafal@quant.(none) +11 -19 Changed definition of Snapshot_image class, now called CS_snapshot. sql/backup/catalog.cc@stripped, 2007-11-30 09:23:29+01:00, rafal@quant.(none) +148 -732 Changed implementation of Archive_info, now called Image_info. Now this class doesn't contain any code for writing parts of backup image. Instead, backup stream library functions are called and the class collects all information needed for them. This file also implements the catalogue services needed by the backup stream library (bcat_*() functions). sql/backup/catalog.h@stripped, 2007-11-30 09:23:29+01:00, rafal@quant.(none) +611 -328 New definition of Archive_info class, now called Image_info and former Image_info, now called Snapshot_info. sql/backup/data_backup.cc@stripped, 2007-11-30 09:23:29+01:00, rafal@quant.(none) +184 -213 Use bstream_rd_table_data() and bstream_wr_table_data() for reading/writng table data blocks. This is done in restore_table_data() function and Block_writer::write_buf() method. sql/backup/kernel.cc@stripped, 2007-11-30 09:23:29+01:00, rafal@quant.(none) +840 -450 - cosmetic changes to execute_backup_command(); - new implementations of Backup_info and Restore_info classes - implementation of rstore-time catalogue services for backup stream library - memory allocation for backup stream library sql/backup/meta_data.cc@stripped, 2007-11-30 09:23:30+01:00, rafal@quant.(none) +93 -283 - meta-data manipulating services for backup stream library. - definitions of meta::Item and derived classes for manipulationg object meta-data sql/backup/meta_data.h@stripped, 2007-11-30 09:23:30+01:00, rafal@quant.(none) +49 -60 Updated definitions of meta::Item and derived classes to make them suitable for implementing services needed by backup stream library. Methods for writing/reading meta-data are removes since now the library functions are used for that purpose (this is done inside bstream_{wr,rd}_meta_data()) sql/backup/stream.cc@stripped, 2007-11-30 09:23:30+01:00, rafal@quant.(none) +249 -205 Changed definitions of IStream and OStream classes. Now they implement the "abstract stream" used by the backup stream library to read/write data. sql/backup/stream.h@stripped, 2007-11-30 09:23:30+01:00, rafal@quant.(none) +39 -586 - removed old code for writing/reading backup stream; - updated definitions of the IStream/OStream classes. sql/backup/stream_services.h@stripped, 2007-11-30 09:23:30+01:00, rafal@quant.(none) +10 -0 New header file defining current version of stream services implemented in the backup kernel. sql/backup/stream_services.h@stripped, 2007-11-30 09:23:30+01:00, rafal@quant.(none) +0 -0 diff -Nrup a/mysql-test/r/backup.result b/mysql-test/r/backup.result --- a/mysql-test/r/backup.result 2007-11-09 22:32:08 +01:00 +++ b/mysql-test/r/backup.result 2007-11-30 09:23:28 +01:00 @@ -57,21 +57,17 @@ INSERT INTO `tasking` VALUES ('333445555 UNLOCK TABLES; BACKUP DATABASE db1,db2 TO 'test.ba'; Backup Summary - header = 52 bytes - meta-data = 807 bytes - data = 639 bytes - -------------- - total 1498 bytes + backed up 4 tables + in 2 databases + using 1 driver DROP DATABASE db1; DROP DATABASE db2; USE mysql; RESTORE FROM 'test.ba'; Restore Summary - header = 52 bytes - meta-data = 807 bytes - data = 639 bytes - -------------- - total 1498 bytes + restored 4 tables + in 2 databases + using 1 driver USE db1; SHOW TABLES; Tables_in_db1 @@ -319,19 +315,15 @@ Chuck@linux:~> mysql -uroot -p Enter password: BACKUP DATABASE bup_default TO "bup_default.bak"; Backup Summary - header = 43 bytes - meta-data = 890 bytes - data = 7368 bytes - -------------- - total 8301 bytes + backed up 4 tables + in 1 database + using 2 drivers DROP DATABASE bup_default; RESTORE FROM "bup_default.bak"; Restore Summary - header = 43 bytes - meta-data = 890 bytes - data = 7368 bytes - -------------- - total 8301 bytes + restored 4 tables + in 1 database + using 2 drivers SELECT * FROM bup_default.t1; a 1 diff -Nrup a/mysql-test/r/backup_commit_blocker.result b/mysql-test/r/backup_commit_blocker.result --- a/mysql-test/r/backup_commit_blocker.result 2007-11-16 02:29:32 +01:00 +++ b/mysql-test/r/backup_commit_blocker.result 2007-11-30 09:23:28 +01:00 @@ -86,11 +86,9 @@ UPDATE bup_commit_blocker.t3 SET col_a = UPDATE bup_commit_blocker.t3 SET col_a = "con4: 06 CHANGED" WHERE col_a LIKE '06%'; COMMIT; Backup Summary - header = 36 bytes - meta-data = 284 bytes - data = 471 bytes - -------------- - total 791 bytes + backed up 3 tables + in 1 database + using 1 driver con1: Showing data after updates and backup SELECT * FROM bup_commit_blocker.t1; col_a @@ -121,11 +119,9 @@ DROP TABLE bup_commit_blocker.t3; con1: Restoring the database RESTORE FROM "bup_commit_blocker.bak"; Restore Summary - header = 36 bytes - meta-data = 284 bytes - data = 471 bytes - -------------- - total 791 bytes + restored 3 tables + in 1 database + using 1 driver con1: Showing the data (no new data should be here). SELECT * FROM bup_commit_blocker.t1; col_a @@ -217,11 +213,9 @@ NULL con6: Completing statement con7: Completing statement Backup Summary - header = 28 bytes - meta-data = 95 bytes - data = 50 bytes - -------------- - total 173 bytes + backed up 1 tables + in 1 database + using 1 driver con1: Showing data after updates and backup SELECT * FROM bup_commit_blocker.t5; col_a @@ -235,11 +229,9 @@ DROP TABLE bup_commit_blocker.t5; con1: Restoring the database RESTORE FROM "bup_commit_blocker.bak"; Restore Summary - header = 28 bytes - meta-data = 95 bytes - data = 50 bytes - -------------- - total 173 bytes + restored 1 tables + in 1 database + using 1 driver con1: Showing the data (no new data should be here). SELECT * FROM bup_commit_blocker.t5; col_a @@ -367,11 +359,9 @@ UPDATE bup_commit_blocker.t3 SET col_a = COMMIT; con7: Completing statement Backup Summary - header = 43 bytes - meta-data = 377 bytes - data = 521 bytes - -------------- - total 941 bytes + backed up 4 tables + in 1 database + using 2 drivers con1: Showing data after updates and backup SELECT * FROM bup_commit_blocker.t1; col_a @@ -409,11 +399,9 @@ DROP TABLE bup_commit_blocker.t5; con1: Restoring the database RESTORE FROM "bup_commit_blocker.bak"; Restore Summary - header = 43 bytes - meta-data = 377 bytes - data = 521 bytes - -------------- - total 941 bytes + restored 4 tables + in 1 database + using 2 drivers con1: Showing the data (no new data should be here). SELECT * FROM bup_commit_blocker.t1; col_a diff -Nrup a/mysql-test/r/backup_no_data.result b/mysql-test/r/backup_no_data.result --- a/mysql-test/r/backup_no_data.result 2007-11-06 19:32:23 +01:00 +++ b/mysql-test/r/backup_no_data.result 2007-11-30 09:23:28 +01:00 @@ -2,18 +2,14 @@ DROP DATABASE IF EXISTS empty_db; CREATE DATABASE empty_db; BACKUP DATABASE empty_db TO 'empty_db.bak'; Backup Summary - header = 11 bytes - meta-data = 2 bytes - data = 0 bytes - -------------- - total 13 bytes + backed up 0 tables + in 1 database + using 0 driver BACKUP DATABASE * TO 'all.bak'; Backup Summary - header = 16 bytes - meta-data = 4 bytes - data = 0 bytes - -------------- - total 20 bytes + backed up 0 tables + in 2 databases + using 0 driver DROP DATABASE empty_db; DROP DATABASE test; SHOW DATABASES; @@ -26,11 +22,9 @@ information_schema mysql RESTORE FROM 'all.bak'; Restore Summary - header = 16 bytes - meta-data = 4 bytes - data = 0 bytes - -------------- - total 20 bytes + restored 0 tables + in 2 databases + using 0 driver SHOW DATABASES; Database information_schema @@ -41,11 +35,9 @@ DROP DATABASE empty_db; DROP DATABASE test; RESTORE FROM 'empty_db.bak'; Restore Summary - header = 11 bytes - meta-data = 2 bytes - data = 0 bytes - -------------- - total 13 bytes + restored 0 tables + in 1 database + using 0 driver SHOW DATABASES; Database information_schema @@ -55,11 +47,9 @@ SHOW TABLES IN empty_db; Tables_in_empty_db RESTORE FROM 'all.bak'; Restore Summary - header = 16 bytes - meta-data = 4 bytes - data = 0 bytes - -------------- - total 20 bytes + restored 0 tables + in 2 databases + using 0 driver SHOW DATABASES; Database information_schema @@ -81,11 +71,9 @@ DROP VIEW IF EXISTS v1; CREATE VIEW v1 AS SELECT * FROM test.t1; BACKUP DATABASE empty_db TO 'empty_db.bak'; Backup Summary - header = 11 bytes - meta-data = 2 bytes - data = 0 bytes - -------------- - total 13 bytes + backed up 0 tables + in 1 database + using 0 driver SHOW DATABASES; Database information_schema @@ -94,11 +82,9 @@ mysql test RESTORE FROM 'empty_db.bak'; Restore Summary - header = 11 bytes - meta-data = 2 bytes - data = 0 bytes - -------------- - total 13 bytes + restored 0 tables + in 1 database + using 0 driver USE empty_db; SHOW TABLES; Tables_in_empty_db diff -Nrup a/mysql-test/r/backup_no_engine.result b/mysql-test/r/backup_no_engine.result --- a/mysql-test/r/backup_no_engine.result 2007-11-06 19:32:23 +01:00 +++ b/mysql-test/r/backup_no_engine.result 2007-11-30 09:23:28 +01:00 @@ -4,20 +4,16 @@ CREATE TABLE db.t1 (a int, b char(32)) ENGINE=myisam; BACKUP DATABASE db TO "db.backup"; Backup Summary - header = 12 bytes - meta-data = 120 bytes - data = 0 bytes - -------------- - total 132 bytes + backed up 1 tables + in 1 database + using 1 driver DROP DATABASE db; CREATE DATABASE db; RESTORE FROM "db.backup"; Restore Summary - header = 12 bytes - meta-data = 120 bytes - data = 0 bytes - -------------- - total 132 bytes + restored 1 tables + in 1 database + using 1 driver SHOW TABLES IN db; Tables_in_db t1 diff -Nrup a/mysql-test/r/backup_security.result b/mysql-test/r/backup_security.result --- a/mysql-test/r/backup_security.result 2007-11-09 22:32:08 +01:00 +++ b/mysql-test/r/backup_security.result 2007-11-30 09:23:28 +01:00 @@ -19,11 +19,9 @@ GRANT SUPER ON *.* TO 'bup_with_rights'@ default: Do backup of database with default test user for later tests. BACKUP DATABASE backup_test to 'backup_test_orig.bak'; Backup Summary - header = 21 bytes - meta-data = 92 bytes - data = 245 bytes - -------------- - total 358 bytes + backed up 1 tables + in 1 database + using 1 driver default: Connect as user with no rights and attempt backup and restore. no_rights: Attempting backup. Should fail with error 1227 BACKUP DATABASE backup_test to 'bup_no_rights.bak'; @@ -50,19 +48,15 @@ Connect as user with rights and attempt no_rights: Attempting backup. Should succeed BACKUP DATABASE backup_test to 'bup_with_rights.bak'; Backup Summary - header = 21 bytes - meta-data = 92 bytes - data = 245 bytes - -------------- - total 358 bytes + backed up 1 tables + in 1 database + using 1 driver no_rights: Attempting restore. Should succeed RESTORE FROM 'bup_with_rights.bak'; Restore Summary - header = 21 bytes - meta-data = 92 bytes - data = 245 bytes - -------------- - total 358 bytes + restored 1 tables + in 1 database + using 1 driver SELECT * FROM t1; a 01 Test #1 - super privilege @@ -75,11 +69,9 @@ a default: Do restore to ensure it still works with default test user. RESTORE FROM 'backup_test_orig.bak'; Restore Summary - header = 21 bytes - meta-data = 92 bytes - data = 245 bytes - -------------- - total 358 bytes + restored 1 tables + in 1 database + using 1 driver SELECT * FROM t1; a 01 Test #1 - super privilege diff -Nrup a/mysql-test/r/backup_snapshot.result b/mysql-test/r/backup_snapshot.result --- a/mysql-test/r/backup_snapshot.result 2007-11-16 02:29:25 +01:00 +++ b/mysql-test/r/backup_snapshot.result 2007-11-30 09:23:29 +01:00 @@ -36,11 +36,9 @@ SELECT release_lock("backup_cs_locked"); release_lock("backup_cs_locked") 1 Backup Summary - header = 29 bytes - meta-data = 184 bytes - data = 320 bytes - -------------- - total 533 bytes + backed up 2 tables + in 1 database + using 2 drivers INSERT INTO bup_snapshot.t1 VALUES("- Yes"); INSERT INTO bup_snapshot.t1 VALUES("- Jethro Tull"); DELETE FROM bup_snapshot.t1 WHERE word LIKE '10%'; @@ -58,11 +56,9 @@ DROP TABLE bup_snapshot.t1; con1: Restoring the database RESTORE FROM "bup_snapshot.bak"; Restore Summary - header = 29 bytes - meta-data = 184 bytes - data = 320 bytes - -------------- - total 533 bytes + restored 2 tables + in 1 database + using 2 drivers con1: Showing the data (no new data should be here). SELECT * FROM bup_snapshot.t1 WHERE word LIKE '-%'; word @@ -98,21 +94,17 @@ SELECT COUNT(*) FROM bup_snapshot.t1; COUNT(*) 12 Backup Summary - header = 29 bytes - meta-data = 184 bytes - data = 320 bytes - -------------- - total 533 bytes + backed up 2 tables + in 1 database + using 2 drivers con1: Dropping the database DROP TABLE bup_snapshot.t1; con1: Restoring the database RESTORE FROM "bup_snapshot.bak"; Restore Summary - header = 29 bytes - meta-data = 184 bytes - data = 320 bytes - -------------- - total 533 bytes + restored 2 tables + in 1 database + using 2 drivers con1: Showing the data (no new data should be here). SELECT * FROM bup_snapshot.t1 WHERE word LIKE '-%'; word diff -Nrup a/sql/backup/CMakeLists.txt b/sql/backup/CMakeLists.txt --- a/sql/backup/CMakeLists.txt 2007-11-29 20:34:05 +01:00 +++ b/sql/backup/CMakeLists.txt 2007-11-30 09:23:29 +01:00 @@ -22,9 +22,9 @@ INCLUDE_DIRECTORIES(${CMAKE_SOURCE_DIR}/ ${CMAKE_SOURCE_DIR}/regex ${CMAKE_SOURCE_DIR}/extra/yassl/include) -SET(BACKUP_SOURCES stream.cc logger.cc string_pool.cc +SET(BACKUP_SOURCES stream.cc logger.cc kernel.cc catalog.cc meta_data.cc data_backup.cc - kernel.cc be_default.cc buffer_iterator.cc + be_default.cc buffer_iterator.cc be_snapshot.cc be_thread.cc) IF(NOT SOURCE_SUBLIBS) @@ -32,5 +32,6 @@ IF(NOT SOURCE_SUBLIBS) ADD_LIBRARY(backupstream stream_v1.c stream_v1_transport.c) ENDIF(NOT SOURCE_SUBLIBS) +TARGET_LINK_LIBRARIES(backup backupstream) ADD_DEPENDENCIES(backup mysys) diff -Nrup a/sql/backup/Makefile.am b/sql/backup/Makefile.am --- a/sql/backup/Makefile.am 2007-11-29 20:34:05 +01:00 +++ b/sql/backup/Makefile.am 2007-11-30 09:23:29 +01:00 @@ -28,16 +28,17 @@ INCLUDES = \ libbackup_la_SOURCES = \ stream.cc \ logger.cc \ - string_pool.cc \ catalog.cc \ + kernel.cc \ meta_data.cc \ data_backup.cc \ - kernel.cc \ be_default.cc \ be_snapshot.cc \ buffer_iterator.cc \ be_thread.cc +libbackup_la_LIBADD = libbackupstream.la + libbackupstream_la_SOURCES= \ stream_v1_transport.c \ stream_v1.c @@ -46,15 +47,16 @@ noinst_HEADERS = \ api_types.h \ backup_engine.h \ backup_kernel.h \ + backup_stream.h \ + stream_services.h \ debug.h \ error.h \ stream.h \ backup_aux.h \ logger.h \ - map.h \ - string_pool.h \ catalog.h \ meta_data.h \ + be_native.h \ be_default.h \ be_snapshot.h \ buffer_iterator.h \ diff -Nrup a/sql/backup/backup_aux.h b/sql/backup/backup_aux.h --- a/sql/backup/backup_aux.h 2007-11-29 20:34:05 +01:00 +++ b/sql/backup/backup_aux.h 2007-11-30 09:23:29 +01:00 @@ -2,6 +2,7 @@ #define _BACKUP_AUX_H #include <backup/api_types.h> +#include <backup_stream.h> namespace backup { @@ -35,8 +36,135 @@ struct LEX_STRING: public ::LEX_STRING str= const_cast<char*>(s.ptr()); length= s.length(); } + + LEX_STRING(byte *begin, byte *end) + { + str= (char*)begin; + if( begin && end > begin) + length= end - begin; + else + length= 0; + } }; +class String: public ::String +{ + public: + + String(const ::String &s): ::String(s) + {} + + String(const ::LEX_STRING &s): + ::String(s.str,s.length,&::my_charset_bin) // FIXME: charset info + {} + + String(byte *begin, byte *end): + ::String((char*)begin,end-begin,&::my_charset_bin) // FIXME: charset info + { + if (!begin) + set((char*)NULL,0,NULL); + } + + String(const char *s): + ::String(s,&::my_charset_bin) + {} + + String(): ::String() + {} +}; + + +/* + + Dynamic_array<Foo> array; + + new (array.get_entry(7)) Foo(....); + + if (foo = array[7]) + { + foo->... + } + + TODO: Look at similar class in sql_array.h +*/ + +template<class X> +class Dynamic_array +{ + ::DYNAMIC_ARRAY m_array; + + public: + + Dynamic_array(uint init_size, uint alloc_increment) + { + my_init_dynamic_array(&m_array,1+sizeof(X),init_size,alloc_increment); + clear_free_space(); + } + + ~Dynamic_array() + { + for (uint pos=0; pos < m_array.elements; ++pos) + { + X *ptr= const_cast<X*>((*this)[pos]); + if (ptr) + ptr->~X(); + } + + delete_dynamic(&m_array); + } + + X* operator[](uint pos) const + { + if (pos >= m_array.elements) + return NULL; + + uchar *ptr= dynamic_array_ptr(&m_array,pos); + + return *ptr == 0xFF ? (X*)(ptr+1) : NULL; + } + + X* get_entry(uint pos) + { + uchar *entry; + + while (pos > m_array.max_element) + { + entry= alloc_dynamic(&m_array); + if (!entry) + break; + } + + clear_free_space(); + + if (pos > m_array.max_element) + return NULL; + + if (pos >= m_array.elements) + m_array.elements= pos+1; + + entry= dynamic_array_ptr(&m_array,pos); + X *ptr= (X*)(entry+1); + + if (*entry == 0xFF) + ptr->~X(); + + *entry= 0xFF; + return new (ptr) X(); + } + + uint size() const + { return m_array.elements; } + + private: + + void clear_free_space() + { + uchar *start= dynamic_array_ptr(&m_array,m_array.elements); + uchar *end= dynamic_array_ptr(&m_array,m_array.max_element); + if (end > start) + bzero(start, end - start); + } +}; TABLE_LIST *build_table_list(const Table_list&,thr_lock_type); @@ -76,6 +204,11 @@ inline int free_table_list(TABLE_LIST *a } return 0; } + +// These functions are implemented in kernel.cc + +int silent_exec_query(THD*, ::String&); +void save_current_time(bstream_time_t &buf); } // backup namespace diff -Nrup a/sql/backup/backup_kernel.h b/sql/backup/backup_kernel.h --- a/sql/backup/backup_kernel.h 2007-11-29 20:34:05 +01:00 +++ b/sql/backup/backup_kernel.h 2007-11-30 09:23:29 +01:00 @@ -3,26 +3,30 @@ #include <backup/api_types.h> #include <backup/catalog.h> -#include <backup/stream.h> #include <backup/logger.h> - -/* - Called from the big switch in mysql_execute_command() to execute - backup related statement - */ -int execute_backup_command(THD*, LEX*); - /** @file Functions and types forming the backup kernel API +*/ - */ + +/** + @brief Size of the buffer used for transfers between backup kernel and + backup/restore drivers. +*/ +#define DATA_BUFFER_SIZE (1024*1024) + +/* + Called from the big switch in mysql_execute_command() to execute + backup related statement +*/ +int execute_backup_command(THD*, LEX*); namespace backup { -class Archive_info; +class Image_info; class Backup_info; class Restore_info; @@ -30,7 +34,6 @@ class Restore_info; // Backup kernel API -int mysql_show_archive(THD*,const backup::Archive_info&); int mysql_backup(THD*, backup::Backup_info&, backup::OStream&); int mysql_restore(THD*, backup::Restore_info&, backup::IStream&); @@ -77,7 +80,7 @@ struct Location /** - Specialization of @c Archive_info which adds methods for selecting items + Specialization of @c Image_info which adds methods for selecting items to backup. When Backup_info object is created it is empty and ready for adding items @@ -86,20 +89,9 @@ struct Location supported). After populating info object with items it should be "closed" with a call to @c close() method. After that it is ready for use as a description of backup archive to be created. - - A linked list of all meta-data items is pointed by @c m_items member. It - consists of three parts: first all the global items, then all per-database - items and finally all per-table items. Inside each part, items are stored in - dependency order so that if item A depends on B then B is before A in the - list (currently dependencies are not checked). One should iterate through the - meta-data item list using @c Backup_info::Item_iterator class. - */ - -class Backup_info: public Archive_info, public Logger +*/ +class Backup_info: public Image_info, public Logger { - class Table_ref; - class Db_ref; - public: Backup_info(THD*); @@ -129,15 +121,11 @@ class Backup_info: public Archive_info, return ok; } - int save(OStream&); - int add_dbs(List< ::LEX_STRING >&); int add_all_dbs(); bool close(); - class Item_iterator; // for iterating over all meta-data items - private: /// State of the info structure. @@ -147,67 +135,64 @@ class Backup_info: public Archive_info, ERROR } m_state; - int find_image(const Table_ref&); - - int default_image_no; ///< Position of the default image in @c images list, -1 if not used. - int snapshot_image_no; ///< Position of the snapshot image in @c images list, -1 if not used. + int find_backup_engine(const ::TABLE *const, const Table_ref&); - Db_item* add_db(const backup::Db_ref&); - Table_item* add_table(const Table_ref&); - - /// Value returned by @c add_table if it decides that the table should be skipped. - static const Table_item *const skip_table; + Table_item* add_table(Db_item&, const Table_ref&); int add_db_items(Db_item&); int add_table_items(Table_item&); THD *m_thd; TABLE *i_s_tables; + String binlog_file_name; ///< stores name of the binlog at VP time - Item *m_items; - Item *m_last_item; - Item *m_last_db; + /** + @brief Storage for table and database names. - friend class Item_iterator; -}; + When adding tables or databases to the backup catalogue, their names + are stored in String objects, and these objects are appended to this + list so that they can be freed when Backup_info object is destroyed. + */ + // FIXME: use better solution, e.g., MEM_ROOT + List<String> name_strings; -class Backup_info::Item_iterator: public Archive_info::Item::Iterator -{ - public: - Item_iterator(const Backup_info &info): - Archive_info::Item::Iterator(info.m_items) - {} + void save_binlog_pos(const ::LOG_INFO &li) + { + binlog_file_name= li.log_file_name; + binlog_file_name.copy(); + binlog_pos.pos= li.pos; + binlog_pos.file= binlog_file_name.c_ptr(); + } + + friend int write_table_data(THD*, Backup_info&, OStream&); }; + /** - Specialization of @c Archive_info which is used to select and restore items - from a backup archive. + Specialization of @c Image_info which is used to select and restore items + from a backup image. - An instance of this class is created by reading backup archive header and it - describes contents of the archive. @c Restore_info methods select which items - should be restored. Instances of @c Restore_info::Item class are created when - reading meta-data info stored in the archive. They are used to restore the - meta-data items (but not the table data, which is done by restore drivers). + An instance of this class is created by reading backup image header and it + describes its contents. @c Restore_info methods select which items + should be restored. @note This class is not fully implemented. Right now it is not possible to select items to restore - always all items are restored. */ -class Restore_info: public Archive_info, public Logger +class Restore_info: public Image_info, public Logger { bool m_valid; + THD *m_thd; + const Db_ref *curr_db; + + CHARSET_INFO *system_charset; + bool same_sys_charset; public: - Restore_info(IStream &s): Logger(Logger::RESTORE), m_valid(TRUE) - { - result_t res= read(s); - if (res == ERROR) - { - report_error(ER_BACKUP_READ_HEADER); - m_valid= FALSE; - } - } + Restore_info(THD*, IStream&); + ~Restore_info(); bool is_valid() const { return m_valid; } @@ -216,8 +201,14 @@ class Restore_info: public Archive_info, { return 0; } /// Determine if given item is selected for restore. - bool selected(const Archive_info::Item&) + bool selected(const Image_info::Item&) { return TRUE; } + + result_t restore_item(Item&, ::String&, byte*, byte*); + + friend int restore_table_data(THD*, Restore_info&, IStream&); + friend int ::bcat_add_item(st_bstream_image_header*, + struct st_bstream_item_info*); }; } // backup namespace diff -Nrup a/sql/backup/backup_stream.h b/sql/backup/backup_stream.h --- /dev/null Wed Dec 31 16:00:00 196900 +++ b/sql/backup/backup_stream.h 2007-11-30 09:23:30 +01:00 @@ -0,0 +1,15 @@ +#ifndef BACKUP_STREAM_H_ +#define BACKUP_STREAM_H_ + +// magic bytes defined in stream.cc +extern const unsigned char backup_magic_bytes[8]; + +extern "C" { + +// We use version 1 of the stream format. + +#include "stream_v1.h" + +} + +#endif /*BACKUP_STREAM_H_*/ diff -Nrup a/sql/backup/be_default.h b/sql/backup/be_default.h --- a/sql/backup/be_default.h 2007-11-29 20:34:05 +01:00 +++ b/sql/backup/be_default.h 2007-11-30 09:23:29 +01:00 @@ -200,22 +200,24 @@ class Restore: public Restore_driver /********************************************************************* - Default image class + Default snapshot class *********************************************************************/ namespace backup { -class Default_image: public Image_info +class Default_snapshot: public Snapshot_info { public: - Default_image(Archive_info &info): Image_info(info) - { ver= 1; } + Default_snapshot() + { + version= 1; + } - image_type type() const - { return DEFAULT_IMAGE; } + enum_snap_type type() const + { return DEFAULT_SNAPSHOT; } const char* name() const { return "Default"; } @@ -224,21 +226,11 @@ class Default_image: public Image_info { return TRUE; }; // accept all tables result_t get_backup_driver(Backup_driver* &ptr) - { return (ptr= new default_backup::Backup(tables,::current_thd, + { return (ptr= new default_backup::Backup(m_tables,::current_thd, TL_READ_NO_INSERT)) ? OK : ERROR; } result_t get_restore_driver(Restore_driver* &ptr) - { return (ptr= new default_backup::Restore(tables,::current_thd)) ? OK : ERROR; } - - result_t do_write_description(OStream&) - { return OK; } // nothing to write - - static result_t - create_from_stream(version_t, Archive_info &info, IStream&, - Image_info* &ptr) - { - return (ptr= new Default_image(info)) ? OK : ERROR; - } + { return (ptr= new default_backup::Restore(m_tables,::current_thd)) ? OK : ERROR; } bool is_valid(){ return TRUE; }; diff -Nrup a/sql/backup/be_native.h b/sql/backup/be_native.h --- /dev/null Wed Dec 31 16:00:00 196900 +++ b/sql/backup/be_native.h 2007-11-30 09:23:30 +01:00 @@ -0,0 +1,74 @@ +#ifndef BE_NATIVE_H_ +#define BE_NATIVE_H_ + +#include <backup_engine.h> + +namespace backup { + +/** + Specialization of @c Image_info for images created by native backup drivers. + */ +class Native_snapshot: public Snapshot_info +{ + const ::handlerton *m_hton; ///< Pointer to storage engine. + Engine *m_be; ///< Pointer to the native backup engine. + const char *m_name; ///< Saved name of storage engine. + unsigned int se_ver; ///< Storage engine version number. + + public: + + Native_snapshot(const ::plugin_ref se): m_hton(NULL), m_be(NULL) + { + m_hton= plugin_data(se,::handlerton*); + se_ver= (*se)->plugin->version; + + DBUG_ASSERT(m_hton); + DBUG_ASSERT(m_hton->get_backup_engine); + + result_t ret= m_hton->get_backup_engine(const_cast< ::handlerton* >(m_hton),m_be); + + if (ret != OK || !m_be) + return; + + version= m_be->version(); + m_name= ::ha_resolve_storage_engine_name(m_hton); + } + + ~Native_snapshot() + { + if (m_be) + m_be->free(); + } + + bool is_valid() + { return m_be != NULL; } + + enum_snap_type type() const + { return NATIVE_SNAPSHOT; } + + const char* name() const + { return m_name; } + + bool accept(const Table_ref&, const ::handlerton *hton) + { return hton == m_hton; }; // this assumes handlertons are single instance objects! + + result_t get_backup_driver(Backup_driver* &drv) + { + DBUG_ASSERT(m_be); + return m_be->get_backup(Driver::PARTIAL,m_tables,drv); + } + + result_t get_restore_driver(Restore_driver* &drv) + { + DBUG_ASSERT(m_be); + return m_be->get_restore(version,Driver::PARTIAL,m_tables,drv); + } + + friend void save_snapshot_info(const Snapshot_info&, + st_bstream_snapshot_info&); +}; + + +} // backup namespace + +#endif /*BE_NATIVE_H_*/ diff -Nrup a/sql/backup/be_snapshot.h b/sql/backup/be_snapshot.h --- a/sql/backup/be_snapshot.h 2007-11-29 20:34:05 +01:00 +++ b/sql/backup/be_snapshot.h 2007-11-30 09:23:29 +01:00 @@ -110,39 +110,31 @@ class Restore: public default_backup::Re namespace backup { -class Snapshot_image: public Image_info +class CS_snapshot: public Snapshot_info { public: - Snapshot_image(Archive_info &info): Image_info(info) - { ver= 1; } + CS_snapshot() + { + version= 1; + } - image_type type() const - { return SNAPSHOT_IMAGE; } + enum_snap_type type() const + { return CS_SNAPSHOT; } const char* name() const { return "Snapshot"; } bool accept(const Table_ref&, const ::handlerton* h) - { - return (h->start_consistent_snapshot != NULL); + { + return (h->start_consistent_snapshot != NULL); }; // accept all tables that support consistent read result_t get_backup_driver(Backup_driver* &ptr) - { return (ptr= new snapshot_backup::Backup(tables,::current_thd)) ? OK : ERROR; } + { return (ptr= new snapshot_backup::Backup(m_tables,::current_thd)) ? OK : ERROR; } result_t get_restore_driver(Restore_driver* &ptr) - { return (ptr= new snapshot_backup::Restore(tables,::current_thd)) ? OK : ERROR; } - - result_t do_write_description(OStream&) - { return OK; } // nothing to write - - static result_t - create_from_stream(version_t, Archive_info &info, IStream&, - Image_info* &ptr) - { - return (ptr= new Snapshot_image(info)) ? OK : ERROR; - } + { return (ptr= new snapshot_backup::Restore(m_tables,::current_thd)) ? OK : ERROR; } bool is_valid(){ return TRUE; }; diff -Nrup a/sql/backup/catalog.cc b/sql/backup/catalog.cc --- a/sql/backup/catalog.cc 2007-11-29 20:34:06 +01:00 +++ b/sql/backup/catalog.cc 2007-11-30 09:23:29 +01:00 @@ -1,823 +1,239 @@ #include "../mysql_priv.h" -/** - @file - - Implementation of @c Archive_info and related classes. - */ - -/* - TODO: - - - Add to Archive_info storage for other meta-data items. - - Make existing storage solutions more rational (e.g., string pool). - - Make reading code resistant to unknown image formats or meta-data types - (or, assume it is handled by format version number). - - Improve Image_info::Tables implementation (use some existing data structure). - - Add more information to backup archive header , for example server's version - string. - - Handle backward compatibility (new code reads archive with earlier version - number) - - Add to Archive_info methods for browsing contents of the archive. - */ - -#if defined(USE_PRAGMA_IMPLEMENTATION) || defined(__APPLE_CC__) -/* - #pragma implementation is needed on powermac platform as otherwise compiler - doesn't create/export vtable for Image_info::Tables class (if you know a - better way for fixing this issue let me know! /Rafal). - - Apparently, configuration macro USE_PRAGMA_IMPLEMENTATION is not set by - ./configure on powermac platform - this is why __APPLE_CC__ is also checked. - */ -#pragma implementation -#endif - -#include "backup_engine.h" +#include <backup_stream.h> #include "backup_aux.h" #include "catalog.h" -#include "be_default.h" #include "be_snapshot.h" +#include "be_default.h" +#include "be_native.h" +/** + @file -/*************************************** - - Implementation of Archive_info class + @brief Implements @c Image_info class and friends. - ***************************************/ + @todo Error reporting + @todo Store endianess info in the image. +*/ namespace backup { -Archive_info::~Archive_info() -{ - for (uint i=0; i<256; ++i) - if (images[i]) - { - delete images[i]; - images[i]= NULL; - } -} - +/* Image_info implementation */ -/** - Write header and catalogue of a backup archive. - - Header forms the first chunk of archive. Currently it contains archive's - format version number followed by a list of table data images used in the - archive. - - Next chunk contains the pool of database names. After that there is one chunk - per image containing list of tables whose data is saved in that image. - @verbatim - ===================== - version number } - --------------------- } header - image descriptions } - ===================== - db names } - ===================== } - tables of image 1 } - ===================== } - } catalogue - ... } - } - ===================== } - tables of image N } - ===================== - @endverbatim - In the picture "====" denotes chunk boundaries. Number of images is known - from the header. - - The format in which image descriptions are saved is determined by - @c Image_info::write_description() method. - - For list of databases and tables the format in which they are saved is - defined by @c StringPool::save() and @c Archive_info::Tables::save() methods - respectively. - */ - -result_t Archive_info::save(OStream &s) +Image_info::Image_info(): + table_count(0) { - DBUG_ENTER("Archive_info::save"); - - size_t start_bytes= s.bytes; - stream_result::value res; - - res= s.write2int(ver); - if (res != stream_result::OK) - { - DBUG_PRINT("backup",("Can't write archive version number (stream_res=%d)",(int)res)); - DBUG_RETURN(ERROR); - } - - // write list of images - DBUG_PRINT("backup",(" writing image list")); - - uint ino; - Image_info *img; - - for (ino=0; ino < img_count ; ++ino) - if ((img= images[ino])) - { - DBUG_PRINT("backup",(" %2d: %s image",ino,img->name())); - if (ERROR == img->write_description(s)) - { - DBUG_PRINT("backup",("Can't write description of %s image (#%d)", - img->name(),ino)); - DBUG_RETURN(ERROR); - } - } - - // close the header chunk - res= s.end_chunk(); - if (res != stream_result::OK) - { - DBUG_PRINT("backup",("Error when closing image list chunk (stream_res=%d)",(int)res)); - DBUG_RETURN(ERROR); - } - + /* initialize st_bstream_image_header members */ + version= 1; - // write catalogue - DBUG_PRINT("backup",(" writing archive catalogue")); - - // db names (one chunk) - if (ERROR == db_names.save(s)) // note: this closes the chunk - { - DBUG_PRINT("backup",("Error saving pool of db names (stream_res=%d)",(int)res)); - DBUG_RETURN(ERROR); - } + /* + The arithmetic below assumes that MYSQL_VERSION_ID digits are arrenged + as follows: HLLRR where + H - major version number + L - minor version number + R - release - // table lists (one chunk per image/list) - for (ino=0; ino < img_count ; ++ino) - if ((img= images[ino])) - { - DBUG_PRINT("backup",(" saving %s image's tables",img->name())); - if (ERROR == img->tables.save(s)) - { - DBUG_PRINT("backup",("Error saving tables (stream_res=%d)",(int)res)); - DBUG_RETURN(ERROR); - } - } + TODO: check if this is correct + */ + DBUG_PRINT("backup",("version %d",MYSQL_VERSION_ID)); + server_version.major= MYSQL_VERSION_ID / 10000; + server_version.minor= (MYSQL_VERSION_ID % 10000) / 100; + server_version.release= MYSQL_VERSION_ID % 100; + server_version.extra.begin= (byte*)MYSQL_SERVER_VERSION; + server_version.extra.end= server_version.extra.begin + + strlen((const char*)server_version.extra.begin); - header_size= s.bytes - start_bytes; + flags= 0; // TODO: set BSTREAM_FLAG_BIG_ENDIAN flag accordingly + snap_count= 0; - DBUG_RETURN(OK); + bzero(&start_time,sizeof(start_time)); + bzero(&end_time,sizeof(end_time)); + bzero(&vp_time,sizeof(vp_time)); + bzero(&binlog_pos,sizeof(binlog_pos)); + bzero(&binlog_group,sizeof(binlog_group)); + bzero(m_snap, sizeof(m_snap)); } -/** - Fill @c Archive_info structure reading data from backup archive header and - catalogue. - - @returns OK or ERROR - */ -result_t Archive_info::read(IStream &s) +Image_info::~Image_info() { - DBUG_ENTER("Archive_info::read"); - - size_t start_bytes= s.bytes; - result_t res; - version_t ver; - - /* - We read archive's header which starts with archive format version number. - If we can't read the version number (end of stream or no data in the chunk) - there is something wrong and we signal error. - */ - - stream_result::value rres= s.read2int(ver); - if ( rres != stream_result::OK) - { - DBUG_PRINT("restore",("Error reading archive version number" - " (stream_res=%d)",(int)rres)); - DBUG_RETURN(ERROR); - } - - if (ver != Archive_info::ver) - { - DBUG_PRINT("restore",("Backup archive version %d not supported",ver)); - DBUG_RETURN(ERROR); - } - - /* - What follows (until the end of the data chunk) is a list of entries - describing data images of the archive. It is read using - Image_info::create_from_stream() function which returns DONE when end of - chunk is reached. - */ - - DBUG_PRINT("restore",(" reading image list")); - - uint ino= 0; - - do - { - Image_info *img; - - res= Image_info::create_from_stream(*this,s,img); - - if (res == OK) - { - DBUG_ASSERT(img); - DBUG_PRINT("restore",(" %2d: %s image",ino,img->name())); - images[ino++]= img; - } - - } while (res == OK && ino < MAX_IMAGES); - - img_count= ino; + // Delete snapshot objects - // If res != DONE we haven't reached end of the chunk - something is wrong - if (res != DONE) + for (uint no=0; no<256; ++no) { - DBUG_PRINT("restore",("Error when reading image list (%d images read)", - img_count)); - DBUG_RETURN(ERROR); - } - - /* - Next chunk starts archive's catalogue. We proceed with reading it. - Note that the catalogue should always contain at least one chunk (db names - pool) and hence we should not hit end of stream here. - */ - - table_count= 0; - - if (s.next_chunk() != stream_result::OK) - { - DBUG_PRINT("restore",("Can't proceed to the catalogue")); - DBUG_RETURN(ERROR); + if (m_snap[no]) + delete m_snap[no]; + m_snap[no]= NULL; } +} - DBUG_PRINT("restore",(" reading catalogue (%d images)",img_count)); +/// Add table to database's table list. +result_t Image_info::Db_item::add_table(Table_item &t) +{ + t.next_table= NULL; + t.base.db= this; - /* - First chunk of the catalogue contains db names pool - we read it and - proceed to the next chunk. We should never hit end of stream here and - so the only acceptable result of db.names.read() is OK. - */ - res= db_names.read(s); - if (res != OK) + if (!m_last_table) { - DBUG_PRINT("restore",("Can't read db names pool (res=%d)",(int)res)); - DBUG_RETURN(ERROR); + m_tables= m_last_table= &t; } - - /* - The following chunks contain lists of tables for each image. There are - as many lists as there are images (possibly 0) and each list occupies - one chunk. - */ - for (uint ino=0; ino < img_count; ++ino) + else { - Image_info *img= images[ino]; - - DBUG_PRINT("restore",(" reading %s image's tables (#%d)",img->name(),ino)); - - /* - There should be as many lists in the stream as there were images in the - header. Thus we should never hit end of stream here. - */ - res= img->tables.read(s); // note: proceeds to the next chunk in the stream - - if (res != OK) - { - DBUG_PRINT("restore",("Can't read table list for %s image (#%d)", - img->name(),ino)); - DBUG_RETURN(ERROR); // neither stream nor chunk should end here - } - - table_count+= img->tables.count(); - - DBUG_PRINT("restore",(" finished reading tables")); + m_last_table->next_table= &t; + m_last_table= &t; } - header_size= s.bytes - start_bytes; - - DBUG_RETURN(OK); -} - -} // backup namespace - - -/********************************** - - Write/read image descriptions - - **********************************/ - -namespace backup { + table_count++; -/** - Write entry describing (format of) a backup driver's image. - - Entry has the form: - @verbatim - | type | version | image description | - @endverbatim - where type is a byte holding Image_info::image_type value, version is 2 byte - integer holding image format version. The format of optional image description - is determined by @c X::do_write_description() method where X is a subclass of - Image_info corresponding to given image type. - */ -result_t -Image_info::write_description(OStream &s) -{ - // TODO: to handle unknown description formats, write description length here - - stream_result::value res= s.writebyte(type()); - - if (res != stream_result::OK) - return ERROR; - - res= s.write2int(ver); - - if (res != stream_result::OK) - return ERROR; - - return do_write_description(s); + return OK; } /** - Create @c Image_info instance from a saved entry describing it. + Locate in the catalogue an object described by the @c st_bstream_item_info + structure. - @retval OK - @retval DONE end of chunk/stream hit - @retval ERROR - */ -result_t -Image_info::create_from_stream(Archive_info &info, IStream &s, Image_info* &ptr) + @todo Handle unknown item types. +*/ +Image_info::Item* +Image_info::locate_item(const st_bstream_item_info *item) const { - uint ver; - byte t; - - stream_result::value res= s.readbyte(t); - - // if we are at end of data chunk or stream, we should tell the caller - if (res != stream_result::OK) - return report_stream_result(res); - - res= s.read2int(ver); - - if (res != stream_result::OK) - return ERROR; - - switch (image_type(t)) { + switch (item->type) { - case NATIVE_IMAGE: - return Native_image::create_from_stream(ver,info,s,ptr); + case BSTREAM_IT_DB: + return get_db(item->pos); - case DEFAULT_IMAGE: - return Default_image::create_from_stream(ver,info,s,ptr); - - case SNAPSHOT_IMAGE: - return Snapshot_image::create_from_stream(ver,info,s,ptr); + case BSTREAM_IT_TABLE: + { + const st_bstream_table_info *ti= reinterpret_cast<const st_bstream_table_info*>(item); + return get_table(ti->snap_no,item->pos); + } default: - DBUG_PRINT("restore",("Unknown image type %d",t)); - return ERROR; + // TODO: warn or report error + return NULL; } } } // backup namespace -/******************* - - Serialization of meta-data items +/* catalogue services for backup stream library */ - *******************/ +extern "C" { -namespace backup { - -/** - Write an entry describing single meta-data item. +/* iterators */ - Entry has format: - @verbatim - | type | id data | create data | - @endverbatim - Type is a single byte holding meta::Item::enum_type value. Id data is - used to determine which item (from the archive catalogue) the entry - corresponds to. Create data is used to create the item. - - The format of id data and create data for item of type X is determined - by methods @c Archive_info::X_item::save_id() and @c meta::X::save(), - respectively. +static uint cset_iter; ///< Used to implement trivial charset iterator. +static uint null_iter; ///< Used to implement trivial empty iterator. - @see @c write_meta_data() for information about the format of the meta-data - section of backup archive. -*/ -result_t -Archive_info::Item::save(THD *thd, OStream &s) +void* bcat_iterator_get(st_bstream_image_header *catalogue, unsigned int type) { - byte b= meta().type(); - - stream_result::value res=s.writebyte(b); + switch (type) { - if (res != stream_result::OK) - return ERROR; + case BSTREAM_IT_PERDB: + case BSTREAM_IT_PERTABLE: + return &null_iter; - if (ERROR == save_id(s)) - return ERROR; - - return meta().save(thd,s); -} + case BSTREAM_IT_CHARSET: + cset_iter= 0; + return &cset_iter; -/** - Create meta-data item from a saved entry. - - This function reads the type byte and calls @c create_from_stream method of - corresponding class to create the item. It stores pointer to the created - item in @c ptr argument. - - @retval OK if new item was created - @retval DONE if end of chunk/stream was reached - @retval ERROR if error has happened - */ -result_t -Archive_info::Item::create_from_stream(const Archive_info &info, - IStream &s, Item* &ptr) -{ - byte b; + case BSTREAM_IT_USER: + return &null_iter; - stream_result::value res= s.readbyte(b); + case BSTREAM_IT_GLOBAL: + // only global items (for which meta-data is stored) are databases + case BSTREAM_IT_DB: + return + new backup::Image_info::Db_iterator(*static_cast<backup::Image_info*>(catalogue)); + // TODO: report error if iterator could not be created - if (res != stream_result::OK) - return report_stream_result(res); - - ptr= NULL; - - result_t res1; - - switch (meta::Item::enum_type(b)) { - - case meta::Item::DB: - res1= Db_item::create_from_stream(info,s,ptr); - break; - - case meta::Item::TABLE: - res1= Table_item::create_from_stream(info,s,ptr); - break; - - default: return ERROR; + default: + return NULL; } - - /* - Note that create_from_stream() should return OK - end of data should not - happen here. - */ - if (res1 != OK || ptr == NULL) - return ERROR; - - return ptr->meta().read(s); -} - -// Db items - -result_t Archive_info::Db_item::save_id(OStream &s) -{ - uint k= key; - DBUG_PRINT("backup",(" saving db-item (%d)",k)); - return stream_result::OK == s.writeint(k) ? OK : ERROR; } -result_t -Archive_info::Db_item::create_from_stream(const Archive_info &i, - IStream &s, - Archive_info::Item* &ptr) +struct st_bstream_item_info* +bcat_iterator_next(st_bstream_image_header *catalogue, void *iter) { - uint k; - stream_result::value res= s.readint(k); - - if (res != stream_result::OK) - return report_stream_result(res); - - return (ptr= new Db_item(i,k)) ? OK : ERROR; -} - -// Table items - -result_t Archive_info::Table_item::save_id(OStream &s) -{ - DBUG_PRINT("backup",(" saving table-item (%d,%d)",img,pos)); - stream_result::value res= s.writeint(img); - - if (res != stream_result::OK) - return ERROR; - - res= s.writeint(pos); - - if (res != stream_result::OK) - return ERROR; - - return OK; -} - -result_t -Archive_info::Table_item::create_from_stream(const Archive_info &i, - IStream &s, - Archive_info::Item* &ptr) -{ - uint img,no; - stream_result::value res= s.readint(img); - - if (res != stream_result::OK) - return report_stream_result(res); + /* If this is the null iterator, return NULL immediately */ + if (iter == &null_iter) + return NULL; - res= s.readint(no); + static bstream_blob name= {NULL, NULL}; - if (res != stream_result::OK) - return ERROR; - - return (ptr= new Table_item(i,img,no)) ? OK : ERROR; -} - -} // backup namespace - - -/********************************** - - Implementation of Image_info::Tables - - **********************************/ - -namespace backup { - -// TODO: use better implementation (red-black tree from mysys?) - -struct Image_info::Tables::node { - StringPool::Key db; - String name; - node *next; - - node(const Image_info::Tables&, - const StringPool::Key &k, - const String &nm): db(k), next(NULL) - { - name.copy(nm); - } -}; - -/// Empty the list. -void Image_info::Tables::clear() -{ - for (node *ptr= m_head; ptr;) + /* + If it is cset iterator then cset_iter variable contains iterator position. + We return only 2 charsets: the utf8 charset used to encode all strings and + the default server charset. + */ + if (iter == &cset_iter) { - node *n=ptr; - ptr= n->next; - delete n; - } - - m_head= m_last= NULL; - m_count= 0; -} - -/** - Add a table to the list. - - @returns Position of the table or -1 if error - */ -int Image_info::Tables::add(const backup::Table_ref &t) -{ - StringPool::Key k= m_db_names.add(t.db().name()); - - if (!k.is_valid()) - return -1; - - return add(k,t.name()); -} - -/// Add table at given position. -int Image_info::Tables::add(const StringPool::Key &k, const String &name) -{ - node *n= new node(*this,k,name); + switch (cset_iter) { + case 0: name.begin= (byte*)::my_charset_utf8_bin.csname; break; + case 1: name.begin= (byte*)::system_charset_info->csname; break; + default: name.begin= NULL; break; + } - if (!n) - return -1; + name.end= name.begin ? name.begin + strlen((char*)name.begin) : NULL; + cset_iter++; - if (m_head == NULL) - { - m_count=1; - m_head= m_last= n; + return name.begin ? (st_bstream_item_info*)&name : NULL; } - else - { - m_count++; - m_last->next= n; - m_last= n; - }; - - return m_count-1; -} - -/** - Locate table at given position. - @returns Pointer to table's list node or NULL if position is not occupied - */ -Image_info::Tables::node* -Image_info::Tables::find_table(uint pos) const -{ - DBUG_ASSERT(pos < m_count); - - node *ptr; - - for (ptr= m_head; ptr && pos; ptr= ptr->next) - pos--; - - //if( !ptr ) ptr= m_last; + /* + In all other cases assume that iter points at instance of + @c Image_info::Iterator and use this instance to get next item. + */ + const backup::Image_info::Item *ptr= (*(backup::Image_info::Iterator*)iter)++; - return ptr; + return ptr ? (st_bstream_item_info*)(ptr->info()) : NULL; } -/// Return table at a given position. -inline -Table_ref Image_info::Tables::operator[](uint pos) const +void bcat_iterator_free(st_bstream_image_header *catalogue, void *iter) { - // Get access to backup::Table_ref protected constructor - - struct Table_ref: public backup::Table_ref - { - Table_ref(const StringPool &db_names, Image_info::Tables::node &n): - backup::Table_ref(db_names[n.db],n.name) - {} - }; + /* + Do nothing for the null and cset iterators, but delete the + @c Image_info::Iterator object otherwise. + */ + if (iter == &null_iter) + return; - node *ptr= find_table(pos); - DBUG_ASSERT(ptr); + if (iter == &cset_iter) + return; - return Table_ref(m_db_names,*ptr); + delete (backup::Image_info::Iterator*)iter; } -/****************** - - Serialization for Image_info::Tables class - - ******************/ +/* db-items iterator */ -/** - Save list of tables in a backup stream. - - The format used assumes that a pool of database names is stored elsewhere. - Thus for each table only the key of the database is stored as var-length - integer followed by table name. Empty list is saved as single NIL value. - - The list is stored in a single stream chunk which determines its end. - - @returns OK or ERROR - */ -result_t -Image_info::Tables::save(OStream &s) +void* bcat_db_iterator_get(st_bstream_image_header *catalogue, struct st_bstream_db_info *db) { - DBUG_ENTER("Image_info::Tables::save"); - stream_result::value res; + using namespace backup; - if (count() == 0) - { - res= s.writenil(); - if (res != stream_result::OK) - DBUG_RETURN(ERROR); - } - else - for (Tables::node *n= m_head ; n ; n= n->next) - { - res= s.writeint(n->db); - if (res != stream_result::OK) - DBUG_RETURN(ERROR); - - res= s.writestr(n->name); - if (res != stream_result::OK) - DBUG_RETURN(ERROR); - }; + Image_info::Db_item *dbi = static_cast<Image_info::Db_item*>(db); - res= s.end_chunk(); - DBUG_RETURN(res == stream_result::ERROR ? ERROR : OK); + return + new Image_info::Ditem_iterator(*static_cast<backup::Image_info*>(catalogue), + *dbi); } -/** - Read a list from a backup stream. - - @pre Stream is positioned at the first entry of the saved list. - @post Stream is positioned at the beginning of next chunk or at its end. - - @retval OK - @retval DONE end of stream or chunk hit (nothing has been read) - @retval ERROR - */ -result_t -Image_info::Tables::read(IStream &s) +struct st_bstream_dbitem_info* +bcat_db_iterator_next(st_bstream_image_header *catalogue, + struct st_bstream_db_info *db, + void *iter) { - DBUG_ENTER("Image_info::Tables::read"); - - stream_result::value res; - uint k,tno=0; - - /* - Read first entry - if it is NIL, we have empty list. Otherwise it should - be db index of the first table. - */ - - res= s.readint(k); - - // If unexpected result, report an error or end of stream/chunk - if (res != stream_result::OK && res != stream_result::NIL) - DBUG_RETURN(report_stream_result(res)); - - // empty the list - clear(); - - if (res == stream_result::OK) // this is non-empty list - do - { - String name; - - res= s.readstr(name); - if (res != stream_result::OK) - break; - - tno= add(k,name); - DBUG_PRINT("restore",("got next table %s.%s (pos %d, dbkey %d)", - (*this)[tno].db().name().ptr(), - (*this)[tno].name().ptr(),tno,k)); - res= s.readint(k); - } - while (res == stream_result::OK); - else - /* - If we have read NIL value, pretend we are at the end of chunk so that - no errors are reported below. - */ - res= stream_result::EOC; + const backup::Image_info::Item *ptr= (*(backup::Image_info::Iterator*)iter)++; - // we should be now at end of chunk/stream - if (res != stream_result::EOC && res != stream_result::EOS) - { - DBUG_PRINT("restore",("Error when reading table no %d in table list",tno)); - DBUG_RETURN(ERROR); - } - - res= s.next_chunk(); - DBUG_RETURN(res == stream_result::ERROR ? ERROR : OK); + return ptr ? (st_bstream_dbitem_info*)ptr->info() : NULL; } -} // backup namespace - -/************************************** - - Native image type definition - - **************************************/ - -namespace backup { - -/* - For native image its format (apart from the version number) is determined - by the storage engine whose backup driver created it. Thus we save the name - of storage engine. - - TODO: add more information here. E.g. the version number of the storage engine. - */ -result_t -Native_image::do_write_description(OStream &s) +void bcat_db_iterator_free(st_bstream_image_header *catalogue, + struct st_bstream_db_info *db, + void *iter) { - String name(::ha_resolve_storage_engine_name(m_hton),&::my_charset_bin); - return stream_result::OK == s.writestr(name) ? OK : ERROR; + delete (backup::Image_info::Ditem_iterator*)iter; } -result_t -Native_image::create_from_stream(version_t ver, - Archive_info &info, - IStream &s, Image_info* &img) -{ - String name; - stream_result::value res= s.readstr(name); - - if (res != stream_result::OK) - return report_stream_result(res); - - LEX_STRING name_lex= name; - - - ::handlerton *hton= plugin_data(::ha_resolve_by_name(::current_thd,&name_lex), - handlerton*); - if (!hton) - return ERROR; - - img= new Native_image(info,hton); - if (!img) - return ERROR; - - if (ver > img->ver) - { - DBUG_PRINT("restore",("Restore diver version %d can't read image version %d", - img->ver,ver)); - return ERROR; - } - - img->ver= ver; - - return OK; } -} // backup namespace diff -Nrup a/sql/backup/catalog.h b/sql/backup/catalog.h --- a/sql/backup/catalog.h 2007-11-29 20:34:06 +01:00 +++ b/sql/backup/catalog.h 2007-11-30 09:23:29 +01:00 @@ -1,500 +1,783 @@ -#ifndef _BACKUP_ARCHIVE_H -#define _BACKUP_ARCHIVE_H +#ifndef CATALOG_H_ +#define CATALOG_H_ + +#include "stream_services.h" +#include "stream.h" +#include "backup_aux.h" +#include <meta_data.h> + +namespace backup { + +class Snapshot_info; /** - @file + Describes contents of a backup image. - Data types used to represent contents of a backup archive and to read/write - its description (catalogue) - */ + This class stores a catalogue of a backup image, that is, description of + all items stored in it (currently only databases and tables). -#if defined(USE_PRAGMA_INTERFACE) || defined(__APPLE_CC__) -/* - #pragma interface is needed on powermac platform as otherwise compiler - doesn't create/export vtable for Image_info::Tables class (if you know a - better way for fixing this issue let me know! /Rafal). + Only item names are stored in the catalogue. Other item data is stored + in the meta-data part of the image and in case of tables, their data is + stored in table data snapshots created by backup drivers. - Apparently, configuration macro USE_PRAGMA_INTERFACE is not set by ./configure, - on powermac platform - this is why __APPLE_CC__ is also checked. + For each snapshot present in the image there is a @c Snapshot_info object + stored in @c m_snap[] array. This object contains list of tables whose + data is stored in it. Note that each table must belong to exactly one + snapshot. + + Contents of the catalogue can be browsed using the iterator classes. + + Info about each object is stored in an instance of a class derived from + @c Image_info::Item. This class determines how to obtain meta-data for the + object and how to create it from the saved meta-data. */ -#pragma interface -#endif +class Image_info: public st_bstream_image_header +{ + public: -#include <backup/api_types.h> -#include <backup/string_pool.h> -#include <backup/stream.h> -#include <backup/backup_engine.h> -#include <backup/meta_data.h> + uint table_count; ///< total number of tables in the archive -namespace backup { + // Classes representing various types of meta-data items. -// Forward declaration for a class describing an image inside backup archive. -class Image_info; + class Item; ///< base class for all item types + class Db_item; + class Table_item; -#define MAX_IMAGES 256 -typedef Image_info* Img_list[MAX_IMAGES]; ///< List (vector) of image descriptions. + class Iterator; ///< base for all iterators + class Db_iterator; ///< iterates over databases in archive + class Ditem_iterator; ///< iterates over per-db items -/** - Structure to hold information about binary log. -*/ -struct st_binlog_info -{ - char binlog_file_name[FN_REFLEN]; ///< the file name - my_off_t position; ///< binlog position -}; + virtual ~Image_info(); -/** - Describes contents of a backup archive. + protected: - This class stores a catalogue of a backup archive, that is, description of - all items stored in the archive (currently only databases and tables). It also - determines how to save and read the catalogue to/from a backup stream. + class Db_ref; + class Table_ref; - Only item names are stored in the catalogue. Other item data is stored - in the meta-data part of an archive and in case of tables, their data is - stored in images created by backup drivers. + /** + Provides storage for the list of databases stored in the catalogue. + */ + class Databases + { + Dynamic_array<Db_item> m_dbs; - The @c images member stores a list of @c Image_info objects describing the - images included in the archive. Each image description contains a list of - tables stored in that image (note that no table can be stored in more than - one image). - - To save space, we have a separate pool of database names (@c db_names member). - In table references, only the key of the database name is stored, not the - whole name. + public: + + Databases(): m_dbs(16,128) + {} + + Db_item* operator[](uint pos) const + { return m_dbs[pos]; } + + uint count() const + { return m_dbs.size(); } + + /// Insert database at given location + Image_info::Db_item* add_db(const Db_ref &db, uint pos); + + /// Insert database at first available position. + Image_info::Db_item* add_db(const Db_ref &db) + { + return add_db(db,count()); + } + + }; + + Databases m_db; ///< list of databases + Snapshot_info *m_snap[256]; ///< list of snapshots + + Image_info(); - When reading or writing backup archive, statistics about the size of its parts - is stored in the members of this class for later reporting. - */ -class Archive_info -{ public: - static const version_t ver=1; - uint img_count; ///< number of images in the archive - uint table_count; ///< total number of tables in the archive + uint db_count() const + { return m_db.count(); } - size_t total_size; ///< size of processed backup archive - size_t header_size; ///< size of archive's header (after reading or writing an archive) - size_t meta_size; ///< size of archive's meta-data (after reading or writing an archive) - size_t data_size; ///< size of archive's table data images (after reading or writing an archive) - - st_binlog_info binlog_information; ///< stores binlog information for PTR - struct tm start_time; ///< the start datetime of the backup - struct tm end_time; ///< the end datetime of the backup - struct tm vp_time; ///< time of validation point + /* + Methods for populating backup catalogue (just wrappers which access m_db + member) + */ - // Classes representing various types of meta-data items. + Db_item* add_db(const Db_ref &db, uint pos) + { + return m_db.add_db(db,pos); + } - class Item; - class Db_item; - class Table_item; + Db_item* add_db(const Db_ref &db) + { + return add_db(db,m_db.count()); + } + + Db_item* get_db(uint pos) const + { + return m_db[pos]; + } /* - Classes which might be used to implement contents browsing. + Inline methods for adding/accessing table items are defined after + Snapshot_info class. + */ + + Table_item* add_table(Db_item&, const Table_ref&, uint, unsigned long int); + Table_item* add_table(Db_item&, const Table_ref&, uint); + Table_item* get_table(uint, unsigned long int) const; - class Item_iterator; // for iterating over all meta-data items - class Db_iterator; // iterates over databases in archive - class Ditem_iterator; // iterates over per-db items + Item* locate_item(const st_bstream_item_info*) const; + + private: + + /** + Buffer for CREATE statement used in @c bcat_get_item_create_query() + + FIXME: find a better solution. */ + String create_stmt_buf; - Img_list images; ///< list of archive's images + // friends - /// Write archive's header and save the catalogue. - result_t save(OStream&); - /// Read the header and catalogue from a stream. - result_t read(IStream&); + friend int ::bcat_add_item(st_bstream_image_header*, struct st_bstream_item_info*); + friend int ::bcat_reset(st_bstream_image_header*); + friend int ::bcat_get_item_create_query(st_bstream_image_header*, + struct st_bstream_item_info*, + bstream_blob *); +}; - virtual ~Archive_info(); - protected: +/* + Provides storage for list of tables of a snapshot. - Archive_info(): - img_count(0), table_count(0), - total_size(0), header_size(0), meta_size(0), data_size(0) - { - for (uint i=0; i<256; ++i) - images[i]= NULL; - } + Implements the Table_list interface. +*/ - // storage for meta-data items +class Tables: public Table_list +{ + Dynamic_array<Image_info::Table_item> m_tables; - StringPool db_names; ///< Pool of database names. + public: - private: + Tables(): m_tables(1024,1024) + {} - friend class Image_info; - friend class Db_item; - friend class Table_item; + backup::Table_ref operator[](uint) const; + + uint count() const + { return m_tables.size(); } + + Image_info::Table_item* add_table(const Table_ref&, unsigned long int); + + Image_info::Table_item* add_table(const Table_ref &t) + { + return add_table(t,count()); + } + + Image_info::Table_item* get_table(unsigned long int pos) + { + return m_tables[pos]; + } }; -/** - Describes an image of table data stored in a backup archive. - An instance of this class: - - informs about the type of image, - - stores list of tables whose data is kept in the image, - - provides methods for creating backup and restore drivers to write/read the - image, - - determines which tables can be stored in the image, - - defines how image's format is described inside backup archive - (via @c do_write_description() method) +/* + Describes table data snapshot stored inside backup image. + + Such snapshot is created by a backup driver and read by a restore driver. */ -class Image_info +class Snapshot_info { + protected: + + Tables m_tables; ///< list of tables whose data is stored in the snapshot + public: - enum image_type {NATIVE_IMAGE, DEFAULT_IMAGE, SNAPSHOT_IMAGE}; + enum enum_snap_type { + NATIVE_SNAPSHOT= BI_NATIVE, ///< snapshot created by native backup driver + DEFAULT_SNAPSHOT= BI_DEFAULT, ///< snapshot created by built-in, blocking driver + CS_SNAPSHOT= BI_CS ///< snapshot created by CS backup driver + }; - virtual image_type type() const =0; ///< Return type of the image. - version_t ver; ///< Image format version. + version_t version; ///< version of snapshot's format + + virtual enum_snap_type type() const =0; /// Check if instance was correctly constructed virtual bool is_valid() =0; + + /// Tell how many tables are stored in the snapshot. + unsigned long int table_count() const + { return m_tables.count(); } + + /// Determine if a table stored in given engine can be saved in this image. + virtual bool accept(const Table_ref&, const ::handlerton*) =0; + /// Create backup driver for the image. virtual result_t get_backup_driver(Backup_driver*&) =0; + /// Create restore driver for the image. virtual result_t get_restore_driver(Restore_driver*&) =0; - size_t init_size; ///< Size of the initial data transfer (estimate). This is - ///< meaningful only after a call to get_backup_driver(). - - /// Write header entry describing the image. - result_t write_description(OStream&); + /// Set snapshot's data format version + virtual result_t set_version(version_t ver) + { + version= ver; + return OK; + } /** - Create instance of @c Image_info described by an entry in backup stream. + Position inside image's snapshot list. - @retval OK entry successfully read - @retval DONE end of chunk or stream has been reached. - @retval ERROR an error was detected - */ - static result_t create_from_stream(Archive_info&, IStream&, Image_info*&); + Starts with 1. M_no == 0 means that this snapshot is not included in the + list. + */ + ushort m_no; - /// Determine if a table stored in given engine can be saved in this image. - virtual bool accept(const Table_ref&, const ::handlerton*) =0; + /** + Size of the initial data transfer (estimate). This is + meaningful only after a call to get_backup_driver(). + */ + size_t init_size; /** - Return name identifying the image in debug messages. + Return name identifying the snapshot in debug messages. The name should fit into "%s backup/restore driver" pattern. */ virtual const char* name() const { return "<Unknown>"; } - virtual ~Image_info() + virtual ~Snapshot_info() {} - /* - Implementation of Table_list interface used to store the - list of tables of an image. Database names are stored in - external StringPool - */ - class Tables: public Table_list - { - public: - - Tables(StringPool &db_names): - m_db_names(db_names), - m_head(NULL),m_last(NULL),m_count(0) - {} - - ~Tables() { clear(); } - - int add(const backup::Table_ref&); - void clear(); + protected: - backup::Table_ref operator[](uint pos) const; - //::TABLE_LIST* get_table_ptr(uint pos) const; + Snapshot_info(): m_no(0), version(0) + {} - uint count() const - { return m_count; } + // Methods for adding and accessing tables stored in the table list. - result_t save(OStream&); - result_t read(IStream&); + Image_info::Table_item* add_table(const Table_ref &t, unsigned long int pos) + { + return m_tables.add_table(t,pos); + } - private: + Image_info::Table_item* add_table(const Table_ref &t) + { + return add_table(t,m_tables.count()); + } - struct node; + Image_info::Table_item* get_table(unsigned long int pos) + { + return m_tables.get_table(pos); + } - int add(const StringPool::Key&, const String&); - node* find_table(uint pos) const; + friend class Image_info; +}; - StringPool &m_db_names; - node *m_head, *m_last; - uint m_count; - friend class Table_ref; - friend class Archive_info::Table_item; - }; +/* + Classes @c Image_info::Db_ref and @c Image_info::Table give access to + protected constructors so that instances of these objects can be created. - Tables tables; ///< List of tables stored in the image. + In the future they can be extended with helper methods and any features + needed to implement the base classes. + */ - protected: +class Image_info::Db_ref: public backup::Db_ref +{ + public: - Image_info(Archive_info &info): - init_size(Driver::UNKNOWN_SIZE), tables(info.db_names) + Db_ref(const String &name): backup::Db_ref(name) {} +}; - /** - Write image specific data describing it. +class Image_info::Table_ref: public backup::Table_ref +{ + public: - Method redefined in subclasses corresponding to different image types. - */ - virtual result_t do_write_description(OStream&) =0; + Table_ref(const backup::Db_ref &db, const String &name): + backup::Table_ref(db.name(),name) + {} }; /** - Represents a meta-data item in a backup archive. + Represents a meta-data item in a backup image. Instances of this class: - - identify a meta-data item inside backup archive, + - identify a meta-data item inside backup image, - provide storage for a corresponding meta::Item instance, - - write item identification data to a backup stream. For each type of meta-data there is a specialized subclass of - @c Archive_info::Item implementing the above tasks. Each subclass has static - @c create_from_stream() method which can create class instance using an - identity stored in a stream. For examples, see @c Archive_info::Table_item - class. - - Class @c Archive_info::Item defines the format of an entry describing a - meta-data item inside the meta-data part of an archive. Such entry is created - by @c Archive_info::save() method. These entries are read by - @c Restore_info::read_item() method. - */ - -class Archive_info::Item + @c Archive_info::Item implementing the above tasks. The subclass also stores + all the item data required by the backup stream library. +*/ +class Image_info::Item { + public: + + virtual ~Item() {} + + virtual const st_bstream_item_info* info() const =0; + + /** + Tells to which database belongs this object. + + Returns NULL for global objects which don't belong to any database. + */ + virtual const Db_ref *in_db(void) + { return NULL; } + + virtual result_t get_create_stmt(::String &buf) + { return meta().get_create_stmt(buf); } + + result_t drop(THD *thd) + { return meta().drop(thd); } + + result_t create(THD *thd, ::String &query, byte *begin, byte *end) + { return meta().create(thd,query,begin,end); } + protected: - /// Pointer to @c Archive_info instance to which this item belongs. - const Archive_info *const m_info; - Item *next; ///< Used to create a linked list of all meta-data items. + String m_name; ///< For storing object's name. - public: + Item() {} - virtual ~Item() {} + private: /// Returns reference to the corresponding @c meta::Item instance. virtual meta::Item& meta() =0; - result_t save(THD*,OStream&); ///< Write entry describing the item. + friend class Image_info; +}; - // Create item from a saved entry. - static result_t create_from_stream(const Archive_info&, IStream&, Item*&); +/** + Specialization of @c Image_info::Item for storing info about a database. +*/ +class Image_info::Db_item + : public st_bstream_db_info, + public Image_info::Item, + public meta::Db, + public Db_ref +{ + Table_item *m_tables; + Table_item *m_last_table; + unsigned long int table_count; - class Iterator; + public: - protected: + Db_item(): + Db_ref(Image_info::Item::m_name), m_tables(NULL), m_last_table(NULL) + { + bzero(&base,sizeof(base)); + base.type= BSTREAM_IT_DB; + } - Item(const Archive_info &i): m_info(&i), next(NULL) - {} + const char* sql_name() const + { return (const char*)base.name.begin; } - /// Save data identifying the item inside the archive. - virtual result_t save_id(OStream&) =0; + /// Store information about database @c db. + Db_item& operator=(const Db_ref &db) + { + Image_info::Item::m_name= db.name(); // save name of the db + /* + setup the name member (inherited from bstream_item_info) to point + at the db name + */ + base.name.begin= (byte*) Image_info::Item::m_name.ptr(); + base.name.end= base.name.begin + Image_info::Item::m_name.length(); + return *this; + } - friend class Archive_info; - friend class Backup_info; - friend class Restore_info; - friend class Iterator; + meta::Item& meta() { return *this; } + + const st_bstream_item_info* info() const { return &base; } + const st_bstream_db_info* db_info() const { return this; } + + result_t add_table(Table_item &t); + + friend class Ditem_iterator; + + friend int ::bcat_add_item(st_bstream_image_header*, struct st_bstream_item_info*); }; /** - Used to iterate over meta-data items. - - Usage: - @code - Item *head; - for (Item::Iterator it(head); it ; it++) - { - it->archive_item_method() - } - @endcode - or - @code - Item *head, *p; - Item::Iterator it(head); - - while ((p=it++)) - { - @<use p here> - } - @endcode - */ -class Archive_info::Item::Iterator + Specialization of @c Image_info::Item for storing info about a table. +*/ +class Image_info::Table_item + : public st_bstream_table_info, + public Image_info::Item, + public meta::Table, + public backup::Table_ref { - Item *m_curr; - Item *m_prev; + Table_item *next_table; + String m_db_name; // FIXME public: - Iterator(Item *const head): m_curr(head), m_prev(NULL) - {} + Table_item(): + Table_ref(m_db_name,Image_info::Item::m_name), + next_table(NULL), tl_entry(NULL) + { + bzero(&base,sizeof(base)); + base.base.type= BSTREAM_IT_TABLE; + } + + const char* sql_name() const + { return (const char*)base.base.name.begin; } - operator bool() const - { return m_curr != NULL; } + meta::Item& meta() { return *this; } - Item* operator++(int) + /// Store information about table @c t. + Table_item& operator=(const Table_ref &t) { - m_prev= m_curr; + m_db_name= t.db().name(); + Image_info::Item::m_name= t.name(); // save name of the db + /* + setup the name member (inherited from bstream_item_info) to point + at the db name + */ + base.base.name.begin= (byte*) Image_info::Item::m_name.ptr(); + base.base.name.end= base.base.name.begin + Image_info::Item::m_name.length(); + return *this; + } - if (m_curr) - m_curr= m_curr->next; + const st_bstream_item_info* info() const { return &base.base; } + const st_bstream_table_info* t_info() const { return this; } - return m_prev; + const Db_ref *in_db(void) + { + return static_cast<Db_item*>(base.db); } - Item* operator->() - { DBUG_ASSERT(m_curr); - return m_curr; } -}; + String create_stmt; ///< buffer to store table's CREATE statement + + /// Build CREATE statement for the table and store it in @c create_stmt + result_t get_create_stmt() + { + if (tl_entry) + return meta::Table::get_create_stmt(create_stmt); + return ERROR; + } + + /// Put in @c buf a CREATE statement for the table. + result_t get_create_stmt(::String &buf) + { + if (create_stmt.is_empty() && tl_entry) + get_create_stmt(); + + buf= create_stmt; + return OK; + } + private: + + /** + Stores pointer to a filled TABLE_LIST structure when the table is opened. + Otherwise should contain NULL. + */ + ::TABLE_LIST *tl_entry; + /// this method is used by meta::Table class to get the CREATE statement + ::TABLE_LIST* get_table_list_entry() + { return tl_entry; } + + friend class Db_item; + friend class Ditem_iterator; + friend class Tables; + friend class Backup_info; + + friend + int ::bcat_add_item(st_bstream_image_header*, struct st_bstream_item_info*); +}; /** - Specialization of @c Archive_info::Item representing a database. + Add table to given snapshot at the indicated location. - A database is identified by a key into Archive_info::db_names string pool. - Using the key one can read database name from the pool. The key is saved - as a var-length coded integer. - */ -class Archive_info::Db_item: - public Archive_info::Item, public meta::Db, public Db_ref + @param db Database to which this table belongs. + @param t Table description. + @param no Snapshot to which it should be added (its position in + @c m_snap[] array) + @param pos Position in snapshot's table list. + + @note Table is also added to the database's table list. + + @todo Report errors. +*/ +inline +Image_info::Table_item* +Image_info::add_table(Db_item &db, const Table_ref &t, uint no, unsigned long int pos) { - StringPool::Key key; + Snapshot_info *snap= m_snap[no]; - Db_item(const Archive_info &i, const StringPool::Key &key): - Archive_info::Item(i), Db_ref(m_info->db_names[key]), key(key) - {} + DBUG_ASSERT(snap); - meta::Item& meta() - { return *this; } + Table_item *ti= snap->add_table(t,pos); + if (!ti) + { + // TODO: report error + return NULL; + } - /// Get the name from @c db_names pool. - const char* sql_name() const - { return m_info->db_names[key].ptr(); } + // If the snapshot was not yet used, assign a new number to it + if (snap->m_no == 0) + snap->m_no= ++snap_count; + + ti->snap_no= snap->m_no-1; + table_count++; + db.add_table(*ti); - public: + return ti; +} - result_t save_id(OStream&); - /// Create instance reading its identity from a stream. - static - result_t create_from_stream(const Archive_info&,IStream&, Archive_info::Item*&); +/** + Add table to given snapshot at first available location. - friend class Archive_info; - friend class Backup_info; -}; + @param db Database to which this table belongs. + @param t Table description. + @param no Snapshot to which it should be added (its position in + @c m_snap[] array) + @note Table is also added to the database's table list. +*/ +inline +Image_info::Table_item* +Image_info::add_table(Db_item &db, const Table_ref &t, uint no) +{ + return add_table(db,t,no,m_snap[no]->table_count()); +} /** - Specialization of @c Archive_info::Item representing a table. + Get @c Table_item object for the table in the given position. - A table is identified by its position inside the table list of - one of archive's images. Its identity is saved as two var-length coded - integers: first being the image number and second the table position inside - image's table list. - */ -class Archive_info::Table_item: - public Archive_info::Item, public meta::Table, public Table_ref + @param no Snapshot to which the table belongs (its position in @c m_snap[] + array) + @param pos Tables position in the snapshot's table list. +*/ +inline +Image_info::Table_item* +Image_info::get_table(uint no, unsigned long int pos) const { - uint img; ///< Image in which this table is saved. - uint pos; ///< Position of the table in image's table list. + uint i; - Table_item(const Archive_info &i, uint no, uint tno): - Archive_info::Item(i), Table_ref(i.images[no]->tables[tno]), - img(no), pos(tno) - {} + // FIXME: avoid the loop + for (i=0; i < 256; ++i) + if (m_snap[i] && m_snap[i]->m_no == no+1) + return m_snap[i]->get_table(pos); + return NULL; +} - public: +/** + Add database to the catalogue, storing it at given position. - meta::Item& meta() - { return *this; } + @param db Database to add. + @param pos Position where it should be stored in the database list. - const char* sql_name() const - { return m_info->images[img]->tables[pos].name().ptr(); } + The position should be not occupied. - /// Table is a per-db item -- indicate to which database it belongs. - const Db_ref in_db() - { return m_info->images[img]->tables[pos].db(); } - - result_t save_id(OStream&); - /// Create instance reading its identity from a stream. - static - result_t create_from_stream(const Archive_info&,IStream&, Archive_info::Item*&); + @todo Report errors. +*/ +inline +Image_info::Db_item* +Image_info::Databases::add_db(const Db_ref &db, uint pos) +{ + Db_item *di= m_dbs.get_entry(pos); + if (!di) + { + // TODO: report error + return NULL; + } + di->base.pos= pos; + *di= db; + return di; +} - friend class Backup_info; -}; +/** + Add table to the list storing it at given position. -} // backup namespace + The position should not be occupied. + @todo Report errors. +*/ +inline +Image_info::Table_item* +Tables::add_table(const Table_ref &t, unsigned long int pos) +{ + Image_info::Table_item *it= m_tables.get_entry(pos); -/************************************************************ + if (!it) + { + // TODO: report error + return NULL; + } - Class describing native backup image + it->base.base.pos= pos; + *it= t; // store table info in the item + return it; +} - ************************************************************/ +/** + Return table at given position. -namespace backup { + The position should not be empty. +*/ +inline +Table_ref Tables::operator[](uint pos) const +{ + DBUG_ASSERT(pos < m_tables.size()); -/** - Specialization of @c Image_info for images created by native backup drivers. - */ -class Native_image: public Image_info + const Image_info::Table_item *ti= m_tables[pos]; + DBUG_ASSERT(ti); + + return *ti; +} + + +class Image_info::Iterator { - const ::handlerton *m_hton; ///< Pointer to storage engine. - Engine *m_be; ///< Pointer to the native backup engine. + protected: - const char *m_name; ///< Used to identify image in debug messages. + const Image_info &m_info; public: - Native_image(Archive_info &info, const ::handlerton *hton): - Image_info(info), m_hton(hton) + Iterator(const Image_info &info): m_info(info) {} + + const Item* operator++(int) { - DBUG_ASSERT(hton); - DBUG_ASSERT(hton->get_backup_engine); + const Item *ptr= get_ptr(); + next(); + return ptr; + } - hton->get_backup_engine(const_cast< ::handlerton* >(hton),m_be); + virtual ~Iterator() {} - if(m_be) - { - ver= m_be->version(); - m_name= ::ha_resolve_storage_engine_name(hton); - } + private: + + virtual const Item* get_ptr() const =0; + virtual bool next() =0; +}; + + +class Image_info::Db_iterator + : public Image_info::Iterator +{ + uint pos; + + public: + + Db_iterator(const Image_info &info): Iterator(info) + { + pos= 0; + find_non_null_pos(); } - bool is_valid() - { return m_be != NULL; } + private: - image_type type() const - { return NATIVE_IMAGE; } + const Item* get_ptr() const + { return m_info.m_db[pos]; } - const char* name() const - { return m_name; } + bool next() + { + pos++; + find_non_null_pos(); + return pos < m_info.m_db.count(); + } - result_t get_backup_driver(Backup_driver* &drv) + void find_non_null_pos() { - DBUG_ASSERT(m_be); - return m_be->get_backup(Driver::PARTIAL,tables,drv); + for(; pos < m_info.m_db.count() && m_info.m_db[pos] == NULL; ++pos); } +}; + - result_t get_restore_driver(Restore_driver* &drv) +class Image_info::Ditem_iterator + : public Image_info::Iterator +{ + Table_item *ptr; + + public: + + Ditem_iterator(const Image_info &info, const Db_item &db): Iterator(info) { - DBUG_ASSERT(m_be); - return m_be->get_restore(ver,Driver::PARTIAL,tables,drv); + ptr= db.m_tables; } - result_t do_write_description(OStream&); - static result_t create_from_stream(version_t,Archive_info&,IStream&,Image_info*&); + private: + + const Item* get_ptr() const + { return ptr; } + + bool next() + { + if (ptr) + ptr= ptr->next_table; + + return ptr != NULL; + } - bool accept(const Table_ref&, const ::handlerton *hton) - { return hton == m_hton; }; // this assumes handlertons are single instance objects! }; } // backup namespace +namespace backup { + +/* + Wrappers around backup stream functions which perform necessary type conversions. + + TODO: report errors +*/ + +inline +result_t +write_preamble(const Image_info &info, OStream &s) +{ + const st_bstream_image_header *hdr= static_cast<const st_bstream_image_header*>(&info); + int ret= bstream_wr_preamble(&s, const_cast<st_bstream_image_header*>(hdr)); + return ret == BSTREAM_ERROR ? ERROR : OK; +} + +inline +result_t +write_summary(const Image_info &info, OStream &s) +{ + const st_bstream_image_header *hdr= static_cast<const st_bstream_image_header*>(&info); + int ret= bstream_wr_summary(&s, const_cast<st_bstream_image_header*>(hdr)); + return ret == BSTREAM_ERROR ? ERROR : OK; +} + +inline +result_t +read_header(Image_info &info, IStream &s) +{ + int ret= bstream_rd_header(&s, static_cast<st_bstream_image_header*>(&info)); + return ret == BSTREAM_ERROR ? ERROR : OK; +} + +inline +result_t +read_catalog(Image_info &info, IStream &s) +{ + int ret= bstream_rd_catalogue(&s, static_cast<st_bstream_image_header*>(&info)); + return ret == BSTREAM_ERROR ? ERROR : OK; +} + +inline +result_t +read_meta_data(Image_info &info, IStream &s) +{ + int ret= bstream_rd_meta_data(&s, static_cast<st_bstream_image_header*>(&info)); + return ret == BSTREAM_ERROR ? ERROR : OK; +} + +inline +result_t +read_summary(Image_info &info, IStream &s) +{ + int ret= bstream_rd_summary(&s, static_cast<st_bstream_image_header*>(&info)); + return ret == BSTREAM_ERROR ? ERROR : OK; +} + +} // backup namespace -#endif +#endif /*CATALOG_H_*/ diff -Nrup a/sql/backup/data_backup.cc b/sql/backup/data_backup.cc --- a/sql/backup/data_backup.cc 2007-11-20 15:14:23 +01:00 +++ b/sql/backup/data_backup.cc 2007-11-30 09:23:29 +01:00 @@ -6,45 +6,19 @@ Code used to backup table data. Function @c write_table_data() and @c restore_table_data() use backup/restore - drivers and protocols to create image of the data stored in the tables being + drivers and protocols to create snapshot of the data stored in the tables being backed up. - */ - -/* - TODO - - - Implement better scheduling strategy in Scheduler::step - - The driver with least data read is chosen. If there are several, - they are polled in round-robin fashion. - - - For consistency, always use OStream for data serialization (?) - - - In Backup_pump::pump(), handle DONE response from backup driver. - - - In case of backup/restore of empty archive, check that all resources - are correctly deallocated. - - - Try to announce block size when initializing restore driver with begin() - method (is it possible?). - - - If an error from driver is ignored (and operation retried) leave trace - of the error in the log. - - - Move the open_and_lock_tables in restore_table_data to before the begin() - calls for the drivers. - - - The table locking code in backup and restore for default and snapshot drivers - should be replaced by something better in version beta. + @todo Implement better scheduling strategy in Scheduler::step + @todo If an error from driver is ignored (and operation retried) leave trace + of the error in the log. */ #include "backup_engine.h" #include "stream.h" #include "backup_kernel.h" #include "debug.h" -#include "be_default.h" -#include "be_snapshot.h" +#include "be_default.h" // needed for table locking code /*********************************************** @@ -65,7 +39,7 @@ struct backup_state { FINISHING, ///< Final data transfer (phase 7). DONE, ///< Backup complete. SHUT_DOWN, ///< After @c end() call. - CANCELLED, ///< After cancelling backup process. + CANCELLED, ///< After cancelling backup process. ERROR, MAX }; @@ -77,16 +51,16 @@ struct backup_state { { Initializer() { - name[INACTIVE]= "INACTIVE"; - name[INIT]= "INIT"; - name[WAITING]= "WAITING"; - name[PREPARING]= "PREPARING"; - name[READY]= "READY"; - name[FINISHING]= "FINISHING"; - name[DONE]= "DONE"; - name[SHUT_DOWN]= "SHUT DOWN"; - name[CANCELLED]= "CANCELLED"; - name[ERROR]= "ERROR"; + name[INACTIVE]= "INACTIVE"; + name[INIT]= "INIT"; + name[WAITING]= "WAITING"; + name[PREPARING]= "PREPARING"; + name[READY]= "READY"; + name[FINISHING]= "FINISHING"; + name[DONE]= "DONE"; + name[SHUT_DOWN]= "SHUT DOWN"; + name[CANCELLED]= "CANCELLED"; + name[ERROR]= "ERROR"; } }; @@ -114,21 +88,23 @@ backup_state::Initializer init; class Block_writer { public: - + enum result_t { OK, NO_RES, ERROR }; result_t get_buf(Buffer &); result_t write_buf(const Buffer&); result_t drop_buf(Buffer&); - Block_writer(size_t size, OStream &s): - m_str(s), buf_size(size) - {} + Block_writer(byte, size_t, OStream&); + ~Block_writer(); private: - OStream &m_str; - size_t buf_size; + byte snap_no; ///< snapshot to which the data belongs + OStream &m_str; ///< stream to which we write + size_t buf_size; ///< size of a single data block + byte *data_buf; ///< pointer to data buffer + bool taken; ///< flag which indicates that the buffer is in use friend class Backup_pump; }; @@ -139,16 +115,17 @@ class Block_writer Poll backup driver for backup data and send it to a stream. Monitors stages of the backup process, keeps track of closed streams etc. - Usage: Initialize using begin() method, then call pump() method repeatedly. - The state member informs about the current state of the backup process. When - done, call end() method. Methods prepare(), lock() and unlock() are forwarded - to backup driver to implement multi-engine synchronization. + Usage: Initialize using @c begin() method, then call @c pump() method + repeatedly. The state @c member informs about the current state of the backup + process. When done, call @c end() method. Methods @c prepare(), @c lock() and + @c unlock() are forwarded to backup driver to implement multi-engine + synchronization. */ class Backup_pump { public: - + backup_state::value state; ///< State of the backup driver. enum { READING, ///< Pump is polling driver for data. @@ -160,7 +137,7 @@ class Backup_pump const char *m_name; ///< Name of the driver (for debug purposes). - Backup_pump(uint, Image_info&, Block_writer&); + Backup_pump(Snapshot_info&, Block_writer&); ~Backup_pump(); bool is_valid() @@ -187,11 +164,9 @@ class Backup_pump private: - static const uint m_buf_size= 2048; ///< Size of buffers used for data transfers. static const uint get_buf_retries= 3; /**< if block writer has no buffers, retry this many times before giving up */ Logger *m_log; ///< Used to report errors if not NULL. - uint m_drv_no; ///< The number of this image in the backup archive. Backup_driver *m_drv; ///< Pointer to the backup driver. Block_writer &m_bw; ///< Block writer used for writing data blocks. Buffer m_buf; ///< Buffer used for data transfers. @@ -207,10 +182,6 @@ class Backup_pump uint m_buf_retries; ///< how many times failed to get a buffer from block writer - bool get_buf(); - bool drop_buf(); - bool write_buf(); - /// Bitmap showing which streams have been closed by the driver. MY_BITMAP m_closed_streams; @@ -298,9 +269,13 @@ class Scheduler::Pump: public Backup_pum public: - Pump(uint no, Image_info &img, OStream &s): - Backup_pump(no,img,bw), start_pos(0), bw(io_buffer_size,s) - {} + // FIXME: how to react if the buffer could not be allocated? + Pump(Snapshot_info &snap, OStream &s): + Backup_pump(snap,bw), start_pos(0), + bw(snap.m_no-1,DATA_BUFFER_SIZE,s) + { + DBUG_ASSERT(snap.m_no > 0); + } size_t pos() const { return start_pos + bytes_in; } @@ -310,7 +285,7 @@ class Scheduler::Pump: public Backup_pum Collect tables from default and snapshot for open and lock tables. There should be at most only 1 of each driver. */ -int get_default_snapshot_tables(backup::Backup_driver *backup_drv, +int get_default_snapshot_tables(backup::Backup_driver *backup_drv, backup::Restore_driver *restore_drv, TABLE_LIST **tables, TABLE_LIST **tables_last) @@ -319,7 +294,7 @@ int get_default_snapshot_tables(backup:: TABLE_LIST *table_list_last= *tables_last; DBUG_ENTER("backup::get_default_snapshot_tables"); - /* + /* If the table list is defined and the last pointer is defined then we are seeing a duplicate of either default or snapshot drivers. There should be at most 1 of each. @@ -327,10 +302,10 @@ int get_default_snapshot_tables(backup:: if (table_list && table_list_last->next_global) { DBUG_PRINT("restore",("Duplicate default or snapshot subimage")); - DBUG_RETURN(ERROR); + DBUG_RETURN(ERROR); } /* - If the table list is empty, use the first one and loop + If the table list is empty, use the first one and loop until the end then record the end of the first one. */ if (!table_list) @@ -360,14 +335,14 @@ int get_default_snapshot_tables(backup:: /** Commit Blocker - The commit blocker ensures storage engines can't commit while in the + The commit blocker ensures storage engines can't commit while in the synchronization phase of Online Backup algorithm. This is needed to - make sure that: + make sure that: 1) backups are consistent between engines, and 2) that the binlog position is consistent with the engine images. - REQUREMENTS + REQUREMENTS A transactional engine needs to block commits during the locking phase of backup kernel. It can not block DDL for long time. @@ -388,14 +363,14 @@ int get_default_snapshot_tables(backup:: 3) Prevents new commits and waits for running commits to finish -- make_global_read_lock_block_commit() - This will make it impossible to enter commit phase in any transaction. + This will make it impossible to enter commit phase in any transaction. This will also wait for any ongoing commit to finish. When the function returns, no transaction is in its commit phase. 4) Read binlog position & do lock calls to all backup drivers This step will read the binlog position and save it in the backup_info - structure. This will occur between the lock() and unlock() calls in + structure. This will occur between the lock() and unlock() calls in the kernel. 5) unlock_global_read_lock() @@ -407,14 +382,14 @@ int get_default_snapshot_tables(backup:: /** Block commits - + This method is used to initiate the first three steps of the commit blocker algorithm (global read lock, close cached tables, make global read lock block commits). - + @param thd (in) the current thread structure. @param tables (in) list of tables to be backed-up. - + @returns 0 on success. */ int block_commits(THD *thd, TABLE_LIST *tables) @@ -426,7 +401,7 @@ int block_commits(THD *thd, TABLE_LIST * */ BACKUP_BREAKPOINT("commit_blocker_step_1"); if (lock_global_read_lock(thd)) - DBUG_RETURN(1); + DBUG_RETURN(1); /* Step 2 - close cached tables. @@ -441,14 +416,14 @@ int block_commits(THD *thd, TABLE_LIST * tables only and only to non-transactional engines. BACKUP_BREAKPOINT("commit_blocker_step_2"); - result= close_cached_tables(thd, 0, tables); + result= close_cached_tables(thd, 0, tables); */ /* Step 3 - make the global read lock to block commits. */ BACKUP_BREAKPOINT("commit_blocker_step_3"); - if (make_global_read_lock_block_commit(thd)) + if (make_global_read_lock_block_commit(thd)) { /* Don't leave things in a half-locked state */ unlock_global_read_lock(thd); @@ -459,11 +434,11 @@ int block_commits(THD *thd, TABLE_LIST * /** Unblock commits - - This method is used to terminate the commit blocker. It calls the last + + This method is used to terminate the commit blocker. It calls the last step of the algorithm (unlock global read lock). - - @param thd (in) the current thread structure. + + @param thd (in) the current thread structure. @returns 0 */ @@ -486,33 +461,27 @@ int unblock_commits(THD *thd) */ int write_table_data(THD* thd, Backup_info &info, OStream &s) { - int error= 0; - my_bool def_or_snap_used= FALSE; // Are default or snapshot used? DBUG_ENTER("backup::write_table_data"); - info.data_size= 0; - - if (info.img_count==0 || info.table_count==0) // nothing to backup + if (info.snap_count==0 || info.table_count==0) // nothing to backup DBUG_RETURN(0); Scheduler sch(s,&info); // scheduler instance List<Scheduler::Pump> inactive; // list of images not yet being created size_t max_init_size=0; // keeps maximal init size for images in inactive list - size_t start_bytes= s.bytes; - DBUG_PRINT("backup/data",("initializing scheduler")); // add unknown "at end" drivers to scheduler, rest to inactive list - for (uint no=0; no < info.img_count; ++no) + for (uint no=0; no < 256; ++no) { - Image_info *i= info.images[no]; + Snapshot_info *i= info.m_snap[no]; if (!i) continue; - Scheduler::Pump *p= new Scheduler::Pump(no,*i,s); + Scheduler::Pump *p= new Scheduler::Pump(*i,s); if (!p || !p->is_valid()) { @@ -545,10 +514,6 @@ int write_table_data(THD* thd, Backup_in sch.init_count, inactive.elements)); - DBUG_PRINT("backup/data",("%u drivers initialized, %u inactive", - sch.init_count, - inactive.elements)); - DBUG_PRINT("backup/data",("-- INIT PHASE --")); BACKUP_BREAKPOINT("data_init"); @@ -629,7 +594,7 @@ int write_table_data(THD* thd, Backup_in Block commits. TODO: Step 2 of the commit blocker has been skipped for this release. - When it is included, developer needs to build a list of all of the + When it is included, developer needs to build a list of all of the non-transactional tables and pass that to block_commits(). */ int error= 0; @@ -648,9 +613,7 @@ int write_table_data(THD* thd, Backup_in { LOG_INFO li; mysql_bin_log.get_current_log(&li); - info.binlog_information.position= li.pos; - memcpy(info.binlog_information.binlog_file_name, - li.log_file_name, strlen(li.log_file_name)); + info.save_binlog_pos(li); DBUG_PRINT("SYNC PHASE - binlog position : ", ("%d", li.pos)); DBUG_PRINT("SYNC PHASE - binlog filename : ", ("%s", li.log_file_name)); } @@ -658,8 +621,7 @@ int write_table_data(THD* thd, Backup_in /* Save VP creation time. */ - time_t skr= my_time(0); - gmtime_r(&skr, &info.vp_time); + save_current_time(info.vp_time); BACKUP_BREAKPOINT("commit_blocker_step_4"); BACKUP_BREAKPOINT("data_unlock"); @@ -685,8 +647,6 @@ int write_table_data(THD* thd, Backup_in DBUG_PRINT("backup/data",("-- DONE --")); } - info.data_size= s.bytes - start_bytes; - DBUG_RETURN(0); error: @@ -710,7 +670,7 @@ namespace backup { class Scheduler::Pump_iterator { public: - + LIST *el; Pump* operator->() @@ -749,13 +709,13 @@ int Scheduler::step() Pump_iterator p(*this); -/* - An attempt to implement more advanced scheduling strategy (not working). + /* + An attempt to implement more advanced scheduling strategy (not working). - for (Pump_ptr it(m_pumps); it; ++it) - if ( !p || it->pos() < p->pos() ) - p= it; -*/ + for (Pump_ptr it(m_pumps); it; ++it) + if ( !p || it->pos() < p->pos() ) + p= it; + */ if (!p) // No active pumps { @@ -859,6 +819,12 @@ int Scheduler::add(Pump *p) p->set_logger(m_log); p->start_pos= avg; + if (p->begin()) + goto error; + + // in case of error, above call should return non-zero code (and report error) + DBUG_ASSERT(p->state != backup_state::ERROR); + DBUG_PRINT("backup/data",("Adding %s to scheduler (at pos %lu)", p->m_name, (unsigned long)avg)); @@ -869,12 +835,6 @@ int Scheduler::add(Pump *p) m_count++; m_total += avg; - if (p->begin()) - goto error; - - // in case of error, above call should return non-zero code (and report error) - DBUG_ASSERT(p->state != backup_state::ERROR); - if (p->init_size != Driver::UNKNOWN_SIZE) { m_init_left += p->init_size; @@ -1041,19 +1001,20 @@ int Scheduler::unlock() **************************************************/ -Backup_pump::Backup_pump(uint no, Image_info &img, Block_writer &bw): +Backup_pump::Backup_pump(Snapshot_info &snap, Block_writer &bw): state(backup_state::INACTIVE), mode(READING), init_size(0), bytes_in(0), bytes_out(0), - m_drv_no(no), m_drv(NULL), m_bw(bw), m_buf_head(NULL), + m_drv(NULL), m_bw(bw), m_buf_head(NULL), m_buf_retries(0) { + DBUG_ASSERT(snap.m_no > 0); m_buf.data= NULL; bitmap_init(&m_closed_streams, NULL, - 1+img.tables.count(), + 1+snap.table_count(), FALSE); // not thread safe - m_name= img.name(); - if (ERROR == img.get_backup_driver(m_drv) || !m_drv) + m_name= snap.name(); + if (ERROR == snap.get_backup_driver(m_drv) || !m_drv) state= backup_state::ERROR; else init_size= m_drv->init_size(); @@ -1072,7 +1033,7 @@ int Backup_pump::begin() state= backup_state::INIT; DBUG_PRINT("backup/data",(" %s enters INIT state",m_name)); - if (ERROR == m_drv->begin(m_buf_size)) + if (ERROR == m_drv->begin(m_bw.buf_size)) { state= backup_state::ERROR; // We check if logger is always setup. Later the assertion can @@ -1185,17 +1146,7 @@ int Backup_pump::cancel() the backup driver is polled for image data or data obtained before is written to the stream. Answers from drivers @c get_data() method are interpreted and the state of the driver is updated accordingly. - - Each block of data obtained from a driver is prefixed with the image - number and the table number stored in @c buf.table_no before it is written to - the stream: - @verbatim - | img no | table no | data from the driver | - @endverbatim - */ - -// Consider: report number of bytes *written* to stream. - +*/ int Backup_pump::pump(size_t *howmuch) { // pumping not allowed in these states @@ -1253,8 +1204,6 @@ int Backup_pump::pump(size_t *howmuch) case Block_writer::OK: m_buf_retries= 0; m_buf_head= m_buf.data; - m_buf.data+= 4; // leave space for image and stream numbers - m_buf.size-= 4; break; case Block_writer::NO_RES: @@ -1297,19 +1246,13 @@ int Backup_pump::pump(size_t *howmuch) m_buf_head= NULL; if ( m_buf.size > 0 ) - { - m_buf.size+= 4; - int2store(m_buf.data,m_drv_no+1); - int2store(m_buf.data+2,m_buf.table_no); mode= WRITING; - } else m_bw.drop_buf(m_buf); break; case PROCESSING: - break; case ERROR: @@ -1319,15 +1262,19 @@ int Backup_pump::pump(size_t *howmuch) state= backup_state::ERROR; return ERROR; - case BUSY: + case DONE: + state= backup_state::DONE; + case BUSY: m_bw.drop_buf(m_buf); m_buf_head=NULL; // thus a new request will be made } } // if (mode == READING) - if (mode == WRITING && state != backup_state::ERROR) + if (mode == WRITING + && state != backup_state::ERROR + && state != backup_state::DONE) { switch (m_bw.write_buf(m_buf)) { @@ -1337,7 +1284,7 @@ int Backup_pump::pump(size_t *howmuch) *howmuch= m_buf.size; DBUG_PRINT("backup/data",(" added %lu bytes from %s to archive (drv_no=%u, table_no=%u)", - (unsigned long)howmuch, m_name, m_drv_no, m_buf.table_no)); + (unsigned long)howmuch, m_name, m_bw.snap_no, m_buf.table_no)); mode= READING; break; @@ -1383,68 +1330,67 @@ int restore_table_data(THD*, Restore_inf enum { READING, SENDING, DONE, ERROR } state= READING; - if (info.img_count==0 || info.table_count==0) // nothing to restore + if (info.snap_count==0 || info.table_count==0) // nothing to restore DBUG_RETURN(0); - Restore_driver* drv[MAX_IMAGES]; + Restore_driver* drv[256]; TABLE_LIST *table_list= 0; TABLE_LIST *table_list_last= 0; - if (info.img_count > MAX_IMAGES) + if (info.snap_count > 256) { - info.report_error(ER_BACKUP_TOO_MANY_IMAGES, info.img_count, MAX_IMAGES); + info.report_error(ER_BACKUP_TOO_MANY_IMAGES, info.snap_count, 256); DBUG_RETURN(ERROR); } // Initializing restore drivers result_t res; - for (uint no=0; no < info.img_count; ++no) + for (uint no=0; no < info.snap_count; ++no) { drv[no]= NULL; - Image_info *img= info.images[no]; + Snapshot_info *snap= info.m_snap[no]; // note: img can be NULL if it is not used in restore. - if (!img) + if (!snap) continue; - res= img->get_restore_driver(drv[no]); + res= snap->get_restore_driver(drv[no]); if (res == backup::ERROR) { - info.report_error(ER_BACKUP_CREATE_RESTORE_DRIVER,img->name()); + info.report_error(ER_BACKUP_CREATE_RESTORE_DRIVER,snap->name()); goto error; }; res= drv[no]->begin(0); if (res == backup::ERROR) { - info.report_error(ER_BACKUP_INIT_RESTORE_DRIVER,img->name()); + info.report_error(ER_BACKUP_INIT_RESTORE_DRIVER,snap->name()); goto error; } /* Collect tables from default and snapshot for open and lock tables. There should be at most only 1 of each driver. */ - if ((img->type() == Image_info::DEFAULT_IMAGE) || - (img->type() == Image_info::SNAPSHOT_IMAGE)) - get_default_snapshot_tables(NULL, (default_backup::Restore *)drv[no], + if ((snap->type() == Snapshot_info::DEFAULT_SNAPSHOT) || + (snap->type() == Snapshot_info::CS_SNAPSHOT)) + get_default_snapshot_tables(NULL, (default_backup::Restore *)drv[no], &table_list, &table_list_last); } { Buffer buf; - uint img_no=0; + uint snap_no=0; uint repeats=0, errors= 0; + int ret; static const uint MAX_ERRORS= 3; static const uint MAX_REPEATS= 7; - size_t start_bytes= s.bytes; - Restore_driver *drvr= NULL; // pointer to the current driver - Image_info *img= NULL; // corresponding restore image object + Snapshot_info *snap= NULL; // corresponding snapshot object /* Open tables for default and snapshot drivers. @@ -1455,7 +1401,7 @@ int restore_table_data(THD*, Restore_inf query_cache.invalidate_locked_for_write(table_list); if (open_and_lock_tables(::current_thd, table_list)) { - DBUG_PRINT("restore", + DBUG_PRINT("restore", ( "error on open tables for default and snapshot drivers!" )); info.report_error(ER_BACKUP_OPEN_TABLES, "restore"); DBUG_RETURN(backup::ERROR); @@ -1466,23 +1412,29 @@ int restore_table_data(THD*, Restore_inf // main data reading loop + st_bstream_data_chunk chunk_info; + while ( state != DONE && state != ERROR ) { switch (state) { case READING: - switch ((int)(s >> buf)) { + bzero(&chunk_info, sizeof(chunk_info)); + ret= bstream_rd_data_chunk(&s,&chunk_info); - case stream_result::EOS: + switch (ret) { + + case BSTREAM_EOS: + case BSTREAM_EOC: state= DONE; break; - case stream_result::OK: + case BSTREAM_OK: state= SENDING; break; - case stream_result::ERROR: + case BSTREAM_ERROR: info.report_error(ER_BACKUP_READ_DATA); default: state= ERROR; @@ -1493,26 +1445,29 @@ int restore_table_data(THD*, Restore_inf if (state != SENDING) break; - DBUG_ASSERT(buf.data); - - img_no= uint2korr(buf.data); - buf.table_no= uint2korr(buf.data+2); - buf.data += 4; - buf.size -= 4; + // data chunk should never be empty + DBUG_ASSERT(chunk_info.data.begin); + DBUG_ASSERT(chunk_info.data.begin < chunk_info.data.end); + + snap_no= chunk_info.snap_no; + buf.table_no= chunk_info.table_no; + buf.last= chunk_info.flags & BSTREAM_FLAG_LAST_CHUNK; + buf.data= chunk_info.data.begin; + buf.size= chunk_info.data.end - chunk_info.data.begin; - if (img_no < 1 || img_no > info.img_count || !(drvr= drv[img_no-1])) + if (snap_no > info.snap_count || !(drvr= drv[snap_no])) { - DBUG_PRINT("restore",("Skipping data for image #%u",img_no)); + DBUG_PRINT("restore",("Skipping data from snapshot #%u",snap_no)); state= READING; break; } - img= info.images[img_no-1]; + snap= info.m_snap[snap_no]; // Each restore driver should have corresponding Image_info object. - DBUG_ASSERT(img); + DBUG_ASSERT(snap); DBUG_PRINT("restore",("Got %lu bytes of %s image data (for table #%u)", - (unsigned long)buf.size, img->name(), buf.table_no)); + (unsigned long)buf.size, snap->name(), buf.table_no)); case SENDING: @@ -1520,37 +1475,21 @@ int restore_table_data(THD*, Restore_inf If we are here, the img pointer should point at the image for which we have next data block and drvr at its restore driver. */ - DBUG_ASSERT(img && drvr); + DBUG_ASSERT(snap && drvr); switch( drvr->send_data(buf) ) { case backup::OK: - switch ((int)s.next_chunk()) { - - case stream_result::OK: - state= READING; - img= NULL; - drvr= NULL; - repeats= 0; - break; - - case stream_result::EOS: - state= DONE; - break; - - case stream_result::ERROR: - info.report_error(ER_BACKUP_NEXT_CHUNK); - default: - state= ERROR; - goto error; - } - + state= READING; + snap= NULL; + drvr= NULL; + repeats= 0; break; case backup::ERROR: if( errors > MAX_ERRORS ) { - info.report_error(ER_BACKUP_SEND_DATA, buf.table_no, img->name()); + info.report_error(ER_BACKUP_SEND_DATA, buf.table_no, snap->name()); state= ERROR; goto error; } @@ -1562,7 +1501,7 @@ int restore_table_data(THD*, Restore_inf default: if( repeats > MAX_REPEATS ) { - info.report_error(ER_BACKUP_SEND_DATA_RETRY, repeats, img->name()); + info.report_error(ER_BACKUP_SEND_DATA_RETRY, repeats, snap->name()); state= ERROR; goto error; } @@ -1579,21 +1518,19 @@ int restore_table_data(THD*, Restore_inf DBUG_PRINT("restore",("End of backup stream")); if (state != DONE) DBUG_PRINT("restore",("state is %d",state)); - - info.data_size= s.bytes - start_bytes; } { // Shutting down drivers String bad_drivers; - for (uint no=0; no < info.img_count; ++no) + for (uint no=0; no < info.snap_count; ++no) { if (!drv[no]) continue; DBUG_PRINT("restore",("Shutting down restore driver %s", - info.images[no]->name())); + info.m_snap[no]->name())); res= drv[no]->end(); if (res == backup::ERROR) { @@ -1601,7 +1538,7 @@ int restore_table_data(THD*, Restore_inf if (!bad_drivers.is_empty()) bad_drivers.append(","); - bad_drivers.append(info.images[no]->name()); + bad_drivers.append(info.m_snap[no]->name()); } drv[no]->free(); } @@ -1622,18 +1559,18 @@ int restore_table_data(THD*, Restore_inf DBUG_PRINT("restore",("Cancelling restore process")); - for (uint no=0; no < info.img_count; ++no) + for (uint no=0; no < info.snap_count; ++no) { if (!drv[no]) continue; - drv[no]->cancel(); drv[no]->free(); } DBUG_RETURN(backup::ERROR); } + } // backup namespace @@ -1645,6 +1582,19 @@ int restore_table_data(THD*, Restore_inf namespace backup { +Block_writer::Block_writer(byte snap_no, size_t size, OStream &s): + m_str(s), buf_size(size), taken(FALSE), snap_no(snap_no) +{ + data_buf= (byte*)my_malloc(buf_size,MYF(0)); +} + +Block_writer::~Block_writer() +{ + if (data_buf) + my_free(data_buf,MYF(0)); + data_buf= NULL; +} + /** Allocate new buffer for data transfer. @@ -1661,11 +1611,18 @@ Block_writer::get_buf(Buffer &buf) buf.table_no= 0; buf.last= FALSE; buf.size= buf_size; - buf.data= m_str.get_window(buf.size); + buf.data= NULL; + + if (taken) + return NO_RES; + + buf.data= data_buf; if (!buf.data) return NO_RES; + taken= TRUE; + return OK; } @@ -1679,9 +1636,23 @@ Block_writer::get_buf(Buffer &buf) Block_writer::result_t Block_writer::write_buf(const Buffer &buf) { - stream_result::value res= m_str.write_window(buf.size); - m_str.end_chunk(); - return res == stream_result::OK ? OK : ERROR; + st_bstream_data_chunk chunk_info; + + chunk_info.table_no= buf.table_no; + chunk_info.data.begin= buf.data; + chunk_info.data.end= buf.data + buf.size; + chunk_info.flags= buf.last ? BSTREAM_FLAG_LAST_CHUNK : 0x00; + chunk_info.snap_no= snap_no; + + int ret= bstream_wr_data_chunk(&m_str,&chunk_info); + + if (ret == BSTREAM_OK) + { + taken= FALSE; + return OK; + } + else + return ERROR; } /** @@ -1694,9 +1665,9 @@ Block_writer::write_buf(const Buffer &bu Block_writer::result_t Block_writer::drop_buf(Buffer &buf) { - m_str.drop_window(); buf.data= NULL; buf.size= 0; + taken= FALSE; return OK; } diff -Nrup a/sql/backup/kernel.cc b/sql/backup/kernel.cc --- a/sql/backup/kernel.cc 2007-11-29 20:34:06 +01:00 +++ b/sql/backup/kernel.cc 2007-11-30 09:23:29 +01:00 @@ -2,22 +2,18 @@ @file Implementation of the backup kernel API. - */ - -/* - TODO: - - Handle SHOW BACKUP ARCHIVE command. - - Add database list to the backup/restore summary. - - Implement meta-data freeze during backup/restore. - - Improve logic selecting backup driver for a table. - - Handle other types of meta-data in Backup_info methods. - - Handle item dependencies when adding new items. - - Lock I_S tables when reading table list and similar (is it needed?) - - When reading table list from I_S tables, use select conditions to - limit amount of data read. (check prepare_select_* functions in sql_help.cc) - - Handle other kinds of backup locations (far future). - - Complete feedback given by BACKUP/RESTORE statements (send_summary function). + @todo Do not overwrite existing backup locations. + @todo Add more error messages. + @todo Use internal table name representation when passing tables to + backup/restore drivers. + @todo Implement meta-data freeze during backup/restore. + @todo Handle other types of meta-data in Backup_info methods. + @todo Handle item dependencies when adding new items. + @todo Lock I_S tables when reading table list and similar (is it needed?) + @todo When reading table list from I_S tables, use select conditions to + limit amount of data read. (check prepare_select_* functions in sql_help.cc) + @todo Handle other kinds of backup locations (far future). */ #include "../mysql_priv.h" @@ -25,9 +21,9 @@ #include "backup_aux.h" #include "stream.h" #include "backup_kernel.h" -#include "meta_data.h" #include "catalog.h" #include "debug.h" +#include "be_native.h" #include "be_default.h" #include "be_snapshot.h" @@ -57,6 +53,13 @@ static bool send_summary(THD*,const Rest // Flag used for testing error reporting bool test_error_flag= FALSE; #endif + +/* + (De)initialize memory allocator for backup stream library. + */ +void prepare_stream_memory(); +void free_stream_memory(); + } /** @@ -70,20 +73,20 @@ bool test_error_flag= FALSE; int execute_backup_command(THD *thd, LEX *lex) { - time_t skr; - DBUG_ENTER("execute_backup_command"); DBUG_ASSERT(thd && lex); BACKUP_BREAKPOINT("backup_command"); + using namespace backup; + /* Check access for SUPER rights. If user does not have SUPER, fail with error. */ if (check_global_access(thd, SUPER_ACL)) DBUG_RETURN(ER_SPECIFIC_ACCESS_DENIED_ERROR); - backup::Location *loc= backup::Location::find(lex->backup_dir); + Location *loc= Location::find(lex->backup_dir); if (!loc) { @@ -91,6 +94,7 @@ execute_backup_command(THD *thd, LEX *le DBUG_RETURN(ER_BACKUP_INVALID_LOC); } + prepare_stream_memory(); int res= 0; switch (lex->sql_command) { @@ -98,7 +102,7 @@ execute_backup_command(THD *thd, LEX *le case SQLCOM_SHOW_ARCHIVE: case SQLCOM_RESTORE: { - backup::IStream *stream= backup::open_for_read(*loc); + IStream *stream= open_for_read(*loc); if (!stream) { @@ -107,7 +111,7 @@ execute_backup_command(THD *thd, LEX *le } else { - backup::Restore_info info(*stream); + Restore_info info(thd,*stream); if (check_info(thd,info)) goto restore_error; @@ -124,7 +128,7 @@ execute_backup_command(THD *thd, LEX *le } else { - info.report_error(backup::log_level::INFO,ER_BACKUP_RESTORE_START); + info.report_error(log_level::INFO,ER_BACKUP_RESTORE_START); // TODO: freeze all DDL operations here @@ -143,8 +147,7 @@ execute_backup_command(THD *thd, LEX *le } else { - info.report_error(backup::log_level::INFO,ER_BACKUP_RESTORE_DONE); - info.total_size += info.header_size; + info.report_error(log_level::INFO,ER_BACKUP_RESTORE_DONE); send_summary(thd,info); } @@ -156,7 +159,7 @@ execute_backup_command(THD *thd, LEX *le restore_error: - res= res ? res : backup::ERROR; + res= res ? res : ERROR; finish_restore: @@ -168,7 +171,9 @@ execute_backup_command(THD *thd, LEX *le case SQLCOM_BACKUP: { - backup::OStream *stream= backup::open_for_write(*loc); + OStream *stream= open_for_write(*loc); + + // TODO: report error if location exists if (!stream) { @@ -177,31 +182,30 @@ execute_backup_command(THD *thd, LEX *le } else { - backup::Backup_info info(thd); + Backup_info info(thd); if (check_info(thd,info)) goto backup_error; - info.report_error(backup::log_level::INFO,ER_BACKUP_BACKUP_START); + info.report_error(log_level::INFO,ER_BACKUP_BACKUP_START); // TODO: freeze all DDL operations here /* Save starting datetime of backup. */ - skr= my_time(0); - gmtime_r(&skr, &info.start_time); + save_current_time(info.start_time); info.save_errors(); if (lex->db_list.is_empty()) { - info.write_message(backup::log_level::INFO,"Backing up all databases"); + info.write_message(log_level::INFO,"Backing up all databases"); info.add_all_dbs(); // backup all databases } else { - info.write_message(backup::log_level::INFO,"Backing up selected databases"); + info.write_message(log_level::INFO,"Backing up selected databases"); info.add_dbs(lex->db_list); // backup databases specified by user } @@ -222,15 +226,14 @@ execute_backup_command(THD *thd, LEX *le } else { - info.report_error(backup::log_level::INFO,ER_BACKUP_BACKUP_DONE); + info.report_error(log_level::INFO,ER_BACKUP_BACKUP_DONE); send_summary(thd,info); } /* Save ending datetime of backup. */ - skr= my_time(0); - gmtime_r(&skr, &info.end_time); + save_current_time(info.end_time); // TODO: unfreeze DDL here } // if (!stream) @@ -239,34 +242,28 @@ execute_backup_command(THD *thd, LEX *le backup_error: - res= res ? res : backup::ERROR; + res= res ? res : ERROR; finish_backup: if (stream) stream->close(); -#ifdef DBUG_BACKUP - backup::IStream *is= backup::open_for_read(*loc); - if (is) - { - dump_stream(*is); - is->close(); - } -#endif - break; } - default: + default: /* execute_backup_command() should be called with correct command id from the parser. If not, we fail on this assertion. */ DBUG_ASSERT(FALSE); - } + + } // switch(lex->sql_command) loc->free(); + free_stream_memory(); + DBUG_RETURN(res); } @@ -281,9 +278,6 @@ execute_backup_command(THD *thd, LEX *le namespace backup { -// defined in meta_backup.cc -int write_meta_data(THD*, Backup_info&, OStream&); - // defined in data_backup.cc int write_table_data(THD*, Backup_info&, OStream&); @@ -300,203 +294,127 @@ int mysql_backup(THD *thd, { DBUG_ENTER("mysql_backup"); + using namespace backup; + // This function should not be called with invalid backup info. DBUG_ASSERT(info.is_valid()); - size_t start_bytes= s.bytes; - BACKUP_BREAKPOINT("backup_meta"); - DBUG_PRINT("backup",("Writing image header")); - - if (info.save(s)) - goto error; - - DBUG_PRINT("backup",("Writing meta-data")); + DBUG_PRINT("backup",("Writing preamble")); - if (backup::write_meta_data(thd,info,s)) + if (write_preamble(info,s)) goto error; DBUG_PRINT("backup",("Writing table data")); BACKUP_BREAKPOINT("backup_data"); - if (backup::write_table_data(thd,info,s)) + if (write_table_data(thd,info,s)) + goto error; + + DBUG_PRINT("backup",("Writing summary")); + + if (write_summary(info,s)) goto error; DBUG_PRINT("backup",("Backup done.")); BACKUP_BREAKPOINT("backup_done"); - info.total_size= s.bytes - start_bytes; - DBUG_RETURN(0); error: - DBUG_RETURN(backup::ERROR); + DBUG_RETURN(ERROR); } namespace backup { -class Backup_info::Table_ref: - public backup::Table_ref -{ - String m_db_name; - String m_name; - ::TABLE *m_table; - - public: - - Table_ref(const Db_item&, const String&); - ~Table_ref() - { close(); } - - ::TABLE* open(THD*); - void close(); - - /** - Return pointer to @c handlerton structure of table's storage engine. - - @return @c NULL if table has not been opened or pointer to the @c handlerton - structure of table's storage engine. - */ - ::handlerton* hton() const - { return m_table && m_table->s ? m_table->s->db_type() : NULL; } - - /// Check if table has been opened. - bool is_open() const - { return m_table != NULL; } -}; - /** - Find image to which given table can be added and add it. - - Creates new images as needed. - - @returns Position of the image found in @c images[] table or -1 on error. - */ - -/* - Logic for selecting image format for a given table location: - - 1. If tables from that location have been already stored in one of the - sub-images then choose that sub-image. + Find backup engine which can backup data of a given table. - 2. If location has "native" backup format, put it in a new sub-image with - that format. + When necessary, a @c Snapshot_info object is created and added to the + @c m_snap[] table. - 3. Otherwise check if one of the existing sub-images would accept table from - this location. + @param t pointer to table's opened TABLE structure + @param tbl Table_ref describing the table - 4. If table has no native backup engine, try a consistent snapshot one. + @return position in @c m_snap[] of the @c Snapshot_info object to which the + table has been added or -1 on error. - 5. When everything else fails, use default (blocking) backup driver. - - Note: 1 is not implemented yet and hence we start with 3. + @todo Add error messages. */ - -int Backup_info::find_image(const Backup_info::Table_ref &tbl) +int Backup_info::find_backup_engine(const ::TABLE *const t, + const Table_ref &tbl) { - DBUG_ENTER("Backup_info::find_image"); - // we assume that table has been opened already - DBUG_ASSERT(tbl.is_open()); - - Image_info *img; - const ::handlerton *hton= tbl.hton(); - // If table has no handlerton something is really bad - we crash here - DBUG_ASSERT(hton); - + handlerton *hton= t->s->db_type(); Table_ref::describe_buf buf; + int no; + + DBUG_ENTER("Backup_info::find_backup_engine"); + DBUG_ASSERT(t); - DBUG_PRINT("backup",("Adding table %s using storage %s to archive%s", + DBUG_PRINT("backup",("Locating backup engine for table %s which uses" + " storage engine %s%s", tbl.describe(buf), - ::ha_resolve_storage_engine_name(hton), + hton ? ::ha_resolve_storage_engine_name(hton) : "(unknown engine)", hton->get_backup_engine ? " (has native backup)." : ".")); - // Point 3: try existing images but not the default or snapshot one. + /* + Note: at backup time the native and CS snapshot info objects are always + located at m_snap[0] and m_snap[1], respectively. They are created in + the Backup_info constructor. + */ - for (uint no=0; no < img_count && no < MAX_IMAGES ; ++no) - { - if (default_image_no >= 0 && no == (uint)default_image_no) - continue; + // try native driver if table has native backup engine - if (snapshot_image_no >= 0 && no == (uint)snapshot_image_no) - continue; + if (hton->get_backup_engine) + { + // see if the snapshot exists already (skip default ones) + for (no=2; no < 256 && m_snap[no] ; ++no) + if (m_snap[no]->accept(tbl,hton)) + DBUG_RETURN(no); - img= images[no]; + if (no == 256) + { + // TODO: report error + DBUG_RETURN(-1); + } - // An image object decides if it can handle given table or not - if( img && img->accept(tbl,hton) ) - DBUG_RETURN(no); - } + // We need to create native snapshot for this table - // Point 2: try native backup of table's storage engine. + m_snap[no]= new Native_snapshot(t->s->db_plugin); - if (hton->get_backup_engine) - { - uint no= img_count; - - // We handle at most MAX_IMAGES many images - DBUG_ASSERT(no<MAX_IMAGES); - - img= images[no]= new Native_image(*this,hton); - - if (!img) - { - report_error(ER_OUT_OF_RESOURCES); - DBUG_RETURN(-1); - } - - DBUG_PRINT("backup",("%s image added to archive",img->name())); - img_count++; - -#ifdef DBUG_OFF - // avoid "unused variable" compilation warning - img->accept(tbl,hton); -#else - // native image should accept all tables from its own engine - bool res= img->accept(tbl,hton); - DBUG_ASSERT(res); -#endif + if (!m_snap[no]) + { + // TODO: report error + DBUG_RETURN(-1); + } - DBUG_RETURN(no); - } + if (!m_snap[no]->accept(tbl,hton)) + { + // TODO: report error + DBUG_RETURN(-1); + } - // Points 4 & 5: try consistent snapshot and default drivers.. + DBUG_RETURN(no); + } - // try snapshot driver first - int ino= snapshot_image_no; - if (hton->start_consistent_snapshot != NULL) - { - if (snapshot_image_no < 0) //image doesn't exist - { - ino= img_count; - snapshot_image_no= img_count; - images[snapshot_image_no]= new Snapshot_image(*this); - img_count++; - DBUG_PRINT("backup",("Snapshot image added to archive")); - } - } - else - ino= default_image_no; //now try default driver - - if (ino < 0) //image doesn't exist - { - ino= img_count; - default_image_no= img_count; - images[default_image_no]= new Default_image(*this); - img_count++; - DBUG_PRINT("backup",("Default image added to archive")); - } - - img= images[ino]; - DBUG_ASSERT(img); - if (img->accept(tbl,hton)) - DBUG_RETURN(ino); // table accepted + /* + Try default drivers in decreasing order of preferrence + (first snapshot, then default) + */ + for (no=1; no >=0; --no) + { + if (!m_snap[no]->accept(tbl,hton)) + continue; - report_error(ER_BACKUP_NO_BACKUP_DRIVER,tbl.describe(buf,sizeof(buf))); - DBUG_RETURN(-1); + DBUG_RETURN(no); + } + + report_error(ER_BACKUP_NO_BACKUP_DRIVER,tbl.describe(buf,sizeof(buf))); + DBUG_RETURN(-1); } } // backup namespace @@ -513,13 +431,6 @@ namespace backup { // Returns tmp table containing records from a given I_S table TABLE* get_schema_table(THD *thd, ST_SCHEMA_TABLE *st); -/* - Backup_info::skip_table pointer is just for indicating that a table - added with Backup_info::add_table() was skipped. It should have value not - possible for regular pointers. - */ -const Backup_info::Table_item *const -Backup_info::skip_table= reinterpret_cast<Backup_info::Table_item*>(1); /** Create @c Backup_info structure and prepare it for populating with meta-data @@ -529,12 +440,13 @@ Backup_info::skip_table= reinterpret_cas These are found by reading INFORMATION_SCHEMA.TABLES table. The table is opened here so that it is ready for use in @c add_db_items() method. It is closed when the structure is closed with the @c close() method. + + @todo Report errors. */ Backup_info::Backup_info(THD *thd): Logger(Logger::BACKUP), - m_state(INIT), default_image_no(-1), snapshot_image_no(-2), - m_thd(thd), i_s_tables(NULL), - m_items(NULL), m_last_item(NULL), m_last_db(NULL) + m_state(INIT), + m_thd(thd), i_s_tables(NULL) { i_s_tables= get_schema_table(m_thd, ::get_schema_table(SCH_TABLES)); if (!i_s_tables) @@ -542,31 +454,61 @@ Backup_info::Backup_info(THD *thd): report_error(ER_BACKUP_LIST_TABLES); m_state= ERROR; } + + // create default and CS snapshot objects + + m_snap[0]= new Default_snapshot(); + if (!m_snap[0]) + { + // TODO: report error + close(); + m_state= ERROR; + } + + m_snap[1]= new CS_snapshot(); + if (!m_snap[1]) + { + // TODO: report error + close(); + m_state= ERROR; + } } Backup_info::~Backup_info() { close(); m_state= DONE; + name_strings.delete_elements(); + // Note: snapshot objects are deleted in ~Image_info() +} - Item_iterator it(*this); - Item *i; +/** + Store information about table data snapshot inside @c st_bstream_snapshot_info + structure. +*/ +void save_snapshot_info(const Snapshot_info &snap, st_bstream_snapshot_info &info) +{ + bzero(&info,sizeof(st_bstream_snapshot_info)); + info.type= enum_bstream_snapshot_type(snap.type()); + info.version= snap.version; + info.table_count= snap.table_count(); - while ((i=it++)) - delete i; + if (snap.type() == Snapshot_info::NATIVE_SNAPSHOT) + { + const Native_snapshot &nsnap= static_cast<const Native_snapshot&>(snap); - m_items= NULL; + info.engine.major= nsnap.se_ver >> 8; + info.engine.minor= nsnap.se_ver & 0xFF; + info.engine.name.begin= (byte*)nsnap.m_name; + info.engine.name.end= info.engine.name.begin + strlen(nsnap.m_name); + } } /** - Close the structure after populating it with items. - - When meta-data is written, we need to open archive's tables to be able - to read their definitions. Tables are opened here so that the - structure is ready for creation of the archive. + Close @c Backup_info object after populating it with items. - FIXME: opening all tables at once consumes too much resources when there - is a lot of tables (opened file descriptors!). Find a better solution. + After this call the @c Backup_info object is ready for use as a catalogue + for backup stream functions such as @c bstream_wr_preamble(). */ bool Backup_info::close() { @@ -576,44 +518,42 @@ bool Backup_info::close() ::free_tmp_table(m_thd,i_s_tables); i_s_tables= NULL; + /* + Go through snapshots and save their descriptions inside snapshots[] table. + */ + for (uint no=0; no < 256; ++no) + { + Snapshot_info *snap= m_snap[no]; + + if (!snap) + continue; + + if (snap->m_no == 0 || snap->table_count() == 0) + { + DBUG_ASSERT(snap->m_no == 0); + DBUG_ASSERT(snap->table_count() == 0); + delete snap; + m_snap[no]= NULL; + continue; + } + + save_snapshot_info(*snap,snapshot[snap->m_no-1]); + } + if (m_state == INIT) m_state= READY; return ok; } -int Backup_info::save(OStream &s) -{ - if (OK != Archive_info::save(s)) - { - report_error(ER_BACKUP_WRITE_HEADER); - return ERROR; - } - - return 0; -} /** - Specialization of @c backup::Db_ref with convenient constructors. - */ - -class Backup_info::Db_ref: public backup::Db_ref -{ - String m_name; + Add to backup image all databases in the list. - public: - - Db_ref(const String &s): backup::Db_ref(m_name), m_name(s) - {} - - Db_ref(const LEX_STRING &s): - backup::Db_ref(m_name), - m_name(s.str,s.length,::system_charset_info) - {} -}; + For each database, all objects stored in that database are also added to + the image. -/** - Add to archive all databases in the list. + @todo Report errors. */ int Backup_info::add_dbs(List< ::LEX_STRING > &dbs) { @@ -623,7 +563,8 @@ int Backup_info::add_dbs(List< ::LEX_STR while ((s= it++)) { - Db_ref db(*s); + String db_name(*s); + Db_ref db(db_name); if (db_exists(db)) { @@ -633,7 +574,10 @@ int Backup_info::add_dbs(List< ::LEX_STR Db_item *it= add_db(db); if (!it) + { + // TODO: report error goto error; + } if (add_db_items(*it)) goto error; @@ -661,9 +605,13 @@ int Backup_info::add_dbs(List< ::LEX_STR } /** - Add to archive all instance's databases (except the internal ones). - */ + Add all databases to backup image (except the internal ones). + For each database, all objects stored in that database are also added to + the image. + + @todo Report errors. +*/ int Backup_info::add_all_dbs() { my_bitmap_map *old_map; @@ -695,19 +643,28 @@ int Backup_info::add_all_dbs() while (!ha->rnd_next(db_table->record[0])) { - String db_name; + String *db_name= new (m_thd->mem_root) String(); - db_table->field[1]->val_str(&db_name); + if (!db_name) + { + report_error(ER_OUT_OF_RESOURCES); + res= ERROR; + goto finish; + } + + db_table->field[1]->val_str(db_name); // skip internal databases - if (db_name == String("information_schema",&::my_charset_bin) - || db_name == String("mysql",&my_charset_bin)) + if (*db_name == String("information_schema") + || *db_name == String("mysql")) { - DBUG_PRINT("backup",(" Skipping internal database %s",db_name.ptr())); + DBUG_PRINT("backup",(" Skipping internal database %s",db_name->ptr())); + delete db_name; continue; } - Db_ref db(db_name); + db_name->copy(); + Db_ref db(*db_name); DBUG_PRINT("backup", (" Found database %s", db.name().ptr())); @@ -716,9 +673,15 @@ int Backup_info::add_all_dbs() if (!it) { res= -3; + delete db_name; goto finish; } + /* + Add name to name_strings list so that it is freed in the destructor. + */ + name_strings.push_back(db_name); + if (add_db_items(*it)) { res= -4; @@ -744,55 +707,14 @@ int Backup_info::add_all_dbs() return res; } -/** - Insert a @c Db_item into the meta-data items list. - */ -Backup_info::Db_item* -Backup_info::add_db(const backup::Db_ref &db) -{ - StringPool::Key key= db_names.add(db.name()); - - if (!key.is_valid()) - { - report_error(ER_OUT_OF_RESOURCES); - return NULL; - } - - Db_item *di= new Db_item(*this,key); - - if (!di) - { - report_error(ER_OUT_OF_RESOURCES); - return NULL; - } - - // insert db item before all table items - - if (!m_last_db) - { - di->next= m_items; - m_items= m_last_db= di; - if (!m_last_item) - m_last_item= m_items; - } - else - { - // since m_last_db is not NULL, m_items list can't be empty - DBUG_ASSERT(m_items); - - di->next= m_last_db->next; - if (m_last_item == m_last_db) - m_last_item= di; - m_last_db->next= di; - } - - return di; -} /** - Add to archive all items belonging to a given database. + Add to archive all objects belonging to a given database. + + @todo Handle other types of objects - not only tables. + @todo Use WHERE clauses when reading I_S.TABLES */ -int Backup_info::add_db_items(Db_item &db) +int Backup_info::add_db_items(Db_item &dbi) { my_bitmap_map *old_map; @@ -800,34 +722,37 @@ int Backup_info::add_db_items(Db_item &d DBUG_ASSERT(is_valid()); // should be valid DBUG_ASSERT(i_s_tables->file); // i_s_tables should be opened - // add all tables from db. - ::handler *ha= i_s_tables->file; /* If error debugging is switched on (see debug.h) then I_S.TABLES access error will be triggered when backing up database whose name starts with 'a'. */ - TEST_ERROR_IF(db.name().ptr()[0]=='a'); + TEST_ERROR_IF(dbi.name().ptr()[0]=='a'); old_map= dbug_tmp_use_all_columns(i_s_tables, i_s_tables->read_set); if (ha->ha_rnd_init(TRUE) || TEST_ERROR) { dbug_tmp_restore_column_map(i_s_tables->read_set, old_map); - report_error(ER_BACKUP_LIST_DB_TABLES,db.name().ptr()); + report_error(ER_BACKUP_LIST_DB_TABLES,dbi.name().ptr()); return ERROR; } int res= 0; + /* + TODO: Instead of looping through all records in the I_S.TABLES table + we could execute an SQL query with appropriate WHERE clause. + */ while (!ha->rnd_next(i_s_tables->record[0])) { String db_name; - String name; String type; String engine; + String *name= new (m_thd->mem_root) String(); + /* Read info about next table/view @@ -835,51 +760,57 @@ int Backup_info::add_db_items(Db_item &d INFORMATION_SCHEMA.TABLES table. */ i_s_tables->field[1]->val_str(&db_name); - i_s_tables->field[2]->val_str(&name); + i_s_tables->field[2]->val_str(name); i_s_tables->field[3]->val_str(&type); i_s_tables->field[4]->val_str(&engine); - if (db_name != db.name()) - continue; // skip tables not from the given database + // skip tables not from the given database + if (db_name != dbi.name()) + { + delete name; + continue; + } // FIXME: right now, we handle only tables - if (type != String("BASE TABLE",&::my_charset_bin)) + if (type != String("BASE TABLE")) { report_error(log_level::WARNING,ER_BACKUP_SKIP_VIEW, - name.c_ptr(),db_name.c_ptr()); + name->c_ptr(),db_name.c_ptr()); + delete name; continue; } - Backup_info::Table_ref t(db,name); + name->copy(); + Backup_info::Table_ref t(dbi,*name); if (engine.is_empty()) { Table_ref::describe_buf buf; report_error(log_level::WARNING,ER_BACKUP_NO_ENGINE,t.describe(buf)); + delete name; continue; } DBUG_PRINT("backup", ("Found table %s for database %s", t.name().ptr(), t.db().name().ptr())); - // We need to open table for add_table() method below - if (!t.open(m_thd)) - { - Table_ref::describe_buf buf; - report_error(ER_BACKUP_TABLE_OPEN,t.describe(buf)); - goto error; - } - // add_table method selects/creates sub-image appropriate for storing given table - Table_item *ti= add_table(t); - - // Close table to free resources - t.close(); + /* + add_table() method selects/creates a snapshot to which this table is added. + The backup engine is chooden in Backup_info::find_backup_engine() method. + */ + Table_item *ti= add_table(dbi,t); if (!ti) + { + delete name; goto error; + } - if (ti == skip_table) - continue; + /* + We store pointers to created String objects so that they all will + be deleted in ~Backup_info(). + */ + name_strings.push_back(name); if (add_table_items(*ti)) goto error; @@ -900,83 +831,68 @@ int Backup_info::add_db_items(Db_item &d return res; } -/* Implementation of Backup_info::Table_ref */ - -Backup_info::Table_ref::Table_ref(const Db_item &db, const String &name): - backup::Table_ref(m_db_name,m_name), m_table(NULL) -{ - m_db_name.append(db.name()); - m_name.append(name); -} /** - Open table and create corresponding @c TABLE structure. - - A pointer to opened @c TABLE structure is stored in @c m_table member. The - structure is owned by @c Table_ref object, to destroy it call @c close() - method. - - This method does nothing if table has been already opened. + Add table to archive's list of meta-data items. - @return Pointer to the opened @c TABLE structure or @c NULL if operation was - not successful. - */ -::TABLE* Backup_info::Table_ref::open(THD *thd) + @todo Correctly handle temporary tables. + @todo Avoid opening tables here - open them only in bcat_get_create_stmt(). +*/ +Image_info::Table_item* +Backup_info::add_table(Db_item &dbi, const Table_ref &t) { - if (is_open()) - return m_table; + Table_ref::describe_buf buf; + // TODO: skip table if it is a tmp one - char path[FN_REFLEN]; - const char *db= m_db_name.ptr(); - const char *name= m_name.ptr(); - ::build_table_filename(path, sizeof(path), db, name, "", 0); - m_table= ::open_temporary_table( - thd, path, db, name, - FALSE /* don't link to thd->temporary_tables */, - OTM_OPEN); + Table_item *ti= NULL; /* - Note: If table couldn't be opened (m_table==NULL), open_temporary_table() - doesn't inform us what was the reason. This makes it difficult to give - precise information in the error log. Currently we just say that table - couldn't be opened. When error reporting is improved, we should try to do - better than that. - */ - return m_table; -} + open table temporarily to: + - get its handlerton + - get a CREATE statement for it + */ -/** - Close previously opened table. + TABLE_LIST entry, *tl= &entry; + bzero(&entry,sizeof(entry)); - Closes table and frees allocated resources. Can be called even when table - has not been opened, in which case it does nothing. - */ -void Backup_info::Table_ref::close() -{ - if (m_table) + // FIXME: table/db name mangling + entry.db= const_cast<char*>(t.db().name().ptr()); + entry.alias= entry.table_name= const_cast<char*>(t.name().ptr()); + + uint cnt; + int res= ::open_tables(m_thd,&tl,&cnt,0); + + if (res || !tl->table) { - // TODO: check if ::free_tmp_table() is not better - ::intern_close_table(m_table); - my_free(m_table, MYF(0)); + report_error(ER_BACKUP_TABLE_OPEN,t.describe(buf)); + return NULL; } - m_table= NULL; - return; -} -/** - Add table to archive's list of meta-data items. + /* + alternative way of opening a single tmp table - but it + doesn't initialize TABLE_LIST structure which we need for getting + CREATE statement. - @pre Table should be opened. - */ + char path[FN_REFLEN]; + const char *db= t.db().name().ptr(); + const char *name= t.name().ptr(); -Backup_info::Table_item* -Backup_info::add_table(const Table_ref &t) -{ - DBUG_ASSERT(t.is_open()); + ::build_table_filename(path, sizeof(path), db, name, "", 0); - // TODO: skip table if it is a tmp one + ::TABLE *table= ::open_temporary_table(m_thd, path, db, name, + FALSE /=* don't link to thd->temporary_tables *=/); + + ... + + ::intern_close_table(table); + my_free(table, MYF(0)); + */ - int no= find_image(t); // Note: find_image reports errors + int no= find_backup_engine(tl->table,t); // Note: reports errors + + DBUG_PRINT("backup",(" table %s backed-up with %s engine", + t.describe(buf), + m_snap[no]->name())); /* If error debugging is switched on (see debug.h) then any table whose @@ -985,14 +901,11 @@ Backup_info::add_table(const Table_ref & TEST_ERROR_IF(t.name().ptr()[0]=='a'); if (no < 0 || TEST_ERROR) - return NULL; + goto end; - /* - If image was found then images[no] should point at the Image_info - object - */ - Image_info *img= images[no]; - DBUG_ASSERT(img); + // add table to the catalogue + + ti= Image_info::add_table(dbi,t,no); /* If error debugging is switched on (see debug.h) then any table whose @@ -1000,39 +913,40 @@ Backup_info::add_table(const Table_ref & */ TEST_ERROR_IF(t.name().ptr()[0]=='b'); - int tno= img->tables.add(t); - if (tno < 0 || TEST_ERROR) + if (!ti || TEST_ERROR) { - Table_ref::describe_buf buf; - report_error(ER_BACKUP_NOT_ACCEPTED,img->name(),t.describe(buf)); - return NULL; + report_error(ER_OUT_OF_RESOURCES); + goto end; } - table_count++; - - Table_item *ti= new Table_item(*this,no,tno); + /* + Note: we obtain and store CREATE statement here because that requires + the table to be opened. Otherwise we could do that when meta-data is + written. + We could still do this (and save memory used to store these statements) + at an expense of opening table again during meta-data write phase. + + TODO: don't open table here, but locate its storage engine based on its + name (which is read from I_S.TABLES table). Then CREATE statement could + be constructed inside meta::Table::get_create_stmt() method. + */ - if (!ti) - { - report_error(ER_OUT_OF_RESOURCES); - return NULL; - } + ti->tl_entry= tl; + ti->get_create_stmt(); + ti->tl_entry= NULL; - // add to the end of items list + end: - ti->next= NULL; - if (m_last_item) - m_last_item->next= ti; - m_last_item= ti; - if (!m_items) - m_items= ti; + ::close_thread_tables(m_thd); return ti; } /** Add to archive all items belonging to a given table. - */ + + @todo Implement this. +*/ int Backup_info::add_table_items(Table_item&) { // TODO: Implement when we handle per-table meta-data. @@ -1052,9 +966,6 @@ int Backup_info::add_table_items(Table_i namespace backup { -// defined in meta_backup.cc -int restore_meta_data(THD*, Restore_info&, IStream&); - // defined in data_backup.cc int restore_table_data(THD*, Restore_info&, IStream&); @@ -1062,34 +973,460 @@ int restore_table_data(THD*, Restore_inf /** - Restore items saved in backup archive. + Restore objects saved in backup image. - @pre archive info has been already read and the stream is positioned - at archive's meta-data + @pre The header and catalogue of backup image has been already read with + @c bstream_rd_header() function and stored inside the @c info object. */ int mysql_restore(THD *thd, backup::Restore_info &info, backup::IStream &s) { DBUG_ENTER("mysql_restore"); - size_t start_bytes= s.bytes; + using namespace backup; + + s.next_chunk(); DBUG_PRINT("restore",("Restoring meta-data")); - if (backup::restore_meta_data(thd, info, s)) - DBUG_RETURN(backup::ERROR); + if (read_meta_data(info, s) == ERROR) + DBUG_RETURN(ERROR); + + s.next_chunk(); DBUG_PRINT("restore",("Restoring table data")); - if (backup::restore_table_data(thd,info,s)) - DBUG_RETURN(backup::ERROR); + // Here restore drivers are created to restore table data + if (restore_table_data(thd,info,s) == ERROR) + DBUG_RETURN(ERROR); DBUG_PRINT("restore",("Done.")); - info.total_size= s.bytes - start_bytes; + if (read_summary(info,s) == ERROR) + DBUG_RETURN(ERROR); DBUG_RETURN(0); } +/**************************** + + Restore_info implementation + + ****************************/ + +namespace backup { + +/** + Initialize @c Restore_info instance and load the catalogue from + the given backup stream. +*/ +Restore_info::Restore_info(THD *thd, IStream &s): + Logger(Logger::RESTORE), m_valid(TRUE), m_thd(thd), curr_db(NULL), + system_charset(NULL), same_sys_charset(TRUE) +{ + int ret= BSTREAM_OK; + + ret= read_header(*this,s); + + if (!(m_valid= (ret != BSTREAM_ERROR))) + return; + + ret= s.next_chunk(); + + if (!(m_valid= (ret == BSTREAM_OK))) + return; + + ret= read_catalog(*this,s); + m_valid= (ret != BSTREAM_ERROR); +} + +Restore_info::~Restore_info() +{} + +/** + Restore an object given the metadata read from backup stream. + + @param it @c Item object describing the object to be restored, + @param stmt that object's CREATE statement stored in the backup image + (can be empty), + @param begin first byte of extra metadata stored in the image, + @param end one after the last byte of the extra metadat. + + If no extra metadata was stored for that object, @c begin and @c end are NULL. + + @note The actual job is done by @c Item::create() and @c Item::drop() + methods, here we only create and maintain correct context for these methods + to work. +*/ +result_t Restore_info::restore_item(Item &it, ::String &stmt, + byte *begin, byte *end) +{ + /* + change the current database if we are going to create a per-db item + and we are not already in the correct one. + */ + const Db_ref *db= it.in_db(); + + if (db && (!curr_db || *db != *curr_db)) + { + DBUG_PRINT("restore",(" changing current db to %s",db->name().ptr())); + curr_db= db; + change_db(m_thd,*curr_db); + } + + /* + note: This is destructive restore, so we drop each object before + creating it. + */ + + result_t ret= it.drop(::current_thd); + + if (ret != OK) + return ret; + + ret= it.create(m_thd, stmt, begin, end); + + return ret; +} + +} // backup namespace + +/************************************************* + + CATALOGUE SERVICES + + *************************************************/ + +/** + Prepare @c Restore_info object for populating the catalogue with items to + restore. + + At this point we know the list of table data snapshots present in the image + (it was read from image's header). Here we create @c Snapshot_info object + for each of them. + + @todo Report errors. +*/ +extern "C" +int bcat_reset(st_bstream_image_header *catalogue) +{ + using namespace backup; + uint no; + + Restore_info *info= static_cast<Restore_info*>(catalogue); + + for (no=0; no < info->snap_count; ++no) + { + st_bstream_snapshot_info *snap= &info->snapshot[no]; + + DBUG_PRINT("restore",("Creating info for snapshot no %d",no)); + + switch (snap->type) { + + case BI_NATIVE: + { + backup::LEX_STRING name_lex(snap->engine.name.begin, snap->engine.name.end); + plugin_ref se= ::ha_resolve_by_name(::current_thd,&name_lex); + handlerton *hton= plugin_data(se,handlerton*); + + if (!hton) + { + // TODO: report error + return BSTREAM_ERROR; + } + + if (!hton->get_backup_engine) + { + // TODO: report error + return BSTREAM_ERROR; + } + + info->m_snap[no]= new Native_snapshot(se); + + break; + } + + case BI_CS: + info->m_snap[no]= new CS_snapshot(); + break; + + case BI_DEFAULT: + info->m_snap[no]= new Default_snapshot(); + break; + + default: + DBUG_PRINT("restore",("Unknown snapshot type %d", + info->snapshot[no].type)); + return BSTREAM_ERROR; + } + + if (!info->m_snap[no]) + { + // TODO: report error + return BSTREAM_ERROR; + } + + if (info->m_snap[no]->set_version(snap->version) != OK) + { + // TODO: report error + return BSTREAM_ERROR; + } + + info->m_snap[no]->m_no= no+1; + + DBUG_PRINT("restore",(" snapshot uses %s engine",info->m_snap[no]->name())); + } + + return BSTREAM_OK; +} + +/** + Called after reading backup image's catalogue and before processing + metadata and table data. + + Nothing to do here. +*/ +extern "C" +int bcat_close(st_bstream_image_header *catalogue) +{ return BSTREAM_OK; } + +/** + Add item to restore catalogue. + + @todo Report errors. +*/ +extern "C" +int bcat_add_item(st_bstream_image_header *catalogue, struct st_bstream_item_info *item) +{ + using namespace backup; + + Restore_info *info= static_cast<Restore_info*>(catalogue); + + backup::String name_str(item->name.begin, item->name.end); + + DBUG_PRINT("restore",("Adding item %s of type %d (pos=%ld)", + item->name.begin, + item->type, + item->pos)); + + switch (item->type) { + + case BSTREAM_IT_DB: + { + Image_info::Db_ref db(name_str); + + Image_info::Db_item *dbi= info->add_db(db,item->pos); + + if (!dbi) + { + // TODO: report error + return BSTREAM_ERROR; + } + + return BSTREAM_OK; + } + + case BSTREAM_IT_TABLE: + { + st_bstream_table_info *it= (st_bstream_table_info*)item; + + DBUG_PRINT("restore",(" table's snapshot no is %d",it->snap_no)); + + Image_info::Db_item *db= info->get_db(it->base.db->base.pos); + + if (!db) + { + // TODO: report error + return BSTREAM_ERROR; + } + + DBUG_PRINT("restore",(" table's database is %s",db->name().ptr())); + + Image_info::Table_ref t(*db,name_str); + + Image_info::Table_item *ti= info->add_table(*db,t,it->snap_no, + item->pos); + if (!ti) + { + // TODO: report error + return BSTREAM_ERROR; + } + + return BSTREAM_OK; + } + + default: + return BSTREAM_OK; + + } // switch (item->type) +} + +/************************************************* + + MEMORY ALLOCATOR FOR BACKUP STREAM LIBRARY + + *************************************************/ + +namespace backup { + +/** + This calss provides memory allocation services for backup stream library. + + An instance of this class must be created and pointer to it stored in the + static @c instance variable during BACKUP/RESTORE operation. This assumes + only one BACKUP/RESTORE operation is running at a time. +*/ +class Mem_allocator +{ + public: + + Mem_allocator(); + ~Mem_allocator(); + + void* alloc(size_t); + void free(void*); + + static Mem_allocator *instance; + + private: + + /// All allocated memory segments are linked into a list using this structure. + struct node + { + node *prev; + node *next; + }; + + node *first; ///< Pointer to the first segment in the list. +}; + + +Mem_allocator::Mem_allocator(): first(NULL) +{} + +/// Deletes all allocated segments which have not been freed. +Mem_allocator::~Mem_allocator() +{ + node *n= first; + + while (n) + { + first= n->next; + my_free(n,MYF(0)); + n= first; + } +} + +/** + Allocate memory segment of given size. + + Extra memory is allocated for a @c node structure which holds pointers + to previous and next segment in the segments list. This is used when + deallocating allocated memory in the destructor. +*/ +void* Mem_allocator::alloc(size_t howmuch) +{ + void *ptr= my_malloc(sizeof(node)+howmuch, MYF(0)); + + if (!ptr) + return NULL; + + node *n= (node*)ptr; + ptr= n + 1; + + n->prev= NULL; + n->next= first; + if (first) + first->prev= n; + first= n; + + return ptr; +} + +/** + Explicit deallocation of previously allocated segment. + + The @c ptr should contain an address which was obtained from + @c Mem_allocator::alloc(). + + The deallocated fragment is removed from the allocated fragments list. +*/ +void Mem_allocator::free(void *ptr) +{ + if (!ptr) + return; + + node *n= ((node*)ptr) - 1; + + if (first == n) + first= n->next; + + if (n->prev) + n->prev->next= n->next; + + if (n->next) + n->next->prev= n->prev; + + my_free(n,MYF(0)); +} + +Mem_allocator *Mem_allocator::instance= NULL; + +/** + This function must be called before @c bstream_alloc() can be used. +*/ +void prepare_stream_memory() +{ + if (Mem_allocator::instance) + delete Mem_allocator::instance; + + Mem_allocator::instance= new Mem_allocator(); +} + +/** + This function should be called when @c bstream_alloc()/ @c bstream_free() + are no longer to be used. + + It destroys the Mem_allocator instance which frees all memory which was + allocated but not explicitly freed. +*/ +void free_stream_memory() +{ + delete Mem_allocator::instance; + Mem_allocator::instance= NULL; +} + +} + +extern "C" { + +/** + Memory allocator for backup stream library. + + @pre @c prepare_stream_memory() has been called (i.e., the Mem_allocator + instance is created. + */ +bstream_byte* bstream_alloc(unsigned long int size) +{ + using namespace backup; + + DBUG_ASSERT(Mem_allocator::instance); + + return (bstream_byte*)Mem_allocator::instance->alloc(size); +} + +/** + Memory deallocator for backup stream library. +*/ +void bstream_free(bstream_byte *ptr) +{ + using namespace backup; + + if (Mem_allocator::instance) + Mem_allocator::instance->free(ptr); +} + +} + /************************************************* BACKUP LOCATIONS @@ -1098,9 +1435,13 @@ int mysql_restore(THD *thd, backup::Rest namespace backup { +/** + Specialization of @c Location class representing a file in the local + filesystem. +*/ struct File_loc: public Location { - String path; + ::String path; enum_type type() const { return SERVER_FILE; } @@ -1145,7 +1486,6 @@ IStream* open_for_read(const Location &l return open_stream<IStream>(loc); } - OStream* open_for_write(const Location &loc) { return open_stream<OStream>(loc); @@ -1232,12 +1572,9 @@ int check_info(THD *thd, Restore_info &i The data about the operation is taken from filled @c Archive_info structure. Parameter @c backup determines if this was backup or restore operation. - - TODO: Add list of databases in the output. */ - static -bool send_summary(THD *thd, const Archive_info &info, bool backup) +bool send_summary(THD *thd, const Image_info &info, bool backup) { Protocol *protocol= ::current_thd->protocol; // client comms List<Item> field_list; // list of fields to send @@ -1261,27 +1598,23 @@ bool send_summary(THD *thd, const Archiv item->maybe_null= TRUE; protocol->send_fields(&field_list, Protocol::SEND_NUM_ROWS | Protocol::SEND_EOF); - my_snprintf(buf,sizeof(buf)," header = %-8lu bytes",(unsigned long)info.header_size); - protocol->prepare_for_resend(); - protocol->store(buf,system_charset_info); - protocol->write(); - - my_snprintf(buf,sizeof(buf)," meta-data = %-8lu bytes",(unsigned long)info.meta_size); + my_snprintf(buf,sizeof(buf)," %s %d tables", + backup ? "backed up" : "restored", + info.table_count); protocol->prepare_for_resend(); protocol->store(buf,system_charset_info); protocol->write(); - my_snprintf(buf,sizeof(buf)," data = %-8lu bytes",(unsigned long)info.data_size); + my_snprintf(buf,sizeof(buf)," in %d database%s", + info.db_count(), + info.db_count() > 1 ? "s" : ""); protocol->prepare_for_resend(); protocol->store(buf,system_charset_info); protocol->write(); - my_snprintf(buf,sizeof(buf)," --------------"); - protocol->prepare_for_resend(); - protocol->store(buf,system_charset_info); - protocol->write(); - - my_snprintf(buf,sizeof(buf)," total %-8lu bytes", (unsigned long)info.total_size); + my_snprintf(buf,sizeof(buf)," using %d driver%s", + info.snap_count, + info.snap_count > 1 ? "s" : ""); protocol->prepare_for_resend(); protocol->store(buf,system_charset_info); protocol->write(); @@ -1321,7 +1654,7 @@ TABLE* get_schema_table(THD *thd, ST_SCH Temporarily set thd->lex->wild to NULL to keep st->fill_table happy. */ - String *wild= thd->lex->wild; + ::String *wild= thd->lex->wild; ::enum_sql_command command= thd->lex->sql_command; thd->lex->wild = NULL; @@ -1380,5 +1713,62 @@ TABLE_LIST *build_table_list(const Table return tl; } + + +/// Execute SQL query without sending anything to client. + +int silent_exec_query(THD *thd, ::String &query) +{ + Vio *save_vio= thd->net.vio; + + DBUG_PRINT("restore",("executing query %s",query.c_ptr())); + + /* + Note: the change net.vio idea taken from execute_init_command in + sql_parse.cc + */ + thd->net.vio= 0; + thd->net.no_send_error= 0; + + thd->query= query.c_ptr(); + thd->query_length= query.length(); + + thd->set_time(time(NULL)); + pthread_mutex_lock(&::LOCK_thread_count); + thd->query_id= ::next_query_id(); + pthread_mutex_unlock(&::LOCK_thread_count); + + const char *ptr; + ::mysql_parse(thd,thd->query,thd->query_length,&ptr); + + thd->net.vio= save_vio; + + if (thd->is_error()) + { + DBUG_PRINT("restore", + ("error executing query %s!", thd->query)); + DBUG_PRINT("restore",("last error (%d): %s",thd->net.last_errno + ,thd->net.last_error)); + return thd->net.last_errno ? (int)thd->net.last_errno : -1; + } + + return 0; +} + +/// Get current time and save it inside bstream_time_t. + +void save_current_time(bstream_time_t &buf) +{ + time_t now= my_time(0); + struct tm time; + gmtime_r(&now,&time); + buf.year= time.tm_year; + buf.mon= time.tm_mon; + buf.mday= time.tm_mday; + buf.hour= time.tm_hour; + buf.min= time.tm_min; + buf.sec= time.tm_sec; +} + } // backup namespace diff -Nrup a/sql/backup/map.h b/sql/backup/map.h --- a/sql/backup/map.h 2007-11-06 19:32:17 +01:00 +++ /dev/null Wed Dec 31 16:00:00 196900 @@ -1,189 +0,0 @@ -#ifndef _UTIL_MAP_H -#define _UTIL_MAP_H - -namespace util { - -/******************************************************* - - Map (associative array) datatype templates. - - *******************************************************/ - -struct key8; - -template<class D, class K=key8> -class Map -{ - public: - - typedef typename D::Element El; - typedef typename K::Key Key; - static const size_t size= K::size; - - Key add(const El &e) - { - Key k= D::hash(e); - return find_el(k,e,TRUE); - } - - El& operator[](const Key &k) const - { - if (!occupied(k)) - return const_cast<El&>(D::null); - - return *(entries[k].el); - } - - Key find(const El &e) const - { - Key k= D::hash(e); - return const_cast<Map<D,K>*>(this)->find_el(k,e); - } - - bool occupied(const Key &k) const - { - return K::valid_key(k) && entries[k].el != NULL; - } - - size_t count() const - { return m_count; } - - Map(): m_count(0) {} - - ~Map() { clear(); } - - void clear() - { - for (uint i=0; i < size; i++) - if (entries[i].el) - delete entries[i].el; - } - -#ifdef MAP_DEBUG - - void print() - { - for(uint i=0 ; i < K::size ; i++ ) - { - node &n= entries[i]; - - if( n.el ) - { - printf("entry %02d (%d,%d): ", i, (int)n.bigger, (int)n.smaller ); - D::print(*n.el); - printf("\n"); - } - } - } - -#endif - - protected: - - // Use binary search tree for storing entries. - - struct node { - El *el; - Key bigger,smaller; - node(): el(NULL), bigger(K::null), smaller(K::null) {}; - void operator=(const El &e) - { - if (el) delete el; - el= D::create(e); - bigger= K::null; - smaller= K::null; - }; - } entries[K::size]; - - uint m_count; - - // PRE: k is valid. - Key find_el(const Key &k, const El &e, bool insert= FALSE) - { - El *x= entries[k].el; - - if (!x) - if (insert) - { - set(k,e); - return k; - } - else return K::null; - - int res; - - if ((res= D::cmp(e,*x)) == 0) - return k; - - Key &k1 = res>0 ? entries[k].bigger : entries[k].smaller; - - if (K::valid_key(k1)) - return find_el(k1,e); - - Key k2; - if (K::valid_key(k2= find_free_loc())) - { - k1= k2; - set(k2,e); - } - - return k2; - } - - Key find_free_loc() const - { - if (m_count >= K::size) - return K::null; - - for (uint k=0; k < size; k++) - if (entries[k].el == NULL) - return k; - - return K::null; - } - - // PRE k is valid - void set(const Key &k, const El &e) - { - entries[k]= e; - m_count++; - } - -}; - - -// 8 bit keys - -struct key8 -{ - typedef key8 Key; - static const size_t size= (size_t)255; - static const unsigned int null= 0xFF; - - operator int() const { return val; }; - bool is_valid() const { return key8::valid_key(*this); }; - static bool valid_key(const Key &k) { return k.val != 0xFF; }; - - key8(): val(0xFF) {}; - key8(unsigned int x) { operator=(x); }; - - Key &operator=(unsigned int x) - { - val= x & 0xFF; - // simple hashing - for (int bits= sizeof(unsigned int) ; bits > 8 ; bits-= 8) - { - x >>= 8; - val ^= x &0xFF; - }; - return *this; - } - - private: - - unsigned char val; -}; - -} // util namespace - -#endif diff -Nrup a/sql/backup/meta_data.cc b/sql/backup/meta_data.cc --- a/sql/backup/meta_data.cc 2007-11-29 20:34:06 +01:00 +++ b/sql/backup/meta_data.cc 2007-11-30 09:23:30 +01:00 @@ -1,228 +1,115 @@ /** @file - Functions used by kernel to save/restore meta-data - */ - -/* - TODO: - - - Handle events, routines, triggers and other item types (use Chuck's code). + @brief Code for handling metadata. - - When saving database info, save its default charset and collation. - Alternatively, save a complete "CREATE DATABASE ..." statement with all - clauses which should work even when new clauses are added in the future. - - - In silent_exec_query() - reset, collect and report errors from statement - execution. - - - In the same function: investigate what happens if query triggers error - - there were signals that the code might hang in that case (some clean-up - needed?) - */ + @todo Handle objects of unknown type + @todo Get and save full CREATE statements also for databases. + @todo Decide how to singal errors in bcat_get_item_create_query. +*/ #include <mysql_priv.h> #include <sql_show.h> -#include "backup_aux.h" -#include "backup_kernel.h" -#include "meta_data.h" +#include <backup_stream.h> +#include <backup_kernel.h> +#include "stream_services.h" +/***************************************************************** -namespace backup { + Services for backup stream library related to meta-data + manipulation. -/** - Write meta-data description to backup stream. + *****************************************************************/ - Meta-data information is stored as a sequence of entries, each describing a - single meta-data item. The format of a single entry depends on the type of - item (see @c Archive_info::Item::save() method). The entries are saved in a - single chunk. +extern "C" +int bcat_create_item(st_bstream_image_header *catalogue, + struct st_bstream_item_info *item, + bstream_blob create_stmt, + bstream_blob other_meta_data) +{ + using namespace backup; - The order of entries is important. Items on which other items depend should - be saved first in the sequence. This order is determined by the implementation - of @c Backup_info::Item_iterator class. + DBUG_ASSERT(catalogue); + DBUG_ASSERT(item); - @note Meta-data description is added to the current chunk of the stream which - is then closed. + Restore_info *info= static_cast<Restore_info*>(catalogue); - @returns 0 on success, error code otherwise. - */ -int write_meta_data(THD *thd, Backup_info &info, OStream &s) -{ - DBUG_ENTER("backup::write_meta_data"); + Image_info::Item *it= info->locate_item(item); - size_t start_bytes= s.bytes; + /* + TODO: Decide what to do when we come across unknown item (locate_item() + returns NULL): break the restore process as it is done now or continue + with a warning? + */ - for (Backup_info::Item_iterator it(info); it; it++) - { - result_t res= it->save(thd,s); // this calls Archive_info::Item::save() method + if (!it) + return BSTREAM_ERROR; // locate_item should report errors - if (res != OK) - { - meta::Item::description_buf buf; - info.report_error(ER_BACKUP_WRITE_META, - it->meta().describe(buf,sizeof(buf))); - DBUG_RETURN(ERROR); - } - } + backup::String stmt(create_stmt.begin, create_stmt.end); - if (stream_result::ERROR == s.end_chunk()) - DBUG_RETURN(ERROR); + DBUG_PRINT("restore",("Creating item of type %d pos %ld: %s", + item->type, item->pos, stmt.ptr())); - info.meta_size= s.bytes - start_bytes; + result_t ret= info->restore_item(*it,stmt, + other_meta_data.begin, + other_meta_data.end); - DBUG_RETURN(0); + return ret == OK ? BSTREAM_OK : BSTREAM_ERROR; } - -/** - Read meta-data items from a stream and create them if they are selected - for restore. - - @pre Stream is at the beginning of a saved meta-data chunk. - @post Stream is at the beginning of the next chunk. - */ -int restore_meta_data(THD *thd, Restore_info &info, IStream &s) +extern "C" +int bcat_get_item_create_query(st_bstream_image_header *catalogue, + struct st_bstream_item_info *item, + bstream_blob *stmt) { - DBUG_ENTER("restore_meta_data"); - Archive_info::Item *it; // save pointer to item read form the stream - Db_ref curr_db; // remember the current database - result_t res; - - size_t start_bytes= s.bytes; - - // read items from the stream until error or end of data is reached. - while (OK == (res= Archive_info::Item::create_from_stream(info,s,it))) - { - DBUG_PRINT("restore",(" got next meta-item.")); - - if (info.selected(*it)) // if the item was selected for restore ... - { - DBUG_PRINT("restore",(" creating it!")); - - /* - change the current database if we are going to create a per-db item - and we are not already in the correct one. - */ - const Db_ref db= it->meta().in_db(); - - if (db.is_valid() && (!curr_db.is_valid() || db != curr_db)) - { - DBUG_PRINT("restore",(" changing current db to %s",db.name().ptr())); - curr_db= db; - change_db(thd,db); - } - - if (OK != (res= it->meta().create(thd))) - { - meta::Item::description_buf buf; - info.report_error(ER_BACKUP_CREATE_META, - it->meta().describe(buf,sizeof(buf))); - DBUG_RETURN(ERROR); - } - - delete it; - } - } + using namespace backup; - // We should reach end of chunk now - if not something went wrong + DBUG_ASSERT(catalogue); + DBUG_ASSERT(item); + DBUG_ASSERT(stmt); - if (res != DONE) - { - info.report_error(ER_BACKUP_READ_META); - DBUG_RETURN(ERROR); - } + Image_info *info= static_cast<Image_info*>(catalogue); - DBUG_ASSERT(res == DONE); + Image_info::Item *it= info->locate_item(item); - if (stream_result::ERROR == s.next_chunk()) + if (!it) { - info.report_error(ER_BACKUP_NEXT_CHUNK); - DBUG_RETURN(ERROR); + // TODO: warn that object was not found (?) + return BSTREAM_ERROR; } - info.meta_size= s.bytes - start_bytes; - - DBUG_RETURN(0); -} - -} // backup namespace + result_t res= it->get_create_stmt(info->create_stmt_buf); + if (res != OK) + return BSTREAM_ERROR; -/********************************************* + stmt->begin= (byte*)info->create_stmt_buf.ptr(); + stmt->end= stmt->begin + info->create_stmt_buf.length(); - Save/restore for different meta-data items. - - *********************************************/ - -namespace backup { - -int silent_exec_query(THD*, String&); - -/** - Write data needed to restore an item. - - By default, a complete DDL CREATE statement for the item is saved. - This statement is constructed using @c meta::X::build_create_stmt() method - where meta::X is the class representing the item. The method stores statement - in the @c create_stmt member. - - @returns OK or ERROR - */ - -result_t -meta::Item::save(THD *thd, OStream &s) -{ - create_stmt.free(); - - if (build_create_stmt(thd)) - return ERROR; - - return stream_result::OK == s.writestr(create_stmt) ? OK : ERROR; + return BSTREAM_OK; } -/** - Read data written by @c save(). - By default, the CREATE statement is read and stored in the @c create_stmt - member. - - @retval OK everything went ok - @retval DONE end of data chunk detected - @retval ERROR error has happened - */ -result_t -meta::Item::read(IStream &s) +extern "C" +int bcat_get_item_create_data(st_bstream_image_header *catalogue, + struct st_bstream_item_info *item, + bstream_blob *data) { - stream_result::value res= s.readstr(create_stmt); - - // Saved string should not be NIL - return res == stream_result::NIL ? ERROR : report_stream_result(res); + /* We don't use any extra data now */ + return BSTREAM_ERROR; } -/** - Create item. - - Default implementation executes the statement stored in the @c create_stmt - member. +/***************************************************************** - @returns OK or ERROR - */ -result_t -meta::Item::create(THD *thd) -{ - if (create_stmt.is_empty()) - { return ERROR; } + Implementation of meta::Item and derived classes. - if (ERROR == drop(thd)) - return ERROR; + *****************************************************************/ - return silent_exec_query(thd,create_stmt) ? ERROR : OK; -} +namespace backup { /** - Destroy item if it exists. + Destroy object if it exists. Default implementation executes SQL statement of the form: @verbatim @@ -235,13 +122,12 @@ meta::Item::create(THD *thd) @returns OK or ERROR */ -result_t -meta::Item::drop(THD *thd) +result_t meta::Item::drop(THD *thd) { const char *ob= sql_object_name(); /* - An item class should define object name for DROP statement + The caller should define object name for DROP statement or redefine drop() method. */ DBUG_ASSERT(ob); @@ -256,125 +142,49 @@ meta::Item::drop(THD *thd) return silent_exec_query(thd,drop_stmt) ? ERROR : OK; } -/**** SAVE/RESTORE DATABASES ***********************************/ - -/** - Save data needed to create a database. - Currently we don't save anything. A database is always created using - "CREATE DATABASE @<name>" statement. - */ -result_t -meta::Db::save(THD*,OStream&) +result_t meta::Db::get_create_stmt(::String &stmt) { - return OK; + // TODO: get a full CREATE statement for a database + return ERROR; } -/** - Read data needed to create a database. - - Nothing to read. We just build the "CREATE DATABASE ..." statement in - @c create_stmt. - */ -result_t -meta::Db::read(IStream&) +result_t meta::Db::create(THD *thd, ::String&, byte*, byte*) { - create_stmt.append("CREATE DATABASE "); - create_stmt.append(sql_name()); - return OK; -} + String stmt; + // TODO: CREATE DATABASE statement should be taken from the backup image. -/**** SAVE/RESTORE TABLES ***************************************/ + stmt.append("CREATE DATABASE "); + stmt.append(sql_name()); -/** - Build a CREATE statement for a table. + return silent_exec_query(thd,stmt) ? ERROR : OK; +} - We use @c store_create_info() function defined in the server. For that - we need to open the table. After building the statement the table is closed to - save resources. Actually, all tables of the thread are closed as we use - @c close_thread_tables() function. - */ -int meta::Table::build_create_stmt(THD *thd) +result_t meta::Table::get_create_stmt(::String &stmt) { - TABLE_LIST t, *tl= &t; + stmt.free(); - bzero(&t,sizeof(TABLE_LIST)); + TABLE_LIST *tl= get_table_list_entry(); - t.db= const_cast<char*>(in_db().name().ptr()); - t.alias= t.table_name= const_cast<char*>(sql_name()); - - uint cnt; - int res= ::open_tables(thd,&tl,&cnt,0); - - if (res) - { - DBUG_PRINT("backup",("Can't open table %s to save its description (error=%d)", - t.alias,res)); - return res; - } + // Table should be opened when this method is called + DBUG_ASSERT(tl); - res= ::store_create_info(thd,&t,&create_stmt,NULL); + int res= ::store_create_info(::current_thd,tl,&stmt,NULL); if (res) DBUG_PRINT("backup",("Can't get CREATE statement for table %s (error=%d)", - t.alias,res)); + tl->alias,res)); - ::close_thread_tables(thd); + stmt.copy(); - return res; -} - -} // backup namespace - - -/************************************************ - - Helper functions - - ************************************************/ - -namespace backup { - -/// Execute SQL query without sending anything to client. - -/* - Note: the change net.vio idea taken from execute_init_command in - sql_parse.cc - */ - -int silent_exec_query(THD *thd, String &query) -{ - Vio *save_vio= thd->net.vio; - - DBUG_PRINT("restore",("executing query %s",query.c_ptr())); - - thd->net.vio= 0; - thd->net.no_send_error= 0; - - thd->query= query.c_ptr(); - thd->query_length= query.length(); - - thd->set_time(time(NULL)); - pthread_mutex_lock(&::LOCK_thread_count); - thd->query_id= ::next_query_id(); - pthread_mutex_unlock(&::LOCK_thread_count); - - const char *ptr; - ::mysql_parse(thd,thd->query,thd->query_length,&ptr); - - thd->net.vio= save_vio; - - if (thd->is_slave_error) - { - DBUG_PRINT("restore", - ("error executing query %s!", thd->query)); - DBUG_PRINT("restore",("last error (%d): %s",thd->net.last_errno - ,thd->net.last_error)); - return thd->net.last_errno ? (int)thd->net.last_errno : -1; - } + /* + TODO: returning error here will just make upper layer think that + the CREATE statement is empty. Change the interface so that real error + can be detected. + */ - return 0; + return res ? ERROR : OK; } } // backup namespace diff -Nrup a/sql/backup/meta_data.h b/sql/backup/meta_data.h --- a/sql/backup/meta_data.h 2007-11-29 20:10:17 +01:00 +++ b/sql/backup/meta_data.h 2007-11-30 09:23:30 +01:00 @@ -1,15 +1,10 @@ -#ifndef _META_BACKUP_H -#define _META_BACKUP_H +#ifndef META_DATA_H_ +#define META_DATA_H_ /** - @file - - Declarations of classes used to handle meta-data items. - */ - -#include <backup/api_types.h> -#include <backup/stream.h> - + @brief Size of a buffer used to describe an object when including it + in error and debug messages. +*/ #define META_ITEM_DESCRIPTION_LEN 128 namespace backup { @@ -17,46 +12,50 @@ namespace backup { namespace meta { /** - Defines how to backup and restore a meta-data item. - - This class provides @c create() and @c drop() methods used to create and - destroy items. The default implementation uses SQL "CREATE ..." and "DROP ..." - statements for that purpose. Its instances determine how to save and read data - needed for item creation and provide any other item specific data. For each - type of meta-data there is a specialized subclass of @c meta::Item which - implements these tasks. - */ + Defines how to backup and restore an object. + This class provides methods for generating object's meta-data and also to + create an object from this data. The class defines default implementations + of these methods, but for specific object types a derived class might + overwrite them. +*/ class Item { public: - /// Possible types of meta-data items. - enum enum_type {DB, TABLE}; + /// Possible types of objects. + enum enum_type { + DB= BSTREAM_IT_DB, + TABLE= BSTREAM_IT_TABLE + }; virtual ~Item() {} - /// Return type of the item. + /// Return type of the object. virtual const enum_type type() const =0; /** - For per-db items return the database to which this item belongs. - For other items returns an invalid db reference. - */ - virtual const Db_ref in_db() - { return Db_ref(); } - - /// Save data needed to create the item. - virtual result_t save(THD*,OStream&); - - /// Read data saved by @c save() method. - virtual result_t read(IStream&); + Obtain a create statement for an object and put it in the given + @c Stream argument. + */ + virtual result_t get_create_stmt(::String&) + { return ERROR; } - /// Destroy the item if it exists. virtual result_t drop(THD*); - /// Create the item. - virtual result_t create(THD*); + /** + Create the object from its meta-data. + + @param query CREATE statement for the object. + @param begin First byte of object's extra meta-data (not used currently). + @param end One byte after the last byte of object's extra meta-data + (not used currently). + */ + virtual result_t create(THD *thd, ::String &query, byte*, byte*) + { + int ret= silent_exec_query(thd,query); + return ret ? ERROR : OK; + } typedef char description_buf[META_ITEM_DESCRIPTION_LEN+1]; @@ -69,13 +68,7 @@ class Item protected: - String create_stmt; /// Storage for a create statement of the item. - - /** - Return SQL name of the object represented by this item like TABLE - or DATABASE. This is used to construct a "DROP ..." statement for - the item. - */ + /// Return SQL name of the object's type such as "TABLE" or "DATABASE". virtual const char* sql_object_name() const { return NULL; } @@ -84,19 +77,10 @@ class Item per-db items the name should *not* be qualified by db name. */ virtual const char* sql_name() const =0; - - /// Store in @c create_stmt a DDL statement which will create the item. - /* - We give a default implementation because an item can not use create - statements and then it doesn't have to worry about this method. - */ - virtual int build_create_stmt(THD*) - { return ERROR; } - }; /** - Specialization of @c meta::Item representing a database. + Specialization of @c meta::Item for database objects. */ class Db: public Item { @@ -106,28 +90,33 @@ class Db: public Item const char* sql_object_name() const { return "DATABASE"; } - // Overwrite default implementations. - result_t save(THD*,OStream&); - result_t read(IStream&); - + result_t get_create_stmt(::String&); + result_t create(THD*, ::String&, byte*, byte*); }; /** - Specialization of @c meta::Item representing a table. + Specialization of @c meta::Item for table objects. */ class Table: public Item { + public: + const enum_type type() const { return TABLE; } const char* sql_object_name() const { return "TABLE"; } - int build_create_stmt(THD*); + result_t get_create_stmt(::String&); + + private: + + virtual ::TABLE_LIST* get_table_list_entry() =0; }; } // meta namespace } // backup namespace -#endif + +#endif /*META_DATA_H_*/ diff -Nrup a/sql/backup/stream.cc b/sql/backup/stream.cc --- a/sql/backup/stream.cc 2007-11-06 19:32:19 +01:00 +++ b/sql/backup/stream.cc 2007-11-30 09:23:30 +01:00 @@ -1,55 +1,128 @@ #include "../mysql_priv.h" +#include <backup_stream.h> #include "stream.h" -/* - TODO +const unsigned char backup_magic_bytes[8]= +{ + 0xE0, // ###..... + 0xF8, // #####... + 0x7F, // .####### + 0x7E, // .######. + 0x7E, // .######. + 0x5F, // .#.##### + 0x0F, // ....#### + 0x03 // ......## +}; - - blocking of OStream output when data window is allocated. - - use my_read instead of read - need to know how to detect EOF. - - remove fixed chunk size limit (backup::Window::buf_size) - - better file buffering (in case of small data chunks) - */ +namespace backup { +/** + Low level write for backup stream library. -// Instantiate templates used in backup stream classes -template class util::IStream< backup::Window >; -template class util::OStream< backup::Window >; + Pointer to this function is stored in @c backup_stream::stream structure + and then used by other stream library function for physical writing of + data. +*/ +extern "C" int stream_write(void *instance, bstream_blob *buf, bstream_blob) +{ + int fd; + int res; -namespace backup { + DBUG_ENTER("backup::IStream::write"); -/************** Window *************************/ + DBUG_ASSERT(instance); + DBUG_ASSERT(buf); -Window::Result Window::set_length(const size_t len) -{ - DBUG_ASSERT(!m_blocked); + OStream *s= (OStream*)instance; + + fd= s->m_fd; + + DBUG_ASSERT(fd >= 0); - m_end= m_head+len; + if (!buf->begin || buf->begin == buf->end) + DBUG_RETURN(BSTREAM_OK); - if (m_end <= last_byte) - return stream_result::OK; + DBUG_ASSERT(buf->end); - m_end= last_byte; - return out_of_bounds(); + size_t howmuch = buf->end - buf->begin; + + res= my_write(fd, buf->begin, howmuch, + MY_NABP /* error if not all bytes written */ ); + + if (res) + DBUG_RETURN(BSTREAM_ERROR); + + s->bytes += howmuch; + + buf->begin= buf->end; + DBUG_RETURN(BSTREAM_OK); } -Window::Result Window::move(const off_t offset) +/** + Low level read for backup stream library. + + Pointer to this function is stored in @c backup_stream::stream structure + and then used by other stream library function for physical reading of + data. +*/ +extern "C" int stream_read(void *instance, bstream_blob *buf, bstream_blob) { - DBUG_ASSERT(!m_blocked); + int fd; + size_t howmuch; - m_head+= offset; + DBUG_ENTER("backup::IStream::read"); - if (m_head > m_end) - m_end= m_head; + DBUG_ASSERT(instance); + DBUG_ASSERT(buf); - if (m_head <= last_byte) - return stream_result::OK; + IStream *s= (IStream*)instance; - m_head= m_end= last_byte; - return out_of_bounds(); + fd= s->m_fd; + + DBUG_ASSERT(fd >= 0); + + if (!buf->begin || buf->begin == buf->end) + DBUG_RETURN(BSTREAM_OK); + + DBUG_ASSERT(buf->end); + + howmuch= buf->end - buf->begin; + + howmuch= my_read(fd, buf->begin, howmuch, MYF(0)); + + /* + How to detect EOF when reading bytes with my_read(). + + We assume that my_read(fd, buf, count, MYF(0)) behaves as POSIX read: + + - if it returns -1 then error has been detected. + - if it returns N>0 then N bytes have been read. + - if it returns 0 then there are no more bytes in the stream (EOS reached). + */ + + if (howmuch == (size_t) -1) + DBUG_RETURN(BSTREAM_ERROR); + + if (howmuch == 0) + DBUG_RETURN(BSTREAM_EOS); + + s->bytes += howmuch; + buf->begin += howmuch; + DBUG_RETURN(BSTREAM_OK); } -/************** Stream *************************/ + +Stream::Stream(const ::String &name, int flags): + m_fd(-1), m_path(name), m_flags(flags) +{ + bzero(&stream, sizeof(stream)); + bzero(&buf, sizeof(buf)); + bzero(&mem, sizeof(mem)); + bzero(&data_buf, sizeof(data_buf)); + block_size= 0; + state= CLOSED; +} bool Stream::open() { @@ -73,258 +146,229 @@ bool Stream::rewind() } -/************** OStream *************************/ - -/* - Implementation of data chunks. - - Data is written to the file in form of data chunks. Each chunk is prefixed with its size stored - in 2 bytes (should it be increased to 4?). - +OStream::OStream(const ::String &name): + Stream(name,O_WRONLY|O_CREAT|O_TRUNC), bytes(0) +{ + stream.write= stream_write; + m_block_size=0; // use default block size provided by the backup stram library +} - OStream instance uses an output buffer of fixed size inherited from Window class. The size of - a chunk is limited by the size of this buffer as a whole chunk is stored inside the buffer - before writing to the file. +/** + Write the magic bytes and format version number at the beginning of a stream. - writing to the file happens when the current chunk is closed with <code>end_chunk()</code> - method. At the time of writing the output, buffer contents is as follows: + Stream should be positioned at its beginning. - ====================== <- m_buf - 2 bytes for chunk size - ====================== <- m_buf+2 (chunk data starts here) + @return Number of bytes written or -1 if error. +*/ +int OStream::write_magic_and_version() +{ + byte buf[10]; - data written to - the chunk + DBUG_ASSERT(m_fd >= 0); - ---------------------- <- m_head + memmove(buf,backup_magic_bytes,8); + // format version = 1 + buf[8]= 0x01; + buf[9]= 0x00; - current output window + int ret= my_write(m_fd, buf, 10, + MY_NABP /* error if not all bytes written */ ); + if (ret) + return -1; // error when writing magic bytes + else + return 10; +} - ====================== <- m_end (this is end of chunk data) +/** + Open and initialize backup stream for writing. - */ + @retval TRUE operation succeeded + @retval FALSE operation failed -byte* OStream::get_window(const size_t len) + @todo Report errors. +*/ +bool OStream::open() { - if (m_blocked || m_end+len > last_byte) - return NULL; - - m_head= m_end; - m_end+= len; - m_blocked= TRUE; + close(FALSE); - return m_head; -} + bool ret= Stream::open(); -void OStream::drop_window() -{ - if (m_blocked) - m_end= m_head; + if (!ret) + return FALSE; - m_blocked= FALSE; -} + // write magic bytes and format version + int len= write_magic_and_version(); -OStream::Result -OStream::write_window(const size_t len) -{ - if (m_blocked) + if (len <= 0) { - DBUG_ASSERT(m_head+len<=m_end); - m_head+=len; - m_end= m_head; + // TODO: report errors + return FALSE; } - m_blocked= FALSE; - - return stream_result::OK; + bytes= 0; + ret= BSTREAM_OK == bstream_open_wr(this,m_block_size,len); + // TODO: report errors + return ret; } +/** + Close backup stream + If @c destroy is TRUE, the stream object is deleted. +*/ void OStream::close(bool destroy) { if (m_fd<0) return; - end_chunk(); - - // write 0 at the end - last_byte=m_buf+2; - Window::reset(); - write2int(0); - - my_write(m_fd,m_buf,2,MYF(0)); - + bstream_close(this); Stream::close(); if (destroy) delete this; } -stream_result::value OStream::end_chunk() -{ - if (m_blocked) - drop_window(); - - DBUG_ASSERT(m_end >= m_buf+2); +/** + Rewind output stream so that it is positioned at its beginning and + ready for writing new image. - size_t len= m_end - m_buf - 2; // length of the chunk - - if (len==0) - { - Window::reset(2); - return stream_result::OK; - } - - // store length of chunk in front of the buffer - Window::reset(); - write2int(len); + @retval TRUE operation succeeded + @retval FALSE operation failed +*/ +bool OStream::rewind() +{ + bstream_close(this); - bytes+= len; + bool ret= Stream::rewind(); - len+= 2; // now len is the number of bytes we want to write + if (!ret) + return FALSE; - uint res= my_write(m_fd,m_buf,len,MY_NABP); + int len= write_magic_and_version(); - Window::reset(2); + if (len <= 0) + return FALSE; - if (res) - return stream_result::ERROR; + ret= BSTREAM_OK == bstream_open_wr(this,m_block_size,len); - return stream_result::OK; + return ret; } -/************** IStream *************************/ -/* - Handling of stream data chunks. +IStream::IStream(const ::String &name): + Stream(name,O_RDONLY), bytes(0) +{ + stream.read= stream_read; +} - Chunks are read into the input buffer inherited from <code>Window</code> class. It is assumed - that a whole chunk will always fit into the buffer (otherwise error is reported). +/** + Check that input stream starts with correct magic bytes and + version number. - When reading a chunk of data, the size of the next chunk is also read-in in the same file access - and stored in the <code>next_chunk_len</code> member. + Stream should be positioned at its beginning. - The input buffer has the following layout: + @return Number of bytes read or -1 if error. +*/ +int IStream::check_magic_and_version() +{ + byte buf[10]; - =================== <- m_buf (start of input buffer) + DBUG_ASSERT(m_fd >= 0); - chunk data + int ret= my_read(m_fd, buf, 10, + MY_NABP /* error if not all bytes read */ ); + if (ret) + return -1; // couldn't read magic bytes - ------------------- <- m_head - current input - window - ------------------- <- m_end + if (memcmp(buf,backup_magic_bytes,8)) + return -1; // wrong magic bytes - chunk data + unsigned int ver = buf[8] + (buf[9]<<8); - =================== <- last_byte (end of chunk data) - size of next chunk - =================== <- last_byte+2 + if (ver != 1) + return -1; // unsupported format version - The first chunk of data is read into the input buffer when stream is opened. Next chunks are - read inside <code>next_chunk()</code> method. + return 10; +} - */ +/** + Open backup stream for reading. -// PRE: there is at least one chunk in the stream. + @retval TRUE operation succeeded + @retval FALSE operation failed -bool IStream::rewind() + @todo Report errors. +*/ +bool IStream::open() { - Stream::rewind(); - Window::reset(); - bytes= 0; - - if (my_read(m_fd, m_buf, 2, MYF(0)) < 2) - return FALSE; - - last_byte= m_head+2; - - read2int(next_chunk_len); + close(FALSE); - Window::reset(); // ignore the 2 bytes containing chunk length - last_byte= m_buf; + bool ret= Stream::open(); - return next_chunk() == stream_result::OK; -} - -stream_result::value IStream::next_chunk() -{ - bytes+= (last_byte-m_buf); // update statistics + if (!ret) + return FALSE; - last_byte= m_buf; + int len= check_magic_and_version(); - if (next_chunk_len == 0) + if (len <= 0) { - Window::reset(); - return stream_result::EOS; + // TODO: report errors + return FALSE; } - size_t len= next_chunk_len+2; - - long int howmuch= 0; // POSIX ssize_t not defined on win platform :| + bytes= 0; - while (len > 0 && (howmuch= ::read(m_fd,m_buf,len)) > 0) - len-= howmuch; + ret= BSTREAM_OK == bstream_open_rd(this,len); + // TODO: report errors + return ret; +} - if (howmuch<0) // error reading file - { - next_chunk_len= 0; - Window::reset(); - return stream_result::ERROR; - } +/** + Close backup stream - if (len == 0) - { - // read length of next chunk (at the end of the buffer) - last_byte+= next_chunk_len+2; - Window::reset(next_chunk_len); - read2int(next_chunk_len); - last_byte-=2; - } - else - { - last_byte+= next_chunk_len+2-len; - next_chunk_len= 0; - } + If @c destroy is TRUE, the stream object is deleted. +*/ +void IStream::close(bool destroy) +{ + if (m_fd<0) + return; - Window::reset(); + bstream_close(this); + Stream::close(); - return howmuch==0 ? stream_result::EOS : stream_result::OK; + if (destroy) + delete this; } -#ifdef DBUG_BACKUP +/** + Rewind input stream so that it can be read again. -// Show data chunks in a backup stream; - -void dump_stream(IStream &s) + @retval TRUE operation succeeded + @retval FALSE operation failed +*/ +bool IStream::rewind() { - stream_result::value res; - byte b; - - DBUG_PRINT("stream",("==========")); + bstream_close(this); - do { + bool ret= Stream::rewind(); - uint chunk_size; - - for( chunk_size=0; (res= s.readbyte(b)) == stream_result::OK ; ++chunk_size ); + if (!ret) + return FALSE; - DBUG_PRINT("stream",(" chunk size= %u",chunk_size)); + int len= check_magic_and_version(); - if( res == stream_result::EOC ) - { - DBUG_PRINT("stream",("----------")); - res= s.next_chunk(); - } + if (len < 0) + return FALSE; - } while ( res == stream_result::OK ); + ret= BSTREAM_OK == bstream_open_rd(this,len); - if (res == stream_result::EOS) - DBUG_PRINT("stream",("==========")); - else - DBUG_PRINT("stream",("== ERROR: %d",(int)res)); + return ret; } -#endif +/// Move to next chunk in the stream. +int IStream::next_chunk() +{ + return bstream_next_chunk(this); +} } // backup namespace diff -Nrup a/sql/backup/stream.h b/sql/backup/stream.h --- a/sql/backup/stream.h 2007-11-06 19:32:20 +01:00 +++ b/sql/backup/stream.h 2007-11-30 09:23:30 +01:00 @@ -1,372 +1,19 @@ #ifndef _BACKUP_STREAM_H_ #define _BACKUP_STREAM_H_ +#include <backup_stream.h> + #include <backup/api_types.h> // for Buffer definition -#include <backup/debug.h> // for definition of DBUG_BACKUP +#include "debug.h" // for definition of DBUG_BACKUP /** @file - Generic Stream Interface for serializing basic types (integers, strings etc). - */ - -namespace backup { - -// TODO: remove the upper limit on chunk size - -const size_t io_buffer_size= 1024*1024; // buffer size for kernel/driver data transfers -const size_t max_chunk_size= 2*1024*1024; // upper limit on a single chunk size - -} - - -namespace util -{ - -typedef unsigned char byte; - -struct stream_result -{ - enum value { OK=0, NIL, ERROR }; -}; - - -/* - Stream window: an abstract object which can be used to access a stream - of bytes through a window in process address space. The methods give pointer - to the beginning of the window and its current size. There is a request for - enlarging the window. We can also move the window beginning to the right - (decreasing window size). If some bytes move outside the window, they can't - be accessed any more. - - ================================================> (stream of bytes) - - |--------------------| - ^ - | - window header - - - 1. Extend window: - - |--------------------............| - ^ - | - window header - - 2. Move window header (shrinks window): - - .....|---------------| - ^ - | - window header + Interface layer between backup kernel and the backup stream library defining + format of the data written/read. - Interface - --------- - - typename Result; - - byte* head() const; // pointer to start of the window - byte* end() const; // pointer to the end of the window - size_t length() const; // current length of the window (in bytes) - Result set_length(const size_t size); // extend the window to have (at least) given length - Result move(const off_t offset); // move window header - //void init(); - //void done(); - - operator int(const Result&); // interpret the result: is it error or other - common situation (end of data). - - Note: window can be used for reading or writing. */ - -/** - Stream classes. - - These classes provide interface for serializing basic data - (numbers, strings) using host independent format. - - Interface is parametrised by a class implementing stream window. - Instance of the window class is used to read/write stream data. - */ - -template<class SWin> -class IStream -{ - SWin *m_win; - - public: - - typedef typename SWin::Result Result; - - IStream(): m_win(NULL) - {} - - // Note: we need to set m_win after instance has been created. - void set_win(SWin *win) - { - m_win= win; - } - - Result readbyte(byte &x); - Result read2int(uint &x); - Result read4int(ulong &x); - Result readint(ulong &x); - Result readint(uint &x) - { - ulong y= 0; - Result res= readint(y); - x= y; - return res; - } - - Result readstr(String &s); -}; - - -template<class SWin> -class OStream -{ - SWin *m_win; - - public: - - typedef typename SWin::Result Result; - - OStream(): m_win(NULL) - {} - - void set_win(SWin *win) - { - m_win= win; - } - - Result writebyte(const byte x); - Result write2int(const int x); - Result write4int(const ulong x); - Result writeint(const ulong x); - Result writestr(const String &s); - - Result writestr(const char *s) - { return writestr(String(s,table_alias_charset)); } - - Result writenil() - { return writebyte(251); } -}; - -template<class SW> -inline -typename IStream<SW>::Result -IStream<SW>::readbyte(byte &x) -{ - Result res; - - DBUG_ASSERT(m_win); - - if ((res= m_win->set_length(1)) != Result(stream_result::OK)) - return res; - - x= *m_win->head(); - - return m_win->move(1); -} - -template<class SW> -inline -typename OStream<SW>::Result -OStream<SW>::writebyte(const byte x) -{ - Result res; - - DBUG_ASSERT(m_win); - - if ((res= m_win->set_length(1)) != Result(stream_result::OK)) - return res; - - (*m_win->head())= x; - - return m_win->move(1); -} - -template<class SW> -inline -typename IStream<SW>::Result -IStream<SW>::read2int(uint &x) -{ - Result res; - - DBUG_ASSERT(m_win); - - if ((res= m_win->set_length(2)) != Result(stream_result::OK)) - return res; - - x= uint2korr(m_win->head()); - - return m_win->move(2); -} - -template<class SW> -inline -typename OStream<SW>::Result -OStream<SW>::write2int(const int x) -{ - Result res; - - DBUG_ASSERT(m_win); - - if ((res= m_win->set_length(2)) != Result(stream_result::OK)) - return res; - - int2store(m_win->head(),x); - - return m_win->move(2); -} - - -template<class SW> -inline -typename IStream<SW>::Result -IStream<SW>::read4int(ulong &x) -{ - Result res; - - DBUG_ASSERT(m_win); - - if ((res= m_win->set_length(4)) != Result(stream_result::OK)) - return res; - - x= uint4korr(m_win->head()); - - return m_win->move(4); -} - -template<class SW> -inline -typename OStream<SW>::Result -OStream<SW>::write4int(const ulong x) -{ - Result res; - - DBUG_ASSERT(m_win); - - if ((res= m_win->set_length(4)) != Result(stream_result::OK)) - return res; - - int4store(m_win->head(),x); - - return m_win->move(4); -} - -// write/read number using variable-length encoding - -template<class SW> -inline -typename IStream<SW>::Result -IStream<SW>::readint(ulong &x) -{ - Result res; - - DBUG_ASSERT(m_win); - - if ((res= m_win->set_length(1)) != Result(stream_result::OK)) - return res; - - x= *m_win->head(); - m_win->move(1); - - switch( x ) { - case 251: - return Result(stream_result::NIL); - - case 252: - { - uint y= 0; - res= read2int(y); - x= y; - return res; - } - - case 253: - return read4int(x); - - default: - return Result(stream_result::OK); - } -} - -template<class SW> -inline -typename OStream<SW>::Result -OStream<SW>::writeint(const ulong x) -{ - Result res; - - DBUG_ASSERT(m_win); - - if ((res= m_win->set_length(1)) != Result(stream_result::OK)) - return res; - - if (x < 251) - return writebyte((byte)x); - - if (x < (1UL<<16)) - { - res= writebyte(252); - return res == Result(stream_result::OK) ? write2int(x) : res; - } - - res= writebyte(253); - return res == Result(stream_result::OK) ? write4int(x) : res; -} - - -// Write/read string using "length coded string" format - -template<class SW> -inline -typename IStream<SW>::Result -IStream<SW>::readstr(String &s) -{ - Result res; - uint len; - - DBUG_ASSERT(m_win); - - if ((res= readint(len)) != Result(stream_result::OK)) - return res; - - if ((res= m_win->set_length(len)) != Result(stream_result::OK)) - return res; - - s.free(); - s.copy((const char*)m_win->head(), len, &::my_charset_bin); - - return m_win->move(len); -} - -template<class SW> -inline -typename OStream<SW>::Result -OStream<SW>::writestr(const String &s) -{ - Result res; - uint len= s.length(); - - if ((res= writeint(len)) != Result(stream_result::OK)) - return res; - - if ((res= m_win->set_length(len)) != Result(stream_result::OK)) - return res; - - memcpy(m_win->head(), s.ptr(), len); - - return m_win->move(len); -} - -} // util namespace - - /************************************************************ Backup Stream Interface @@ -385,102 +32,16 @@ namespace backup { struct stream_result { enum value { - OK= util::stream_result::OK, - NIL= util::stream_result::NIL, - ERROR= util::stream_result::ERROR, - EOC, // end of chunk - EOS // end of stream + OK= BSTREAM_OK, + EOC= BSTREAM_EOC, + EOS= BSTREAM_EOS, + ERROR= BSTREAM_ERROR }; }; -/** - Convert stream_result into a corresponding result_t value. - - End of stream or chunk is reported as DONE, errors as ERROR and - other stream results (OK,NIL) as OK. - */ -inline -result_t report_stream_result(const stream_result::value &res) -{ - switch (res) { - - case stream_result::ERROR: - return ERROR; - - case stream_result::EOC: - case stream_result::EOS: - return DONE; - - default: - return OK; - } -} - -/** - Implementation of stream window interface to be used by util::{I,O}Stream - templates. - - It provides a window inside a static data buffer <code>m_buf</code> of - fixed size <code>buf_size</code>. The window starts at <code>m_head</code> and ends - at <code>m_end</code>. Window can be moved and resized using <code>move()</code> - and <code>set_length()</code> methods. These methods take into account - size of <code>m_buf</code> and report overflows accordingly. - - The window can be used by the util::{I,O}Stream templates to read/write - basic data types in a uniform, host independent way. - */ - -class Window -{ - public: - - typedef stream_result::value Result; - - byte* head() - { return m_head; } - - byte* end() - { return m_end; } - - size_t length() - { return m_end - m_head; } - - Result set_length(const size_t); ///< resize window to the given size - Result move(const off_t); ///< move beginning of the window (shrinks size) - - virtual ~Window() {} - - protected: - - Window(): - last_byte(m_buf+buf_size), m_head(m_buf), m_end(m_buf), m_blocked(FALSE) - {} - - static const size_t buf_size= max_chunk_size; ///< data buffer size - byte m_buf[buf_size]; - - byte *last_byte; ///< points at the byte after the last byte of m_buf - byte *m_head; ///< points at first byte of the current window - byte *m_end; ///< points at the byte after the last byte of the current window - - bool m_blocked; ///< If true, set_length() and move() are blocked (return ERROR). - - /// Create empty window at offset in m_buf - void reset(off_t offset=0) - { m_head= m_end= m_buf+offset; } - - /// Define the result which should be returned in case of buffer overflow. - virtual Result out_of_bounds() const - { return stream_result::ERROR; } - - friend class util::OStream<Window>; - friend class util::IStream<Window>; -}; - -} // backup namespace - -namespace backup { +extern "C" int stream_write(void *instance, bstream_blob *buf, bstream_blob); +extern "C" int stream_read(void *instance, bstream_blob *buf, bstream_blob); /**************************************************** @@ -488,7 +49,15 @@ namespace backup { ****************************************************/ -class Stream +/** + Base for @c OStream and @c IStream. + + It stores file descriptor and provides basic methods for operating on + it. It also inherits from (and correctly fills) the backup_stream structure + so that an instance of @c Stream class can be passed to backup stream library + functions. +*/ +class Stream: public backup_stream { public: @@ -497,7 +66,7 @@ class Stream bool rewind(); /// Check if stream is opened - operator bool() + bool is_open() const { return m_fd>0; } ~Stream() @@ -505,172 +74,56 @@ class Stream protected: - Stream(const String &name, int flags): - m_fd(-1), m_path(name), m_flags(flags) {} + Stream(const ::String&, int); int m_fd; String m_path; int m_flags; ///< flags used when opening the file + size_t m_block_size; + friend int stream_write(void*,bstream_blob*,bstream_blob); + friend int stream_read(void*,bstream_blob*,bstream_blob); }; -/** - Implements backup stream which writes data to a file. - - This class inherits from util::OStream which defines methods for serialization of - basic datatypes (strings and integer). It also implements the concept of data chunks. Data - is stored in chunks - writing to an IStream appends data to the current chunk. A chunk is - closed and a new one is started with <code>end_chunk()</code> method. - - A client of this class can ask an instance for an output buffer with <code>get_window()</code> - method. After filling the buffer its contents can be written to the stream. This is to avoid - double buffering and unnecessary copying of data. However, once an output buffer is allocated, - all output to the stream is blocked until the buffer is written with <code>write_window()</code> - or dropped with <code>drop_window()</code>. - */ - +/// Used to write to backup stream. class OStream: - private Window, - public Stream, - public util::OStream< Window > + public Stream { - typedef util::OStream< Window > Base3; - public: - typedef stream_result::value Result; // disambiguate - size_t bytes; ///< number of bytes written - bool open() - { - close(FALSE); - return Stream::open() && rewind(); - } + OStream(const ::String&); + bool open(); void close(bool destroy=TRUE); + bool rewind(); - bool rewind() - { - Stream::rewind(); - Window::reset(2); - bytes= 0; - return TRUE; - } - - Result end_chunk(); - - /** - Ask stream for output buffer of given size. - - If buffer is allocated, stream is blocked for other operations until - either write_window() or drop_window() is called. - - @note Writing to stream using output buffer doesn't create chunks - boundaries. Explicit call to end_chunk() is needed. - */ - byte* get_window(const size_t); - Result write_window(const size_t); - void drop_window(); - - stream_result::value operator <<(const String &str) - { - return writestr(str); - } - - OStream(const String &name): - Stream(name,O_WRONLY|O_CREAT|O_TRUNC), Base3() - { set_win(this); } + private: + int write_magic_and_version(); }; - -/** - Implements backup stream reading data from a file. - - This class inherits from util::IStream which defines methods for serialization of - basic datatypes (strings and integer). It also handles chunk boundaries as created by - the OStream class. When reading data at the end of a data chunk, <code>stream_result::EOC</code> - is returned. To access data in next chunk, <code>next_chunk()</code> must be called. - */ - +/// Used to read from backup stream. class IStream: - private Window, - public Stream, - public util::IStream< Window > + public Stream { - typedef util::IStream< Window > Base3; - public: - typedef stream_result::value Result; // disambiguate - size_t bytes; ///< number of bytes read - bool open() - { - close(FALSE); - return Stream::open() && rewind(); - } - - void close(bool destroy=TRUE) - { - Stream::close(); - if (destroy) - delete this; - } + IStream(const ::String &name); + bool open(); + void close(bool destroy=TRUE); bool rewind(); - Result next_chunk(); - - stream_result::value operator >>(String &str) - { - return readstr(str); - } - - /** - Return current chunk. - - Will return the same chunk until next_chunk() is called. - */ - stream_result::value operator >>(Buffer &buf) - { - m_end= last_byte; - - if (last_byte == m_buf) // empty chunk means end of stream - return stream_result::EOS; - - buf.data= m_buf; - buf.size= last_byte - m_buf; - - return stream_result::OK; - } - - IStream(const String &name): - Stream(name,O_RDONLY), Base3(), bytes(0) - { - set_win(this); - last_byte= m_buf; - } + int next_chunk(); private: - /// Length of next chunk of data or 0 if there are no more. - uint next_chunk_len; // we use 2 bytes to store chunk len - - Result out_of_bounds() const - { return next_chunk_len > 0 ? stream_result::EOC : stream_result::EOS; } - - friend class stream_instances; + int check_magic_and_version(); }; - -#ifdef DBUG_BACKUP - -// Function for debugging backup stream implementation. -void dump_stream(IStream &); - -#endif } // backup namespace diff -Nrup a/sql/backup/stream_services.h b/sql/backup/stream_services.h --- /dev/null Wed Dec 31 16:00:00 196900 +++ b/sql/backup/stream_services.h 2007-11-30 09:23:30 +01:00 @@ -0,0 +1,10 @@ +#ifndef STREAM_SERVICES_H_ +#define STREAM_SERVICES_H_ + +extern "C" { + +#include "stream_v1_services.h" + +} + +#endif /*STREAM_SERVICES_H_*/ diff -Nrup a/sql/backup/string_pool.cc b/sql/backup/string_pool.cc --- a/sql/backup/string_pool.cc 2007-11-06 19:32:20 +01:00 +++ /dev/null Wed Dec 31 16:00:00 196900 @@ -1,240 +0,0 @@ -#include "../mysql_priv.h" - -#include "string_pool.h" - -extern const String my_null_string; - -namespace util { - -String *StringDom::create(const Element &s) -{ - String *ns= new (::current_thd->mem_root) String(); - ns->copy(s); - return ns; -} - -String &StringDom::null= const_cast< String& >(::my_null_string); - -// Map template instance used as StringPool implementation. -template class Map<StringDom>; - -} // util namespace - - -namespace backup { - -// Serialization of StringPool - -/* - Pool - - 0: "foo" - 1: - - 2: - - 3: "bar" - 4: "baz" - ... - - is saved as - - | foo | NIL : 2 | bar | baz | ... - - where the (NIL:2) entry describes a "hole" in the pool of width 2. - - Empty pool is saved as | NIL : 0 |. -*/ - -result_t StringPool::save(OStream &str) -{ - DBUG_ENTER("StringPool::save()"); - - if (count() == 0) // empty pool - { - DBUG_PRINT("string_pool",("saving empty pool")); - if (stream_result::ERROR == str.writenil() - || stream_result::ERROR == str.writeint(0)) - DBUG_RETURN(ERROR); - } - else - { - String buf; - uint skip=0, i=0; - - DBUG_PRINT("string_pool",("saving pool of size %lu",(unsigned long)count())); - - for (i=0 ; TRUE ; ++i) - { - if (i >= count() || entries[i].el != NULL) // non-empty or last entry - { - /* - If skip > 0 this entry closes a hole of skip entries. - Write hole width to finish its description. - */ - if (skip > 0) - { - DBUG_PRINT("string_pool",("noting hole of width %d",skip)); - if (stream_result::ERROR == str.writeint(skip)) - { - DBUG_PRINT("string_pool",("error when writing hole width")); - DBUG_RETURN(ERROR); - } - } - - if (i >= count()) - break; - - skip= 0; // the hole has ended now - - DBUG_PRINT("string_pool",("writing entry %s at position %d", - (entries[i].el)->c_ptr(),i)); - // save the entry - if (stream_result::ERROR == str.writestr(*entries[i].el)) - { - DBUG_PRINT("string_pool",("error when writing string")); - DBUG_RETURN(ERROR); - } - - continue; - } - - // we have an empty entry, if skip == 0 it starts a new hole which - // is marked by NIL value - - if( skip == 0 ) - if (stream_result::ERROR == str.writenil()) - DBUG_RETURN(ERROR); - - skip++; - } - - } // if (count()==0) else ... - - DBUG_PRINT("string_pool",("saved")); - - DBUG_RETURN(str.end_chunk() == stream_result::ERROR ? ERROR : OK); -} - -/** - @retval OK - @retval DONE end of stream/chunk hit - @retval ERROR - */ -result_t StringPool::read(IStream &str) -{ - DBUG_ENTER("StringPool::read"); - - stream_result::value res; - String buf; - uint skip= 0, i=0; - - /* - Read first entry. If it is NIL then we either have an empty pool - or a pool which starts with a hole. This is decided by the number following - NIL value. - */ - - res= str.readstr(buf); - - if (res != stream_result::OK && res != stream_result::NIL) - { - // In that case there is an error or we hit end of data - if (res != stream_result::ERROR) - DBUG_PRINT("string_pool",("End of data hit")); - DBUG_RETURN(report_stream_result(res)); - } - - clear(); - - if (res == stream_result::NIL) - { - res= str.readint(skip); - - if (res != stream_result::OK) - DBUG_RETURN(ERROR); - - if (skip == 0) // this is empty pool - nothing more to read - { - DBUG_PRINT("string_pool",("Found empty pool")); - goto finish; - } - - DBUG_PRINT("sting_pool",("Starts with hole of width %d",skip)); - - /* - this is non-empty pool starting with a hole - read the following - string - */ - - res= str.readstr(buf); - - if (res == stream_result::ERROR) - { - DBUG_PRINT("string",("Error reading string")); - DBUG_RETURN(ERROR); - } - } - - /* - If we are here, we have a non-empty pool and are going to populate it - with strings read from the stream. The first entry is already read. - */ - - while (res == stream_result::OK) - { - /* - If skip > 0 then we have a hole of that size preceding the next - string entry. - */ - if (skip > 0) - i+= skip; - - skip= 0; - - if (i >= size) // Not enough space to store the string - { - DBUG_PRINT("string_pool",("Not enough space to store string at pos %d" - " (size=%lu)",i,(unsigned long)size)); - DBUG_RETURN(ERROR); - } - - DBUG_PRINT("string_pool",("Adding string %s at position %d",buf.c_ptr(),i)); - // store the string and increase position - set(i++,buf); - - // read next entry from stream - res= str.readstr(buf); - - if (res == stream_result::NIL) // this is hole description - { - // read size of the hole - res= str.readint(skip); - DBUG_PRINT("string_pool",("Found hole of width %d",skip)); - - if (res != stream_result::OK) - { - DBUG_PRINT("string_pool",("Found hole - error reading its width")); - DBUG_RETURN(ERROR); - } - - // read the following string - res= str.readstr(buf); - } - } - - // Now we should be at the end of chunk - if (res != stream_result::EOC && res != stream_result::EOS) - { - DBUG_PRINT("string_pool",("Error when reading strings (res=%d)",(int)res)); - DBUG_RETURN(ERROR); - } - - DBUG_PRINT("string_pool",("Finished reading strings")); - - finish: - - res= str.next_chunk(); - - DBUG_RETURN(res == stream_result::ERROR ? ERROR: OK); -} - -} // backup namespace diff -Nrup a/sql/backup/string_pool.h b/sql/backup/string_pool.h --- a/sql/backup/string_pool.h 2007-11-06 19:32:21 +01:00 +++ /dev/null Wed Dec 31 16:00:00 196900 @@ -1,59 +0,0 @@ -#ifndef _STRING_POOL_H -#define _STRING_POOL_H - -#include "map.h" -#include "stream.h" - -/** - @file - - Implementation of a string pool which is a collection of - String objects indexed by 8 bit keys. - */ - -/* - TODO - - - change memory allocation scheme - - use mysql library functions - - simplify by not using Map template - */ - -namespace util { - -/// A domain of String values which can by used by Map template. - -struct StringDom -{ - typedef String Element; - static Element &null; - - static Element *create(const Element &s); - - static int cmp(const Element &x, const Element &y) - { return stringcmp(&x,&y); } - - static unsigned int hash(const Element&) - { return 0; } -}; - -typedef Map<StringDom> StringPool; - -} // util namespace - - -namespace backup { - -/** - Adds serialization to util::StringPool class - */ -class StringPool: public util::StringPool -{ - public: - result_t save(OStream&); - result_t read(IStream&); -}; - -} // backup namespace - -#endif
| Thread | ||
|---|---|---|
| • bk commit into 6.0 tree (rafal:1.2669) WL#4060 | rsomla | 30 Nov |
© 1995-2008 MySQL AB, 2008-2009 Sun Microsystems, Inc.
Page generated in 2.155 sec. using MySQL 5.1.14-beta-log