#At file:///ext/mysql/bzr/backup/stream-doc/ based on revid:hema@stripped
2868 Rafal Somla 2009-09-08
Bug #40587 - Doxygen documentation of backup image format is not up to date
This patch updates the doxygen documentation for the backup image format
and fixes/clarifies issues found there. The format of the documentation is
changed to the one which was proposed by Paul for the internals manual.
@ sql/backup/kernel.cc
- Add the main documentation page.
- Include description of how to use kernel API into doxygen docs.
@ sql/backup/stream.cc
- Add page describing general backup stream format used. Backup image format
is described on a separate page.
@ sql/backup/stream_v1.c
Add pages describing general structure of image format v1 and the image layer
format as implemented by functions in this file. Format of the image layer
documentation has been changed to the one proposed by Paul for the internals
manual.
Also renamed bstream_rd_image_info() -> bstream_rd_snapshot_info() to follow
new terminology and make it consistent with bstream_wr_snapshot_info().
@ sql/backup/stream_v1_transport.c
Added pages documenting the transport layer format of a backup image.
Documentation format is changed to the one proposed by Paul for internals
manual.
modified:
sql/backup/kernel.cc
sql/backup/stream.cc
sql/backup/stream_v1.c
sql/backup/stream_v1_transport.c
=== modified file 'sql/backup/kernel.cc'
--- a/sql/backup/kernel.cc 2009-08-21 15:43:19 +0000
+++ b/sql/backup/kernel.cc 2009-09-08 09:29:08 +0000
@@ -3,6 +3,54 @@
@brief Implementation of the backup kernel API.
+ @todo Use internal table name representation when passing tables to
+ backup/restore drivers.
+ @todo Handle other types of in Backup_info methods.
+ @todo Handle item dependencies when adding new items.
+ @todo Handle other kinds of backup locations (far future).
+*/
+
+/**
+ @mainpage
+
+ @section Structure Structure of the MySQL Backup System
+
+ @verbatim
+
+ | Online Backup Module
+ |
+ (1) | +---------------+ +------------------------+
+ Server ---> | Backup Kernel | <--> | Backup/Restore drivers |
+ | +---------------+ (3) +------------------------+
+ Core <--- | Stream layer |
+ (2) | +---------------+
+ | (4) ^ (5)
+ | | |
+ +--------|---|----------------------------
+ | v
+ | Backup Stream Library
+ |
+ @endverbatim
+
+ Backup stream library is autonomous so that external applications can link
+ against it to be able to read backup images created by the system.
+ The format of backup images is described @ref stream_format "here".
+
+ The components of the system communicate with each other using well defined
+ interfaces:
+ -# @ref KernelAPI "Backup Kernel API"
+ -# Object Services API and Backup Log API
+ -# Backup Driver API
+ -# @ref streamlib "Backup Stream Library API"
+ -# Backup Stream Library Callback API
+*/
+
+/**
+ @defgroup KernelAPI Backup kernel API.
+
+ Backup kernel API is the interface between the backup system and its external
+ users.
+
@section s1 How to use backup kernel API to perform backup and restore operations
To perform backup or restore operation an appropriate context must be created.
@@ -52,11 +100,6 @@
} // if code jumps here, context destructor will do the clean-up automatically
@endcode
- @todo Use internal table name representation when passing tables to
- backup/restore drivers.
- @todo Handle other types of meta-data in Backup_info methods.
- @todo Handle item dependencies when adding new items.
- @todo Handle other kinds of backup locations (far future).
*/
#include "../mysql_priv.h"
@@ -613,7 +656,7 @@ int Backup_restore_ctx::prepare(::String
if (!mem_alloc)
return fatal_error(report_error(ER_OUT_OF_RESOURCES));
- // Freeze all meta-data.
+ // Freeze all .
DEBUG_SYNC(m_thd, "before_backup_ddl_block");
ret= obs::bml_get(m_thd);
@@ -638,7 +681,7 @@ int Backup_restore_ctx::prepare(::String
@note This function reports errors.
- @note It is important that changes of meta-data are blocked as part of the
+ @note It is important that changes of are blocked as part of the
preparations. The set of server objects and their definitions should not
change after the backup context has been prepared and before the actual backup
is performed using @c do_backup() method.
@@ -672,7 +715,7 @@ Backup_restore_ctx::prepare_for_backup(S
/*
Do preparations common to backup and restore operations. After call
- to prepare() all meta-data changes are blocked.
+ to prepare() all changes are blocked.
*/
DEBUG_SYNC(m_thd, "before_backup_common_prepare");
if (prepare(backupdir, orig_loc)) // Logs errors and detects interruptions
@@ -810,7 +853,7 @@ Backup_restore_ctx::prepare_for_restore(
/*
Do preparations common to backup and restore operations. After this call
- changes of meta-data are blocked.
+ changes of are blocked.
*/
DEBUG_SYNC(m_thd, "before_restore_common_prepare");
if (prepare(backupdir, orig_loc))
@@ -1060,7 +1103,7 @@ void Backup_restore_ctx::unlock_tables()
This should reverse all settings made when context was created and prepared.
If it was requested, the backup/restore location is removed. Also, the backup
stream memory allocator is shut down. Any other allocated resources are
- deleted in the destructor. Changes to meta-data are unblocked.
+ deleted in the destructor. Changes to are unblocked.
@returns 0 or error code if error was detected.
@@ -1131,7 +1174,7 @@ int Backup_restore_ctx::close()
// unlock tables if they are still locked
unlock_tables(); // Never errors
- // unfreeze meta-data
+ // unfreeze
obs::bml_release(); // Never errors
// restore thread options
@@ -1414,7 +1457,7 @@ int Backup_restore_ctx::do_restore(bool
if (report_killed())
DBUG_RETURN(fatal_error(ER_QUERY_INTERRUPTED));
- DBUG_PRINT("restore", ("Restoring meta-data"));
+ DBUG_PRINT("restore", ("Restoring "));
// unless RESTORE... OVERWRITE: return error if database already exists
if (!overwrite)
@@ -2083,7 +2126,7 @@ void bcat_db_iterator_free(st_bstream_i
/*****************************************************************
- Services for backup stream library related to meta-data
+ Services for backup stream library related to
manipulation.
*****************************************************************/
@@ -2318,9 +2361,9 @@ int bcat_get_item_create_query(st_bstrea
}
/**
- Get extra meta-data (if any) for a given object.
+ Get extra (if any) for a given object.
- @note Extra meta-data is not used currently.
+ @note Extra is not used currently.
*/
extern "C"
int bcat_get_item_create_data(st_bstream_image_header *catalogue,
=== modified file 'sql/backup/stream.cc'
--- a/sql/backup/stream.cc 2009-08-21 15:43:19 +0000
+++ b/sql/backup/stream.cc 2009-09-08 09:29:08 +0000
@@ -1,3 +1,65 @@
+/**
+ @page stream_format Backup Stream Format
+
+ A backup stream is a sequence of bytes that consists of a 10-byte prefix
+ followed by a backup image.
+ @verbatim
+
+ backup_stream: stream_prefix backup_image
+ stream_prefix: magic_number version
+ @endverbatim
+
+ @c Magic_number consists of eight bytes. They are as follows:
+ @verbatim
+
+ 0xE0, // ###.....
+ 0xF8, // #####...
+ 0x7F, // .#######
+ 0x7E, // .######.
+ 0x7E, // .######.
+ 0x5F, // .#.#####
+ 0x0F, // ....####
+ 0x03 // ......##
+ @endverbatim
+
+ @c Version is stored in the following two bytes, the least
+ significant byte first.
+
+ @c Backup_image that follows @c backup_prefix is stored in the format
+ indicated by the version number. Currently only one backup image format
+ is used -- @ref image_format1.
+
+ The stream format does not provide direct support for compression or
+ encryption. But a complete backup stream can be compressed and/or encrypted
+ externally. In that case, it will be wrapped in the compression/encryption
+ format used, with additional headers added in front of it. Before processing
+ such a stream, it should be first uncompressed/decrypted and then the 10 byte
+ backup prefix should be seen.
+
+ @note
+ The prefix containing the magic number and version number is @em not part of
+ the backup image. It is stored in the backup stream to make two things
+ possible:
+ - detection of where the backup image starts inside a backup stream,
+ - determination of the format of the backup image that follows the prefix.
+
+ @par
+ For example, suppose that the backup image is not stored at the beginning
+ of a stream, but is preceded by some other information. An application can
+ scan the stream looking for the backup magic number. Once found, the
+ application will know with high probability that the version number and a
+ backup image follows.
+
+ @par
+ In other scenarios, the 10-byte prefix might not be needed. Suppose that
+ the backup image is sent from some kind of a server to a client using a custom
+ protocol. The client knows that what server sends is a backup image. In the
+ initial handshake, the server tells the client the version of the backup
+ image format to follow. Thus, after opening the communication channel, the
+ client can directly read the bytes of the backup image and there is no need
+ to send the 10-byte prefix.
+*/
+
#include "../mysql_priv.h"
#include "my_dir.h"
=== modified file 'sql/backup/stream_v1.c'
--- a/sql/backup/stream_v1.c 2009-06-30 20:37:35 +0000
+++ b/sql/backup/stream_v1.c 2009-09-08 09:29:08 +0000
@@ -9,12 +9,61 @@
@brief
Implementation of the high-level functions for writing and reading backup
- image using version 1 of backup stream format.
+ image using version 1 of backup image format (see @ref image_format1).
@todo handle errors when creating iterators in functions like bstream_wr_catalogue()
@todo use data chunk sequence numbers to detect discontinuities in backup stream.
*/
+/**
+ @page image_format1 Backup Image Format (v1)
+
+ The backup image structure can be thought of at two levels:
+ - The transport layer consists of blocks, all of which have the same size
+ (except possibly the last). The size is given at the beginning of the first
+ block.
+ - The image layer consists of chunks, which have varying size.
+
+ At the transport layer, blocks are read and decomposed to obtain fragments
+ (one or more per block). Fragments are assembled to obtain chunks.
+ A chunk might be smaller than a block, or it might contain information from
+ multiple blocks.
+
+ At the higher level, backup image is seen as a sequence of data chunks. How to
+ interpret contents of these chunks is explained on @ref image_layer page.
+
+ Page @ref transport_layer describes the low level structure of a backup image
+ -- how fixed size blocks are decomposed into fragments from which data chunks
+ are assembled.
+*/
+
+/**
+ @page image_layer Image Layer Format
+
+ On the top level, backup image consists of a premable, followed by several
+ chunks with table data and closed with a summary chunk.
+
+@verbatim
+ backup_image: preamble table_data ... summary
+@endverbatim
+
+ @c Preamble describes the image and objects that it contains. @c Table_data
+ chunks contain table contents.
+ @c Summary contains information which is known only when backup image has
+ been created. Currently this is mainly information about the validity point of
+ the image and data needed for point-in-time recovery operations.
+*/
+
+/**
+ @defgroup streamlib Backup Stream Library
+
+ This library defines version 1 of backup image format described
+ @ref image_format1 "here".
+ It provides functions for writing and reading streams using this format.
+ The functions are declared in the stream_v1.h header.
+*/
+
+
#ifdef DBUG_OFF
# define ASSERT(X)
#else
@@ -23,13 +72,6 @@
# define ASSERT(X) assert(X)
#endif
-/**
- @page streamlib Backup Stream Library
-
- This library defines version 1 of backup stream format. It provides functions
- for writing and reading streams using this format. The functions are declared
- in the stream_v1.h header.
-*/
/* local types */
@@ -150,23 +192,29 @@ byte get_byte_size_t(size_t value)
*************************************************************************/
/**
- @page stream_format Backup Stream Format (v1)
-
- Backup image consists of 3 main parts: preamble, table data and summary.
- @verbatim
-
- [backup image]= [ preamble | table data | 0x00 ! summary(*) ]
- @endverbatim
-
- The 0x00 byte separates table data chunks from the summary chunk. This works
- because no table data chunk can start with 0x00.
-
- Optionally, summary can be included in the preamble, instead of being stored
- at the end of the image. This is indicated by flags in the header.
- @verbatim
+ @page image_layer
+ @section preamble Preamble Format
+
+ The preamble contains global information about the image and describes
+ objects it contains.
- [preamble]= [ header | summary (*) | catalogue | meta data ]
- @endverbatim
+@verbatim
+ preamble: header snapshot_description ... catalog metadata
+@endverbatim
+
+ @c Preamble consists of several chunks, which contain the following
+ information:
+ - Header information, such as global image flags, image creation time,
+ and server version number
+ - Information about table data snapshots stored in the image
+ - A catalog listing all objects stored in the image
+ - Metadata for the objects
+
+ Backup image can include several table data snapshots storing data from the
+ tables. The snapshots correspond to backup drivers which were used to create
+ them. Snapshots are numbered, starting from 1, according to the order in
+ which their descriptions appear in the preamble. The number of snapshots
+ present in the image is given by the @c snapshot_count field in the header.
*/
int bstream_wr_header(backup_stream*, struct st_bstream_image_header*);
@@ -193,12 +241,13 @@ int bstream_wr_preamble(backup_stream *s
}
/**
- Read backup image preamble creating all items stored in it
+ Read backup image preamble creating all objects stored in it
@retval BSTREAM_ERROR Error while reading preamble
@retval BSTREAM_OK Read successful
@retval BSTREAM_EOS Preamble has been read and there are no more chunks in
the stream.
+ @see @ref preamble.
*/
int bstream_rd_preamble(backup_stream *s, struct st_bstream_image_header *hdr)
{
@@ -233,27 +282,12 @@ int bstream_rd_preamble(backup_stream *s
return ret;
}
-/**
- @page stream_format
-
- @section summary Summary section
-
- @verbatim
- [summary]= [ vp time ! end time ! binlog pos ! binlog group pos ]
- @endverbatim
+/**
+ Save binlog position.
- Summary starts with 0x00 byte to distinguish it from table data chunks which
- never start with that value.
- @verbatim
-
- [binlog pos]= [ pos:4 ! binlog file name ]
-
- [binlog group pos] uses the same format as [binlog pos].
- @endverbatim
+ @see @ref summary.
*/
-
-/** Save binlog position. */
int bstream_wr_binlog_pos(backup_stream *s, struct st_bstream_binlog_info pos)
{
blob name;
@@ -276,6 +310,8 @@ int bstream_wr_binlog_pos(backup_stream
@retval BSTREAM_OK Read successful
@retval BSTREAM_EOC Read successful and end of chunk has been reached
@retval BSTREAM_EOS Read successful and end of stream has been reached
+
+ @see @ref summary.
*/
int bstream_rd_binlog_pos(backup_stream *s, struct st_bstream_binlog_info *pos)
{
@@ -299,6 +335,8 @@ int bstream_rd_binlog_pos(backup_stream
This function assumes that all members of the header (such as @c vp_time) are
already filled. It also stores the 0x00 byte separating summary from the
preceding table data chunks.
+
+ @see @ref summary
*/
int bstream_wr_summary(backup_stream *s, struct st_bstream_image_header *hdr)
{
@@ -325,6 +363,8 @@ int bstream_wr_summary(backup_stream *s,
@retval BSTREAM_OK Read successful
@retval BSTREAM_EOC Read successful and end of chunk has been reached
@retval BSTREAM_EOS Read successful and end of stream has been reached
+
+ @see @ref summary
*/
int bstream_rd_summary(backup_stream *s, struct st_bstream_image_header *hdr)
{
@@ -348,29 +388,64 @@ int bstream_rd_summary(backup_stream *s,
*************************************************************************/
int bstream_wr_snapshot_info(backup_stream*, struct st_bstream_snapshot_info*);
-int bstream_rd_image_info(backup_stream*, struct st_bstream_snapshot_info*);
+int bstream_rd_snapshot_info(backup_stream*, struct st_bstream_snapshot_info*);
/**
- @page stream_format
-
- @section header Header
+ @page image_layer
+ @subsection header Header Format
- @verbatim
+@verbatim
+ header: flags creation_time snapshot_count server_version extra_data
+@endverbatim
+
+ @c Header chunk contains the following fields:
+ - @c flags (2 bytes). Integer image flags.
+ - @c creation_time (6 bytes). Time of image creation.
+ - @c snapshot_count (1 byte). Number of snapshots included in the image.
+ - @c server_version (variable length). Version of the server which created the
+ image.
+ - @c extra_data (variable length). The remaining bytes to the end of the chunk
+ are reserved for storing additional information - currently they are
+ ignored.
+
+ Bits in @c flags field have the following meanings:
+ - Bit 0: Reserved.
+ - Bit 1 (BSTREAM_FLAG_BIG_ENDIAN)
+ - If set, the server that created the backup has big-endian architecture.
+ - If clear, the server has little-endian architecture.
+ - Bit 2 (BSTREAM_FLAG_BINLOG)
+ - If set, the binlog_coords and binlog_group_coords fields in the summary
+ contain valid values.
+ - If clear, binlog_coords and binlog_group_coords should be ignored (they are
+ present in the summary but the values are not useful).
+ - Bits 3-15: Reserved.
+
+ @c Server_version is encoded as follows:
+ - 1 byte. Integer major number
+ - 1 byte. Integer minor number
+ - 1 byte. Integer release number
+ - Variable length. String representation of the version
+ .
+ For example, a server version of "6.0.8-alpha" is stored using this
+ byte sequence:
+ 0x06 0x00 0x08 0x0b 0x36 0x2e 0x30 0x2e 0x38 0x2d 0x61 0x6c 0x70 0x68 0x61
- [header]= [ flags:2 ! creation time ! #of snapshots:1 ! server version !
- extra data | snapshot descriptions ]
- @endverbatim
+ @note
+ The big-endian flag has no bearing on storage of values in the backup image
+ itself. It might be of use in the case where an image was written by some
+ native driver but fails to restore on a machine other than the one on which
+ it was created. If one checks the big-endian flag and finds that the backup
+ host byte order differs from the restore host byte order, that might indicate
+ that the native backup and restore drivers did not correctly deal with
+ different host byte orders. A possible workaround would be to attempt the
+ restore on a host with the same byte order as the backup host.
+*/
- [snapshot descriptions] contains descriptions of the table data snapshots
- present in the image. Each description is stored in a separate chunk
- (number of snapshots is given in the header).
- @verbatim
+/**
+ Write header of backup image
- [snapshot descriptions]= [ snapshot description | ... | snapshot description ]
- @endverbatim
+ @see @ref header.
*/
-
-/** Write header of backup image */
int bstream_wr_header(backup_stream *s, struct st_bstream_image_header *hdr)
{
int ret= BSTREAM_OK;
@@ -404,6 +479,8 @@ int bstream_wr_header(backup_stream *s,
@retval BSTREAM_OK Read successful
@retval BSTREAM_EOC Read successful and end of chunk has been reached
@retval BSTREAM_EOS Read successful and end of stream has been reached
+
+ @see @ref header.
*/
int bstream_rd_header(backup_stream *s, struct st_bstream_image_header *hdr)
{
@@ -425,7 +502,7 @@ int bstream_rd_header(backup_stream *s,
return BSTREAM_ERROR;
CHECK_RD_OK(bstream_next_chunk(s));
- CHECK_RD_RES(bstream_rd_image_info(s, &hdr->snapshot[i]));
+ CHECK_RD_RES(bstream_rd_snapshot_info(s, &hdr->snapshot[i]));
}
rd_error:
@@ -434,37 +511,50 @@ int bstream_rd_header(backup_stream *s,
}
/**
- @page stream_format
-
- @subsection snapshot Snapshot description entry
-
- @verbatim
-
- [snapshot description] = [ image type:1 ! format version: 2 ! global options:2 !
- #of tables ! backup engine info ! extra data ]
- @endverbatim
+ @page image_layer
+ @subsection snapshot Snapshot Description Format
- [image type] is encoded as follows:
-
- - 0 = snapshot created by native backup driver (BI_NATIVE),
- - 1 = snapshot created by built-in blocking driver (BI_DEFAULT),
- - 2 = snapshot created using created by built-in driver using consistent
- read transaction (BI_CS).
- - 3 = snapshot created by built-in no data driver (BI_NODATA),
+ Table data snapshots are created by backup drivers and store the data from
+ tables handled by these drivers. Each snapshot present in the image is
+ described by a @c snapshot_description chunk in the preamble.
+
+@verbatim
+ snapshot_description: snapshot_type format_version global_options
+ table_count [storage_engine_info] extra_data
+
+ backup_engine_info: engine_name major_version minor_version
+@endverbatim
- Format of [backup engine info] depends on snapshot type. It is empty for the
- default and CS snapshots. For native snapshots it has format
- @verbatim
+ Each @c snapshot_description chunk contains the following information:
+ - @c snapshot_type (1 byte). Integer snapshot type, encoded as follows:
+ - 0 (BI_NATIVE): Snapshot was created by native backup driver.
+ - 1 (BI_DEFAULT): Snapshot was created by the built-in blocking driver.
+ - 2 (BI_CS): Snapshot was created by the built-in driver that uses
+ a consistent read transaction.
+ - 3 (BI_NODATA): Phony snapshot used for tables whose data is not stored
+ in the image.
+ - @c format_version (2 bytes). Integer snapshot format version.
+ - @c global_options (2 bytes). Integer global options. Reserved for future use.
+ - @c table_count (variable length). Integer count of the tables stored
+ in the snapshot.
+ - @c storage_engine_info (variable length). Information about storage engine
+ which provided the native backup driver (present only for native snapshots).
+ - @c extra_data (variable length). Extra data. Reserved for future use;
+ currently empty.
- [backup engine info (native)] = [ storage engine name ! storage engine version ]
+ The @c backup_engine_info field is present only for native snapshots, that is,
+ snapshots of type 0 (BI_NATIVE). It has this format:
+
+ - @c engine_name (variable-length). String engine name.
+ - @c major_version (1 byte). Integer major version of the engine.
+ - @c minor_version (1 byte). Integer minor version of the engine.
+*/
- [server version] = [ major:1 ! minor:1 ! release:1 ! extra string ]
+/**
+ Save description of table data snapshot
- [engine version] = [ major:1 ! minor:1 ]
- @endverbatim
+ @see @ref snapshot
*/
-
-/** Save description of table data snapshot */
int bstream_wr_snapshot_info(backup_stream *s, struct st_bstream_snapshot_info *info)
{
int ret= BSTREAM_OK;
@@ -509,8 +599,10 @@ int bstream_wr_snapshot_info(backup_stre
@note
This function allocates memory to store snapshot info. The caller is
responsible for freeing this memory.
+
+ @see @ref snapshot
*/
-int bstream_rd_image_info(backup_stream *s, struct st_bstream_snapshot_info *info)
+int bstream_rd_snapshot_info(backup_stream *s, struct st_bstream_snapshot_info *info)
{
unsigned short int type;
int ret= BSTREAM_OK;
@@ -555,75 +647,107 @@ int bstream_rd_image_info(backup_stream
*************************************************************************/
/**
- @page stream_format
-
- @section catalogue Image catalogue
-
- The catalogue describes what items are stored in the image. Note that it
- doesn't contain any meta-data, only item names and other information needed to
- identify and select them.
- @verbatim
-
- [catalogue]= [ charsets ! 0x00 ! users ! 0x00 ! tablespaces ! 0x00 !
- databases | db catalogue | ... | db catalogue ]
- @endverbatim
-
- Catalogue starts with list of charsets where each charset is identified by its
- name. In other places of the image, charsets can be identified by their
- positions in this list. Number of charsets is limited to 256 so that one byte
- is enough to identify a charset.
- @verbatim
-
- [charsets]= [ charset name ! ... ! charset name ]
- @endverbatim
-
- Two first entries in [charsets] have special meaning and should be always
- present.
-
- First charset is the charset used to encode all strings stored in
- the preamble. This should be a universal charset like UTF8, capable
- of representing any string.
+ @page image_layer
- Second charset in the list is the default charset of the server on which
- image was created. It can be the same as the first charset.
+ @section catalog Catalog Format
- The following charsets are any charsets used by the items stored in the image
- and thus needed to restore these items.
-
- @verbatim
-
- [users]= [ user name ! ... ! user name ]
- @endverbatim
-
- User list contains users for which any privileges are stored in the image.
-
- Following user list is a list of tablespaces used by the tables stored in
- the backup image. Only tablespace names are listed here, their definitions
- are stored in the meta-data section.
- @verbatim
-
- [tablespaces]= [ ts name ! ... ! ts name ]
- @endverbatim
-
- Finally, a list of all databases follows. If the list is empty, it
- consists of a single null string. Otherwise it has format:
- @verbatim
-
- [databases]= [ db info ! ... ! db info ]
-
- [db info]= [ db name ! db flags:1 ! optional extra data ]
- [db flags]= [ has extra data:.1 ! unused:.7 ]
- [optional extra data]= [data len:2 ! the data:(data len) ]
- @endverbatim
-
- [optional extra data] is present only if indicated in the flags.
-
- If there are no databases in the image, the database list is empty and there
- are no database catalogues.
- @verbatim
-
- [catalogue (no databases)] = [ charsets ! 0x00 ! users ! 0x00 ! tablespaces ! 0x00 ]
- @endverbatim
+ The catalog describes what objects are stored in the backup image. It contains
+ no metadata because that is stored in a separate section of the image.
+ The catalog only lists the objects and provides the information needed to
+ identify and select them. It consists of a header chunk followed by zero or
+ more chunks that each describe objects in a single database:
+
+@verbatim
+ catalog: catalog_header [db_catalog ...]
+@endverbatim
+
+ @subsection catalog_header Catalog Header Format
+
+@verbatim
+ catalog_header: charsets 0x00 [users] 0x00 [tablespaces] 0x00 databases
+@endverbatim
+
+ The charsets, users, and tablespaces sections of the @c catalog_header chunk
+ each contain a list of strings and are terminated by a 0x00 byte
+ (an empty string). The databases section contains database-information items
+ and extends to the end of the @c catalog_header chunk.
+
+ The @c catalog_header starts with a list of character sets identified by name:
+
+@verbatim
+ charsets: charset_name ...
+@endverbatim
+
+ Character set names are string values represented using ASCII characters.
+ The first two @c charset_name entries have a special meaning and should
+ always be present:
+ -# The character set used to encode all strings stored in
+ the preamble following the charsets list. This should be a universal
+ character set capable of representing any string, such as UTF8.
+ -# The default character set of the server on which the backup image was
+ created. It can be the same as the first character set.
+
+ @c charset_name values following the first two, if present, are any character
+ sets used by the objects stored in the image and thus needed to restore those
+ objects.
+
+ References to character sets at other locations in the backup image are as
+ 0-based positions within the character set list. The number of character sets
+ is limited to 256 so that one byte is sufficient to identify a character set
+ by its number.
+
+ @note Currently, even if some objects (such as tables or databases) use
+ character sets in their definition, these character sets will not be listed in
+ the image. On restore, all entries in this list are ignored.
+ Collation information is not stored currently, but might be in the future.
+
+ @note Character set and collation numbers used within the image catalog are
+ internal to the backup image and have nothing to do with the IDs used in the
+ server. Backup image-processing code must translate between image IDs and
+ internal server IDs if necessary.
+
+ After the character set names, a list of users follows:
+
+@verbatim
+ users: user_name ...
+@endverbatim
+
+ The users list is currently always empty. It is reserved for future use when
+ there will be a need to refer to users inside backup image (e.g., when user
+ definitions are included).
+
+ After users, a list of names of all tablespaces stored in the image follows:
+
+@verbatim
+ tablespace: tablespace_name ...
+@endverbatim
+
+ Each @c tablespace_name value is a string.
+
+ After tablespaces, a list of all databases follows. If the list is empty, it
+ consists of a single empty string. Otherwise, it is a list of one or more
+ @c db_info entries:
+
+@verbatim
+ databases: 0x00 | db_info ...
+@endverbatim
+
+ @c db_info entries each contain fields that describe a single database:
+
+@verbatim
+ db_info: db_name db_flags [extra_data]
+@endverbatim
+ - @c db_name (variable length). String database name.
+ - @c db_flags (1 byte). Integer database flags.
+ - @c extra_data (variable length). Extra data. Optional; present only if
+ indicated in the flags. It consists of:
+ - Integer data length (2 bytes).
+ - Data bytes, as many as specified by the length.
+
+ Bits in the @c db_flags field are used as follows:
+ - Bits 0-6: Reserved
+ - Bit 7 (BSTREAM_FLAG_HAS_EXTRA_DATA): Set if the extra_data field is present
+ in the db_info entry.
*/
/** Definition of extra data flag bits. */
@@ -640,7 +764,7 @@ int bstream_rd_db_catalogue(backup_strea
The contents of the image is read from the @c cat object using iterators
and @c bcat_*() functions defined by the program using this library.
- @see @c bcat_iterator_get(), @c bcat_iterator_next(), @c bcat_iterator_free()
+ @see @ref catalog
*/
int bstream_wr_catalogue(backup_stream *s, struct st_bstream_image_header *cat)
{
@@ -753,7 +877,7 @@ int bstream_wr_catalogue(backup_stream *
@retval BSTREAM_EOC Read successful and end of chunk has been reached
@retval BSTREAM_EOS Read successful and end of stream has been reached
- @see @c bcat_add_item()
+ @see @ref catalog
*/
int bstream_rd_catalogue(backup_stream *s, struct st_bstream_image_header *cat)
{
@@ -906,28 +1030,11 @@ int bstream_rd_catalogue(backup_stream *
return ret;
}
-/**
- @page stream_format
-
- Encoding of item types used in a backup image.
-
- - 1 = character set,
- - 2 = user,
- - 3 = privilege,
- - 4 = database,
- - 5 = table,
- - 6 = view.
- - 7 = stored procedure.
- - 8 = stored function.
- - 9 = event.
- - 10 = trigger.
- - 11 = table space.
-
- Value 0 doesn't encode a valid item type and is used as item list separator.
- */
/**
- Save item type.
+ Save object type.
+
+ See @ref basic_data_itype for encoding of object types.
@retval BSTREAM_OK type was saved successfully
@retval BSTREAM_ERROR error writing or attempt to save unknown type.
@@ -952,7 +1059,9 @@ int bstream_wr_item_type(backup_stream *
}
/**
- Read item type.
+ Read object type.
+
+ See @ref basic_data_itype for encoding of object types.
@retval BSTREAM_ERROR Error while reading or non-recognized type found.
@retval BSTREAM_OK Read successful
@@ -989,52 +1098,62 @@ int bstream_rd_item_type(backup_stream *
}
/**
- @page stream_format
+ @page image_layer
- @subsection db_catalogue Database catalogue
+ @subsection db_catalog Database Catalog Format
- Database catalogue lists all tables and other per-db items belonging to that
- database.
- @verbatim
+ Each @c db_catalog chunk lists all tables and other per-database objects
+ belonging to a single database. If there are no databases in the image, the
+ databases list in the @c catalog_header chunk is empty and the catalog
+ contains no @c db_catalog chunks following the @c catalog_header chunk.
- [db catalogue]= [ db-item info ! ... ! db-item info ]
- @endverbatim
+ If a database is empty, its @c db_catalog chunk consists of two 0x00 bytes.
+ Otherwise, @c db_catalog contains lists of @c table_info and @c db_item_info
+ entries:
- Each entry in the catalogue describes a single item, which can be a table or
- of other kind.
- @verbatim
+@verbatim
+ db_catalog: 0x00 0x00 | db_tables db_other_items
- [db-item info]= [ type:2 ! name ! optional item data ]
- @endverbatim
+ db_tables: table_info ...
+ db_other_items: db_item_info ...
+@endverbatim
- [optional item data] is used only for tables:
+ Each item entry in @c db_catalog starts with 2 byte integer indicating type of
+ the object. All tables are stored before other per-database objects. This
+ is important for addressing the per-database objects because the coordinates
+ for such objects refer to the position in the @c db_other_items list.
+ (For example, per-database object 0 is the object described by the first
+ item in @c db_other_items.)
- @verbatim
+@verbatim
+ table_info: type table_name flags snapshot_num table_pos
+@endverbatim
- [optional item data (table)]= [ flags:1 ! snapshot no.:1 ! pos !
- optional extra data ]
- @endverbatim
+ - @c type (2 bytes): Integer object type (always 0x05 0x00 for a table).
+ - @c table_name (variable length): String table name.
+ - @c flags (1 byte): Integer flags.
+ - @c snapshot_num (1 byte): Integer indicating which snapshot contains
+ table's data (0-based).
+ - @c table_pos (variable length): Integer position of the table within
+ its snapshot (0-based).
- [snapshot no.] tells which snapshot contains tables data and [pos] tells what
- is the position of the table in this snapshot.
+ The @c flags field is reserved for future use.
- Presence of extra data is indicated by a flag.
- @verbatim
+@verbatim
+ db_item_info:= type name
+@endverbatim
- [flags]= [ has_extra_data:.1 ! unused:.7 ]
+ - @c type (2 bytes): Integer object type.
+ - @c name (variable length): String object name.
- [optional extra data]= [ data_len:1 ! extra data:(data_len) ]
- @endverbatim
-
- If database is empty, it stores two 0x00 bytes.
- @verbatim
-
- [db catalogue (empty)] = [ 0x00 0x00 ]
- @endverbatim
+ Allowable object type values are given in @ref basic_data_itype.
*/
+/**
+ Save catalogue of objects belonging to given database.
-/** Save catalogue of items belonging to given database. */
+ @see @ref db_catalog
+*/
int bstream_wr_db_catalogue(backup_stream *s, struct st_bstream_image_header *cat,
struct st_bstream_db_info *db_info)
{
@@ -1057,7 +1176,7 @@ int bstream_wr_db_catalogue(backup_strea
if (item->base.type == BSTREAM_IT_TABLE)
{
- CHECK_WR_RES(bstream_wr_byte(s,0x00)); /* flags: we don't use extra data */
+ CHECK_WR_RES(bstream_wr_byte(s,0x00)); /* flags */
CHECK_WR_RES(bstream_wr_byte(s,((struct st_bstream_table_info*)item)->snap_num));
CHECK_WR_RES(bstream_wr_num(s,item->base.pos));
}
@@ -1090,7 +1209,7 @@ int bstream_wr_db_catalogue(backup_strea
@retval BSTREAM_EOC Read successful and end of chunk has been reached
@retval BSTREAM_EOS Read successful and end of stream has been reached
- @see @c bcat_add_item()
+ @see @ref db_catalog
*/
int bstream_rd_db_catalogue(backup_stream *s, struct st_bstream_image_header *cat,
struct st_bstream_db_info *db_info)
@@ -1145,69 +1264,59 @@ int bstream_rd_db_catalogue(backup_strea
/*************************************************************************
*
- * META DATA
+ * METADATA
*
*************************************************************************/
/**
- @page stream_format
-
- @section meta_data Meta data section
-
- Meta data section contains meta-data for items which need to be created
- when restoring data. It is divided into three main parts, storing meta data
- for global items, tables and other items (per-db and per-table).
- @verbatim
-
- [meta data]= [ global items | tables | other items ]
- @endverbatim
-
- The only global items for which we store meta-data information are tablespaces
- and databases. Tablespace definitions should come before database definitions
- on the [global items] list.
-
- [Tables] section contains all tables which are grouped on per-database basis
- (this is for easier skipping of tables upon selective restore).
- @verbatim
-
- [tables] = [ tables from db1 | ... | tables from dbN ]
- @endverbatim
+ @page image_layer
- [Other items] has two parts for all per-database items (except tables) and
- all per-table items.
- @verbatim
+ @section metadata Metadata Format
- [other items]= [ per-db items ! 0x00 0x00 ! per-table items ]
- @endverbatim
-
- The per-database items other than tables can not be grouped by database
- because of possible inter-database dependenciens. This is why they are stored
- in a separate section.
-
- If there are no databases in the image, [meta data] consists of [global items]
- only.
- @verbatim
-
- [meta data (no databases)]= [ global items ]
- @endverbatim
-
- Meta data item lists can be empty or consist of several item entries. Empty
- item list consist of two 0x00 bytes which can not start any valid
- [item entry].
- @verbatim
-
- [item list] = [ item entry ! ... ! item entry ]
- [item list (empty)]= [ 0x00 0x00 ]
- @endverbatim
+ The metadata section contains information for objects that need to be created
+ when restoring data. It has three main sections:
+ - Metadata for global objects (single chunk)
+ - Metadata for tables (one chunk per a database)
+ - Metadata for other per-database objects (single chunk)
+
+@verbatim
+ metadata: global_items [tables ... other_items]
+@endverbatim
+
+ If there are no databases in the image, @c metadata consists of the
+ @c global_items chunk only. Otherwise, for each database stored in the image
+ one @c tables chunk with table metadata follows. Finally, there is
+ @c other_items chunk containing metadata for per-database objects other than
+ tables.
+
+ Each of these chunks contains a list of metadata entries, each entry containing
+ information required for restoring a single object.
+ The order of metadata entries is relevant. They are stored in such an order
+ that objects can be created while reading these entries without breaking any
+ dependencies. An exception is metadata for tables which are grouped by
+ database, which can break dependencies. However, this is OK since only
+ foreign key constraints can introduce dependencies between tables and they are
+ disabled during restore. If the list stored in a chunk is empty, then the
+ chunk consists of two 0x00 bytes.
+
+ Currently the @c global_items chunk contains metadata only for tablespaces and
+ databases, but it might be used to store other global objects in the future.
+ Tablespace definitions precede database definitions.
+
+ The @c other_items chunk stores metadata for per-database objects other than
+ tables. Currently these are stored routines, views, triggers, events and
+ grants. They are not grouped by a database but rather stored in a dependency
+ preserving order. For historical reasons, there are always two 0x00 bytes at
+ the end of @c other_items chunk.
*/
-/** different formats in which item positions are stored */
+/** different formats in which object positions are stored */
enum enum_bstream_meta_item_kind {
- GLOBAL_ITEM, /**< only item position is stored */
+ GLOBAL_ITEM, /**< only object position is stored */
TABLE_ITEM, /**< only table position is stored (database is implicit) */
- PER_DB_ITEM, /**< item position followed by it's database position */
+ PER_DB_ITEM, /**< object position followed by its database position */
/**
- Item position followed by it's table's database position followed by the
+ Item position followed by its table's database position followed by the
table's position inside that database
*/
PER_TABLE_ITEM
@@ -1228,7 +1337,11 @@ int bstream_wr_item_def(backup_stream*,
int read_and_create_items(backup_stream *s, struct st_bstream_image_header *cat,
enum enum_bstream_meta_item_kind kind);
-/** Write meta-data section of a backup image */
+/**
+ Write metadata section of a backup image
+
+ @see @ref metadata
+*/
int bstream_wr_meta_data(backup_stream *s, struct st_bstream_image_header *cat)
{
void *iter= NULL, *titer= NULL;
@@ -1238,7 +1351,7 @@ int bstream_wr_meta_data(backup_stream *
bool item_written= FALSE;
bool has_db= FALSE;
- /* global items (this includes databases) */
+ /* global objects (this includes databases) */
iter= bcat_iterator_get(cat,BSTREAM_IT_GLOBAL);
@@ -1304,7 +1417,7 @@ int bstream_wr_meta_data(backup_stream *
if (!has_db)
return BSTREAM_OK;
- /* other per-db items */
+ /* other per-db objects */
CHECK_WR_RES(bstream_end_chunk(s));
@@ -1349,14 +1462,16 @@ wr_error:
}
/**
- Read backup image meta-data section.
+ Read backup image metadata section.
- All items read are created using @c bstream_create_item() function.
+ All objects read are created using @c bstream_create_item() function.
@retval BSTREAM_ERROR Error while reading
@retval BSTREAM_OK Read successful
@retval BSTREAM_EOC Read successful and end of chunk has been reached
@retval BSTREAM_EOS Read successful and end of stream has been reached
+
+ @see @ref metadata
*/
int bstream_rd_meta_data(backup_stream *s, struct st_bstream_image_header *cat)
{
@@ -1365,7 +1480,7 @@ int bstream_rd_meta_data(backup_stream *
int ret=BSTREAM_OK;
bool has_db= FALSE;
- /* global items */
+ /* global objects */
CHECK_RD_RES(read_and_create_items(s,cat,GLOBAL_ITEM));
@@ -1398,7 +1513,7 @@ int bstream_rd_meta_data(backup_stream *
if (!has_db)
return ret;
- /* other per-db item */
+ /* other per-db objects */
if (ret != BSTREAM_EOC)
return BSTREAM_ERROR;
@@ -1408,13 +1523,13 @@ int bstream_rd_meta_data(backup_stream *
/*
If we hit end of chunk/stream, there is nothing more to read
- (no per-table items)
+ (no per-table objects)
*/
if (ret != BSTREAM_OK)
return ret;
- /* per-table items */
+ /* per-table objects */
CHECK_RD_RES(read_and_create_items(s,cat,PER_TABLE_ITEM));
@@ -1429,50 +1544,113 @@ int bstream_rd_meta_data(backup_stream *
/**
- @page stream_format
+ @page image_layer
+
+ @subsection metadata_entry Single metadata entry format
- @subsection item_entry Single item entry
+ A single metadata entry has the following format:
- Item list is a sequence of meta data item entries, each having the
- following format:
- @verbatim
-
- [item entry]= [ type:2 ! flags:1 ! position in the catalogue !
- optional extra data ! optional CREATE statement ]
- @endverbatim
-
- Item meta-data contains a CREATE statement or other data in unspecified format
- or both. [flags] inform about which meta-data elements are present in the
- entry.
- @verbatim
-
- [flags]= [ has_extra_data:.1 ! has_create_stmt:.1 ! unused:.6 ]
- @endverbatim
-
- The position in the catalogue is represented by 1 to 3 numbers, depending on
- in which part of catalogue the entry lies.
- @verbatim
-
- [item position (global)]= [db no.]
- [item position (table)]= [ snap no. ! pos in snapshot's table list ]
- [item position (other per-db item)]= [ pos in db item list ! db no. ]
- [item position (per-table item)] = [ pos in table's item list ! db no. ! table pos ]
- @endverbatim
-
- Note that table is identified by its position inside the snapshot to which it
- belongs.
- @verbatim
+@verbatim
+ item_entry: type flags catalog_pos [extra_data] [object_metadata]
+@endverbatim
+
+ - @c type (2 bytes): Integer object type.
+ - @c flags (1 byte): Integer flags.
+ - @c catalog_pos (variable length): Catalog coordinates of the object - exact
+ format depends on the type of the object.
+ - @c extra_data (variable length): Optional; present only if
+ BSTREAM_FLAG_HAS_EXTRA_DATA is set in the flags. (Currently not used.) If
+ present it consists of
+ - Integer data length (2 bytes).
+ - Data bytes, as many as specified by the length.
+ - @c object_metadata (variable length): String with object's serialization.
+ Optional; present only if BSTREAM_FLAG_HAS_CREATE_STMT is set in the flags.
- [optional extra data]= [ data_len:2 ! extra data:(data_len) ]
- @endverbatim
+ Allowable object type values are given in @ref basic_data_itype. The format of
+ the catalog coordinates depends on the object type, as described below.
+
+ Object metadata is a serialization of the object as a string, as
+ provided by server's object services API. Usually, it is an SQL CREATE
+ statement for the object, but it can also contain additional SQL statements
+ and other information. There is also a possibility of storing arbitrary
+ additional metadata in the optional @c extra_data field, however this field
+ is not used currently.
+
+ The @c flags field indicates which metadata elements are present in
+ the entry:
+ - Bits 0-5: Reserved
+ - Bit 6 (BSTREAM_FLAG_HAS_CREATE_STMT): The object_metadata field is present.
+ - Bit 7 (BSTREAM_FLAG_HAS_EXTRA_DATA): The extra_data field is present.
+
+ @subsubsection catalog_pos_global Catalog coordinates for global objects
+
+ These are used to identify global objects whose metadata is stored in
+ @c global_items chunk of metadata section. Coordinate of a global object
+ consists of a single variable length integer:
+
+@verbatim
+ catalog_pos: pos_in_the_list
+@endverbatim
+
+ It is the 0-based position of the object in the corresponding list
+ in the @c catalog_header chunk of the catalog. The catalog is capable of
+ storing four types of global objects: databases, tablespaces, users, and
+ character sets. For all these global objects, @c catalog_pos is a single
+ number pointing at the entry on the appropriate list in the @c catalog_header
+ chunk:
+ - Database: The @c databases list.
+ - Tablespace: The @c tablespaces list.
+ - User: The @c users list.
+ - Character set: The @c charsets list.
+
+ @note Only tablespace and database metadata is stored in the @c global_items
+ chunk of metadata section thus only tablespace and database coordinates are
+ currently used.
+
+ @subsubsection catalog_pos_table Catalog coordinates for tables
+
+ These are used for identifying tables inside table metadata entries in @c
+ tables chunks of metadata section.
+
+@verbatim
+ catalog_pos: table_pos snapshot_num
+@endverbatim
+ - @c table_pos (variable length): Integer position of the table within the
+ snapshot storing this table's data (0-based).
+ - @c snapshot_num (1 byte). Integer snapshot number (0-based).
+
+ Table coordinates (K,N) indicate the table described by the @c table_info
+ entry with @c snapshot_num = N and @c table_pos = K. This @c table_info entry
+ can be found in the @c db_catalog chunk corresponding to the database
+ containing the table.
+
+ @subsubsection catalog_pos_perdb Catalog coordinates for other per-database objects
+
+ They are used for identifying per-database objects for which metadata is
+ stored in the @c other_items chunk of the metadata section.
+
+@verbatim
+ catalog_pos: item_pos db_num
+@endverbatim
+ - @c item_pos (variable length): 0-based position in the @c db_other_items
+ list from the @c db_catalog chunk corresponding to the given database.
+ - @c db_num (variable length): 0-based position of the database in the
+ @c databases list in @c catalog_header.
+
+ The other per-database objects represent views, stored procedures,
+ stored functions, events, triggers, and grants. Coordinates (K,N) indicate
+ the object described by the K-th @c db_item_info entry in the
+ @c db_other_items list inside the N-th @c db_catalog chunk.
*/
/** Definition of create statement flag bits. */
#define BSTREAM_FLAG_HAS_CREATE_STMT 0x40
/**
- Write entry describing single item but without CREATE statement or
+ Write entry describing single object but without CREATE statement or
other meta data.
+
+ @see @ref metadata_entry
*/
int bstream_wr_meta_item(backup_stream *s,
enum enum_bstream_meta_item_kind kind,
@@ -1486,7 +1664,7 @@ int bstream_wr_meta_item(backup_stream *
CHECK_WR_RES(bstream_wr_item_type(s,item->type));
CHECK_WR_RES(bstream_wr_byte(s,flags));
- /* save item's position in the catalogue */
+ /* save object's position in the catalogue */
CHECK_WR_RES(bstream_wr_num(s,item->pos));
@@ -1502,23 +1680,22 @@ int bstream_wr_meta_item(backup_stream *
if (kind == PER_TABLE_ITEM)
CHECK_WR_RES(bstream_wr_num(s,
((struct st_bstream_titem_info*)item)->table->base.base.pos));
-
wr_error:
return ret;
}
/**
- Read initial part of item entry and locate that item in the catalogue.
+ Read initial part of object entry and locate that object in the catalogue.
- Pointer to an appropriate structure describing the located item is stored in
+ Pointer to an appropriate structure describing the located object is stored in
@c (*item). This description is not persistent - next call to this function
- can overwrite it with description of another item.
+ can overwrite it with description of another object.
@param[in] s the backup stream
- @param[in] kind format in which item coordinates are stored
+ @param[in] kind format in which object coordinates are stored
@param[in] flags the flags saved in the entry are stored in that location
- @param[in] item pointer to a structure describing item found is stored here.
+ @param[in] item pointer to a structure describing object found is stored here.
@retval BSTREAM_ERROR Error while reading
@retval BSTREAM_OK Read successful
@@ -1526,7 +1703,9 @@ int bstream_wr_meta_item(backup_stream *
@retval BSTREAM_EOS Read successful and end of stream has been reached
@note If function returns BSTREAM_OK and @c (*item) is set to NULL, it means
- that we are looking at an empty item list.
+ that we are looking at an empty object list.
+
+ @see @ref metadata_entry
*/
int bstream_rd_meta_item(backup_stream *s,
enum enum_bstream_meta_item_kind kind,
@@ -1564,7 +1743,7 @@ int bstream_rd_meta_item(backup_stream *
CHECK_RD_OK(bstream_rd_byte(s,flags));
- /* read item's position */
+ /* read object's position */
CHECK_RD_RES(bstream_rd_num(s,&item_buf.any.pos));
@@ -1611,14 +1790,16 @@ int bstream_rd_meta_item(backup_stream *
/**
- Write entry with given item's meta-data.
+ Write entry with given object's metadata.
@param[in] s backup stream.
@param[in] cat image catalogue.
- @param[in] kind determines format in which item's coordinates are saved.
+ @param[in] kind determines format in which object's coordinates are saved.
@param[in] item stream item to write.
@returns Status of operation.
+
+ @see @ref metadata_entry
*/
int bstream_wr_item_def(backup_stream *s,
struct st_bstream_image_header *cat,
@@ -1637,7 +1818,7 @@ int bstream_wr_item_def(backup_stream *s
query.end= 0;
/*
- Fetch item's create query and/or extra metadata data. Note that
+ Fetch object's create query and/or extra metadata data. Note that
the BSTREAM_EOS reply from bcat_get_item_create_*() functions
indicates lack of the corresponding piece of metadata.
*/
@@ -1654,7 +1835,7 @@ int bstream_wr_item_def(backup_stream *s
else if (ret == BSTREAM_ERROR)
goto wr_error;
- /* save the header of metadata entry, containing item coordinates */
+ /* save the header of metadata entry, containing object coordinates */
ret= bstream_wr_meta_item(s,kind,flags,item);
if (ret == BSTREAM_ERROR)
@@ -1674,18 +1855,20 @@ int bstream_wr_item_def(backup_stream *s
}
/**
- Read list of meta-data entries and create the corresponding items.
+ Read list of metadata entries and create the corresponding objects.
The entries are read until the end of chunk or 0x00 marker is hit.
- After reading meta-data for each item (CREATE statement and/or extra meta-data)
- the item is created using @c bstream_create_item() function which should be
+ After reading metadata for each object (CREATE statement and/or extra metadata)
+ the object is created using @c bstream_create_item() function which should be
implemented by the program using this library.
@retval BSTREAM_ERROR error while reading
- @retval BSTREAM_EOC either list was empty or all items were read and created
+ @retval BSTREAM_EOC either list was empty or all objects were read and created
successfully
- @retval BSTREAM_EOS all items read and created successfully and end of
+ @retval BSTREAM_EOS all objects read and created successfully and end of
stream has been reached
+
+ @see @ref metadata_entry
*/
int read_and_create_items(backup_stream *s, struct st_bstream_image_header *cat,
enum enum_bstream_meta_item_kind kind)
@@ -1741,29 +1924,66 @@ int read_and_create_items(backup_stream
*************************************************************************/
/**
- @page stream_format
+ @page image_layer
- @section data Table data section
+ @section table_data Table Data Format
- Format of table data section of backup image.
- @verbatim
-
- [table data]= [ table data chunk | ... | table data chunk ]
-
- [table data chunk]= [ snapshot no.:1 ! seq no.:2 ! flags:1 ! table no. ! data ]
- @endverbatim
-
- Data chunks of each snapshot are numbered by consecutive numbers. This can be
- used to detect discontinuities in a backup stream. Currently only one flag
- is used, indicating last data chunk for a given table.
- @verbatim
-
- [flags]= [ unused:.7 ! last data block:.1 ]
- @endverbatim
+ A snapshot of data from several tables is created by a backup driver
+ and used by restore driver to restore these table data. Each table is handled
+ by one of the drivers and each driver has a list of tables which it handles.
+ The number of snapshots contained in a backup image is given in the
+ @c header chunk (see @ref header). Each snapshot is described by a single
+ @c snapshot_description chunk in the preamble (see @ref snapshot). Snapshots
+ are numbered by consecutive numbers starting with 1.
+
+ Table data snapshot consists of several data chunks. Usually, each data chunk
+ is assigned to one of the tables within the snapshot, but there can also
+ be "common data" chunks which are not tied to any particular table.
+ Data chunks from different snapshots can be mixed in the image.
+
+ A single @c table_data chunk within backup image contains a data chunk from
+ one of the snapshots. It has the following format:
+
+@verbatim
+ table_data: snapshot_num sequence_num flags table_num data
+@endverbatim
+ - @c snapshot_num (1 byte): Integer snapshot number (1-based).
+ - @c sequence_num (2 bytes): Integer sequence number. Currently not used and
+ is always 0.
+ - @c flags (1 byte): Integer flags.
+ - @c table_num (variable length): Integer table number (1-based).
+ - @c data (variable length): Table data, to end of chunk.
+
+ The snapshot to which this data chunk belongs is identified in the first byte
+ of the chunk (@c snapshot_num field). Since snapshot numbers begin with 1,
+ this byte must be non-zero. This allows detecting the end of the @c table_data
+ chunks sequence in the stream as the final @c summary chunk starts with 0x00
+ byte (see @ref summary).
+
+ In the future, sequence numbers will be used to detect discontinuities in a
+ backup stream. However, this is not currently implemented and a sequence
+ number is always 0 (two 0x00 bytes). Upon restore, sequence numbers are
+ currently ignored.
+
+ Bits in the flags field are used as follows:
+ - Bit 0 (BSTREAM_FLAG_LAST_CHUNK): Set if the chunk is the last data chunk
+ for the table.
+ - Bits 1-7. Reserved.
+
+ Field @c table_num identifies the table to which this data chunk belongs. If
+ @c table_num is 0 then this is a common data chunk not associated with any
+ particular table. Otherwise (@c table_num-1, @c snapshot_num-1) are the
+ catalog coordinates of the table as described in @ref catalog_pos_table.
+
+ The format of @c data field is determined by the backup driver which created
+ given snapshot, as specified in the corresponding @c snapshot_description
+ chunk in the preamble.
*/
/**
Write chunk with data from backup driver.
+
+ @see @ref table_data
*/
int bstream_wr_data_chunk(backup_stream *s,
struct st_bstream_data_chunk *chunk)
@@ -1808,6 +2028,8 @@ int bstream_wr_data_chunk(backup_stream
Return value @c BSTREAM_EOC indicates that all table data chunks have been read.
The rest of the backup stream can contain image summary block.
+
+ @see @ref table_data
*/
int bstream_rd_data_chunk(backup_stream *s,
struct st_bstream_data_chunk *chunk)
@@ -1919,11 +2141,70 @@ int bstream_rd_data_chunk(backup_stream
return ret;
}
+
+/**
+ @page image_layer
+
+ @section summary Summary Format
+
+ The summary chunk stores information which are known only at the end of backup
+ process. Specifically the binary log coordinates of image's validity point,
+ which enable point-in-time recovery operations that combine use of the backup
+ image and a log of modifications made after the validity point.
+
+ The summary chunk appears at the end of the backup image, after all table data
+ chunks. It always starts with 0x00 byte, to distinguish it from table data
+ chunks which never begin with 0x00 (see @ref table_data).
+
+@verbatim
+ summary: 0x00 vp_time end_time binlog_coords binlog_group_coords
+@endverbatim
+ - 0x00 (1 byte): To distinguishing the summary from preceding table data
+ chunks.
+ - @c vp_time (6 bytes): Time of validity point.
+ - @c end_time (6 bytes): Time when backup ended.
+ - @c binlog_coords (variable length): Binary log coordinates, stored as:
+ - @c pos (4 bytes): Integer binary log position.
+ - @c binlog_file (variable length): String binary log filename.
+ - @c binlog_group_coords (variable length): Binary log group coordinates.
+ The format is same as for @c binlog_coords. This field is reserved for
+ future use.
+*/
+
+
/*********************************************************************
*
* WRITING/READING BASIC TYPES
*
*********************************************************************/
+/**
+ @page image_layer
+
+ @section basic_data Basic Data Type Storage
+
+ This section discusses storage for basic data types:
+ - Fixed-length and variable-length integers
+ - Strings
+ - Times
+
+ @subsection basic_data_int Integer Storage
+
+ Only unsigned integers are stored.
+
+ Fixed-length integers are stored as 1-byte, 2-byte, or 4-byte values, least
+ significant byte first.
+
+ Variable-length integers are stored as a sequence of bytes, each byte
+ storing 7 bits (least significant bits first). The most significant bit in
+ a byte is set if more bytes follow. There is no maximum number of bytes that
+ a variable-length integer can contain. A program that reads them must be
+ prepared to deal with arbitrarily large values, or to throw an error if
+ it encounters a value larger than what it can handle.
+
+ Example: The binary number 100101011101011010110100011110 will be split
+ into 7-bit groups 10 0101011 1010110 1011010 0011110 and then stored in
+ 5 bytes (least significant bytes first): 0x9E 0xDA 0xD6 0xAB 0x02
+*/
/** Write single byte to backup stream */
int bstream_wr_byte(backup_stream *s, unsigned short int x)
@@ -2065,6 +2346,8 @@ int bstream_rd_int4(backup_stream *s, un
significant bit in a byte tells if there are more bytes to follow
(if it is set) or if current byte is the last one (if it is not set).
The bits are saved starting with least significant ones.
+
+ @see @ref basic_data_int
*/
int bstream_wr_num(backup_stream *s, unsigned long int x)
{
@@ -2090,6 +2373,8 @@ int bstream_wr_num(backup_stream *s, uns
@retval BSTREAM_OK Read successful
@retval BSTREAM_EOC Read successful and end of chunk has been reached
@retval BSTREAM_EOS Read successful and end of stream has been reached
+
+ @see @ref basic_data_int
*/
int bstream_rd_num(backup_stream *s, unsigned long int *x)
{
@@ -2113,17 +2398,28 @@ int bstream_rd_num(backup_stream *s, uns
return (b & 0x80) ? BSTREAM_ERROR : ret;
}
-/*
- String format.
+/**
+ @page image_layer
- [string]= [ size ! string bytes:(size) ]
+ @subsection basic_data_string String Storage
+
+ Strings are stored as counted strings: A byte count indicating the length of
+ the string, followed by the bytes in the string. The length is stored as a
+ variable-length integer using the encoding just described.
+
+ Strings are encoded using a universal encoding listed as the first item in
+ the @c charsets list in the @c catalog_header chunk.
+
+ An empty string (or "null string") is represented as a single 0x00 byte.
- All strings are stored using the same universal character set, which is listed
- in image's catalogue as the first entry.
+ Example: "abcd" is encoded as 0x04 0x61 0x62 0x63 0x64
+ Example: "" (the empty string) is encoded as 0x00
*/
/**
Write a string.
+
+ @see @ref basic_data_string
*/
int bstream_wr_string(backup_stream *s, bstream_blob str)
{
@@ -2152,6 +2448,8 @@ int bstream_wr_string(backup_stream *s,
@note
Caller of this function is responsible for freeing memory allocated to store
the string.
+
+ @see @ref basic_data_string
*/
int bstream_rd_string(backup_stream *s, bstream_blob *str)
{
@@ -2178,16 +2476,39 @@ int bstream_rd_string(backup_stream *s,
return bstream_read_blob(s, *str);
}
-/*
- Time format:
+/**
+ @page image_layer
+
+ @subsection basic_data_time Time Storage
- [time]= [ year and month:2 ! mday:1 ! hour:1 ! min:1 ! sec:1 ]
+ Times are stored in UTC. A time value takes six bytes:
- [year and month]= [ year:.12 ! month:.4 ]
+ - 2 bytes. Years since 1900 and month (0..11)
+ - Bits 0-7, 12-15. Year:
+ - High 8 bits from bits 0-7 of byte 0
+ - Low 4 bits from bits 4-8 of byte 1
+ - Bits 8-11. Month (bits 0-3 of byte 1)
+ - 1 byte. Day of month (1..31)
+ - 1 byte. Hour (0..23)
+ - 1 byte. Minute (0..59)
+ - 1 byte. Second (0..60)
+ Example: 2008-10-11 15:28:17 is encoded as 0x06 0xc9 0x0b 0x0f 0x1c 0x11
+ - Year = (0x06 << 4) + (0xc9 >> 4) = 0x60 + 0x0c = 0x6c = 108 (that is, 2008 - 1900)
+ - Month = 0xc9 & 0x0f = 9
+ - Day = 0x0b = 11
+ - Hour = 0x0f = 15
+ - Minute = 0x1c = 28
+ - Second = 0x11 = 17
+
+ A time value may consist entirely of 0x00 bytes, which means "no date."
*/
-/** Write time entry */
+/**
+ Write time entry
+
+ @see @ref basic_data_time
+*/
int bstream_wr_time(backup_stream *s, bstream_time_t *time)
{
byte buf[6];
@@ -2214,6 +2535,8 @@ int bstream_wr_time(backup_stream *s, bs
@retval BSTREAM_OK Read successful
@retval BSTREAM_EOC Read successful and end of chunk has been reached
@retval BSTREAM_EOS Read successful and end of stream has been reached
+
+ @see @ref basic_data_time
*/
int bstream_rd_time(backup_stream *s, bstream_time_t *time)
{
@@ -2238,3 +2561,29 @@ int bstream_rd_time(backup_stream *s, bs
return BSTREAM_OK;
}
+
+/**
+ @page image_layer
+
+ @subsection basic_data_itype Object Type Encoding
+
+ Object type encoding is used for @c db_item values in @c db_catalog chunks,
+ and for object types in metadata item lists.
+
+ Allowable object types are encoded as follows:
+ - 1 = character set
+ - 2 = user
+ - 3 = privilege
+ - 4 = database
+ - 5 = table
+ - 6 = view
+ - 7 = stored procedure
+ - 8 = stored function
+ - 9 = event
+ - 10 = trigger
+ - 11 = tablespace
+
+ Object types are encoded using two bytes. A value of 0 is not a valid object
+ type, so a 0x00 0x00 sequence can be used to separate one object list from
+ the next.
+*/
=== modified file 'sql/backup/stream_v1_transport.c'
--- a/sql/backup/stream_v1_transport.c 2009-08-03 07:59:51 +0000
+++ b/sql/backup/stream_v1_transport.c 2009-09-08 09:29:08 +0000
@@ -11,11 +11,92 @@
Implementation of the low-level I/O functions from backup stream library.
These functions form the transport layer of the 1st version of backup stream
- format. They split stream into a sequence of data chunks.
+ format. They split stream into a sequence of data chunks.
+ See @ref transport_layer.
@todo free internal buffer memory in case of errors.
*/
+/**
+ @page transport_layer Transport Layer (Block Level) Format
+
+ The transport layer consists of fixed-sized blocks. Blocks within a given
+ backup image are the same size (except possibly the last), but the block
+ size is encoded within the first few blocks of the image itself. Blocks are
+ encoded such that variable-size backup image chunks can be recognized and
+ extracted from the byte stream.
+
+ The backup stream prefix (magic number and image format version) is not part
+ of the sequence of blocks comprising the backup image. If present, the prefix
+ must be read separately.
+
+@verbatim
+ backup_image: first_block [initial_block ...] [regular_block ...]
+ first_block: block_size initial_block_count block_data
+ initial_block: block_size block_data
+ regular_block: block_data
+@endverbatim
+
+ The first few blocks are special:
+ - The first block contains the block size for all following blocks, the
+ number N of "initial" blocks to follow, and some data.
+ - The next N blocks following the first block (the "initial" blocks) contain
+ the block size and some data.
+ - The blocks following the first and initial blocks contain only data.
+
+ The first and inital blocks all contain the block size. The redundancy serves
+ to enable detection of data corruption.
+
+ Fields in the first block of a stream:
+ - @c block_size (4 bytes): An integer indicating how large blocks are.
+ - @c initial_block_count (1 byte): An integer indicating how many initial
+ blocks follow the first block.
+ - @c block_data (block_size-5 bytes): Stream data to the end of the block.
+
+ The size of @c block_data is block_size-5 because the first 5 bytes of the
+ block contain the @c block_size and @c initial_block_count fields.
+
+ Fields in the initial blocks of a stream:
+ - @c block_size (4 bytes): An integer indicating how large blocks are.
+ - @c block_data (block_size-4 bytes): Stream data to the end of the block.
+
+ The size of @c block_data is block_size-4 because the first 4 bytes of the
+ block contain the @c block_size field.
+ The @c block_size value in all @c initial_block blocks must match the
+ @c block_size in @c first_block.
+
+ Regular blocks contain only @c block_data without any additional fields.
+
+ Because the block size is unknown initially, the block reader must begin
+ by reading the first four bytes of @c first_block to get the block size
+ before it can read the rest of the block.
+
+ @note
+ The last block in a backup image might not be the full block size, and there
+ might not be as many initial blocks as indicated in the first block. For
+ example, the first block might indicate a @c block_size of 16384 bytes and
+ @c initial_block_count of 2. But if the entire backup image size is 3000
+ bytes, the first block will contain only 3000 bytes and there will be no
+ initial blocks following it. The block-reading level must be prepared to deal
+ with this and handle what block data bytes are actually present, interpreting
+ them according to the rules governing @c block_data content described in
+ @ref block_data.
+
+ @note
+ It is possible to have external data at the beginning of the first block of
+ a backup image, This is for situations where, e.g., backup image storage
+ requires storing additional headers and still wants to preserve block
+ aligmnet of all the data. For example, when backup image is stored in a file
+ then it is preceded by a 10 byte prefix containing magic number and image
+ version number (see @ref stream_format). This 10 byte prefix is stored at
+ the beginning of the first block and is not considered a part of the backup
+ image. The @c block_size field of the first block starts at byte 11 in that
+ case and the size of @c block_data equals block_size-15. If such external
+ prefix is present in the first block, its presence is not indicated in the
+ image itself. It must be determined by other means (e.g., by a convention as
+ in the above example).
+*/
+
#ifdef DBUG_OFF
# define ASSERT(X)
#else
@@ -121,52 +202,6 @@ extern byte get_byte_short(short value);
*/
extern byte get_byte_size_t(size_t value);
-/*
- Carrier format
- --------------
-
- [backup stream] = [ block ! ... ! block ]
-
- [block] = [ fragment ! ... ! fragment ]
-
- where [fragment] is one of
-
- [EOC marker] = [ 0x80 ]
- [EOS marker] = [ 0xC0 ]
- [fragment extending to the end of block, last in a chunk] = [ 0x00 ! payload ]
- [fragment extending to the end of block, more follow] = [ 0x40 ! payload ]
- [limited size fragment] = [ header:1 ! payload ]
-
- The header of [limited size fragment] consists of two bit fragment type info
- followed by 6 bit, non-zero value encoding length of the fragment:
-
- [fragment header] = [ type:.2 ! value:.6 ]
-
- There are four types of fragments determined by the first 2 bits of the header:
-
- 00 - small fragment which is the last fragment of a chunk,
- 01 - small fragment with more fragments following it,
- 10 - big fragment (more fragments follow),
- 11 - huge fragment (more fragments follow).
-
- Encoding of the size of the fragment depends on its type:
-
- - for small fragments: size= value
- - for big fragments: size= value << 6
- - for huge fragments: size= value << 12
-
- For small fragments, second bit of the header determines if it is the last
- fragment of a chunk or there are more fragments to come. Chunk can't end with
- a big or huge fragment and thus for these fragments we always expect more
- fragments to come. [EOC marker] ends a chunk, even if the last fragment said
- that more fragments will follow.
-
- The biggest fragment size is 64*2^12 ~= 250 Kb. The format of fragment header
- puts constraints on possible fragment sizes. If a chunk of data has size not
- possible to encode by a single fragment header, it is split into several
- fragments of correct sizes.
-*/
-
/*************************************************************************
*
@@ -174,6 +209,83 @@ extern byte get_byte_size_t(size_t value
*
*************************************************************************/
+/**
+ @page transport_layer
+
+ @section block_data Block Data Format (Transport Layer)
+
+ When a block is read in the transport layer, its @c block_data bytes are
+ extracted and then interpreted to find the fragments that it contains.
+ (No fragment ever crosses a block boundary.) These fragments are assembled
+ into chunks for use by the image layer, which operates on chunk units
+ (see @ref image_format1). There is no fixed relationship between chunks and
+ blocks. A chunk can be constructed from fragments spanning several blocks,
+ or a chunk might require fragments from only part of a block.
+
+ The @c block_data part of a given block consists of one or more fragments:
+
+@verbatim
+ block_data_stream: fragment [fragment ...]
+ fragment: EOC | EOS | frag_header payload
+ EOC: 0x80 (end of chunk)
+ EOS: 0xc0 (end of stream)
+@endverbatim
+
+ An EOC marker ends a chunk even if the preceding fragment indicates that
+ more fragments follow. EOS indicates end of the stream - all data folowing it
+ do not belong to the backup image. A regular fragment consists of 1 byte
+ header and data payload.
+
+@verbatim
+ frag_header: frag_type frag_size
+ payload: data bytes (size depends on frag_header contents)
+@endverbatim
+
+ A @c frag_header fragment header byte indicates the type and size of the
+ fragment:
+ - Bits 0-5: The fragment size. Interpretation of this value depends on the
+ fragment type.
+ - Bits 6-7: The fragment type.
+
+ Thus:
+@verbatim
+ frag_type = (frag_header & 0xc0)
+ frag_size = (frag_header & 0x3f)
+@endverbatim
+
+ Possible frag_type values:
+ - 0x00: Small fragment, more fragments in chunk to follow
+ - 0x40: Small fragment, last fragment in chunk
+ - 0x80: Big fragment, more fragments in chunk to follow
+ - 0xc0: Huge fragment, more fragments in chunk to follow
+
+ For a small fragment, the second bit of the @c frag_type bits determine
+ whether there are more fragments in the chunk.
+ A chunk cannot end with a big or huge fragment, but an EOC marker can follow
+ indicating end of the chunk.
+
+ The @c frag_size value must be interpreted to determine the actual fragment
+ size:
+ - Small fragment:
+@code
+ size = rest of block if frag_size == 0;
+ size = frag_size otherwise
+@endcode
+ - Big fragment:
+@code
+ size = (frag_size << 6)
+@endcode
+ - Huge fragment:
+@code
+ size = (frag_size << 12)
+@endcode
+
+ The biggest fragment size is 64<<12 ~= 250 Kb. The format of fragment
+ header puts constraints on possible fragment sizes. Data of arbitrary length
+ can be divided into several fragments of appropriate sizes.
+ Each fragment adds one byte overhead for the fragment header.
+*/
+
#define FR_EOC 0x80 /**< bits for EOC fragment */
#define FR_EOS 0xC0 /**< bits for EOS fragment */
#define FR_MORE 0x00 /**< bits for MORE fragment */
Attachment: [text/bzr-bundle] bzr/rafal.somla@sun.com-20090908092908-t82h1nbcoox7o2r1.bundle
| Thread |
|---|
| • bzr commit into mysql-6.0-backup branch (Rafal.Somla:2868) Bug#40587 | Rafal Somla | 8 Sep |
