| List: | Commits | « Previous MessageNext Message » | |
| From: | mcbrown | Date: | November 7 2006 1:39pm |
| Subject: | svn commit - mysqldoc@docsrva: r3867 - trunk/refman-5.1 | ||
| View as plain text | |||
Author: mcbrown Date: 2006-11-07 13:39:47 +0100 (Tue, 07 Nov 2006) New Revision: 3867 Log: Adding a Falcon SE stub to the 5.1 tree. Added: trunk/refman-5.1/se-falcon.xml Added: trunk/refman-5.1/se-falcon.xml =================================================================== --- trunk/refman-5.1/se-falcon.xml (rev 0) +++ trunk/refman-5.1/se-falcon.xml 2006-11-07 12:39:47 UTC (rev 3867) Changed blocks: 1, Lines Added: 754, Lines Deleted: 0; 24665 bytes @@ -0,0 +1,754 @@ +<?xml version="1.0" encoding="utf-8"?> +<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.3//EN" +"http://www.oasis-open.org/docbook/xml/4.3/docbookx.dtd" +[ + <!ENTITY % fixedchars.entities SYSTEM "../refman-common/fixedchars.ent"> + %fixedchars.entities; + <!ENTITY % urls.entities SYSTEM "../refman-common/urls.ent"> + %urls.entities; + <!ENTITY % versions.entities SYSTEM "versions.ent"> + %versions.entities; +]> +<section id="se-falcon"> + + <title>Falcon Storage Engine</title> + + <para> + The Falcon Storage Engine has been designed with modern database + requirements in mind, and particularly for use within a web + serving or other environment that requires high performance, while + still supporting the transactional and logging functionality + required in this environment. + </para> + + <para> + Falcon is currently available only for the Windows, Linux, Mac OS + X and Solaris operating systems, running on either 32-bit or + 64-bit architectures. + </para> + + <section id="falcon-features"> + + <title>Falcon Features</title> + + <para> + Falcon is primarily a memory-based storage engine and has been + specially developed for systems that are able to support larger + memory architectures and multi-threaded or multi-core CPU + environments. Most 64-bit architectures are ideal platforms for + the Falcon engine, where there is a larger available memory + space and 2-,4- or 8-core CPUs available. It can also be + deployed within a standard 32-bit environment. + </para> + + <para> + The Falcon storage engine is designed to work within + high-traffic transactional applications. It supports a number of + key features that make this possible: + </para> + + <itemizedlist> + + <listitem> + <para> + True Multi Version Concurrency Control (MVCC) enables + records and tables to be updated without the overhead + associated with row-level locking mechanisms. The MVCC + implementation virtually eliminates the need to lock tables + or rows during the update process. + </para> + </listitem> + + <listitem> + <para> + Optimized for multiple threads allowing multiple connections + to read and update the data. + </para> + </listitem> + + <listitem> + <para> + Transaction-safe (fully ACID-compliant). + </para> + </listitem> + + <listitem> + <para> + Serial Log provides high performance and recovery + capabilities without sacrificing performance. + </para> + </listitem> + + <listitem> + <para> + Advanced B-Tree indexes. + </para> + </listitem> + + <listitem> + <para> + Server enforced referential integrity ensures data validity + at all times. + </para> + </listitem> + + <listitem> + <para> + Data compression stores the information on disk in a + compressed format, resulting in smaller and more efficient + physical data sizes. + </para> + </listitem> + + <listitem> + <para> + Configurable page sizes enable you to select the page size + according to your data needs. + </para> + </listitem> + + <listitem> + <para> + Intelligent disk management automatically manages disk file + size, extensions and space reclamation. + </para> + </listitem> + + <listitem> + <para> + Data and index caching provides quick access to data without + the requirement to load index data from disk. + </para> + </listitem> + + <listitem> + <para> + Implicity savepoints ensure data integrity during + transactions. + </para> + </listitem> + + </itemizedlist> + + </section> + + <section id="falcon-principles"> + + <title>Principles and Terminology</title> + + <para> + To get the best out of the Falcon engine you should understand + the following basic principles and terminology. + </para> + + <para>The MySQL Falcon architecture combines advanced techniques with a + simplified structure that results in a high-performance transactional + database that requires little maintenance or troubleshooting by the + database administrator.</para> + + <para>The structure of the Falcon storage engine </para> + + <section id="falcon-principles-tablespace"> + + <title>Falcon Tablespace and File Structure</title> + + <para> + Within Falcon, all data is stored within a single tablespace, + which in turn is stored within a single file within the MySQL + directory structure. The single tablespace stores all record + data, indexes, database structure and other information. The + individual information is stored within a series of pages. + </para> + + <para> + Pages describe the internal storage allocation block within + the Falcon storage engine. Pages are used to store data and + index information. The page size and how the Falcon engine + caches and allocates pages for use when storing information + affect the performance of the engine depending on the records + that are being stored, index complexity + </para> + + <para> + Pages cached in memory are used to store indexes, blobs and + the structural data for a given tablespace. Active records + (those read or updated are stored within a separate record + cache. + </para> + + <para> + All transactions on the database are logged and stored within + a separate log file. The logfile is automatically flushed and + the changes written to disk when there is a + <literal>COMMIT</literal> command, when auto-commit is + enabled, or automatically every 30 seconds when transactions + are not being employed. + </para> + + </section> + + <section id="falcon-principles-logging"> + + <title>Falcon Serial Log</title> + + <para> + Falcon uses a Serial Log to hold certain types of information + before that data is finally committed to the database. The log + is used to store the following types of information: + </para> + + <itemizedlist> + + <listitem> + <para> + Data records during the commit phase. + </para> + </listitem> + + <listitem> + <para> + Physical database changes required for data recovery after + a crash. + </para> + </listitem> + + <listitem> + <para> + Logical database changes required for resource recovery + after a crash. + </para> + </listitem> + + <listitem> + <para> + Transaction state changes for all active transactions + (active to committed, active to rolled back, active to + limbo). + </para> + </listitem> + + </itemizedlist> + + <para> + All transactions within Falcon are written to the Falcon + Serial Log and then committed to the database, either + automatically when <literal>AUTOCOMMIT</literal> is enabled, + or manually when the <literal>COMMIT</literal> command is + used. + </para> + + <para> + Logging information is stored in memory and unwritten changes + to the log are periodically flushed to disk. A background + thread processes the contents of the log, committing the log + changes to the database. The commit process sets the final + status of all records and pages, regardless of any intervening + states; only the final state is actually written to disk. + </para> + + <para> + Note, however, that the serial log commit process only updates + the record data through the in-memory page cache. The actual + record data will be written to disk when the checkpoint + process occurs. The exception to this rule are index and blog + entries, which are immediately written to disk as part of the + commit process. + </para> + + <para> + Falcon creates two serial log files. The first log file is + used to store the serial log data until the log reaches a + specified size. Once that size has been reached, logging is + switched to the second serial log file. The commit process + continues to read from the first log file until all + transactions have been written to the database. The first log + file is then released and recreated. + </para> + + <para> + Log entries in the second file are then processed until all + transactions in the log have been completed. That file is then + released and recreated, ready to be pressed into use as soon + as the first log file is full or becomes locked for commits. + </para> + + <section id="falcon-principles-logging-rollback"> + + <title>Rollback Process</title> + + <para> + Transaction rollbacks are handled by the thread for that + transaction. The rollback process performs the following + actions: + </para> + + <itemizedlist> + + <listitem> + <para> + Backing out index updates. + </para> + </listitem> + + <listitem> + <para> + Backing out any blob data created by the transaction. + </para> + </listitem> + + <listitem> + <para> + Releasing allocated record slots. + </para> + </listitem> + + <listitem> + <para> + Backing out record versions created in memory. + </para> + </listitem> + + </itemizedlist> + + </section> + + <section id="falcon-principles-logging-groupcommit"> + + <title>Group Commits</title> + + <para> + For performance, Falcon uses a group commit system that + ensures that all pending updates to the serial log are + written to disk at the same time. Falcon can have multiple + active transactions, but only one transactions writes all + the pending changes into the serial log on disk, reducing + the number of disk writes and improving the overall + performance of the serial log. + </para> + + <para> + For example: + </para> + + <orderedlist> + + <listitem> + <para> + Transaction 1 commits, creates all the necessary log + entries, and starts to write the log to disk. + </para> + </listitem> + + <listitem> + <para> + Whilte Transaction 1 commits are being written, + Transacations 2 and 3 write their log entries into the + serial log. + </para> + </listitem> + + <listitem> + <para> + Once Transaction 1 has finished the physical write, + either transaction 2 or 3 (but not both) will write out + the unwritten portion of the in-memory log to disk. + Because both transacations have occurred since the last + disk-write of the serial log, the informaiton for both + is written to the disk at the same time. + </para> + </listitem> + + <listitem> + <para> + While transactions 2 and 3 are writing, transactions 4, + 5 and 6 are being written to the in-memory log. When the + write for 2 and 3 completes, the entries for 4, 5 and 6 + are written. + </para> + </listitem> + + </orderedlist> + + <para> + The result of the above process is that there are only three + physical writes to disk, even though there are six + transactions in the sequence: + </para> + + <itemizedlist> + + <listitem> + <para> + Transaction 1 + </para> + </listitem> + + <listitem> + <para> + Transactions 2 and 3 + </para> + </listitem> + + <listitem> + <para> + Transactions 4, 5 and 6 + </para> + </listitem> + + </itemizedlist> + + <para> + The process continues, with just one transaction writing all + the in-memory serial log entries to disk since the last + write. The entire system ensures that the in-memory and the + disk log are kept in synchronization, with the lowest amount + of physical disk writes + </para> + + <para> + The above process works in tandem with the use of the two + serial log files to ensure that the in-memory information + and on disk information are kiept up to date. + </para> + + </section> + + </section> + + <section id="falcon-principles-recovery"> + + <title>Falcon Crash Recovery</title> + + <para> + The Falcon Serial Log is used automatically during startup to + recover transactions and update the database. When + transactions and changes are written to the Serial Log the log + includes entries that record changes to all areas of the + database, including the indexes, changes to + <literal>BLOB</literal> data, and any structural changes to + the database. + </para> + + <para> + During crash recovery, Falcon examines the serial log and + identifies the first entry that has not been committed to the + database. The recovery process wriutes all unwritten data, + changes index and blob data, releasing any necessary record + slots (from deleted records) and committing any structural + changes. + </para> + + </section> + + <section id="falcon-principles-compression"> + + <title>Data Compression</title> + + <para> + Data stored in the Falcon tablespace is compressed on disk, + but is stored in an uncompressed format in memory. Compression + occurs automatically when data is committed to disk. + </para> + + </section> + + <section id="falcon-principles-recordslot"> + + <title>Record Slot</title> + + <para> + A record slot is an internal record identifier that is used to + find records in memory and on disk. It is essentially a + pointer to the pages that contain the data for a particular + record. A new record slot is created for each record for the + duration of that record's existence. The record slot is only + relinquished when the record is erased from the database. + </para> + + </section> + + <section id="falcon-principles-pagecache"> + + <title>Page Cache</title> + + <para> + Falcon stores data in pages, and pages can contain a variety + of information, including record data, index data and table + schemas. Pages are recorded on disk and a cache of the most + recently used or accessed pages are also kept in memory. Note + that the in-memory page cache only contains basic record data; + blog information is stored only on disk, or in the serial log. + </para> + + <para> + The page cache is only flushed during the checkpoint process + (approximately every 30 seconds). Changes to index and blob + information occurs during the serial log commital process. + </para> + + </section> + + </section> + + <section id="falcon-configuration"> + + <title>Configuration Parameters</title> + +<!-- TODO: [MC] Current falcon.conf will change in the future --> + + <para> + Parameters are configured through the + <filename>falcon.conf</filename> file. Parameters can be + configured by specifying the parameter name and the + corresponding value, separated by a space. Memory values can be + specified in bytes, or with a number followed by + <literal>kb</literal>, <literal>mb</literal> or + <literal>gb</literal>. + </para> + +<!-- <para>Configuration parameters within the Falcon database are + configurable for each individual tablespace. The + <literal>falcon_page_size</literal> parameter can only be + configured when the tablespace is created and cannot be changed. + The remaining paramters can be set and altered dynamically while + <literal>mysqld</literal> is running by setting the + appropriate system variable.</para>--> + +<!-- TODO: [MC] These literals are based on the current wording of + the configurable parameters in falcon.conf. + They will almost certainly change in the future. --> + + <itemizedlist> + + <listitem> + <para> + <literal>falcon_page_size</literal> (Page Size) controls the + size of the pages used to store information within the + tablespace. Valid sizes are 1, 2, 4, 8, 16 and 32 KB. + </para> + </listitem> + + <listitem> + <para> + <literal>falcon_min_record_memory</literal> (Record Cache Base) + sets the minimum amount of memory that will be allocated for + caching record data. When cache memory is scavenged, the process + will stop when the cache usage reaches this value. The default is + <literal>falcon_max_record_memory</literal>/2 (10MB). + </para> + </listitem> + + <listitem> + <para> + <literal>falcon_max_record_memory</literal> (Record Cache Top) sets + the maximum size of memory that will be allocated for + caching record data. The default is 20MB. + </para> + </listitem> + + <listitem> + <para> + <literal>falcon_page_cache_size</literal> (Page Cache Size) sets + the amount of memory that will be allocated for caching + pages from the tablespace file. The default is 4MB. + </para> + </listitem> + + <listitem> + <para> + <literal>maximum_file_size</literal> (Maximum File Size) + sets the maximum size of the tablespace file used to hold + record data. You can use this parameter to limit the amount + of space that will automatically be used by the storage + engine for storing information. + </para> + </listitem> + + <listitem> + <para> + <literal>logfile_location</literal> (Log file location) sets + the directory for storing the serial log. The filenames used + by the serial log (two files are create for storing serial + data) are allocated according to the name of the tablespace. + The default location is in the same directory as the + tablespace file. + </para> + </listitem> + + <listitem> + <para> + <literal>log_file_memory_use</literal> (Log file memory use) + configures the amount of memory allocated to providing + access to the physical logfile stored on disk. Currently the + log file memory is fixed at 8MB. + </para> + </listitem> + + </itemizedlist> + + <para> + The relationship between the record cache and the page cache is + driven by the information that is cached by each system. Whole + records that are in active use (being read or updated) are + stored within the record cache, however, <literal>BLOB</literal> + data is stored only within the page cache. + </para> + + <para> + The page cache is used to store database metadata, + <literal>BLOB</literal> data and table indexes. + </para> + + <para> + Falcon parameters can be also be set on the command-line to + <command>mysqld</command> using the following command-line + options: + </para> + + <itemizedlist> + + <listitem> + <para> + <literal>--falcon-max-record-memory=#</literal> + </para> + </listitem> + + <listitem> + <para> + <literal>--falcon-min-record-memory=#</literal> + </para> + </listitem> + + <listitem> + <para> + <literal>--falcon-page-cache-size=#</literal> + </para> + </listitem> + + <listitem> + <para> + <literal>--falcon-debug-server</literal> + </para> + </listitem> + + <listitem> + <para> + <literal>--falcon-page-size=#</literal> + </para> + </listitem> + + <listitem> + <para> + <literal>--falcon-log-dir=<replaceable>name</replaceable></literal> + </para> + </listitem> + + </itemizedlist> + + <para> + You can also enable and disable the Falcon storage engine at + startup by supplying these options to <command>mysqld</command>, + providing that the <literal>mysqld</literal> binary includes the + Falcon Storage Engine. + </para> + + <itemizedlist> + + <listitem> + <para> + <literal>--falcon</literal> enables the Falcon storage + engine. + </para> + </listitem> + + <listitem> + <para> + <literal>--skip-falcon</literal> disables the Falcon Storage + Engine. + </para> + </listitem> + + </itemizedlist> + + </section> + + <section id="falcon-limits"> + + <title>Limits</title> + + <para> + There are a number of limits in the current version of Falcon: + </para> + + <itemizedlist> + + <listitem> + <para> + There is a limit of 2<superscript>32</superscript> (4.29 + billion) records for a single table. By using multiple + tables within the same tablespace you can have more than + this number of records. + </para> + </listitem> + + <listitem> + <para> + Pages can be a maximum of 32768 bytes in size. + </para> + </listitem> + + <listitem> + <para> + Each tablespace has a limit of + 2<superscript>32</superscript> pages within a single + tablespace. Through a combination of the page size and the + maximum number of pages in a tablespace, there is a limit of + 140,737,488,355,328 bytes (128 TB) to a single tablespace. + </para> + </listitem> + + </itemizedlist> + + <para> + Although the maximum available storage within a tablespace is + 128TB, the true number of records and quantity of data that you + can store is dependent on a number of factors: + </para> + + <itemizedlist> + + <listitem> + <para> + Record storage requirements + </para> + </listitem> + + <listitem> + <para> + Index storage requirements + </para> + </listitem> + + <listitem> + <para> + Compression ratio of stored data + </para> + </listitem> + + </itemizedlist> + + <para> + Because of the complex relationship between the storage, + indexing and compression facilities it is impossible to predict + or calculate the disk storage space required for a specific + dataset. + </para> + + </section> + +<!--> <section id="falcon-roadmap"> + <title>Falcon Roadmap</title> + <para>Falcon will have...</para> + </section>--> + +</section>
| Thread | ||
|---|---|---|
| • svn commit - mysqldoc@docsrva: r3867 - trunk/refman-5.1 | mcbrown | 7 Nov |
© 1995-2008 MySQL AB, 2008-2009 Sun Microsystems, Inc.
Page generated in 0.737 sec. using MySQL 5.1.14-beta-log