Author: mcbrown
Date: 2006-11-08 15:53:37 +0100 (Wed, 08 Nov 2006)
New Revision: 3881
Log:
Updated Falcon with more background and principle information, structured for inclusion in manual, move about sections in prep for additions.
Modified:
trunk/refman-5.1/se-falcon.xml
Modified: trunk/refman-5.1/se-falcon.xml
===================================================================
--- trunk/refman-5.1/se-falcon.xml 2006-11-08 12:09:46 UTC (rev 3880)
+++ trunk/refman-5.1/se-falcon.xml 2006-11-08 14:53:37 UTC (rev 3881)
Changed blocks: 14, Lines Added: 305, Lines Deleted: 229; 25344 bytes
@@ -15,19 +15,22 @@
<para>
The Falcon Storage Engine has been designed with modern database
- requirements in mind, and particularly for use within a web
+ requirements in mind, and particularly for use within high-volume web
serving or other environment that requires high performance, while
still supporting the transactional and logging functionality
required in this environment.
</para>
+ <warning><para>Falcon is currently an Alpha release and should not be used in
+ production environments.</para></warning>
+
<para>
- Falcon is currently available only for the Windows, Linux, Mac OS
- X and Solaris operating systems, running on either 32-bit or
- 64-bit architectures.
+ Falcon is currently available only for the Windows and Linux operating
+ systems running on either 32-bit or
+ 64-bit architectures.
</para>
- <section id="falcon-features">
+ <section id="se-falcon-features">
<title>Falcon Features</title>
@@ -55,20 +58,27 @@
records and tables to be updated without the overhead
associated with row-level locking mechanisms. The MVCC
implementation virtually eliminates the need to lock tables
- or rows during the update process.
+ or rows during the update process.
</para>
</listitem>
<listitem>
<para>
- Optimized for multiple threads allowing multiple connections
- to read and update the data.
+ Flexible locking, including flexible locking levels, smart deadlock
+ detection and lock timeouts keep data protected and transactions and
+ operations flowing at full speed.
</para>
</listitem>
+ <listitem><para>Optimized for modern CPUs and environments to support
+ multiple threads allowing multiple transactions and fast transaction
+ handling.
+ </para>
+ </listitem>
<listitem>
<para>
- Transaction-safe (fully ACID-compliant).
+ Transaction-safe (fully ACID-compliant) and able to handle multiple
+ concurrent transactions.
</para>
</listitem>
@@ -95,7 +105,8 @@
<listitem>
<para>
Data compression stores the information on disk in a
- compressed format, resulting in smaller and more efficient
+ compressed format, compressing and decompressing data on the fly.
+ The result is in smaller and more efficient
physical data sizes.
</para>
</listitem>
@@ -123,7 +134,7 @@
<listitem>
<para>
- Implicity savepoints ensure data integrity during
+ Implicit savepoints ensure data integrity during
transactions.
</para>
</listitem>
@@ -131,9 +142,181 @@
</itemizedlist>
</section>
+
+ <section id="se-falcon-configuration">
+
+ <title>Configuration Parameters</title>
+
+ <para>
+ Parameters are configured through the standard
+ <filename>my.cnf</filename> or <filename>my.ini</filename> file. Parameters can be
+ configured by specifying the parameter name and the
+ corresponding value, separated by a space. Memory values can be
+ specified in bytes, or with a number followed by
+ <literal>kb</literal>, <literal>mb</literal> or
+ <literal>gb</literal>.</para>
+
+ <itemizedlist>
+
+ <listitem>
+ <para>
+ <literal>falcon_page_size</literal> (Page Size) controls the
+ size of the pages used to store information within the
+ tablespace. Valid sizes are 1, 2, 4, 8, 16 and 32 KB.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ <literal>falcon_min_record_memory</literal> (Record Cache Base)
+ sets the minimum amount of memory that will be allocated for
+ caching record data. When cache memory is scavenged, the process
+ will stop when the cache usage reaches this value. The default is
+ <literal>falcon_max_record_memory</literal>/2 (10MB).
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ <literal>falcon_max_record_memory</literal> (Record Cache Top) sets
+ the maximum size of memory that will be allocated for
+ caching record data. The default is 20MB.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ <literal>falcon_page_cache_size</literal> (Page Cache Size) sets
+ the amount of memory that will be allocated for caching
+ pages from the tablespace file. The default is 4MB.
+ </para>
+ </listitem>
+
- <section id="falcon-principles">
+ <!--
+ <remark>
+ [MC] Not currently implemented
+ </remark>
+ <listitem>
+ <para>
+ <literal>maximum_file_size</literal> (Maximum File Size)
+ sets the maximum size of the tablespace file used to hold
+ record data. You can use this parameter to limit the amount
+ of space that will automatically be used by the storage
+ engine for storing information.
+ </para>
+ </listitem>-->
+
+ <listitem>
+ <para>
+ <literal>falcon_log_dir</literal> (Log file location) sets
+ the directory for storing the serial log. The filenames used
+ by the serial log (two files are create for storing serial
+ data) are allocated according to the name of the tablespace.
+ The default location is in the same directory as the
+ tablespace file.
+ </para>
+ </listitem>
+
+
+ <!--
+ <remark>
+ [MC] Not currently implemented
+ </remark>
+ <listitem>
+ <para>
+ <literal>log_file_memory_use</literal> (Log file memory use)
+ configures the amount of memory allocated to providing
+ access to the physical logfile stored on disk. Currently the
+ log file memory is fixed at 8MB.
+ </para>
+ </listitem>-->
+
+ </itemizedlist>
+
+ <para>
+ The relationship between the record cache and the page cache is
+ driven by the information that is cached by each system. Whole
+ records that are in active use (being read or updated) are
+ stored within the record cache, however, <literal>BLOB</literal>
+ data is stored only within the page cache.
+ </para>
+
+ <para>
+ The page cache is used to store database metadata,
+ <literal>BLOB</literal> data and table indexes.
+ </para>
+
+ <para>
+ Falcon parameters can be also be set on the command-line to
+ <command>mysqld</command> using the following command-line
+ options:
+ </para>
+
+ <itemizedlist>
+
+ <listitem>
+ <para>
+ <literal>--falcon-max-record-memory=#</literal>
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ <literal>--falcon-min-record-memory=#</literal>
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ <literal>--falcon-page-cache-size=#</literal>
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ <literal>--falcon-page-size=#</literal>
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ <literal>--falcon-log-dir=<replaceable>name</replaceable></literal>
+ </para>
+ </listitem>
+
+ </itemizedlist>
+
+ <para>
+ You can also enable and disable the Falcon storage engine at
+ startup by supplying these options to <command>mysqld</command>,
+ providing that the <literal>mysqld</literal> binary includes the
+ Falcon Storage Engine.
+ </para>
+
+ <itemizedlist>
+
+ <listitem>
+ <para>
+ <literal>--falcon</literal> enables the Falcon storage
+ engine.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ <literal>--skip-falcon</literal> disables the Falcon Storage
+ Engine.
+ </para>
+ </listitem>
+
+ </itemizedlist>
+
+ </section>
+
+ <section id="se-falcon-principles">
+
<title>Principles and Terminology</title>
<para>
@@ -146,20 +329,57 @@
database that requires little maintenance or troubleshooting by the
database administrator.</para>
- <para>The structure of the Falcon storage engine </para>
+ <para></para>
- <section id="falcon-principles-tablespace">
+
+ <itemizedlist>
+ <listitem><para><emphasis role="bold">User datafile</emphasis> — stores the Falcon data.</para></listitem>
+ <listitem><para><emphasis role="bold">Falcon serial log</emphasis> — contains recently committed
+ data changes, index changes and transactional information. Also
+ provides data recovery facilities. </para></listitem>
+ <listitem><para><emphasis role="bold">Page cache</emphasis> — holds database pages being read or written.</para></listitem>
+ <listitem><para><emphasis role="bold">Record cache</emphasis> — holds copies of active and uncommitted
+ records.</para></listitem>
+ <listitem><para><emphasis role="bold">System memory</emphasis> — contains transaction context
+ information, index accelerators and system metadata.</para></listitem>
+ <listitem><para><emphasis role="bold">Work Threads</emphasis> — move data from the Falcon Serial Log into
+ the database page cache and from the page cache to disk. </para></listitem>
+ </itemizedlist>
+
+
+ <section id="se-falcon-principles-tablespace">
- <title>Falcon Tablespace and File Structure</title>
+ <title>Falcon datafile and data structures</title>
<para>
Within Falcon, all data is stored within a single tablespace,
which in turn is stored within a single file within the MySQL
- directory structure. The single tablespace stores all record
+ directory structure. A single Falcon database will create three main
+ files. One file contains the Falcon data and will be stored in a file with
+ the name of the Falcon database with the extension
+ <filename>.fts</filename>. For example, Falcon tables defined within
+ the database <literal>test</literal> will be stored within the file
+ <filename>test.fts</filename> within the main MySQL data directory.</para>
+
+ <para>Two further files contain the on-disk copy of the Falcon serial
+ log. These are also created within the realm of the corresponding
+ database. So with the above example datafile
+ <filename>test.fts</filename> the log files will be named
+ <filename>test.fl1</filename> and <filename>test.fl2</filename>.</para>
+
+ <para>Table definitions are, as with with the other MySQL engines,
+ stored within a <literal>.frm</literal> file within a database
+ specific directory. For example, the table
+ <literal>falcontest</literal> within the <literal>test</literal>
+ database will create the table definition file
+ <filename>falcontest.frm</filename>.</para>
+
+ <para>A single Falcon database file stores all record
data, indexes, database structure and other information. The
individual information is stored within a series of pages.
</para>
-
+
+
<para>
Pages describe the internal storage allocation block within
the Falcon storage engine. Pages are used to store data and
@@ -187,7 +407,7 @@
</section>
- <section id="falcon-principles-logging">
+ <section id="se-falcon-principles-logging">
<title>Falcon Serial Log</title>
@@ -272,7 +492,7 @@
as the first log file is full or becomes locked for commits.
</para>
- <section id="falcon-principles-logging-rollback">
+ <section id="se-falcon-principles-logging-rollback">
<title>Rollback Process</title>
@@ -312,7 +532,7 @@
</section>
- <section id="falcon-principles-logging-groupcommit">
+ <section id="se-falcon-principles-logging-groupcommit">
<title>Group Commits</title>
@@ -415,7 +635,7 @@
</section>
- <section id="falcon-principles-recovery">
+ <section id="se-falcon-principles-recovery">
<title>Falcon Crash Recovery</title>
@@ -440,8 +660,67 @@
</section>
- <section id="falcon-principles-compression">
+ <section id="se-falcon-principles-caches">
+ <title>Falcon Memory Caches</title>
+
+ <para>Falcon was designed to perform best on systems with generous amounts of memory. The memory
+ caches utilized by Falcon are similar in some respects with other RDBMS’s and MySQL engines;
+ however, the cache structures offer a number of improvements over traditional memory caching
+ strategies. The mechanisms used by Falcon with respect to memory caching include:</para>
+
+ <itemizedlist> <listitem><para><emphasis role="bold">Log Cache</emphasis> — log information is kept in memory and flushed to the Falcon Log when
+ transactions commit. Falcon keeps eight windows into the log file for reading and writing, and
+ each window is 1MB. </para> </listitem>
+ <listitem> <para><emphasis role="bold">System and Index Cache</emphasis> — data needed by Falcon (table and field definitions, transaction
+ state, etc.) are also maintained in memory for quick reference. In addition, local index
+ accelerators represent index segments created by a running transaction are also stored in the
+ system memory. When a transaction changes indexed fields, it builds an index accelerator
+ section in system memory, representing its changes. On commit, all index changes for the
+ transaction are written to the serial log in sorted order and later merged with the permanent
+ index by the worker thread. </para> </listitem>
+ <listitem> <para><emphasis role="bold">Page Cache</emphasis> — database pages read from disk for a particular database. The page cache
+ size is controlled by the <literal>falcon_page_cache_size</literal> parameter, the default of which is 4MB,
+ and is set in the my.cnf file. Although record and index changes go to the serial log before
+ being written to database pages, blob data is written directly into the page cache. This avoids
+ logging large data items that are rarely referenced or changed by the transaction that creates
+ them.</para> </listitem>
+ <listitem> <para><emphasis role="bold">Record Cache</emphasis> — the record cache is a memory region devoted to holding rows that have
+ been requested by end-user queries for a particular database or created by active
+ transactions. Note that this cache differs from traditional data caches in that only specific
+ rows needed by applications reside in the cache as opposed to entire data pages (which may
+ contain only subsets of needed information). The record cache can hold several versions of
+ records that have been modified or deleted. This technique guarantees that active data
+ needed to satisfy user requests is in memory, shortens row access time, and reduces cache
+ bloat by not including un-requested information. The record cache also assists in supporting
+ the multi-version concurrency control (MVCC) mechanisms of the Falcon engine. The record
+ cache is controlled by two parameters. The <literal>falcon_min_record_memory</literal> parameter
+ (default 10MB) determines the minimum amount of RAM supplied to the record cache, and
+ the <literal>falcon_max_record_memory</literal> (default 20MB) limits the total amount of memory
+ available to the cache.</para> </listitem>
+
+ <listitem><para>Because of the support the record cache supplies to transactions, a scavenge thread is used to
+ ensure only “hot” data resides in the cache. When the <literal>falcon_max_record_memory</literal> limit is
+ reached, Falcon surveys the demographics of the generational data in the cache, and removes the
+ oldest generations. This process is more complicated than the standard LRU algorithm used by many
+ database systems, but it is more efficient and faster.</para> </listitem></itemizedlist>
+
+ </section>
+
+ <section id="se-falcon-principles-threads">
+ <title>Falcon Threads</title>
+
+ <para>Falcon uses two worker threads to process information within the
+ Falcon structures. One thread, the "gopher" thread, is devoted to
+ moving committed data changes from the Falcon log to data pages and to
+ merge index changes with permanent index data. The second thread handles
+ the periodic flushing of the page cache and scavenges space allocated
+ within the record cache. </para>
+
+ </section>
+
+ <section id="se-falcon-principles-compression">
+
<title>Data Compression</title>
<para>
@@ -452,7 +731,7 @@
</section>
- <section id="falcon-principles-recordslot">
+ <section id="se-falcon-principles-recordslot">
<title>Record Slot</title>
@@ -467,213 +746,10 @@
</section>
- <section id="falcon-principles-pagecache">
-
- <title>Page Cache</title>
-
- <para>
- Falcon stores data in pages, and pages can contain a variety
- of information, including record data, index data and table
- schemas. Pages are recorded on disk and a cache of the most
- recently used or accessed pages are also kept in memory. Note
- that the in-memory page cache only contains basic record data;
- blog information is stored only on disk, or in the serial log.
- </para>
-
- <para>
- The page cache is only flushed during the checkpoint process
- (approximately every 30 seconds). Changes to index and blob
- information occurs during the serial log commital process.
- </para>
-
- </section>
-
</section>
+
+ <section id="se-falcon-limits">
- <section id="falcon-configuration">
-
- <title>Configuration Parameters</title>
-
-<!-- TODO: [MC] Current falcon.conf will change in the future -->
-
- <para>
- Parameters are configured through the
- <filename>falcon.conf</filename> file. Parameters can be
- configured by specifying the parameter name and the
- corresponding value, separated by a space. Memory values can be
- specified in bytes, or with a number followed by
- <literal>kb</literal>, <literal>mb</literal> or
- <literal>gb</literal>.
- </para>
-
-<!-- <para>Configuration parameters within the Falcon database are
- configurable for each individual tablespace. The
- <literal>falcon_page_size</literal> parameter can only be
- configured when the tablespace is created and cannot be changed.
- The remaining paramters can be set and altered dynamically while
- <literal>mysqld</literal> is running by setting the
- appropriate system variable.</para>-->
-
-<!-- TODO: [MC] These literals are based on the current wording of
- the configurable parameters in falcon.conf.
- They will almost certainly change in the future. -->
-
- <itemizedlist>
-
- <listitem>
- <para>
- <literal>falcon_page_size</literal> (Page Size) controls the
- size of the pages used to store information within the
- tablespace. Valid sizes are 1, 2, 4, 8, 16 and 32 KB.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>falcon_min_record_memory</literal> (Record Cache Base)
- sets the minimum amount of memory that will be allocated for
- caching record data. When cache memory is scavenged, the process
- will stop when the cache usage reaches this value. The default is
- <literal>falcon_max_record_memory</literal>/2 (10MB).
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>falcon_max_record_memory</literal> (Record Cache Top) sets
- the maximum size of memory that will be allocated for
- caching record data. The default is 20MB.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>falcon_page_cache_size</literal> (Page Cache Size) sets
- the amount of memory that will be allocated for caching
- pages from the tablespace file. The default is 4MB.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>maximum_file_size</literal> (Maximum File Size)
- sets the maximum size of the tablespace file used to hold
- record data. You can use this parameter to limit the amount
- of space that will automatically be used by the storage
- engine for storing information.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>logfile_location</literal> (Log file location) sets
- the directory for storing the serial log. The filenames used
- by the serial log (two files are create for storing serial
- data) are allocated according to the name of the tablespace.
- The default location is in the same directory as the
- tablespace file.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>log_file_memory_use</literal> (Log file memory use)
- configures the amount of memory allocated to providing
- access to the physical logfile stored on disk. Currently the
- log file memory is fixed at 8MB.
- </para>
- </listitem>
-
- </itemizedlist>
-
- <para>
- The relationship between the record cache and the page cache is
- driven by the information that is cached by each system. Whole
- records that are in active use (being read or updated) are
- stored within the record cache, however, <literal>BLOB</literal>
- data is stored only within the page cache.
- </para>
-
- <para>
- The page cache is used to store database metadata,
- <literal>BLOB</literal> data and table indexes.
- </para>
-
- <para>
- Falcon parameters can be also be set on the command-line to
- <command>mysqld</command> using the following command-line
- options:
- </para>
-
- <itemizedlist>
-
- <listitem>
- <para>
- <literal>--falcon-max-record-memory=#</literal>
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>--falcon-min-record-memory=#</literal>
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>--falcon-page-cache-size=#</literal>
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>--falcon-debug-server</literal>
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>--falcon-page-size=#</literal>
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>--falcon-log-dir=<replaceable>name</replaceable></literal>
- </para>
- </listitem>
-
- </itemizedlist>
-
- <para>
- You can also enable and disable the Falcon storage engine at
- startup by supplying these options to <command>mysqld</command>,
- providing that the <literal>mysqld</literal> binary includes the
- Falcon Storage Engine.
- </para>
-
- <itemizedlist>
-
- <listitem>
- <para>
- <literal>--falcon</literal> enables the Falcon storage
- engine.
- </para>
- </listitem>
-
- <listitem>
- <para>
- <literal>--skip-falcon</literal> disables the Falcon Storage
- Engine.
- </para>
- </listitem>
-
- </itemizedlist>
-
- </section>
-
- <section id="falcon-limits">
-
<title>Limits</title>
<para>
@@ -746,7 +822,7 @@
</section>
-<!--> <section id="falcon-roadmap">
+<!--> <section id="se-falcon-roadmap">
<title>Falcon Roadmap</title>
<para>Falcon will have...</para>
</section>-->
| Thread |
|---|
| • svn commit - mysqldoc@docsrva: r3881 - trunk/refman-5.1 | mcbrown | 8 Nov |
