List:Falcon Storage Engine« Previous MessageNext Message »
From:Ann W. Harrison Date:June 8 2009 7:25pm
Subject:Re: Worklog #4284 and on-line DDL
View as plain text  
Kostja,
>> ...
>>    Adding a column on-line should not block concurrent access to the
>> table, including inserts, updates, and deletes.
>>
>>    Because of the way Falcon handles table descriptions, dropping
>> and altering a column can be done without blocking DML operations.
>> (Tables have formats.  Records carry the version number of the
>> format under which they were created.  Requests include the format
>> version they expect.  Falcon converts between formats on the fly
>> in memory.  Records on disk change format when they are updated.
...
> 
> No, at the moment there is no provision for a truly online DDL
> operations in the server, WL#4284 doesn't change that, yet it
> doesn't make matters worse.

Actually, for Falcon, it does.
> 
> The way ALTER TABLE works today is:
> 
> 1) Get a shared metadata lock on the table.
> 2) Create a copy table that contains the new table definition.
> Copy data between tables.
> 3) Upgrade the shared metadata lock to an exclusive one.
> 4) Replace the old table with a copy.
> 
> The only difference after WL#4284 is that step 4 waits for all
> pending transactions against the subject table to complete.

Which is completely unnecessary in the case of Falcon.  Adding a
column to a table doesn't change the table at all - just Falcon's
internal description of it.  After the change, Falcon will construct
MySQL records in the new format for new transactions.

> No new transactions can start using the table while step 4 is in
> progress -- they block and wait till step 4 to complete.
> 
> Prior to WL#4284, step 4 would ignore active transactions.

Which is fine with Falcon.  Active transactions continue to get
records in their original format.  New transactions get records
in the new format.  Everybody runs (or could run) happily in
parallel.

> 
> In order to achieve a truly online ALTER TABLE, one needs to add
> support for multi-versioning on the server layer, for table
> definitions. With multi-versioning, we won't have to take an
> exclusive lock at step 4, it will be sufficient to make sure that
> old transactions continue to use the old definition of the table. 
> 

If the engine handles the transformation, MySQL doesn't need to
have a multi-generational record format.  For what little it's worth,
here's a mid-level description of how it all works:

1) Falcon has descriptive system tables that represent tables,
fields, and fields in tables.  These are not multi-generational.
The tables contain all the normal stuff - names, data types, etc.
The 'fields in tables' table has a value for position and values
for field identifier and table identifier.  The position is the
logical value for the ordinal position of the field - where the
user expects to see it in a SELECT * ...  The field identifier
is the physical value for the position of the field in the
stored record.

2) The table identifier and field identifier are two of the three
parts of the unique key of a system table called 'formats'.  The
format record is a bare-bones description of the table format.
Every table has at least one format record.  Tables that have
been altered to add, change, or drop field have multiple format
records.

3) Every stored record has a format version as part of its header.

4) When a transaction references a table, it gets the format number
then current.  It will always see records in that format.  If the
engine finds a record that has a newer or older format, the record
is converted, if necessary through several steps.

There are some fine points - for example, dropping a field requires
that all transactions that have the old format complete before
any transaction can modify a record in the new format - otherwise
data will disappear from a transaction's visibility.  Another
issue is what changes to allow e.g. shrinking fields then growing
them back - do you give an error if the old value fits in the
final format, or not?  Or just don't allows fields to shrink?


Regards,

Ann




Thread
Re: Worklog #4284 and on-line DDLAnn W. Harrison8 Jun