From: Jon Olav Hauglid
Date: February 1 2012 11:52am
Subject: bzr push into mysql-trunk-wl5534-stage branch (jon.hauglid:3811 to 3812)
 WL#5534
List-Archive: http://lists.mysql.com/commits/142693
Message-Id: <201202011152.q11Bq9Gh018620@acsmt357.oracle.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit

 3812 Jon Olav Hauglid	2012-02-01
      WL#5534 Online ALTER, Phase 1
      
      This worklog extends the handler API to add general support
      for online table changes and rewrites the ALTER TABLE
      implementation to it. The handler API extensions are based
      on Cluster's online ALTER implementation and therefore also
      narrows the gap between the Server and Cluster codebases.
      
      Using this new API, the server supports concurrent reads
      and writes during the main phase of in-place ALTER TABLE
      operations. In this first phase the focus is on index
      creation, but the API is general enough to be extended to
      other ALTER TABLE operations.
      
      The ALTER TABLE syntax is extended to allow users to specify
      concurrency requirements as well as select between the in-place
      and copy algorithms.
      
      Execution of an ALTER TABLE statement has three phases:
      1) Initialization
         In this phase the algorithm and locking level is determined.
         This is done based on requirements from the user, capabilities
         of the given storage engine and characteristics of the
         ALTER TABLE operation.
         The new handler API calls for this phase has a default
         implementation that converts them to the old API calls so
         existing engines can be used without changes.
      
      2) Execution
         In this phase, the table definition is changed. This is done
         either directly on the current table (i.e. in-place) or using
         a temporary table copy. The new API calls only concern the
         the former. In this patch, a compatibility layer converts the
         new API calls for in-place execution to the old API calls for
         index creation/deletion. However these implementations will be
         removed later once the InnoDB changes that makes use of the
         new API is in place.
      
      3) Finalization
         The final phase, updates the data dictionary, installs the
         new .FRM and notifies the storage engine.
      
      The metadata subsystem is extended with a new lock type,
      MDL_SHARED_UPGRADABLE, which is used for phases when ALTER TABLE
      allows concurrent reads and writes.
      
      The patch also updates the partitioning implementation to use the
      new handler API functions.
     @ include/mysql_com.h
        Added deprecation comment for old flag.
     @ mysql-test/suite/rpl/t/rpl_slave_grp_exec.test
        Added missing sync_slave_with_master which caused
        problems made visible by WL#5534.
     @ mysql-test/t/alter_table.test
        Added single connection regression test for WL#5534
     @ mysql-test/t/flush_read_lock.test
        ALTER TABLE on temporary transactional tables is now
        compatible with FTWRL - updated test.
     @ mysql-test/t/innodb_mysql_lock.test
        Added regression test for Bug#11750045 fixed by WL#5534.
     @ mysql-test/t/innodb_mysql_sync.test
        Updated existing tests with changed debug sync points.
        Added multi connection tests for WL#5534.
     @ mysql-test/t/lock.test
        Added MERGE related test coverage for MDL-related regression
        introduced during WL#5534 development.
     @ mysql-test/t/mdl_sync.test
        Updated existing tests with changed debug sync points.
        Updated existing tests with changed ALTER TABLE locking behavior.
        Added test coverage for new MDL_SHARED_UPGRADABLE lock.
     @ mysql-test/t/partition_debug_sync.test
        Updated existing tests with changed debug sync points.
     @ mysql-test/t/truncate_coverage.test
        Updated existing tests with changed debug sync points.
     @ sql/ha_partition.cc
        Replaced old online ALTER API implemenation with
        implementation of new online ALTER API.
     @ sql/ha_partition.h
        Replaced old online ALTER API implemenation with
        implementation of new online ALTER API.
     @ sql/handler.cc
        handler::print_keydup_error() now takes KEY rather than
        key number as parameter, to account for situations where
        the key number is not known.
        Added compatibility layer implementations of new handler
        API functions for online ALTER TABLE execution. This allows
        storage engines to be used without changes. Note that parts
        of the compatibility layer that implements execution of 
        in-place add/drop index will be removed once InnoDB has been
        updated to use the new API functions.
     @ sql/handler.h
        Added new handler API calls for implementing online ALTER
        TABLE operations.
        Added Alter_inplace_info class which describes the changes
        to be done by ALTER TABLE. It is used as a paramenter for
        the new API calls.
        Marked parts of the old API for online add/drop index as
        deprected.
        Added public wrappers for new handler API functions to allow
        for enforcing asserts etc. regardless for handler implementations
        of the new API functions.
     @ sql/key.cc
        key_unpack() now takes KEY rather than key number as
        parameter, to account for situations where the key number
        is not known.
     @ sql/key.h
        key_unpack() now takes KEY rather than key number as
        parameter, to account for situations where the key number
        is not known.
     @ sql/mdl.cc
        Added new lock type MDL_SHARED_UPGRADABLE. It's an upgradable
        shared metadata lock that allows concurrent reads and writes
        to table data. The lock is not compatible with itself.
        It is used by the first phase of ALTER TABLE and during
        execution of ALTER TABLE operations if supported by
        the storage engine.
        Updates to MDL lock type compatibility matrices.
        Minor adjstments to the API to account for this new lock type.
     @ sql/mdl.h
        Added new lock type MDL_SHARED_UPGRADABLE. It's an upgradable
        shared metadata lock that allows concurrent reads and writes
        to table data. The lock is not compatible with itself.
        It is used by the first phase of ALTER TABLE and during
        execution of ALTER TABLE operations if supported by
        the storage engine.
        Minor adjstments to the API to account for this new lock type.
     @ sql/share/errmsg-utf8.txt
        Add new error messages ER_UNKNOWN_ALTER_ALGORITHM
        and ER_UNKNOWN_ALTER_LOCK used during parsing of
        ALTER TABLE syntax extension.
     @ sql/sql_admin.cc
        Updated with ALTER TABLE command flags moving to Alter_info class.
     @ sql/sql_alter.cc
        Moved Alter_info class here.
        Moved ALTER TABLE command flags to Alter_info.
        Updated and documented the individual flags.
        Added support for setting requested levels of concurrency
        and algorithm from new ALTER TABLE syntax.
        Added new ALter_table_ctx class for maintaining runtime
        context during execution of ALTER TABLE statements.
        Added new Sql_cmd class for IMPORT/DISCARD TABLESPACE.
     @ sql/sql_alter.h
        Moved Alter_info class here.
        Moved ALTER TABLE command flags to Alter_info.
        Updated and documented the individual flags.
        Added support for setting requested levels of concurrency
        and algorithm from new ALTER TABLE syntax.
        Added new ALter_table_ctx class for maintaining runtime
        context during execution of ALTER TABLE statements.
        Added new Sql_cmd class for IMPORT/DISCARD TABLESPACE.
     @ sql/sql_base.cc
        Updated with changes to MDL API and introduction of
        MDL_SHARED_UPGRADABLE lock.
        rm_temporary_table() now takes path parameter without extension.
     @ sql/sql_base.h
        open_table_uncached() now takes additional parameter telling
        if table should be opened by storage engine.
        Removed unused Alter_info member from
        Alter_table_prelocking_strategy.
     @ sql/sql_class.h
        enum_enable_or_disable moved to Alter_info.
        open_table_uncached() now takes additional parameter telling
        if table should be opened by storage engine.
     @ sql/sql_insert.cc
        Adjusted parameters to mysql_create_table_no_lock()
        and quick_rm_table().
     @ sql/sql_lex.cc
        Alter_info moved to sql_alter.
     @ sql/sql_lex.h
        Alter_info and ALTER_* flags moved to sql_alter.
        Alter_table_change_level no longer needed, removed.
     @ sql/sql_partition.cc
        Adjusted parameters for prep_alter_part_table(), ALTER TABLE
        context is now stored in Alter_table_ctx rather than passed
        separately.
        Updated with changes to MDL API.
        open_table_uncached() now takes additional parameter telling
        if table should be opened by storage engine.
     @ sql/sql_partition.h
        Adjusted parameters for prep_alter_part_table(), ALTER TABLE
        context is now stored in Alter_table_ctx rather than passed
        separately.
     @ sql/sql_partition_admin.cc
        Adjusted for changes to mysql_compare_tables()
        Updated with changes to MDL API
     @ sql/sql_table.cc
        build_tmptable_filename() now returns a path to .FRM
        without extension.
        Simplified mysql_compare_tables() since it's now only
        used by partitioning code.
        Made mysql_discard_or_import_tablespace() a global function
        since it's now called by a separate Sql_cmd.
        quick_rm_table() now takes a the new NO_HA_TABLE flag
        as parameter for when the table should not be removed in
        the storage engine.
        mysql_create_table_no_lock() is now a wrapper to
        create_table_impl() which does the real work.
        create_table_impl() now returns info about KEY objects
        in the table create, which prevents redundant calls to
        mysql_prepare_create_table().
        mysql_rename_table() now takes NO_HA_TABLE flag if table
        should not be renamed in engine.
        Added fill_alter_inplace_info() which figures out which
        operations are to be done by ALTER TABLE and initializes
        structures to be used during execution.
        Added is_inplace_alter_impossible() which determines
        if the characteristics for the pending ALTER TABLE operations
        are incompatible with in-place execution.
        Added mysql_inplace_alter_table() for execution of 
        in-place ALTER TABLE operations.
        Extracted simple_rename_or_index_change() from existing
        ALTER TABLE operation.
        Rewrite of mysql_alter_table():
        - No thr_lock.c lock is now taken when the table is
          first opened. It is taken later once the metadata
          lock has been upgraded. This avoids deadlock scenarios.
        - Name handling moved to new Alter_table_ctx class.
        - Simplified control flow (e.g. reduced number of gotos).
        - Added handling of new LOCK and ALGORITHM syntax.
        - Extracted code to new simple_rename_or_index_change() function.
        - In-place code generalized and extracted to new 
          mysql_inplace_alter_table() function.
        - For in-place ALTER we now open a TABLE instance from
          the .FRM created for the new version of the table and
          use this TABLE instance of in-place handler API calls.
        - Errors during clean-up phase (e.g. deletion of temporary
          files) are no longer ignored.
     @ sql/sql_table.h
        Made build_tmptable_filename() a global function.
        Made mysql_discard_or_import_tablespace() a global function
        since it's now called by a separate Sql_cmd.
        Added NO_HA_TABLE flag.
        Simplified mysql_compare_tables() since it's now only
        used by partitioning code.
        Adjusted quick_rm_table() parameters.
     @ sql/sql_trigger.cc
        Updated with changes to MDL API.
     @ sql/sql_truncate.cc
        open_table_uncached() now takes additional parameter telling
        if table should be opened by storage engine.
        Updated with changes to MDL API.
     @ sql/sql_yacc.yy
        Added LOCK and ALGORITHM clauses to ALTER TABLE statement.
        Added separate Sql_cmd for ALTER TABLE IMPORT/DISCARD TABLESPACE.
        Changed MDL lock type used for ALTER TABLE to MDL_SHARED_UPGRADABLE.
     @ sql/unireg.cc
        Added no_ha_table flag to rea_create_table() indicating that
        only .FRM(/.PAR) needs to be created. Used by ALTER TABLE.
     @ sql/unireg.h
        Added no_ha_table flag to rea_create_table() indicating that
        only .FRM(/.PAR) needs to be created. Used by ALTER TABLE.
     @ storage/innobase/handler/ha_innodb.cc
        Fixed regression found during WL#5534 development.
        Patch from Marko.
     @ storage/innobase/handler/ha_innodb.h
        Updated with changes in handler API.
     @ storage/innobase/handler/handler0alter.cc
        handler_add_index class renamed inplace_alter_handler_ctx.
        Updated with changes in handler API.
     @ storage/myisammrg/ha_myisammrg.cc
        ALTER TABLE now takes MDL_SHARED_UPGRADABLE lock when
        opening the table to be altered.
     @ unittest/gunit/mdl-t.cc
        Updated tests with changes to MDL API and introduction
        of MDL_SHARED_UPGRADABLE lock.
     @ unittest/gunit/mdl_mytap-t.cc
        Updated tests with changes to MDL API and introduction
        of MDL_SHARED_UPGRADABLE lock.

    modified:
      include/mysql_com.h
      mysql-test/include/commit.inc
      mysql-test/r/alter_table.result
      mysql-test/r/archive.result
      mysql-test/r/commit_1innodb.result
      mysql-test/r/ctype_utf8mb4.result
      mysql-test/r/flush_read_lock.result
      mysql-test/r/innodb_mysql_lock.result
      mysql-test/r/innodb_mysql_sync.result
      mysql-test/r/lock.result
      mysql-test/r/mdl_sync.result
      mysql-test/r/partition_debug_sync.result
      mysql-test/r/truncate_coverage.result
      mysql-test/suite/innodb/r/innodb-index.result
      mysql-test/suite/innodb/r/innodb_index_large_prefix.result
      mysql-test/suite/innodb/r/innodb_index_large_prefix_4k.result
      mysql-test/suite/innodb/r/innodb_index_large_prefix_8k.result
      mysql-test/suite/innodb/r/innodb_mysql.result
      mysql-test/suite/innodb/r/innodb_prefix_index_liftedlimit.result
      mysql-test/suite/perfschema/r/stage_mdl_table.result
      mysql-test/suite/rpl/t/rpl_slave_grp_exec.test
      mysql-test/t/alter_table.test
      mysql-test/t/flush_read_lock.test
      mysql-test/t/innodb_mysql_lock.test
      mysql-test/t/innodb_mysql_sync.test
      mysql-test/t/lock.test
      mysql-test/t/mdl_sync.test
      mysql-test/t/partition_debug_sync.test
      mysql-test/t/truncate_coverage.test
      sql/ha_partition.cc
      sql/ha_partition.h
      sql/handler.cc
      sql/handler.h
      sql/key.cc
      sql/key.h
      sql/mdl.cc
      sql/mdl.h
      sql/share/errmsg-utf8.txt
      sql/sql_admin.cc
      sql/sql_alter.cc
      sql/sql_alter.h
      sql/sql_base.cc
      sql/sql_base.h
      sql/sql_class.h
      sql/sql_insert.cc
      sql/sql_lex.cc
      sql/sql_lex.h
      sql/sql_partition.cc
      sql/sql_partition.h
      sql/sql_partition_admin.cc
      sql/sql_table.cc
      sql/sql_table.h
      sql/sql_trigger.cc
      sql/sql_truncate.cc
      sql/sql_yacc.yy
      sql/unireg.cc
      sql/unireg.h
      storage/innobase/handler/ha_innodb.cc
      storage/innobase/handler/ha_innodb.h
      storage/innobase/handler/handler0alter.cc
      storage/myisammrg/ha_myisammrg.cc
      unittest/gunit/mdl-t.cc
      unittest/gunit/mdl_mytap-t.cc
 3811 Jon Olav Hauglid	2012-01-31
      Change tree name.

    modified:
      .bzr-mysql/default.conf

Diff too large for email (11162 lines, the limit is 10000).
No bundle (reason: useless for push emails).