List:Commits« Previous MessageNext Message »
From:Andrei Elkin Date:March 30 2012 11:17am
Subject:bzr push into mysql-trunk branch (andrei.elkin:3824 to 3825) Bug#13893310
View as plain text  
 3825 Andrei Elkin	2012-03-30
      BUG#13893310 checkpoint_group size wrong at recovery after cold restart
      
      The issue with checkpoint_group at MTS recovery is that after the server restart
      MTS recovery gaps collecting algorithm initialized the recovery bitmap with the default 512 size 
      rather than with a correct one with size of not less than of Worker group_executed of the last 
      slave session.
      
      That is corrected. The max possible size is used in the gaps collecting.
      opt_mts_checkpoint_group 's update step is made as 8 (bits).
      Some refactoring is done as well.
      
      Also MTS recovery gaps collecting is deployed on a common to 
      START-SLAVE and --skip-start-slave=0 execution path.
     @ mysql-test/suite/rpl/r/rpl_mts_debug.result
        results updated.
     @ mysql-test/suite/rpl/t/rpl_mts_debug.test
        cleanup: making the test to looking uniformally (@save).
     @ sql/rpl_info_dummy.cc
        Rpl handler interface is extended with reset_info().
     @ sql/rpl_info_dummy.h
        Rpl handler interface is extended with reset_info().
     @ sql/rpl_info_factory.cc
        submerging worker->init_info() into create_worker().
        Signature of both changed to propagate more of the caller context such
        as recovery mode.
     @ sql/rpl_info_factory.h
        signature is changed.
     @ sql/rpl_info_file.cc
        Rpl handler interface is extended with reset_info().
     @ sql/rpl_info_file.h
        Rpl handler interface is extended with reset_info().
     @ sql/rpl_info_handler.h
        Rpl handler interface is extended with reset_info().
     @ sql/rpl_info_table.cc
        Rpl handler interface is extended with reset_info().
     @ sql/rpl_info_table.h
        Rpl handler interface is extended with reset_info().
     @ sql/rpl_rli.cc
        recovery_groups bitmap life time is defined depending whether the previous parallel slave
        session left gaps. It is initialized in mts_recovery_group() if that's the case;
        making mts_recovery_groups() to be called in a common for START-SLAVE and init_slave()
        execution path;
        Relay_log_info::reset_workers_recovered() is added to be called at the end of
        mts-recovery execution phase (aka gaps filling).
     @ sql/rpl_rli.h
        a new flag is added to indicate recovery_groups is initialized;
        reset_workers_recovered() is added.
     @ sql/rpl_rli_pdb.cc
        Relocating bitmap_init() for Worker::group_* bitmaps into
        Worker::init_info();
        Two possible value for the size depends on the last argument of
        create_worker();
        In the gaps collecting phase of recovery the size must be set to the max
        possible value;
        Slave_worker::reset_info() is added for RLI::reset_workers_recovered().
     @ sql/rpl_rli_pdb.h
        Worker::init_info() signature accepts a new argument;
        reset_info() to the set of methods is added.
     @ sql/rpl_slave.cc
        making mts_recovery_groups() to be called in a common for START-SLAVE and init_slave()
        execution path, notice Change-Master forces to have it in a separate invocation point 
        in RLI::init_info();
        changing mts_recovery_groups() to corresponds to error code convention for func:s
        called by its caller;
        initialization of rli->recovery_groups bitmap to the max size of the Worker group_executed;
        a little refinement of asserts and logics in mts_recovery_groups() is done;
        fixing too early (so incorrect) updating for rli->recovery_parallel_workers in the error branch
        of slave_start_workers();
        Cleanup for rli->recovery_groups at the end of the Coordinator (SQL) thread to complete
        its life time;
        remove_workers() is replaced by Rpl_info_factory::reset_workers();
        deploying RLI::reset_workers_recovered() at two points corresponding to
        the end of mts-recovery.
     @ sql/rpl_slave.h
        changing signature for mts_recovery_groups().
     @ sql/share/errmsg-utf8.txt
        A new error message is added to correspond to an error branch of impossible MTS recovery
        at time of START SLAVE.
     @ sql/sys_vars.cc
        making mts_checkpoint_group to vary by 8 bits to correspond
        to the Worker info table schema that contains the number of bytes (not bits)
        field.

    modified:
      mysql-test/suite/rpl/r/rpl_mts_debug.result
      mysql-test/suite/rpl/t/rpl_mts_debug.test
      sql/rpl_info_dummy.cc
      sql/rpl_info_dummy.h
      sql/rpl_info_factory.cc
      sql/rpl_info_factory.h
      sql/rpl_info_file.cc
      sql/rpl_info_file.h
      sql/rpl_info_handler.h
      sql/rpl_info_table.cc
      sql/rpl_info_table.h
      sql/rpl_rli.cc
      sql/rpl_rli.h
      sql/rpl_rli_pdb.cc
      sql/rpl_rli_pdb.h
      sql/rpl_slave.cc
      sql/rpl_slave.h
      sql/share/errmsg-utf8.txt
      sql/sys_vars.cc
 3824 Joerg Bruehe	2012-03-30 [merge]
      Null upmerge of a version number change from 5.5 to 5.6

=== modified file 'mysql-test/suite/rpl/r/rpl_mts_debug.result'
--- a/mysql-test/suite/rpl/r/rpl_mts_debug.result	2012-03-23 20:11:19 +0000
+++ b/mysql-test/suite/rpl/r/rpl_mts_debug.result	2012-03-30 11:15:17 +0000
@@ -41,6 +41,9 @@ include/stop_slave.inc
 call mtr.add_suppression("option 'slave_checkpoint_group': unsigned value 524281 adjusted to 524280");
 call mtr.add_suppression("Failed during slave worker thread create");
 call mtr.add_suppression("Slave SQL: Failed during slave workers initialization, Error_code: 1593");
+call mtr.add_suppression("Mismatch between the number of bytes configured to store checkpoint information and the previously stored information");
+set @save.slave_checkpoint_group= @@global.slave_checkpoint_group;
+set @save.slave_parallel_workers= @@global.slave_parallel_workers;
 SET GLOBAL slave_parallel_workers= 2;
 SET GLOBAL slave_checkpoint_group=524281;
 Warnings:
@@ -54,8 +57,8 @@ SET GLOBAL debug= "d,inject_init_worker_
 START SLAVE SQL_THREAD;
 include/wait_for_slave_sql_error.inc [errno=1593]
 SET GLOBAL debug="";
-SET GLOBAL slave_checkpoint_group= 512;
-SET GLOBAL slave_parallel_workers= 0;
+set @@global.slave_checkpoint_group= @save.slave_checkpoint_group;
+set @@global.slave_parallel_workers= @save.slave_parallel_workers;
 include/start_slave.inc
 include/rpl_reset.inc
 include/rpl_end.inc

=== modified file 'mysql-test/suite/rpl/t/rpl_mts_debug.test'
--- a/mysql-test/suite/rpl/t/rpl_mts_debug.test	2012-03-05 16:35:12 +0000
+++ b/mysql-test/suite/rpl/t/rpl_mts_debug.test	2012-03-30 11:15:17 +0000
@@ -106,9 +106,10 @@ source include/start_slave.inc;
 call mtr.add_suppression("option 'slave_checkpoint_group': unsigned value 524281 adjusted to 524280");
 call mtr.add_suppression("Failed during slave worker thread create");
 call mtr.add_suppression("Slave SQL: Failed during slave workers initialization, Error_code: 1593");
+call mtr.add_suppression("Mismatch between the number of bytes configured to store checkpoint information and the previously stored information");
 
---let $saved_slave_checkpoint_group= `SELECT @@global.slave_checkpoint_group`
---let $saved_slave_parallel_workers= `SELECT @@global.slave_parallel_workers`
+set @save.slave_checkpoint_group= @@global.slave_checkpoint_group;
+set @save.slave_parallel_workers= @@global.slave_parallel_workers;
 SET GLOBAL slave_parallel_workers= 2;
 SET GLOBAL slave_checkpoint_group=524281;
 
@@ -140,8 +141,8 @@ START SLAVE SQL_THREAD;
 --eval SET GLOBAL debug="$saved_debug"
 
 ## clean up
---eval SET GLOBAL slave_checkpoint_group= $saved_slave_checkpoint_group
---eval SET GLOBAL slave_parallel_workers= $saved_slave_parallel_workers
+set @@global.slave_checkpoint_group= @save.slave_checkpoint_group;
+set @@global.slave_parallel_workers= @save.slave_parallel_workers;
 --source include/start_slave.inc
 --source include/rpl_reset.inc
 

=== modified file 'sql/rpl_info_dummy.cc'
--- a/sql/rpl_info_dummy.cc	2011-10-13 14:01:50 +0000
+++ b/sql/rpl_info_dummy.cc	2012-03-30 11:15:17 +0000
@@ -64,6 +64,12 @@ void Rpl_info_dummy::do_end_info(const u
   return;
 }
 
+int Rpl_info_dummy::do_reset_info()
+{
+  if (abort) DBUG_ASSERT(0);
+  return 0;
+}
+
 int Rpl_info_dummy::do_remove_info(const ulong *uidx __attribute__((unused)),
                                   const uint nidx __attribute__((unused)))
 {

=== modified file 'sql/rpl_info_dummy.h'
--- a/sql/rpl_info_dummy.h	2011-10-13 14:01:50 +0000
+++ b/sql/rpl_info_dummy.h	2012-03-30 11:15:17 +0000
@@ -40,6 +40,7 @@ private:
   int do_flush_info(const ulong *uidx, const uint nidx,
                     const bool force);
   int do_remove_info(const ulong *uidx, const uint nidx);
+  int do_reset_info();
 
   int do_prepare_info_for_read(const uint nidx);
   int do_prepare_info_for_write(const uint nidx);

=== modified file 'sql/rpl_info_factory.cc'
--- a/sql/rpl_info_factory.cc	2012-03-16 15:04:13 +0000
+++ b/sql/rpl_info_factory.cc	2012-03-30 11:15:17 +0000
@@ -277,8 +277,49 @@ err:
   DBUG_RETURN(TRUE);
 }
 
+bool Rpl_info_factory::reset_workers(Relay_log_info *rli, bool cold)
+{
+  Rpl_info_handler *handler_file= NULL;
+  Rpl_info_handler *handler_table= NULL;
+  char search_fname[FN_REFLEN];
+  bool error= true;
+
+  DBUG_ENTER("Rpl_info_factory::create_workers");
+
+  if (rli->recovery_parallel_workers == 0)
+    DBUG_RETURN(0);
+
+  char *pos= strmov(search_fname, relay_log_info_file);
+  strmov(pos, ".");
+
+  if (!(handler_file= new Rpl_info_file(Slave_worker::get_number_worker_fields(),
+                                        search_fname)))
+    goto err;
+
+  if (!(handler_table= new Rpl_info_table(Slave_worker::get_number_worker_fields() + 2,
+                                          MYSQL_SCHEMA_NAME.str, WORKER_INFO_NAME.str)))
+    goto err;
+
+  if (handler_file->reset_info())
+    goto err; 
+
+  if (handler_table->reset_info())
+    goto err;
+
+  error= false;
+
+err:
+  if (cold)
+    rli->recovery_parallel_workers= 0; // no mts_recovery_groups() next time
+
+  delete handler_file;
+  delete handler_table;
+  DBUG_RETURN(error);
+}
+
 Slave_worker *Rpl_info_factory::create_worker(uint worker_option, uint worker_id,
-                                              Relay_log_info *rli)
+                                              Relay_log_info *rli,
+                                              bool gaps_collecting)
 {
   char info_fname[FN_REFLEN];
   char info_name[FN_REFLEN];
@@ -326,6 +367,12 @@ Slave_worker *Rpl_info_factory::create_w
   if (decide_repository(worker, worker_option, &handler_src, &handler_dest,
                         &msg))
     goto err;
+       
+  if (worker->init_info(gaps_collecting))
+  {
+    worker->end_info();
+    goto err;
+  }
 
   DBUG_RETURN(worker);
 

=== modified file 'sql/rpl_info_factory.h'
--- a/sql/rpl_info_factory.h	2012-01-11 12:23:17 +0000
+++ b/sql/rpl_info_factory.h	2012-03-30 11:15:17 +0000
@@ -44,7 +44,8 @@ class Rpl_info_factory
   static bool change_rli_repository(Relay_log_info *mi, const uint mi_option,
                                     const char **msg);
   static Slave_worker *create_worker(uint rli_option, uint worker_id,
-                                     Relay_log_info *rli);
+                                     Relay_log_info *rli, bool gaps_collecting);
+  static bool reset_workers(Relay_log_info *rli, bool cold);
 private:
   static bool decide_repository(Rpl_info *info,
                                 uint option,

=== modified file 'sql/rpl_info_file.cc'
--- a/sql/rpl_info_file.cc	2012-01-06 20:00:48 +0000
+++ b/sql/rpl_info_file.cc	2012-03-30 11:15:17 +0000
@@ -192,6 +192,37 @@ int Rpl_info_file::do_remove_info(const 
   DBUG_RETURN(error);
 }
 
+int Rpl_info_file::do_reset_info()
+{
+  uint i= 0;
+  struct st_my_dir *dir_info= NULL;
+  struct fileinfo *file_info= NULL;
+  char dir_name[FN_REFLEN];
+  size_t dir_size= 0;
+  char* file_name= NULL;
+  size_t file_size= 0;
+  int error= FALSE;
+
+  DBUG_ENTER("Rpl_info_file::do_reset_info");
+
+  file_name= info_fname + dirname_part(dir_name, info_fname, &dir_size);
+  file_size= strlen(file_name);
+
+  if (!(dir_info= my_dir(dir_name, MYF(MY_DONT_SORT))))
+    DBUG_RETURN(TRUE);
+
+  file_info= dir_info->dir_entry;
+  for (i= dir_info->number_off_files ; i-- ; file_info++)
+  {
+    if (!strncmp(file_info->name, file_name, file_size) &&
+        my_delete(file_info->name, MYF(MY_WME)))
+      error= TRUE;
+  }
+  my_dirend(dir_info);
+
+  DBUG_RETURN(error);
+}
+
 bool Rpl_info_file::do_set_info(const int pos, const char *value)
 {
   return (my_b_printf(&info_file, "%s\n", value) > (size_t) 0 ?

=== modified file 'sql/rpl_info_file.h'
--- a/sql/rpl_info_file.h	2011-10-13 14:01:50 +0000
+++ b/sql/rpl_info_file.h	2012-03-30 11:15:17 +0000
@@ -50,6 +50,7 @@ private:
   int do_flush_info(const ulong *uidx, const uint nidx,
                     const bool force);
   int do_remove_info(const ulong *uidx, const uint nidx);
+  int do_reset_info();
 
   int do_prepare_info_for_read(const uint nidx);
   int do_prepare_info_for_write(const uint nidx);

=== modified file 'sql/rpl_info_handler.h'
--- a/sql/rpl_info_handler.h	2011-10-13 14:01:50 +0000
+++ b/sql/rpl_info_handler.h	2012-03-30 11:15:17 +0000
@@ -89,7 +89,7 @@ public:
   }
 
   /**
-    Deletes any information in the repository.
+    Deletes a specific information in the repository.
 
     @param[in] uidx Array of fields that identifies an object
     @param[in] nidx Number of fields in the array
@@ -103,6 +103,17 @@ public:
   }
 
   /**
+    Deletes all information in the repository.
+
+    @retval FALSE No error
+    @retval TRUE  Failure
+  */
+  int reset_info()
+  {
+    return do_reset_info();
+  }
+
+  /**
     Closes access to the repository.
 
     @param[in] uidx Array of fields that identifies an object
@@ -346,6 +357,7 @@ private:
   virtual int do_flush_info(const ulong *uidx, const uint nidx,
                             const bool force)= 0;
   virtual int do_remove_info(const ulong *uidx, const uint nidx)= 0;
+  virtual int do_reset_info()= 0;
   virtual void do_end_info(const ulong *uidx, const uint nidx)= 0;
   virtual int do_prepare_info_for_read(const uint nidx)= 0;
   virtual int do_prepare_info_for_write(const uint nidx)= 0;

=== modified file 'sql/rpl_info_table.cc'
--- a/sql/rpl_info_table.cc	2012-01-31 15:16:16 +0000
+++ b/sql/rpl_info_table.cc	2012-03-30 11:15:17 +0000
@@ -219,6 +219,50 @@ end:
   DBUG_RETURN(error);
 }
 
+int Rpl_info_table::do_reset_info()
+{
+  int error= 1;
+  TABLE *table= NULL;
+  ulong saved_mode;
+  Open_tables_backup backup;
+
+  DBUG_ENTER("Rpl_info_table::do_reset_info");
+
+  THD *thd= access->create_thd();
+
+  saved_mode= thd->variables.sql_mode;
+  tmp_disable_binlog(thd);
+
+  /*
+    Opens and locks the rpl_info table before accessing it.
+  */
+  if (access->open_table(thd, str_schema, str_table,
+                         get_number_info(), TL_WRITE,
+                         &table, &backup))
+    goto end;
+
+  /*
+    Deletes a row in the rpl_info table.
+  */
+  if ((error= table->file->truncate()))
+  {
+     table->file->print_error(error, MYF(0));
+     goto end;
+  }
+
+  error= 0;
+
+end:
+  /*
+    Unlocks and closes the rpl_info table.
+  */
+  access->close_table(thd, table, &backup, error);
+  reenable_binlog(thd);
+  thd->variables.sql_mode= saved_mode;
+  access->drop_thd(thd);
+  DBUG_RETURN(error);
+}
+
 int Rpl_info_table::do_remove_info(const ulong *uidx, const uint nidx)
 {
   int error= 1;

=== modified file 'sql/rpl_info_table.h'
--- a/sql/rpl_info_table.h	2011-10-13 14:01:50 +0000
+++ b/sql/rpl_info_table.h	2012-03-30 11:15:17 +0000
@@ -65,6 +65,7 @@ private:
   int do_flush_info(const ulong *uidx, const uint nidx,
                     const bool force);
   int do_remove_info(const ulong *uidx, const uint nidx);
+  int do_reset_info();
 
   int do_prepare_info_for_read(const uint nidx);
   int do_prepare_info_for_write(const uint nidx);

=== modified file 'sql/rpl_rli.cc'
--- a/sql/rpl_rli.cc	2012-03-28 18:01:14 +0000
+++ b/sql/rpl_rli.cc	2012-03-30 11:15:17 +0000
@@ -89,7 +89,8 @@ Relay_log_info::Relay_log_info(bool is_s
    rows_query_ev(NULL), last_event_start_time(0),
    slave_parallel_workers(0),
    recovery_parallel_workers(0), checkpoint_seqno(0),
-   checkpoint_group(opt_mts_checkpoint_group), mts_recovery_group_cnt(0),
+   checkpoint_group(opt_mts_checkpoint_group), 
+   recovery_groups_inited(false), mts_recovery_group_cnt(0),
    mts_recovery_index(0), mts_recovery_group_seen_begin(0),
    mts_group_status(MTS_NOT_IN_GROUP), reported_unsafe_warning(false),
    sql_delay(0), sql_delay_end(0), m_flags(0), row_stmt_start_timestamp(0),
@@ -108,7 +109,6 @@ Relay_log_info::Relay_log_info(bool is_s
     group_master_log_name[0]= 0;
   until_log_name[0]= ign_master_log_name_end[0]= 0;
   set_timespec_nsec(last_clock, 0);
-  bitmap_init(&recovery_groups, NULL, checkpoint_group, FALSE);
   memset(&cache_buf, 0, sizeof(cache_buf));
   cached_charset_invalidate();
 
@@ -152,7 +152,6 @@ Relay_log_info::~Relay_log_info()
 {
   DBUG_ENTER("Relay_log_info::~Relay_log_info");
 
-  bitmap_free(&recovery_groups);
   mysql_mutex_destroy(&log_space_lock);
   mysql_cond_destroy(&log_space_cond);
   mysql_mutex_destroy(&pending_jobs_lock);
@@ -249,6 +248,16 @@ void Relay_log_info::reset_notified_chec
   }
 }
 
+void Relay_log_info::reset_workers_recovered()
+{
+  for (uint i= 0; i < workers.elements; i++)
+  {
+    Slave_worker *w= *(Slave_worker **) dynamic_array_ptr(&workers, i);
+    w->reset_info();
+  }
+  recovery_parallel_workers= slave_parallel_workers;
+}
+
 static inline int add_relay_log(Relay_log_info* rli,LOG_INFO* linfo)
 {
   MY_STAT s;
@@ -1531,9 +1540,13 @@ int Relay_log_info::init_info()
 
     if (hot_log)
       mysql_mutex_unlock(log_lock);
-
-    DBUG_RETURN(recovery_parallel_workers && !is_mts_recovery() ?
-                mts_recovery_groups(this, &recovery_groups) : 0);
+    DBUG_RETURN((recovery_parallel_workers && !mi->rli->is_mts_recovery()) ?
+                /* 
+                   though mts_recovery_groups() is reentrant don't call 
+                   it again if a previous run was not followed (completed)
+                   with the gaps filling.
+                */
+                mts_recovery_groups(this) : 0);
   }
 
   cur_log_fd = -1;

=== modified file 'sql/rpl_rli.h'
--- a/sql/rpl_rli.h	2012-03-28 15:24:17 +0000
+++ b/sql/rpl_rli.h	2012-03-30 11:15:17 +0000
@@ -512,6 +512,7 @@ public:
   uint checkpoint_seqno;  // counter of groups executed after the most recent CP
   uint checkpoint_group;  // cache for ::opt_mts_checkpoint_group 
   MY_BITMAP recovery_groups;  // bitmap used during recovery
+  bool recovery_groups_inited;
   ulong mts_recovery_group_cnt; // number of groups to execute at recovery
   ulong mts_recovery_index;     // running index of recoverable groups
   bool mts_recovery_group_seen_begin;
@@ -623,6 +624,11 @@ public:
   */
   void reset_notified_checkpoint(ulong, time_t, bool);
 
+  /**
+     Called when gaps execution is ended to it is crash-safe
+     to reset the last session info which the method does for each Worker.
+  */
+  void reset_workers_recovered();
   /*
    * End of MTS section ******************************************************/
 

=== modified file 'sql/rpl_rli_pdb.cc'
--- a/sql/rpl_rli_pdb.cc	2012-03-05 16:35:12 +0000
+++ b/sql/rpl_rli_pdb.cc	2012-03-30 11:15:17 +0000
@@ -86,8 +86,6 @@ Slave_worker::Slave_worker(Relay_log_inf
   checkpoint_master_log_name[0]= 0;
   my_init_dynamic_array(&curr_group_exec_parts, sizeof(db_worker_hash_entry*),
                         SLAVE_INIT_DBS_IN_GROUP, 1);
-  bitmap_init(&group_executed, NULL, c_rli->checkpoint_group, FALSE);
-  bitmap_init(&group_shifted, NULL, c_rli->checkpoint_group, FALSE);
   mysql_mutex_init(key_mutex_slave_parallel_worker, &jobs_lock,
                    MY_MUTEX_INIT_FAST);
   mysql_cond_init(key_cond_slave_parallel_worker, &jobs_cond, NULL);
@@ -96,10 +94,14 @@ Slave_worker::Slave_worker(Relay_log_inf
 Slave_worker::~Slave_worker() 
 {
   delete_dynamic(&curr_group_exec_parts);
-  bitmap_free(&group_executed);
-  bitmap_free(&group_shifted);
   mysql_mutex_destroy(&jobs_lock);
   mysql_cond_destroy(&jobs_cond);
+  if (inited)
+  {
+    bitmap_free(&group_executed);
+    bitmap_free(&group_shifted);
+  }
+  inited= 0;
 }
 
 /**
@@ -119,7 +121,7 @@ int Slave_worker::init_worker(Relay_log_
   Slave_job_item empty= {NULL};
 
   c_rli= rli;
-  if (init_info() || 
+  if (init_info(false) || 
       DBUG_EVALUATE_IF("inject_init_worker_init_info_fault", true, false))
     DBUG_RETURN(1);
 
@@ -153,7 +155,21 @@ int Slave_worker::init_worker(Relay_log_
   DBUG_RETURN(0);
 }
 
-int Slave_worker::init_info()
+/**
+   A part of Slave worker iitializer that provides a 
+   minimum context for MTS recovery.
+
+   @param gaps_collecting   clarifies what state the caller
+          executes this method from. When it's @c true
+          that is @c mts_recovery_groups() and Worker should
+          restore the last time info.
+          Whet it's @c false Worker should not read the last
+          session time stale info. It will be reset once
+          recovery execution gets done.
+          
+   @return 0 on success, non-zero for a failure
+*/
+int Slave_worker::init_info(bool gaps_collecting)
 {
   enum_return_check return_check= ERROR_CHECKING_REPOSITORY;
 
@@ -163,23 +179,35 @@ int Slave_worker::init_info()
     DBUG_RETURN(0);
 
   /*
+    Worker bitmap size depends on recovery mode.
+    If it is gaps collecting the bitmaps must be capable to accept
+    up to MTS_MAX_BITS_IN_GROUP of bits.
+  */
+  size_t num_bits= gaps_collecting ?
+    MTS_MAX_BITS_IN_GROUP : c_rli->checkpoint_group;
+  /*
     This checks if the repository was created before and thus there
     will be values to be read. Please, do not move this call after
     the handler->init_info(). 
   */
   return_check= check_info(); 
-  if (return_check == ERROR_CHECKING_REPOSITORY)
+  if (return_check == ERROR_CHECKING_REPOSITORY ||
+      (return_check == REPOSITORY_DOES_NOT_EXIST && gaps_collecting))
     goto err;
 
   if (handler->init_info(uidx, nidx))
     goto err;
 
-  if (return_check == REPOSITORY_EXISTS && read_info(handler))
+  bitmap_init(&group_executed, NULL, num_bits, FALSE);
+  bitmap_init(&group_shifted, NULL, num_bits, FALSE);
+  
+  if (gaps_collecting && read_info(handler))
+  {
+    bitmap_free(&group_executed);
+    bitmap_free(&group_shifted);
     goto err;
-
+  }
   inited= 1;
-  if (flush_info(TRUE))
-    goto err;
 
   DBUG_RETURN(0);
 
@@ -269,6 +297,9 @@ bool Slave_worker::read_info(Rpl_info_ha
       from->get_info((ulong *) &temp_checkpoint_seqno,
                      (ulong) 0) ||
       from->get_info(&nbytes, (ulong) 0) ||
+
+      (DBUG_ASSERT(nbytes <= (group_executed.n_bits + 7) / 8), 0) ||
+
       from->get_info(buffer, (size_t) nbytes,
                      (uchar *) 0))
     DBUG_RETURN(TRUE);
@@ -289,6 +320,8 @@ bool Slave_worker::write_info(Rpl_info_h
   ulong nbytes= (ulong) no_bytes_in_map(&group_executed);
   uchar *buffer= (uchar*) group_executed.bitmap;
 
+  DBUG_ASSERT(nbytes <= c_rli->checkpoint_group / 8);
+
   if (to->prepare_info_for_write(nidx) ||
       to->set_info(group_relay_log_name) ||
       to->set_info((ulong) group_relay_log_pos) ||
@@ -306,6 +339,20 @@ bool Slave_worker::write_info(Rpl_info_h
   DBUG_RETURN(FALSE);
 }
 
+/**
+   Crean up valueable for gaps collecting info.
+   This worker won't contribute to recovery bitmap at future
+   slave restart (see @c mts_recovery_groups).
+*/
+bool Slave_worker::reset_info()
+{
+  set_group_master_log_name("");
+  set_group_master_log_pos(0);
+  flush_info(true);
+
+  return FALSE;
+}
+
 size_t Slave_worker::get_number_worker_fields()
 {
   return sizeof(info_slave_worker_fields)/sizeof(info_slave_worker_fields[0]);

=== modified file 'sql/rpl_rli_pdb.h'
--- a/sql/rpl_rli_pdb.h	2011-11-21 17:46:02 +0000
+++ b/sql/rpl_rli_pdb.h	2012-03-30 11:15:17 +0000
@@ -345,12 +345,13 @@ public:
   en_running_state volatile running_status;
 
   int init_worker(Relay_log_info*, ulong);
-  int init_info();
+  int init_info(bool);
   void end_info();
   int flush_info(bool force= FALSE);
-  size_t get_number_worker_fields();
+  static size_t get_number_worker_fields();
   void slave_worker_ends_group(Log_event*, int);
   bool commit_positions(Log_event *evt, Slave_job_group *ptr_g, bool force);
+  bool reset_info();
 
 protected:
 

=== modified file 'sql/rpl_slave.cc'
--- a/sql/rpl_slave.cc	2012-03-28 15:24:17 +0000
+++ b/sql/rpl_slave.cc	2012-03-30 11:15:17 +0000
@@ -194,7 +194,6 @@ static int terminate_slave_thread(THD *t
 static bool check_io_slave_killed(THD *thd, Master_info *mi, const char *info);
 int slave_worker_exec_job(Slave_worker * w, Relay_log_info *rli);
 static int mts_event_coord_cmp(LOG_POS_COORD *id1, LOG_POS_COORD *id2);
-static bool remove_workers(Relay_log_info* rli);
 
 /*
   Find out which replications threads are running
@@ -398,7 +397,7 @@ int init_recovery(Master_info* mi, const
 
       We need to improve this. /Alfranio.
     */
-    error= mts_recovery_groups(rli, &rli->recovery_groups);
+    error= mts_recovery_groups(rli);
     if (rli->mts_recovery_group_cnt)
     {
       error= 1;
@@ -513,7 +512,7 @@ int remove_info(Master_info* mi)
   mi->end_info();
   mi->rli->end_info();
 
-  if (mi->remove_info() || remove_workers(mi->rli) ||
+  if (mi->remove_info() || Rpl_info_factory::reset_workers(mi->rli, true) ||
       mi->rli->remove_info())
     goto err;
 
@@ -559,41 +558,6 @@ int flush_master_info(Master_info* mi, b
   DBUG_RETURN (err);
 }
 
-/*
-  Remove worker's entries from the repositories.
-*/
-bool remove_workers(Relay_log_info* rli)
-{
-  Slave_worker *worker= NULL;
-
-  for (uint id= 0; id < rli->recovery_parallel_workers; id++)
-  {
-    if (!(worker=
-          Rpl_info_factory::create_worker(opt_rli_repository_id, id, rli)))
-      goto err;
-
-    if (worker->init_info())
-    {
-      delete worker;
-      goto err;
-    }
-
-    worker->end_info();
-
-    if (worker->remove_info())
-    {
-      delete worker;
-      goto err;
-    }
-
-    delete worker;
-  }
-  return false;
-
-err:
-  return true;
-}
-
 /**
   Convert slave skip errors bitmap into a printable string.
 */
@@ -1102,14 +1066,21 @@ int start_slave_threads(bool need_slave_
                               mi);
   if (!error && (thread_mask & SLAVE_SQL))
   {
-    error= start_slave_thread(
+    /*
+      MTS-recovery gaps gathering is placed onto common execution path
+      for either START-SLAVE and --skip-start-slave= 0 
+    */
+    if (mi->rli->recovery_parallel_workers != 0 && !mi->rli->is_mts_recovery())
+      error= mts_recovery_groups(mi->rli);
+    if (!error)
+      error= start_slave_thread(
 #ifdef HAVE_PSI_INTERFACE
-                              key_thread_slave_sql,
+                                key_thread_slave_sql,
 #endif
-                              handle_slave_sql, lock_sql, lock_cond_sql,
-                              cond_sql,
-                              &mi->rli->slave_running, &mi->rli->slave_run_id,
-                              mi);
+                                handle_slave_sql, lock_sql, lock_cond_sql,
+                                cond_sql,
+                                &mi->rli->slave_running, &mi->rli->slave_run_id,
+                                mi);
     if (error)
       terminate_slave_threads(mi, thread_mask & SLAVE_IO, !need_slave_mutex);
   }
@@ -3237,13 +3208,16 @@ int apply_event_and_update_pos(Log_event
   {
     reason= ev->shall_skip(rli);
   }
-
-  DBUG_PRINT("mts", ("Mts is recovering %d, number of bits set %d, "
-             "bitmap is set %d, index %lu.\n",
-             rli->is_mts_recovery(), bitmap_bits_set(&rli->recovery_groups),
-             bitmap_is_set(&rli->recovery_groups, rli->mts_recovery_index),
-             rli->mts_recovery_index));
-
+  if (rli->is_mts_recovery())
+  {
+    DBUG_PRINT("mts", ("Mts is recovering %d, number of bits set %d, "
+                       "bitmap is set %d, index %lu.\n",
+                       rli->is_mts_recovery(),
+                       bitmap_bits_set(&rli->recovery_groups),
+                       bitmap_is_set(&rli->recovery_groups,
+                                     rli->mts_recovery_index),
+                       rli->mts_recovery_index));
+  }
   if (reason == Log_event::EVENT_SKIP_COUNT)
   {
     sql_slave_skip_counter= --rli->slave_skip_counter;
@@ -3412,12 +3386,20 @@ int apply_event_and_update_pos(Log_event
         rli->mts_recovery_index++;
         if (--rli->mts_recovery_group_cnt == 0)
         {
-          rli->recovery_parallel_workers= rli->slave_parallel_workers;
           rli->mts_recovery_index= 0;
+          sql_print_information("Slave: MTS Recovery has completed at "
+                                "relay log %s, position %llu "
+                                "master log %s, position %llu.",
+                                rli->get_group_relay_log_name(),
+                                rli->get_group_relay_log_pos(),
+                                rli->get_group_master_log_name(),
+                                rli->get_group_master_log_pos());
+          // reset the Worker tables to remove last slave session time info
+          rli->reset_workers_recovered();
         }
         rli->mts_recovery_group_seen_begin= false;
-
-        error= rli->flush_info(TRUE);
+        if (!error)
+          error= rli->flush_info(TRUE);
       }
     }
 
@@ -4353,7 +4335,7 @@ int mts_event_coord_cmp(LOG_POS_COORD *i
          (poscmp  < 0  ? -1 : (poscmp  > 0  ?  1 : 0))));
 }
 
-bool mts_recovery_groups(Relay_log_info *rli, MY_BITMAP *groups)
+int mts_recovery_groups(Relay_log_info *rli)
 { 
   Log_event *ev= NULL;
   const char *errmsg= NULL;
@@ -4367,10 +4349,11 @@ bool mts_recovery_groups(Relay_log_info 
   File file;
   LOG_INFO linfo;
   my_off_t offset= 0;
+  MY_BITMAP *groups= &rli->recovery_groups;
 
   DBUG_ENTER("mts_recovery_groups");
-  DBUG_ASSERT(rli->recovery_parallel_workers > 0);
 
+  DBUG_ASSERT(rli->slave_parallel_workers == 0);
   /*
     Save relay log position to compare with worker's position.
   */
@@ -4390,12 +4373,13 @@ bool mts_recovery_groups(Relay_log_info 
     above_lwm_jobs in asc ordered by the master binlog coordinates.
   */
   my_init_dynamic_array(&above_lwm_jobs, sizeof(Slave_job_group),
-                        rli->recovery_parallel_workers, rli->recovery_parallel_workers);
+                        rli->recovery_parallel_workers,
+                        rli->recovery_parallel_workers);
 
   for (uint id= 0; id < rli->recovery_parallel_workers; id++)
   {
     Slave_worker *worker=
-      Rpl_info_factory::create_worker(opt_rli_repository_id, id, rli);
+      Rpl_info_factory::create_worker(opt_rli_repository_id, id, rli, true);
 
     if (!worker)
     {
@@ -4403,7 +4387,6 @@ bool mts_recovery_groups(Relay_log_info 
       goto err;
     }
 
-    worker->init_info();
     LOG_POS_COORD w_last= { const_cast<char*>(worker->get_group_master_log_name()),
                             worker->get_group_master_log_pos() };
     if (mts_event_coord_cmp(&w_last, &cp) > 0)
@@ -4447,8 +4430,12 @@ bool mts_recovery_groups(Relay_log_info 
         while(!eof);
         continue;
   */
-
-  bitmap_clear_all(groups);
+  if (above_lwm_jobs.elements != 0)
+  {
+    bitmap_init(groups, NULL, MTS_MAX_BITS_IN_GROUP, FALSE);
+    rli->recovery_groups_inited= true;
+    bitmap_clear_all(groups);
+  }
   rli->mts_recovery_group_cnt= 0;
   for (uint it_job= 0; it_job < above_lwm_jobs.elements; it_job++)
   {
@@ -4458,14 +4445,14 @@ bool mts_recovery_groups(Relay_log_info 
                             w->get_group_master_log_pos() };
     bool checksum_detected= FALSE;
 
-    sql_print_information("Recovery relay log info based on Worker-Id %lu, "
-                          "group_relay_log_name %s, group_relay_log_pos %lu "
-                          "group_master_log_name %s, group_master_log_pos %lu",
+    sql_print_information("Slave: MTS group recovery relay log info based on Worker-Id %lu, "
+                          "group_relay_log_name %s, group_relay_log_pos %llu "
+                          "group_master_log_name %s, group_master_log_pos %llu",
                           w->id,
                           w->get_group_relay_log_name(),
-                          (ulong) w->get_group_relay_log_pos(),
+                          w->get_group_relay_log_pos(),
                           w->get_group_master_log_name(),
-                          (ulong) w->get_group_master_log_pos());
+                          w->get_group_master_log_pos());
 
     recovery_group_cnt= 0;
     not_reached_commit= true;
@@ -4548,10 +4535,10 @@ bool mts_recovery_groups(Relay_log_info 
           flag_group_seen_begin= false;
           recovery_group_cnt++;
 
-          sql_print_information("Group Recoverying relay log info "
-                                "group_master_log_name %s, event_master_log_pos %llu.",
-                                rli->get_group_master_log_name(),
-                                ev->log_pos);
+          sql_print_information("Slave: MTS group recovery relay log info "
+                                "group_master_log_name %s, "
+                                "event_master_log_pos %llu.",
+                                rli->get_group_master_log_name(), ev->log_pos);
           if ((ret= mts_event_coord_cmp(&ev_coord, &w_last)) == 0)
           {
 #ifndef DBUG_OFF
@@ -4600,8 +4587,7 @@ bool mts_recovery_groups(Relay_log_info 
       recovery_group_cnt : rli->mts_recovery_group_cnt);
   }
 
-  DBUG_ASSERT(rli->mts_recovery_group_cnt < groups->n_bits);
-  DBUG_ASSERT(rli->mts_recovery_group_cnt < rli->checkpoint_group);
+  DBUG_ASSERT(rli->mts_recovery_group_cnt <= groups->n_bits);
 
 err:
   
@@ -4614,7 +4600,7 @@ err:
 
   delete_dynamic(&above_lwm_jobs);
 
-  DBUG_RETURN(error);
+  DBUG_RETURN(error ? ER_MTS_RECOVERY_FAILURE : 0);
 }
 
 /**
@@ -4775,7 +4761,7 @@ int slave_start_single_worker(Relay_log_
   Slave_worker *w= NULL;
 
   if (!(w=
-      Rpl_info_factory::create_worker(opt_rli_repository_id, i, rli)))
+        Rpl_info_factory::create_worker(opt_rli_repository_id, i, rli, false)))
   {
     sql_print_error("Failed during slave worker thread create");
     error= 1;
@@ -4900,12 +4886,14 @@ int slave_start_workers(Relay_log_info *
       goto err;
   }
 
-err:
   rli->slave_parallel_workers= rli->workers.elements;
   // end recovery right now if mts_recovery_groups() did not find any gaps
   if (rli->mts_recovery_group_cnt == 0)
-    rli->recovery_parallel_workers= rli->slave_parallel_workers;
+  {
+    rli->reset_workers_recovered();
+  }
 
+err:
   return error;
 }
 
@@ -5360,6 +5348,8 @@ llstr(rli->get_group_master_log_pos(), l
  err:
 
   slave_stop_workers(rli, &mts_inited); // stopping worker pool
+  if (rli->recovery_groups_inited)
+    bitmap_free(&rli->recovery_groups);
 
   /*
     Some events set some playgrounds, which won't be cleared because thread
@@ -8094,7 +8084,7 @@ err:
   if (ret == FALSE)
   {
     if (mts_remove_workers)
-      remove_workers(mi->rli);
+      Rpl_info_factory::reset_workers(mi->rli, true);
     my_ok(thd);
   }
   DBUG_RETURN(ret);

=== modified file 'sql/rpl_slave.h'
--- a/sql/rpl_slave.h	2012-03-26 19:30:00 +0000
+++ b/sql/rpl_slave.h	2012-03-30 11:15:17 +0000
@@ -250,7 +250,7 @@ extern char *master_ssl_cipher, *master_
        
 extern I_List<THD> threads;
 
-bool mts_recovery_groups(Relay_log_info *rli, MY_BITMAP *groups);
+int mts_recovery_groups(Relay_log_info *rli);
 bool mts_checkpoint_routine(Relay_log_info *rli, ulonglong period,
                             bool force, bool locked);
 #endif /* HAVE_REPLICATION */

=== modified file 'sql/share/errmsg-utf8.txt'
--- a/sql/share/errmsg-utf8.txt	2012-03-28 15:24:17 +0000
+++ b/sql/share/errmsg-utf8.txt	2012-03-30 11:15:17 +0000
@@ -6742,6 +6742,9 @@ ER_UNKNOWN_ALTER_LOCK
 
 ER_MTS_CHANGE_MASTER_CANT_RUN_WITH_GAPS
   eng "CHANGE MASTER cannot be executed when the slave was stopped with an error or killed in MTS mode. Consider using RESET SLAVE or START SLAVE UNTIL."
+ER_MTS_RECOVERY_FAILURE
+  eng "Cannot recover after SLAVE errored out in parallel execution mode. Additional error messages can be found in the MySQL error log."
+
 #
 #  End of 5.6 error messages.
 #

=== modified file 'sql/sys_vars.cc'
--- a/sql/sys_vars.cc	2012-03-28 15:24:17 +0000
+++ b/sql/sys_vars.cc	2012-03-30 11:15:17 +0000
@@ -3772,9 +3772,9 @@ static Sys_var_uint Sys_checkpoint_mts_g
        "before a checkpoint operation is called to update progress status.",
        GLOBAL_VAR(opt_mts_checkpoint_group), CMD_LINE(REQUIRED_ARG),
 #ifndef DBUG_OFF
-       VALID_RANGE(1, MTS_MAX_BITS_IN_GROUP), DEFAULT(512), BLOCK_SIZE(1));
+       VALID_RANGE(1, MTS_MAX_BITS_IN_GROUP), DEFAULT(512), BLOCK_SIZE(8));
 #else
-       VALID_RANGE(512, MTS_MAX_BITS_IN_GROUP), DEFAULT(512), BLOCK_SIZE(1));
+       VALID_RANGE(512, MTS_MAX_BITS_IN_GROUP), DEFAULT(512), BLOCK_SIZE(8));
 #endif /* DBUG_OFF */
 #endif /* HAVE_REPLICATION */
 

No bundle (reason: useless for push emails).
Thread
bzr push into mysql-trunk branch (andrei.elkin:3824 to 3825) Bug#13893310Andrei Elkin31 Mar