List:Internals« Previous MessageNext Message »
From:Scott Wiersdorf Date:November 17 2000 11:21pm
Subject:Re: mysqlhotcopy patch
View as plain text  
On Fri, Nov 17, 2000 at 04:38:44PM +0200, Michael Widenius wrote:
> 
> >>>>> "Scott" == Scott Wiersdorf <scott@stripped> writes:
> 
> Scott> MySQL people,
> Scott> (I couldn't find a 'contrib' email address; Tim Bunce thought it
> Scott> should be sent to support).
> 
> The right address for MySQL additions is internals@stripped

I've subscribed myself to this list now; I've got another patch as
per your comments below about the tilde (~) in the table name space.

I neglected to praise you, Monty, for MySQL (I let Tim have some
thanks for his cool tool, but the cool tool wouldn't be cool without
MySQL being even cooler!). God bless you.

The new mysqlhotcopy is attached along with the diff the last one I sent.

Thanks again,

Scott

> Scott> I have really appreciated mysqlhotcopy; what a cool tool! (I sent the
> Scott> original copy of this messages to Tim Bunce, so he's seen it already).
> 
> Scott> I've hacked on it a bit and added (I've tried to keep to the original
> Scott> style) better support for scp and including (or excluding) a set of
> Scott> tables via a regex.
> 
> Scott> I made two dangerous assumptions in adding table regex support: a
> Scott> table will not have a dot (.) or a tilde (~) in the table name (not
> Scott> the filename). I use the dot to separate a database from a table name:
> 
> Scott>     mysqlhotcopy db_1./^table_name$/
> 
> Scott> and the tilde to negate a regex:
> 
> Scott>     mysqlhotcopy db_1./~^bad_tables/
> 
> Scott> which would copy everything in db_1 except tables starting with
> Scott> 'bad_tables'. Are these safe assumptions?
> 
> . is safe but ~ is not.  The only characters MySQL guarantees will
> never be in a table name are '.', '/' and '\0'.  On the other hand, I
> think ~ is extremely uncommon in a table name so this shouldn't be a
> real problem (as long you have added a note in the documentation about
> that)
> 
> Scott> I've taken the liberty to update the documentation and add a
> Scott> sprinkling of comments in the source itself. I do not believe my
> Scott> hacking has removed previous functionality! ;o)
> 
> Scott> Attached is the diff (-C2) and source. I hope this helps someone
> Scott> (other than me!).
> 
> Scott> Scott 
> 
> Thanks.
> 
> I have now added your copy to the MySQL distribution ; I also bumbed
> the version number to 1.8
> 
> Regards,
> Monty
> 

-- 
Scott Wiersdorf
scott@stripped
http://www.ipartner.net/
#!/usr/bin/perl5

use strict;
use Getopt::Long;
use Data::Dumper;
use File::Basename;
use File::Path;
use DBI;

=head1 NAME

mysqlhotcopy - fast on-line hot-backup utility for local MySQL databases

=head1 SYNOPSIS

  mysqlhotcopy db_name

  mysqlhotcopy --suffix=_copy db_name_1 ... db_name_n

  mysqlhotcopy db_name_1 ... db_name_n /path/to/new_directory

  mysqlhotcopy db_name./regex/

  mysqlhotcopy db_name./^\(foo\|bar\)/

  mysqlhotcopy db_name./~regex/

  mysqlhotcopy db_name_1./regex_1/ ... db_name_n./regex_n/ /path/to/new_directory

  mysqlhotcopy --method='scp -Bq -i /usr/home/foo/.ssh/identity' --user=root \
               --password=secretpassword db_1./^nice_table/ \
               user@stripped:~/path/to/new_directory

WARNING: THIS IS VERY MUCH A FIRST-CUT ALPHA. Comments/patches welcome.

=cut

# Documentation continued at end of file

my $VERSION = "1.8";
my $REVISION = '$Revision$';
my $OPTIONS = <<"_OPTIONS";

Usage: $0 db_name [new_db_name | directory]

  -?, --help           display this helpscreen and exit
  -u, --user=#         user for database login if not current user
  -p, --password=#     password to use when connecting to server
  -P, --port=#         port to use when connecting to local server
  -S, --socket=#       socket to use when connecting to local server

  --allowold           don't abort if target already exists (rename it _old)
  --keepold            don't delete previous (now renamed) target when done
  --indices            include index files in copy
  --method=#           method for copy (only "cp" currently supported)

  -q, --quiet          be silent except for errors
  --debug              enable debug
  -n, --dryrun         report actions without doing them

  --regexp=#           copy all databases with names matching regexp
  --suffix=#           suffix for names of copied databases
  --checkpoint=#       insert checkpoint entry into specified db.table
  --flushlog           flush logs once all tables are locked 

  Try 'perldoc $0' for more complete documentation

_OPTIONS

sub usage {
    die @_, $OPTIONS;
}

my %opt = (
    user	=> getpwuid($>),
    indices	=> 1,	# for safety
    allowold	=> 0,	# for safety
    keepold	=> 0,
    method	=> "cp",
    flushlog    => 0,
);
Getopt::Long::Configure(qw(no_ignore_case)); # disambuguate -p and -P
GetOptions( \%opt,
    "help",
    "user|u=s",
    "password|p=s",
    "port|P=s",
    "socket|S=s",
    "allowold!",
    "keepold!",
    "indices!",
    "method=s",
    "debug",
    "quiet|q",
    "mv!",
    "regexp=s",
    "suffix=s",
    "checkpoint=s",
    "flushlog",
    "dryrun|n",
) or usage("Invalid option");

# @db_desc
# ==========
# a list of hash-refs containing:
#
#   'src'     - name of the db to copy
#   't_regex' - regex describing tables in src
#   'target'  - destination directory of the copy
#   'tables'  - array-ref to list of tables in the db
#   'files'   - array-ref to list of files to be copied
#

my @db_desc = ();
my $tgt_name = undef;

if ( $opt{regexp} || $opt{suffix} || @ARGV > 2 ) {
    $tgt_name   = pop @ARGV unless ( exists $opt{suffix} );
    @db_desc = map { s{^([^\.]+)\./(.+)/$}{$1}; { 'src' => $_, 't_regex' => ( $2 ?
$2 : '.*' ) } } @ARGV;
}
else {
    usage("Database name to hotcopy not specified") unless ( @ARGV );

    $ARGV[0] =~ s{^([^\.]+)\./(.+)/$}{$1};
    @db_desc = ( { 'src' => $ARGV[0], 't_regex' => ( $2 ? $2 : '.*' ) } );

    if ( @ARGV == 2 ) {
	$tgt_name   = $ARGV[1];
    }
    else {
	$opt{suffix} = "_copy";
    }
}

my $mysqld_help;
my %mysqld_vars;
my $start_time = time;
$0 = $1 if $0 =~ m:/([^/]+)$:;
$opt{quiet} = 0 if $opt{debug};
$opt{allowold} = 1 if $opt{keepold};

# --- connect to the database ---
my $dsn = ";host=localhost";
$dsn .= ";port=$opt{port}" if $opt{port};
$dsn .= ";mysql_socket=$opt{socket}" if $opt{socket};

my $dbh = DBI->connect("dbi:mysql:$dsn;mysql_read_default_group=mysqlhotcopy",
                        $opt{user}, $opt{password},
{
    RaiseError => 1,
    PrintError => 0,
    AutoCommit => 1,
});


# --- check that checkpoint table exists if specified ---
if ( $opt{checkpoint} ) {
    eval { $dbh->do( qq{ select time_stamp, src, dest, msg 
			 from $opt{checkpoint} where 1 != 1} );
       };

    die "Error accessing Checkpoint table ($opt{checkpoint}): $@"
      if ( $@ );
}


# --- get variables from database ---
my $sth_vars = $dbh->prepare("show variables like 'datadir'");
$sth_vars->execute;
while ( my ($var,$value) = $sth_vars->fetchrow_array ) {
    $mysqld_vars{ $var } = $value;
}
my $datadir = $mysqld_vars{'datadir'}
    || die "datadir not in mysqld variables";
$datadir =~ s:/$::;


# --- get target path ---
my ($tgt_dirname, $to_other_database);
$to_other_database=0;
if ($tgt_name =~ m:^\w+$: && @db_desc <= 1)
{
    $tgt_dirname = "$datadir/$tgt_name";
    $to_other_database=1;
}
elsif ($tgt_name =~ m:/: || $tgt_name eq '.') {
    $tgt_dirname = $tgt_name;
}
elsif ( $opt{suffix} ) {
    print "copy suffix $opt{suffix}\n" unless $opt{quiet};
}
else {
    die "Target '$tgt_name' doesn't look like a database name or directory path.\n";
}


# --- resolve database names from regexp ---
if ( defined $opt{regexp} ) {
    my $sth_dbs = $dbh->prepare("show databases");
    $sth_dbs->execute;
    while ( my ($db_name) = $sth_dbs->fetchrow_array ) {
	push @db_desc, { 'src' => $db_name } if ( $db_name =~ m/$opt{regexp}/o );
    }
}


# --- get list of tables to hotcopy ---
my $hc_locks = "";
my $hc_tables = "";
my $num_tables = 0;
my $num_files = 0;

foreach my $rdb ( @db_desc ) {
    my $db = $rdb->{src};
    eval { $dbh->do( "use $db" ); };
    die "Database '$db' not accessible: $@"  if ( $@ );
    my @dbh_tables = $dbh->func( '_ListTables' );

    ## generate regex for tables/files
    my $t_regex = $rdb->{t_regex};                  ## assign temporary regex
    my $negated = $t_regex =~ s/^~(.+)$/$1/;        ## remove negation operator: a table
name may not begin with ~
    $t_regex = qr/$t_regex/;                        ## make regex string from user regex

    ## filter (out) tables specified in t_regex
    print "Filtering tables with '$t_regex'\n" if $opt{debug};
    @dbh_tables = ( $negated 
		    ? grep { $_ !~ $t_regex } @dbh_tables 
		    : grep { $_ =~ $t_regex } @dbh_tables );

    ## get list of files to copy
    my $db_dir = "$datadir/$db";
    opendir(DBDIR, $db_dir ) 
      or die "Cannot open dir '$db_dir': $!";

    my %db_files;
    map { ( /(.+)\.\w+$/ ? { $db_files{$_} = $1 } : () ) } readdir(DBDIR);
    unless( keys %db_files ) {
	warn "'$db' is an empty database\n";
    }
    closedir( DBDIR );

    ## filter (out) files specified in t_regex
    my @db_files = sort ( $negated 
			  ? grep { $db_files{$_} !~ $t_regex } keys %db_files
			  : grep { $db_files{$_} =~ $t_regex } keys %db_files );

    ## remove indices unless we're told to keep them
    unless ($opt{indices}) {
	@db_files = grep { not /\.(ISM|MYI)$/ } @db_files;
    }

    $rdb->{files}  = [ @db_files ];
    my @hc_tables = map { "$db.$_" } @dbh_tables;
    $rdb->{tables} = [ @hc_tables ];

    $hc_locks .= ", "  if ( length $hc_locks && @hc_tables );
    $hc_locks .= join ", ", map { "$_ READ" } @hc_tables;
    $hc_tables .= ", "  if ( length $hc_tables && @hc_tables );
    $hc_tables .= join ", ", @hc_tables;

    $num_tables += scalar @hc_tables;
    $num_files  += scalar @{$rdb->{files}};
}


# --- resolve targets for copies ---
my @targets = ();

if (length $tgt_name ) {
    # explicit destination directory specified

    # GNU `cp -r` error message
    die "copying multiple databases, but last argument ($tgt_dirname) is not a
directory\n"
      if ( @db_desc > 1 && !(-e $tgt_dirname && -d $tgt_dirname ) );

    if ($to_other_database)
    {
      foreach my $rdb ( @db_desc ) {
	$rdb->{target} = "$tgt_dirname";
      }
    }
    elsif ($opt{method} =~ /^scp\b/) 
    {   # we have to trust scp to hit the target
	foreach my $rdb ( @db_desc ) {
	    $rdb->{target} = "$tgt_dirname/$rdb->{src}";
	}
    }
    else
    {
      die "Last argument ($tgt_dirname) is not a directory\n"
	if (!(-e $tgt_dirname && -d $tgt_dirname ) );
      foreach my $rdb ( @db_desc ) {
	$rdb->{target} = "$tgt_dirname/$rdb->{src}";
      }
    }
  }
else {
  die "Error: expected \$opt{suffix} to exist" unless ( exists $opt{suffix} );

  foreach my $rdb ( @db_desc ) {
    $rdb->{target} = "$datadir/$rdb->{src}$opt{suffix}";
  }
}

print Dumper( \@db_desc ) if ( $opt{debug} );


# --- bail out if all specified databases are empty ---
die "No tables to hot-copy" unless ( length $hc_locks );


# --- create target directories ---
my @existing = ();
foreach my $rdb ( @db_desc ) {
    push @existing, $rdb->{target} if ( -d  $rdb->{target} );
}

die "Can't hotcopy to '", join( "','", @existing ), "' because already exist and
--allowold option not given.\n"
  if ( @existing && !$opt{allowold} );

retire_directory( @existing ) if ( @existing );

foreach my $rdb ( @db_desc ) {
    my $tgt_dirpath = $rdb->{target};
    if ( $opt{dryrun} ) {
	print "mkdir $tgt_dirpath, 0750\n";
    }
    elsif ($opt{method} =~ /^scp\b/) {
	## assume it's there?
	## ...
    }
    else {
	mkdir($tgt_dirpath, 0750)
	  or die "Can't create '$tgt_dirpath': $!\n";
    }
}

##############################
# --- PERFORM THE HOT-COPY ---
#
# Note that we try to keep the time between the LOCK and the UNLOCK
# as short as possible, and only start when we know that we should
# be able to complete without error.

# read lock all the tables we'll be copying
# in order to get a consistent snapshot of the database

if ( $opt{checkpoint} ) {
    # convert existing READ lock on checkpoint table into WRITE lock
    unless ( $hc_locks =~ s/$opt{checkpoint}\s+READ/$opt{checkpoint} WRITE/ ) {
	$hc_locks .= ", $opt{checkpoint} WRITE";
    }
}

my $hc_started = time;	# count from time lock is granted

if ( $opt{dryrun} ) {
    print "LOCK TABLES $hc_locks\n";
    print "FLUSH TABLES /*!32323 $hc_tables */\n";
    print "FLUSH LOGS\n" if ( $opt{flushlog} );
}
else {
    my $start = time;
    $dbh->do("LOCK TABLES $hc_locks");
    printf "Locked $num_tables tables in %d seconds.\n", time-$start unless $opt{quiet};
    $hc_started = time;	# count from time lock is granted

    # flush tables to make on-disk copy uptodate
    $start = time;
    $dbh->do("FLUSH TABLES /*!32323 $hc_tables */");
    printf "Flushed tables ($hc_tables) in %d seconds.\n", time-$start unless $opt{quiet};
    $dbh->do( "FLUSH LOGS" ) if ( $opt{flushlog} );
}

my @failed = ();

foreach my $rdb ( @db_desc ) {
    my @files = map { "$datadir/$rdb->{src}/$_" } @{$rdb->{files}};
    next unless @files;
    eval { copy_files($opt{method}, \@files, $rdb->{target} ); };

    push @failed, "$rdb->{src} -> $rdb->{target} failed: $@"
      if ( $@ );

    if ( $opt{checkpoint} ) {
	my $msg = ( $@ ) ? "Failed: $@" : "Succeeded";

	eval {
	    $dbh->do( qq{ insert into $opt{checkpoint} (src, dest, msg) 
			  VALUES ( '$rdb->{src}', '$rdb->{target}', '$msg' )
			} ); 
	};

	if ( $@ ) {
	    warn "Failed to update checkpoint table: $@\n";
	}
    }
}

if ( $opt{dryrun} ) {
    print "UNLOCK TABLES\n";
    if ( @existing && !$opt{keepold} ) {
	my @oldies = map { $_ . '_old' } @existing;
	print "rm -rf @oldies\n" 
    }
    $dbh->disconnect();
    exit(0);
}
else {
    $dbh->do("UNLOCK TABLES");
}

my $hc_dur = time - $hc_started;
printf "Unlocked tables.\n" unless $opt{quiet};

#
# --- HOT-COPY COMPLETE ---
###########################

$dbh->disconnect;

if ( @failed ) {
    # hotcopy failed - cleanup
    # delete any @targets 
    # rename _old copy back to original

    print "Deleting @targets \n" if $opt{debug};
    rmtree([@targets]);
    if (@existing) {
	print "Restoring @existing from back-up\n" if $opt{debug};
        foreach my $dir ( @existing ) {
	    rename("${dir}_old", $dir )
	      or warn "Can't rename ${dir}_old to $dir: $!\n";
	}
    }

    die join( "\n", @failed );
}
else {
    # hotcopy worked
    # delete _old unless $opt{keepold}

    if ( @existing && !$opt{keepold} ) {
	my @oldies = map { $_ . '_old' } @existing;
	print "Deleting previous copy in @oldies\n" if $opt{debug};
	rmtree([@oldies]);
    }

    printf "$0 copied %d tables (%d files) in %d second%s (%d seconds overall).\n",
	    $num_tables, $num_files,
	    $hc_dur, ($hc_dur==1)?"":"s", time - $start_time
	unless $opt{quiet};
}

exit 0;


# ---

sub copy_files {
    my ($method, $files, $target) = @_;
    my @cmd;
    print "Copying ".@$files." files...\n" unless $opt{quiet};

    if ($method =~ /^s?cp\b/) { # cp or scp with optional flags
	@cmd = ($method);
	# add option to preserve mod time etc of copied files
	# not critical, but nice to have
	push @cmd, "-p" if $^O =~ m/^(solaris|linux|freebsd)$/;

	# add files to copy and the destination directory
	push @cmd, @$files, $target;
    }
    else {
	die "Can't use unsupported method '$method'\n";
    }

    if ( $opt{dryrun} ) {
	print "@cmd\n";
	next;
    }

    ## for some reason system fails but backticks works ok for scp...
    ## if anyone knows why we have to use backticks for scp to work,
    ## please let me know (scott@stripped)
    print "Executing '@cmd'\n" if $opt{debug};
    my $cp_status = system @cmd;
    if ($cp_status != 0) {
	warn "Burp ('scuse me). Trying backtick execution...\n" if $opt{debug}; #'
	## try something else
	`@cmd` && die "Error: @cmd failed ($cp_status) while copying files.\n";
    }
}

sub retire_directory {
    my ( @dir ) = @_;

    foreach my $dir ( @dir ) {
	my $tgt_oldpath = $dir . '_old';
	if ( $opt{dryrun} ) {
	    print "rmtree $tgt_oldpath\n" if ( -d $tgt_oldpath );
	    print "rename $dir, $tgt_oldpath\n";
	    next;
	}

	if ( -d $tgt_oldpath ) {
	    print "Deleting previous 'old' hotcopy directory ('$tgt_oldpath')\n" unless
$opt{quiet};
	    rmtree([$tgt_oldpath])
	}
	rename($dir, $tgt_oldpath)
	  or die "Can't rename $dir=>$tgt_oldpath: $!\n";
	print "Existing hotcopy directory renamed to '$tgt_oldpath'\n" unless $opt{quiet};
    }
}

__END__

=head1 DESCRIPTION

mysqlhotcopy is designed to make stable copies of live MySQL databases.

Here "live" means that the database server is running and the database
may be in active use. And "stable" means that the copy will not have
any corruptions that could occur if the table files were simply copied
without first being locked and flushed from within the server.

=head1 OPTIONS

=over 4

=item --checkpoint checkpoint-table

As each database is copied, an entry is written to the specified
checkpoint-table.  This has the happy side-effect of updating the
MySQL update-log (if it is switched on) giving a good indication of
where roll-forward should begin for backup+rollforward schemes.

The name of the checkpoint table should be supplied in database.table format.
The checkpoint-table must contain at least the following fields:

=over 4

  time_stamp timestamp not null
  src varchar(32)
  dest varchar(60)
  msg varchar(255)

=back

=item --suffix suffix

Each database is copied back into the originating datadir under
a new name. The new name is the original name with the suffix
appended. 

If only a single db_name is supplied and the --suffix flag is not
supplied, then "--suffix=_copy" is assumed.

=item --allowold

Move any existing version of the destination to a backup directory for
the duration of the copy. If the copy successfully completes, the backup 
directory is deleted - unless the --keepold flag is set.  If the copy fails,
the backup directory is restored.

The backup directory name is the original name with "_old" appended.
Any existing versions of the backup directory are deleted.

=item --keepold

Behaves as for the --allowold, with the additional feature 
of keeping the backup directory after the copy successfully completes.

=item --flushlog

Rotate the log files by executing "FLUSH LOGS" after all tables are
locked, and before they are copied.

=item --regexp pattern

Copy all databases with names matching the pattern

=item db_name./pattern/

Copy only tables matching the regular expression 'pattern'. Shell
metacharacters ( (, ), |, !, [, ], etc.) have to be escaped to protect
them from shell interpolation.  For example, to select all tables in
database db1 whose names begin with 'foo' or 'bar':

    mysqlhotcopy --indices --method=cp db1./^\(foo\|bar\)/

Alternatively (and probably what you really want to do), protect the
table name regex with single quotes ('') to prevent shell
interpolation on UN*Xish machines:

    mysqlhotcopy --indices --method=cp 'db1./^(foo|bar)/'

See the Perl regular expression man page for more details (perldoc
perlre) on what a valid regular expression looks like.

=item db_name./~pattern/

Copy only tables not matching pattern. For example, to copy tables
that do not begin with foo nor bar:

    mysqlhotcopy --indices --method=cp db1./~^\(foo\|bar\)/

While the tilde (~) is a vaild character in the MySQL table name
space, if you're using the table regex matching feature to copy only
selected tables, none of those tables' names may begin with a tilde.
If you happen to have a table whose name begins with a tilde, you may
still use mysqlhotcopy to copy the entire database.

See the Perl regular expression man page for more details (perldoc
perlre) on what a vaild regular expression looks like.

=item -?, --help

Display helpscreen and exit

=item -u, --user=#         

user for database login if not current user

=item -p, --password=#     

password to use when connecting to server

=item -P, --port=#         

port to use when connecting to local server

=item -S, --socket=#         

UNIX domain socket to use when connecting to local server

=item  --indices          

include index files in copy

=item  --method=#           

method for copy (only "cp" currently supported). Alpha support for
"scp" was added in November 2000. Your experience with the scp method
will vary with your ability to understand how scp works. 'man scp'
and 'man ssh' are your friends.

The destination directory _must exist_ on the target machine using
the scp method. Liberal use of the --debug option will help you figure
out what's really going on when you do an scp.

Note that using scp will lock your tables for a _long_ time unless
your network connection is _fast_. If this is unacceptable to you,
use the 'cp' method to copy the tables to some temporary area and then
scp or rsync the files at your leisure.

=item -q, --quiet              

be silent except for errors

=item  --debug

Debug messages are displayed 

=item -n, --dryrun

Display commands without actually doing them

=back

=head1 WARRANTY

This software is free and comes without warranty of any kind. You
should never trust backup software without studying the code yourself.
Study the code inside this script and only rely on it if I<you> believe
that it does the right thing for you.

Patches adding bug fixes, documentation and new features are welcome.

=head1 TO DO

Extend the individual table copy to allow multiple subsets of tables
to be specified on the command line:

  mysqlhotcopy db newdb  t1 t2 /^foo_/ : t3 /^bar_/ : +

where ":" delimits the subsets, the /^foo_/ indicates all tables
with names begining with "foo_" and the "+" indicates all tables
not copied by the previous subsets.

newdb is either another not existing database or a full path to a directory
where we can create a directory 'db'

Add option to lock each table in turn for people who don't need
cross-table integrity.

Add option to FLUSH STATUS just before UNLOCK TABLES.

Add support for other copy methods (eg tar to single file?).

Add support for forthcoming MySQL ``RAID'' table subdirectory layouts.

Add option to only copy the first 65KB of index files. That simplifies
recovery (recovery with no index file at all is complicated).

=head1 AUTHOR

Tim Bunce

Martin Waite - added checkpoint, flushlog, regexp and dryrun options

Ralph Corderoy - added synonyms for commands

Scott Wiersdorf - added table regex and scp support

=cut

Index: mysqlhotcopy
===================================================================
RCS file: /home/cvs/cvsroot/contrib/mysqlhotcopy/mysqlhotcopy,v
retrieving revision 1.2
diff -C2 -r1.2 mysqlhotcopy
*** mysqlhotcopy	2000/11/16 19:10:32	1.2
--- mysqlhotcopy	2000/11/17 23:11:29
***************
*** 26,33 ****
    mysqlhotcopy db_name./~regex/
  
!   mysqlhotcopy db_name_1./regex_1/ db_name_1./regex_2/ ... db_name_n./regex_n/
/path/to/new_directory
  
!   mysqlhotcopy --method='scp -Bq -i /usr/home/foo/.ssh/identity' --user=root
--password=secretpassword \
!          db_1./^nice_table/ user@stripped:~/path/to/new_directory
  
  WARNING: THIS IS VERY MUCH A FIRST-CUT ALPHA. Comments/patches welcome.
--- 26,34 ----
    mysqlhotcopy db_name./~regex/
  
!   mysqlhotcopy db_name_1./regex_1/ ... db_name_n./regex_n/ /path/to/new_directory
  
!   mysqlhotcopy --method='scp -Bq -i /usr/home/foo/.ssh/identity' --user=root \
!                --password=secretpassword db_1./^nice_table/ \
!                user@stripped:~/path/to/new_directory
  
  WARNING: THIS IS VERY MUCH A FIRST-CUT ALPHA. Comments/patches welcome.
***************
*** 63,67 ****
    --flushlog           flush logs once all tables are locked 
  
!   Try 'perldoc $0 for more complete documentation'
  _OPTIONS
  
--- 64,69 ----
    --flushlog           flush logs once all tables are locked 
  
!   Try 'perldoc $0' for more complete documentation
! 
  _OPTIONS
  
***************
*** 151,154 ****
--- 153,157 ----
  });
  
+ 
  # --- check that checkpoint table exists if specified ---
  if ( $opt{checkpoint} ) {
***************
*** 161,164 ****
--- 164,168 ----
  }
  
+ 
  # --- get variables from database ---
  my $sth_vars = $dbh->prepare("show variables like 'datadir'");
***************
*** 190,193 ****
--- 194,198 ----
  }
  
+ 
  # --- resolve database names from regexp ---
  if ( defined $opt{regexp} ) {
***************
*** 199,204 ****
  }
  
- # --- get list of tables to hotcopy ---
  
  my $hc_locks = "";
  my $hc_tables = "";
--- 204,209 ----
  }
  
  
+ # --- get list of tables to hotcopy ---
  my $hc_locks = "";
  my $hc_tables = "";
***************
*** 213,219 ****
  
      ## generate regex for tables/files
!     my $t_regex = $rdb->{t_regex};        ## assign temporary regex
!     my $negated = $t_regex =~ tr/~//d;    ## remove and count negation operator: we
don't allow ~ in table names
!     $t_regex = qr/$t_regex/;              ## make regex string from user regex
  
      ## filter (out) tables specified in t_regex
--- 218,224 ----
  
      ## generate regex for tables/files
!     my $t_regex = $rdb->{t_regex};                  ## assign temporary regex
!     my $negated = $t_regex =~ s/^~(.+)$/$1/;        ## remove negation operator: a table
name may not begin with ~
!     $t_regex = qr/$t_regex/;                        ## make regex string from user regex
  
      ## filter (out) tables specified in t_regex
***************
*** 258,263 ****
  }
  
- # --- resolve targets for copies ---
  
  my @targets = ();
  
--- 263,268 ----
  }
  
  
+ # --- resolve targets for copies ---
  my @targets = ();
  
***************
*** 300,309 ****
  print Dumper( \@db_desc ) if ( $opt{debug} );
  
- # --- bail out if all specified databases are empty ---
  
  die "No tables to hot-copy" unless ( length $hc_locks );
  
- # --- create target directories ---
  
  my @existing = ();
  foreach my $rdb ( @db_desc ) {
--- 305,314 ----
  print Dumper( \@db_desc ) if ( $opt{debug} );
  
  
+ # --- bail out if all specified databases are empty ---
  die "No tables to hot-copy" unless ( length $hc_locks );
  
  
+ # --- create target directories ---
  my @existing = ();
  foreach my $rdb ( @db_desc ) {
***************
*** 464,470 ****
  	push @cmd, "-p" if $^O =~ m/^(solaris|linux|freebsd)$/;
  
- 	# add recursive option for scp
- 	push @cmd, "-r" if $^O =~ /m^(solaris|linux|freebsd)$/ && $method =~ /^scp\b/;
- 
  	# add files to copy and the destination directory
  	push @cmd, @$files, $target;
--- 469,472 ----
***************
*** 480,483 ****
--- 482,487 ----
  
      ## for some reason system fails but backticks works ok for scp...
+     ## if anyone knows why we have to use backticks for scp to work,
+     ## please let me know (scott@stripped)
      print "Executing '@cmd'\n" if $opt{debug};
      my $cp_status = system @cmd;
***************
*** 579,588 ****
  =item db_name./pattern/
  
! Copy only tables matching pattern. Shell metacharacters ( (, ), |, !,
! etc.) have to be escaped (e.g. \). For example, to select all tables
! in database db1 whose names begin with 'foo' or 'bar':
  
      mysqlhotcopy --indices --method=cp db1./^\(foo\|bar\)/
  
  =item db_name./~pattern/
  
--- 583,602 ----
  =item db_name./pattern/
  
! Copy only tables matching the regular expression 'pattern'. Shell
! metacharacters ( (, ), |, !, [, ], etc.) have to be escaped to protect
! them from shell interpolation.  For example, to select all tables in
! database db1 whose names begin with 'foo' or 'bar':
  
      mysqlhotcopy --indices --method=cp db1./^\(foo\|bar\)/
  
+ Alternatively (and probably what you really want to do), protect the
+ table name regex with single quotes ('') to prevent shell
+ interpolation on UN*Xish machines:
+ 
+     mysqlhotcopy --indices --method=cp 'db1./^(foo|bar)/'
+ 
+ See the Perl regular expression man page for more details (perldoc
+ perlre) on what a valid regular expression looks like.
+ 
  =item db_name./~pattern/
  
***************
*** 591,594 ****
--- 605,617 ----
  
      mysqlhotcopy --indices --method=cp db1./~^\(foo\|bar\)/
+ 
+ While the tilde (~) is a vaild character in the MySQL table name
+ space, if you're using the table regex matching feature to copy only
+ selected tables, none of those tables' names may begin with a tilde.
+ If you happen to have a table whose name begins with a tilde, you may
+ still use mysqlhotcopy to copy the entire database.
+ 
+ See the Perl regular expression man page for more details (perldoc
+ perlre) on what a vaild regular expression looks like.
  
  =item -?, --help

Thread
Re: mysqlhotcopy patchScott Wiersdorf18 Nov