Re: bzr commit into mysql-6.0-backup branch (luis.soares:2773)

[Rafal Somla]

Hi Luis,

I have tried your NUTS patch and I have a couple of comments (numbers in [] 
refer to comments inlined below in your patch).

1. The command line options of nuts.pl should be documented - otherwise it is 
not possible to use the utility.

2. Also, it is very important to document the feedback given by the utility - 
what logs it produces, how does it report errors etc, what entries can be found 
in logs and reports and what they mean. Just guessing is not good enough.

3. It seems to me that in backup_nuts.test you run backup suite located in the 
NUTS distribution dir. This is not good I think - there should be a way of 
placing NUTS tests in the MySQL source tree and run them from backup_nuts (see [3]).

4. I think it should be possible to use NUTS from mtr without setting PATH 
variable (see [1]).

5. When looking at errors reported from backup_nuts, I have no idea which line 
of which test script triggered each error. And this makes it very difficult to 
trace the problem.

6. Looking inside logs and reports produced by NUTS, it is hard to distinguish 
errors/problem from other debug information present there.

When I run backup_nuts from mtr I get this:

> TEST                                      RESULT   TIME (ms)
> ------------------------------------------------------------
> 
> Nuts 0.1

Is it intentional that this prints out?

> nuts.backup_nuts                         [ fail ]
>         Test ended at 2009-03-04 08:16:01
> 
> CURRENT_TEST: nuts.backup_nuts
> [10:11:21] backup::backup_and_restore....DBI
> connect('host=127.0.0.1:port=15000:user=root:passwd=:database=test','HASH(0x8c7d790)',...)
> failed: Can't connect to MySQL server on '127.0.0.1' (111) at
> /ext/mysql/bzr/nuts/lib/My/Nuts/Instance/Server.pm line 163
> DBI
> connect('host=127.0.0.1:port=15000:user=root:passwd=:database=test','HASH(0x8c7d790)',...)
> failed: Can't connect to MySQL server on '127.0.0.1' (111) at
> /ext/mysql/bzr/nuts/lib/My/Nuts/Instance/Server.pm line 163
> DBI
> connect('host=127.0.0.1:port=15000:user=root:passwd=:database=test','HASH(0x8c7d790)',...)
> failed: Can't connect to MySQL server on '127.0.0.1' (111) at
> /ext/mysql/bzr/nuts/lib/My/Nuts/Instance/Server.pm line 163

How do I know which line of which test script triggered this error?
...
> Test Summary Report
> -------------------
> backup::rpl_backup        (Wstat: 0 Tests: 69 Failed: 4)
>   Failed tests:  18, 40, 58, 61

How can I find out which test is the test #18?

> Files=2, Tests=77, 272 wallclock secs ( 0.45 usr  1.84 sys +  6.72 cusr 17.35 csys =
> 26.36 CPU)

What Files means?

> Result: FAIL
> print() on closed filehandle GEN1 at /usr/share/perl/5.8/Test/Builder.pm line 1199.
> mysqltest: At line 3: command "nuts.pl --build .. --suite=backup --work-dir=var"
> failed

What follows are inlined comments to your patch.

Luis Soares wrote:
> #At file:///home/lsoares/Workspace/mysql-server/tests/6.0-bkp/ based on
> revid:jorgen.loland@stripped
> 
>  2773 Luis Soares	2009-02-27
>       Integrates NUTS with MTR.
>       
>       This is experimental. To tryout integration do:
>       
>       1. clone NUTS
>       2. add $NUTS_HOME/bin to PATH
>       3. execute MTR: mtr --suite=nuts backup_nuts
>      @ mysql-test/include/have_nuts.inc
>         MTR include file that checks whether NUTS is installed or not.
>      @ mysql-test/suite/nuts
>         Added nuts suite.
>      @ mysql-test/suite/nuts/t/backup_nuts.test
>         Sample test on how to integrate effortlessly NUTS into MTR.
>         This one runs NUTS on its backup test suite.
> 
>     added:
>       mysql-test/include/have_nuts.inc
>       mysql-test/suite/nuts/
>       mysql-test/suite/nuts/t/
>       mysql-test/suite/nuts/t/backup_nuts.test
> === added file 'mysql-test/include/have_nuts.inc'
> --- a/mysql-test/include/have_nuts.inc	1970-01-01 00:00:00 +0000
> +++ b/mysql-test/include/have_nuts.inc	2009-02-27 16:02:47 +0000
> @@ -0,0 +1,25 @@
> +# Define the name of the file for checking whether nuts exists or not.
> +let $mtr_nuts_exists_file= var/tmp/nuts_exists_$MTR_BUILD_THREAD;
> +
> +# pass it to perl execution
> +let NUTS_EXISTS_FILE= $mtr_nuts_exists_file;
> +
> +perl;
> +my $success= (system("nuts.pl", "--version") == 0);
> +my $file= $ENV{'NUTS_EXISTS_FILE'};
> +open FH, "> $file";
> +print FH "let \$nuts_exists= $success;";
> +close FH;
> +EOF
> +
> +# evaluates the result of perl execution.
> +source $mtr_nuts_exists_file;
> +
> +# remove the file from the tmp dir
> +#remove_file $mtr_nuts_exists_file;

Why commented out?

> +
> +# $nut_exists is defined by the perl execution, now we just need to check it
> +if(!$nuts_exists)
> +{
> +  skip Unable to find NUTS. Install (bzr clone
> bzr+ssh://bk-internal.mysql.com/bzrroot/nuts) and add "$NUTS_HOME/bin/" to PATH.;
> +}

[1] Suggestion: give a user option to point at the NUTS distr with variable, 
without a need to set PATH.

> 
> === added directory 'mysql-test/suite/nuts'
> === added directory 'mysql-test/suite/nuts/t'
> === added file 'mysql-test/suite/nuts/t/backup_nuts.test'
> --- a/mysql-test/suite/nuts/t/backup_nuts.test	1970-01-01 00:00:00 +0000
> +++ b/mysql-test/suite/nuts/t/backup_nuts.test	2009-02-27 16:02:47 +0000
> @@ -0,0 +1,4 @@
> +--source include/have_nuts.inc
> +let NUTS_DEPLOYER_THREAD= $MTR_BUILD_THREAD;
> +--exec nuts.pl --build .. --suite=backup --work-dir=var

[2] Suggestion: "--exec perl nuts.pl ..." - should be more portable.


[3] I have guessed that --suite=backup option means that tests from the backup 
suite in the NUTS distribution are executed. My first thought was that nuts.pl 
will look at mtr's backup suite and locate the test to run there. My suggestion 
would be to implement this behaviour for better integration with mtr. More 
precisely:

1. NUTS could use the same dir layout for suits as mtr. I.e., test suite foo 
will be located in suite/foo, say in nuts subdir.

2. There is env variable or option for the root dir of the test suites. By 
default it is set to the dist. dir of NUTS, so that the tests located there will 
be executed. But in mtr, it could be set to mysql-test dir in the source tree 
and then the tests distributed with the tree will be executed.

Rafal