Hi Rafal, all:
Find some comments in-line.
Rafal Somla wrote:
> <div class="moz-text-flowed" style="font-family: -moz-fixed">Hi Luis,
>
> I have tried your NUTS patch and I have a couple of comments (numbers
> in [] refer to comments inlined below in your patch).
>
> 1. The command line options of nuts.pl should be documented -
> otherwise it is not possible to use the utility.
Sure. We are working on this.
>
> 2. Also, it is very important to document the feedback given by the
> utility - what logs it produces, how does it report errors etc, what
> entries can be found in logs and reports and what they mean. Just
> guessing is not good enough.
Sure. We are working on this.
>
> 3. It seems to me that in backup_nuts.test you run backup suite
> located in the NUTS distribution dir. This is not good I think - there
> should be a way of placing NUTS tests in the MySQL source tree and run
> them from backup_nuts (see [3]).
We need to discuss this because our idea was to keep them in different
directories.
>
> 4. I think it should be possible to use NUTS from mtr without setting
> PATH variable (see [1]).
Sure. Our fault.
>
> 5. When looking at errors reported from backup_nuts, I have no idea
> which line of which test script triggered each error. And this makes
> it very difficult to trace the problem.
Which error in particular? By looking into the logs we can see which
line in the test is causing an error.
>
> 6. Looking inside logs and reports produced by NUTS, it is hard to
> distinguish errors/problem from other debug information present there.
I agree. Any suggestions on how to improve it?
Different logs?
>
> When I run backup_nuts from mtr I get this:
>
>> TEST RESULT TIME (ms)
>> ------------------------------------------------------------
>>
>> Nuts 0.1
>
> Is it intentional that this prints out?
I agree this need to be removed.
>
>> nuts.backup_nuts [ fail ]
>> Test ended at 2009-03-04 08:16:01
>>
>> CURRENT_TEST: nuts.backup_nuts
>> [10:11:21] backup::backup_and_restore....DBI
>>
> connect('host=127.0.0.1:port=15000:user=root:passwd=:database=test','HASH(0x8c7d790)',...)
>> failed: Can't connect to MySQL server on '127.0.0.1' (111) at
>> /ext/mysql/bzr/nuts/lib/My/Nuts/Instance/Server.pm line 163
>> DBI
>>
> connect('host=127.0.0.1:port=15000:user=root:passwd=:database=test','HASH(0x8c7d790)',...)
>> failed: Can't connect to MySQL server on '127.0.0.1' (111) at
>> /ext/mysql/bzr/nuts/lib/My/Nuts/Instance/Server.pm line 163
>> DBI
>>
> connect('host=127.0.0.1:port=15000:user=root:passwd=:database=test','HASH(0x8c7d790)',...)
>> failed: Can't connect to MySQL server on '127.0.0.1' (111) at
>> /ext/mysql/bzr/nuts/lib/My/Nuts/Instance/Server.pm line 163
>
> How do I know which line of which test script triggered this error?
>
We need to fix/hide this message. In fact this is not an error.
The framework is testing if a server is ready by trying to connect to it.
If the server is not ready after a number of attempts an error is
reported in the log.
>> Test Summary Report
>> -------------------
>> backup::rpl_backup (Wstat: 0 Tests: 69 Failed: 4)
>> Failed tests: 18, 40, 58, 61
>
> How can I find out which test is the test #18?
Counting the number of ok(s) in the test. :(
There is a verbose alternative that shows exactly which "ok" assertion
is failing.
I think that a detailed report is being generated. I am not sure though.
Luis?
>
>> Files=2, Tests=77, 272 wallclock secs ( 0.45 usr 1.84 sys + 6.72
>> cusr 17.35 csys = 26.36 CPU)
>
> What Files means?
Number of tests.
>
>> Result: FAIL
>> print() on closed filehandle GEN1 at
>> /usr/share/perl/5.8/Test/Builder.pm line 1199.
>> mysqltest: At line 3: command "nuts.pl --build .. --suite=backup
>> --work-dir=var" failed
>
> What follows are inlined comments to your patch.
>
> Luis Soares wrote:
>> #At file:///home/lsoares/Workspace/mysql-server/tests/6.0-bkp/ based
>> on revid:jorgen.loland@stripped
>>
>> 2773 Luis Soares 2009-02-27
>> Integrates NUTS with MTR.
>> This is experimental. To tryout integration do:
>> 1. clone NUTS
>> 2. add $NUTS_HOME/bin to PATH
>> 3. execute MTR: mtr --suite=nuts backup_nuts
>> @ mysql-test/include/have_nuts.inc
>> MTR include file that checks whether NUTS is installed or not.
>> @ mysql-test/suite/nuts
>> Added nuts suite.
>> @ mysql-test/suite/nuts/t/backup_nuts.test
>> Sample test on how to integrate effortlessly NUTS into MTR.
>> This one runs NUTS on its backup test suite.
>>
>> added:
>> mysql-test/include/have_nuts.inc
>> mysql-test/suite/nuts/
>> mysql-test/suite/nuts/t/
>> mysql-test/suite/nuts/t/backup_nuts.test
>> === added file 'mysql-test/include/have_nuts.inc'
>> --- a/mysql-test/include/have_nuts.inc 1970-01-01 00:00:00 +0000
>> +++ b/mysql-test/include/have_nuts.inc 2009-02-27 16:02:47 +0000
>> @@ -0,0 +1,25 @@
>> +# Define the name of the file for checking whether nuts exists or not.
>> +let $mtr_nuts_exists_file= var/tmp/nuts_exists_$MTR_BUILD_THREAD;
>> +
>> +# pass it to perl execution
>> +let NUTS_EXISTS_FILE= $mtr_nuts_exists_file;
>> +
>> +perl;
>> +my $success= (system("nuts.pl", "--version") == 0);
>> +my $file= $ENV{'NUTS_EXISTS_FILE'};
>> +open FH, "> $file";
>> +print FH "let \$nuts_exists= $success;";
>> +close FH;
>> +EOF
>> +
>> +# evaluates the result of perl execution.
>> +source $mtr_nuts_exists_file;
>> +
>> +# remove the file from the tmp dir
>> +#remove_file $mtr_nuts_exists_file;
>
> Why commented out?
>
>> +
>> +# $nut_exists is defined by the perl execution, now we just need to
>> check it
>> +if(!$nuts_exists)
>> +{
>> + skip Unable to find NUTS. Install (bzr clone
>> bzr+ssh://bk-internal.mysql.com/bzrroot/nuts) and add
>> "$NUTS_HOME/bin/" to PATH.;
>> +}
>
> [1] Suggestion: give a user option to point at the NUTS distr with
> variable, without a need to set PATH.
>
>>
>> === added directory 'mysql-test/suite/nuts'
>> === added directory 'mysql-test/suite/nuts/t'
>> === added file 'mysql-test/suite/nuts/t/backup_nuts.test'
>> --- a/mysql-test/suite/nuts/t/backup_nuts.test 1970-01-01 00:00:00
>> +0000
>> +++ b/mysql-test/suite/nuts/t/backup_nuts.test 2009-02-27 16:02:47
>> +0000
>> @@ -0,0 +1,4 @@
>> +--source include/have_nuts.inc
>> +let NUTS_DEPLOYER_THREAD= $MTR_BUILD_THREAD;
>> +--exec nuts.pl --build .. --suite=backup --work-dir=var
>
> [2] Suggestion: "--exec perl nuts.pl ..." - should be more portable.
>
>
> [3] I have guessed that --suite=backup option means that tests from
> the backup suite in the NUTS distribution are executed. My first
> thought was that nuts.pl will look at mtr's backup suite and locate
> the test to run there. My suggestion would be to implement this
> behaviour for better integration with mtr. More precisely:
>
> 1. NUTS could use the same dir layout for suits as mtr. I.e., test
> suite foo will be located in suite/foo, say in nuts subdir.
>
> 2. There is env variable or option for the root dir of the test
> suites. By default it is set to the dist. dir of NUTS, so that the
> tests located there will be executed. But in mtr, it could be set to
> mysql-test dir in the source tree and then the tests distributed with
> the tree will be executed.
>
> Rafal
>
> </div>
