My comments:
------------
Manual:
-------
[M1] This does not make sense to me:
"In particular, by enabling to have fault-injection either at the
network level, by changing network packets flow, or at the application
level, by instrumenting the server with crash-points, it easily evaluate
recovery issues."
Disclaimer: I noticed Lars had many comments about the manual, so I
didn't do a thorough review at this point.
Testing:
--------
[T1] plan tests = X - won't this just add maintenance cost?
[T2] nuts tests seem to take awfully long to execute. A modified
one-threaded version
of backup_and_restore.pm takes ~20 seconds while performing the same
test in an mtr script takes ~300ms. This is not a startup cost; copying
the pm-file to three different files in the nuts_home increases
execution time 3 times.
This is a blocker.
[T3] As it is now, when executing "./mtr --suite=nuts backup_nuts", only
one test is reported as passed in mtr:
TEST RESULT TIME (ms)
------------------------------------------------------------
* Nuts backup suite successfully executed *
nuts.backup_nuts [ pass ] 59553
------------------------------------------------------------
I see quite a few problems with this:
4.1) You don't get error reports as individual pm test files fail -
you have to wait until all the tests have executed. As more and
more tests are converted to nuts, mtr will look like it hangs.
4.2) Quite often, I'm interested in starting bug fixing immediately
if a test fails. I may even want to stop the tests because I
need to rerun the tests once I have fixed something. I can't do
this because of 4.1.
4.3) To remedy 4.1 and 4.2, the code tree (e.g., mysql-6.0-backup)
seemingly has to contain one test file in suite/nuts/t/ for
each test in the $NUTS_HOME repository. This is duplication of
code and will lead to errors
These problems will probably go away if nuts is merged into main.
I consider this a blocker.
[T4]
--exec perl ../../nuts/bin/nuts.pl --verbose --build .. --suite=backup
--work-dir=var
Should be executed in $NUTS_HOME instead of the relative path.
Questions:
----------
[Q1] Can returned result sets be processed programatically in nuts?
Examples:
- Check that an error has a given number, but ignore the error text
- Check the content of a BLOB
- Write a method that can check if a result set has size X and
contains a record Y?
Regards,
Jørgen
Luis Soares wrote:
> #At file:///home/lsoares/Workspace/mysql-server/tests/6.0-bkp/ based on
> revid:jorgen.loland@stripped
>
> 2773 Luis Soares 2009-03-06
> Integrates NUTS with MTR.
>
> This is experimental. To try out integration do:
>
> 1. clone NUTS
> 2. put nuts in the same directory level as your mysql tree is
> ( Should be in: (...) mysql-6.0-backup/mysql-test$ ../../nuts )
> 3. execute MTR: mtr --suite=nuts backup_nuts
>
> This commits addresses some of the issues Rafal mentioned:
>
> 1. removed the print out of nuts version
> 2. removed the erroneous commented instruction to remove temp file
> 3. removed the need to set path (set as Lars instructed to). In the
> future the have_nuts include could probably take a parameter with which
> the
> test writer would set NUTS_HOME, eg:
>
> let $nuts_home= somewhere_in_the_filesystem
> --source include/have_nuts.inc
>
> If $nuts_home is not defined then falls back to: mysql-test$ ../../nuts
>
> 4. Documentation, containing: instructions on how to analyze failures and
> log
> files; command line interface description; configuration files
> description;
> has been updated in NUTS manual:
>
> https://inside.mysql.com/wiki/NUTS_-_MANUAL
> @ mysql-test/include/have_nuts.inc
> MTR include file that checks whether NUTS is installed or not.
> @ mysql-test/suite/nuts
> Added nuts suite.
> @ mysql-test/suite/nuts/t/backup_nuts.test
> Sample test on how to integrate effortlessly NUTS into MTR.
> This one runs NUTS on its backup test suite.
>
> added:
> mysql-test/include/have_nuts.inc
> mysql-test/suite/nuts/
> mysql-test/suite/nuts/t/
> mysql-test/suite/nuts/t/backup_nuts.test
> === added file 'mysql-test/include/have_nuts.inc'
> --- a/mysql-test/include/have_nuts.inc 1970-01-01 00:00:00 +0000
> +++ b/mysql-test/include/have_nuts.inc 2009-03-06 01:39:59 +0000
> @@ -0,0 +1,27 @@
> +# Define the name of the file for checking whether nuts exists or not.
> +let $mtr_nuts_exists_file= var/tmp/nuts_exists_$MTR_BUILD_THREAD;
> +
> +# pass it to perl execution
> +let NUTS_EXISTS_FILE= $mtr_nuts_exists_file;
> +
> +perl;
> +my $cmd= "../../nuts/bin/nuts.pl --version";
> +my $output= `$cmd`;
> +my $success= ($? == 0);
> +my $file= $ENV{'NUTS_EXISTS_FILE'};
> +open FH, "> $file";
> +print FH "let \$nuts_exists= $success;";
> +close FH;
> +EOF
> +
> +# evaluates the result of perl execution.
> +source $mtr_nuts_exists_file;
> +
> +# remove the file from the tmp dir
> +remove_file $mtr_nuts_exists_file;
> +
> +# $nut_exists is defined by the perl execution, now we just need to check it
> +if(!$nuts_exists)
> +{
> + skip Unable to find NUTS. Install using 'bzr clone
> bzr+ssh://bk-internal.mysql.com/bzrroot/nuts ../../nuts' .;
> +}
>
> === added directory 'mysql-test/suite/nuts'
> === added directory 'mysql-test/suite/nuts/t'
> === added file 'mysql-test/suite/nuts/t/backup_nuts.test'
> --- a/mysql-test/suite/nuts/t/backup_nuts.test 1970-01-01 00:00:00 +0000
> +++ b/mysql-test/suite/nuts/t/backup_nuts.test 2009-03-06 01:39:59 +0000
> @@ -0,0 +1,7 @@
> +--source include/have_nuts.inc
> +let NUTS_DEPLOYER_THREAD= $MTR_BUILD_THREAD;
> +--exec perl ../../nuts/bin/nuts.pl --verbose --build .. --suite=backup
> --work-dir=var
> +
> +--echo * Nuts backup suite successfully executed *
> +
> +--exit
>
>
>
> ------------------------------------------------------------------------
>
>
