Re: Testing 3rd party engines

[Ronald Bradford]

Hi Paul,

Your post certainly hits all the present hotspots with mysql-test. I've 
been looking into this and here are my observations to date that may 
assist you and others.

MySQL Tests does provide a level of management for the new storage 
engines.  The present approach that some tests are now employing is a 
level of refactoring. Here are some existing examples using this 
approach.  In summary:

    * Refactor the actual test to an include file
    * Build separate test wrappers and results per storage engine
    * Use have_* handling


Two tests to look at in the 5.1 code are that use this are mix2_*.test 
and the ps_*.test.

At the recent MySQL Camp the comment was made we could see 50 to 100 
storage engines so I can easily see that this approach, while practical 
in the early stages does not scale well without manual work.  In 
addition, the amount of specific tests within say for example 
include/mix2.inc does limit certain functionality that may not be 
implemented in a particular engine.  Shorter tests, and more specific 
tests would improve the portability for future engines, and certainly 
help particularly when there is an early failure within a long test.

Another thing I've found while trolling the mysql_test_run.pl code is 
the support for test suites. There is also some reference in the docs 
for syntax.

You could create the following as a demo:

cd $MYSQL_HOME/mysql-test
mkdir -p suite/pbxt/{r,t}
cp r/alias.test suite/pbxt/r
cp r/alias.result suite/pbxt/r
./mysql-test-run.pl --suite=pbxt

Effectively you can have an entire t/ and r/ directory structure under a 
suite/[name] subdirectory.  This doesn't help in the refactoring process 
but does provide an area for extending tests specific for your storage 
engine.  I've found an issue here with the use of includes, they don't 
work if you copy the include directory, you need to repath accordingly.

When it comes to things like FULLTEXT and GEOMETRY I've found the 
quickest solution is to add these tests to disabled.def initially 
tackling all other test first.

These points really only look at creative ways to help with present 
issues, this is certainly a topic that could benefit from a revised 
analysis period.

Good Luck.

Regards

Ronald

Paul McCullagh wrote:
> Hi All,
>
> I am just writing a blog on the subject of testing 3rd party engines 
> (http://pbxt.blogspot.com/), and was wondering if this subject has 
> come up at all before.
>
> When using mysql-test-run to test a different engine, for example:
>
> $ ./mysql-test-run --mysqld=--default-storage-engine=pbxt
>
> you get a lot of failures which are not actually bugs.
>
> For example wherever SHOW CREATE TABLE is used in a test, the result 
> verifies that ENGINE=MyISAM.
>
> In my blog I mentioned this and other things that would make testing 
> 3rd party engines easier by eliminating unnecessary differences.
>
> If you have a moment, please check it out: http://pbxt.blogspot.com/
>
> Best regards,
>
> Paul
>
>
> --MySQL Internals Mailing List
> For list archives: http://lists.mysql.com/internals
> To unsubscribe:    
> http://lists.mysql.com/internals?unsub=1
>


-- 
Ronald Bradford, Senior Consultant
MySQL Inc, www.mysql.com
1.551.689.2361 (cell)

Are you MySQL certified?  www.mysql.com/certification