List:Internals« Previous MessageNext Message »
From:Stewart Smith Date:August 13 2008 11:46am
Subject:Re: Testing the Storage Engine API
View as plain text  
On Tue, 2008-07-15 at 13:31 +0530, Nidhi Shrotriya wrote: 
> I would like to put forward some ideas here that I believe should fall
> in the same line of thought.
> Please have a look at the very first draft of the framework design
> coming out on these lines.
> https://inside.mysql.com/wiki/SEATeamPage/SE_API_Testing_Framework

<pasting from url here so I can comment in place>

Summary of my points is: a different approach would be better.

> [edit] How to Test Storage Engine API 
>       * API in our context -> handler.h (the minimum) 
> for ex:
>   ha_write_row() 
>   These functions represent the public interface to *users* of the handler class,
> hence they are *not* virtual. These are wrappers over private virtual APIs.
> write_row()
>   For the inheritance interface. Storage Engines implement those.
> [edit] What to Test? 
> We are testing -
> 
> 
> 
>       * Server Contract -> The appropriate API got called on
>         performing the operation. 

I don't think this is something useful to test.

If the server doesn't tell the handler to write the row, it's not going
to be able to read the row. A simple INSERT, SELECT 4 statement
(including create and drop table) test will show this.

>       * It performed the operation successfully (SE used the
>         parameters correctly, SE dependent). Should be tested by
>         making the TestSE work like some SE. 

A test SE isn't going to bring much to the table. To be remotely useful,
it has to be amazingly non-trivial. It takes approximately two years to
integrate a non-trivial storage engine with the MySQL server so that
there aren't easily discoverable bugs by knowledgeable people.

I think a test storage engine is a really bad idea.

Use real engines. If the tests are engine independent, then you're
providing something engine developers have been wanting for a few years
now. An engine test suite has merit and real benefits now.

Engines do have different features - so providing some method of input
to the test suite of these would be useful too.

Durability tests also, kill the server (in all sorts of places), check
that data is there.


Success is also not what you want to test. The existing test suite does
an okay job of making sure that INSERT means that a row got written.
What we're really lacking (outside of NDB kernel at least) is testing of
failures.

I've been meaning to take 15 minutes "spare time" and develop a
LD_PRELOAD that wraps all IO calls and on each one, forks with one
returning a temporary error, the other a temporary error and see how
badly things break. Anecdotal (and more.. like reading the source)
evidence shows that in some parts of the server instead of throwing
error messages we just loose data/ignore errors/harm puppies.


>       * The actual flow of calls made from server->SE. We should look
>         into the feasibility of this in terms of comparing the server
>         traces. 

Comparing traces (if you mean debug traces) is not very useful.

It's also only really valid if you're only running one thread.

Tests that involve concurrency are currently hard/impossible to write
with mysql-test. Extra stuff that makes it easy is useful. See my SoC
project for last year (dynamic autogenerated and verifiable sql for
mysqlslap) for some ideas. Also look at the NDB Hugo classes and the
tests run in autotest (i can explain more, perhaps via phone).

> I also understand it has been tested through its usage in storage
> engines and the distribution of tests in /mysql-test contain
> -Tests for particular SEs.
> -Common set of tests with test wrappers to change the SEs.

only for a small number of tests.

We've had to make a fair amount of changes to tests in drizzle because
the default engine is now InnoDB. Some tests do really "smart" things
like load thousands of rows with autocommit on (wrapping begin and
commit statements around the data load functions took minutes off the
test run time).

A serious effort to fix all of this up would be very useful.

> Though the organization of tests makes it a bit difficult to say we have
> comprehensive standalone engine suits or a complete common set for
> different SEs. I believe it needs some effort there too to have it more
> streamlined.

We don't have a standalone engine test suite.

We should have any engine test suite in the source tree, not separate -
or it will never be run.


hope this helps,
stewart
-- 
Stewart Smith, Senior Software Engineer (MySQL Cluster)
Sun Microsystems Inc
Office: +61398640020  Mob: +61438844332  VoIP: 6616@stripped

Attachment: [application/pgp-signature] This is a digitally signed message part signature.asc
Attachment: [application/pgp-signature] This is a digitally signed message part signature.asc
Thread
Testing the Storage Engine APIAmit k. Saha26 Jun
  • Re: Testing the Storage Engine APIMARK CALLAGHAN26 Jun
    • Re: Testing the Storage Engine APIAmit k. Saha26 Jun
      • Re: Testing the Storage Engine APIPaul McCullagh10 Jul
        • Re: Testing the Storage Engine APIAmit k. Saha10 Jul
          • Re: Testing the Storage Engine APIMark Leith10 Jul
            • Re: Testing the Storage Engine APIBrian Aker10 Jul
  • Re: Testing the Storage Engine APIChad MILLER10 Jul
Re: Testing the Storage Engine APINidhi Shrotriya15 Jul
  • Re: Testing the Storage Engine APIStewart Smith13 Aug
    • Re: Testing the Storage Engine APIMARK CALLAGHAN24 Aug
      • Re: Testing the Storage Engine APIDavi Arnaut26 Aug
        • Re: Testing the Storage Engine APIkentoku20 Jan
          • Re: Testing the Storage Engine APIkentoku30 Jan