List:Internals« Previous MessageNext Message »
From:MARK CALLAGHAN Date:August 24 2009 7:44pm
Subject:Re: Testing the Storage Engine API
View as plain text  
Is there a storage engine independent test suite? I thought one was to
be provided when 5.1.37 was released, but I don't see many diffs in
mysql-test/t for that release.

http://forge.mysql.com/wiki/EngineIndependentTestSuite
http://forge.mysql.com/wiki/Storage_Engine_API_Testing_Framework

I am interested in tests that can be run against any storage engine
(more like EngineIndependentTestSuite than
Storage_Engine_API_Testing_Framework).

Mark

On Wed, Aug 13, 2008 at 4:46 AM, Stewart Smith<stewart@stripped> wrote:
> On Tue, 2008-07-15 at 13:31 +0530, Nidhi Shrotriya wrote:
>> I would like to put forward some ideas here that I believe should fall
>> in the same line of thought.
>> Please have a look at the very first draft of the framework design
>> coming out on these lines.
>> https://inside.mysql.com/wiki/SEATeamPage/SE_API_Testing_Framework
>
> <pasting from url here so I can comment in place>
>
> Summary of my points is: a different approach would be better.
>
>> [edit] How to Test Storage Engine API
>>       * API in our context -> handler.h (the minimum)
>> for ex:
>>   ha_write_row()
>>   These functions represent the public interface to *users* of the handler
> class, hence they are *not* virtual. These are wrappers over private virtual APIs.
>> write_row()
>>   For the inheritance interface. Storage Engines implement those.
>> [edit] What to Test?
>> We are testing -
>>
>>
>>
>>       * Server Contract -> The appropriate API got called on
>>         performing the operation.
>
> I don't think this is something useful to test.
>
> If the server doesn't tell the handler to write the row, it's not going
> to be able to read the row. A simple INSERT, SELECT 4 statement
> (including create and drop table) test will show this.
>
>>       * It performed the operation successfully (SE used the
>>         parameters correctly, SE dependent). Should be tested
> by
>>         making the TestSE work like some SE.
>
> A test SE isn't going to bring much to the table. To be remotely useful,
> it has to be amazingly non-trivial. It takes approximately two years to
> integrate a non-trivial storage engine with the MySQL server so that
> there aren't easily discoverable bugs by knowledgeable people.
>
> I think a test storage engine is a really bad idea.
>
> Use real engines. If the tests are engine independent, then you're
> providing something engine developers have been wanting for a few years
> now. An engine test suite has merit and real benefits now.
>
> Engines do have different features - so providing some method of input
> to the test suite of these would be useful too.
>
> Durability tests also, kill the server (in all sorts of places), check
> that data is there.
>
>
> Success is also not what you want to test. The existing test suite does
> an okay job of making sure that INSERT means that a row got written.
> What we're really lacking (outside of NDB kernel at least) is testing of
> failures.
>
> I've been meaning to take 15 minutes "spare time" and develop a
> LD_PRELOAD that wraps all IO calls and on each one, forks with one
> returning a temporary error, the other a temporary error and see how
> badly things break. Anecdotal (and more.. like reading the source)
> evidence shows that in some parts of the server instead of throwing
> error messages we just loose data/ignore errors/harm puppies.
>
>
>>       * The actual flow of calls made from server->SE. We
> should look
>>         into the feasibility of this in terms of comparing
> the server
>>         traces.
>
> Comparing traces (if you mean debug traces) is not very useful.
>
> It's also only really valid if you're only running one thread.
>
> Tests that involve concurrency are currently hard/impossible to write
> with mysql-test. Extra stuff that makes it easy is useful. See my SoC
> project for last year (dynamic autogenerated and verifiable sql for
> mysqlslap) for some ideas. Also look at the NDB Hugo classes and the
> tests run in autotest (i can explain more, perhaps via phone).
>
>> I also understand it has been tested through its usage in storage
>> engines and the distribution of tests in /mysql-test contain
>> -Tests for particular SEs.
>> -Common set of tests with test wrappers to change the SEs.
>
> only for a small number of tests.
>
> We've had to make a fair amount of changes to tests in drizzle because
> the default engine is now InnoDB. Some tests do really "smart" things
> like load thousands of rows with autocommit on (wrapping begin and
> commit statements around the data load functions took minutes off the
> test run time).
>
> A serious effort to fix all of this up would be very useful.
>
>> Though the organization of tests makes it a bit difficult to say we have
>> comprehensive standalone engine suits or a complete common set for
>> different SEs. I believe it needs some effort there too to have it more
>> streamlined.
>
> We don't have a standalone engine test suite.
>
> We should have any engine test suite in the source tree, not separate -
> or it will never be run.
>
>
> hope this helps,
> stewart
> --
> Stewart Smith, Senior Software Engineer (MySQL Cluster)
> Sun Microsystems Inc
> Office: +61398640020  Mob: +61438844332  VoIP: 6616@stripped
>



-- 
Mark Callaghan
mdcallag@stripped
Thread
Testing the Storage Engine APIAmit k. Saha26 Jun
  • Re: Testing the Storage Engine APIMARK CALLAGHAN26 Jun
    • Re: Testing the Storage Engine APIAmit k. Saha26 Jun
      • Re: Testing the Storage Engine APIPaul McCullagh10 Jul
        • Re: Testing the Storage Engine APIAmit k. Saha10 Jul
          • Re: Testing the Storage Engine APIMark Leith10 Jul
            • Re: Testing the Storage Engine APIBrian Aker10 Jul
  • Re: Testing the Storage Engine APIChad MILLER10 Jul
Re: Testing the Storage Engine APINidhi Shrotriya15 Jul
  • Re: Testing the Storage Engine APIStewart Smith13 Aug
    • Re: Testing the Storage Engine APIMARK CALLAGHAN24 Aug
      • Re: Testing the Storage Engine APIDavi Arnaut26 Aug
        • Re: Testing the Storage Engine APIkentoku20 Jan
          • Re: Testing the Storage Engine APIkentoku30 Jan