From: MARK CALLAGHAN Date: August 24 2009 7:44pm Subject: Re: Testing the Storage Engine API List-Archive: http://lists.mysql.com/internals/37278 Message-Id: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Is there a storage engine independent test suite? I thought one was to be provided when 5.1.37 was released, but I don't see many diffs in mysql-test/t for that release. http://forge.mysql.com/wiki/EngineIndependentTestSuite http://forge.mysql.com/wiki/Storage_Engine_API_Testing_Framework I am interested in tests that can be run against any storage engine (more like EngineIndependentTestSuite than Storage_Engine_API_Testing_Framework). Mark On Wed, Aug 13, 2008 at 4:46 AM, Stewart Smith wrote: > On Tue, 2008-07-15 at 13:31 +0530, Nidhi Shrotriya wrote: >> I would like to put forward some ideas here that I believe should fall >> in the same line of thought. >> Please have a look at the very first draft of the framework design >> coming out on these lines. >> https://inside.mysql.com/wiki/SEATeamPage/SE_API_Testing_Framework > > > > Summary of my points is: a different approach would be better. > >> [edit] How to Test Storage Engine API >> =A0 =A0 =A0 * API in our context -> handler.h (the minimum) >> for ex: >> =A0 ha_write_row() >> =A0 These functions represent the public interface to *users* of the han= dler class, hence they are *not* virtual. These are wrappers over private v= irtual APIs. >> write_row() >> =A0 For the inheritance interface. Storage Engines implement those. >> [edit] What to Test? >> We are testing - >> >> >> >> =A0 =A0 =A0 * Server Contract -> The appropriate API got called on >> =A0 =A0 =A0 =A0 performing the operation. > > I don't think this is something useful to test. > > If the server doesn't tell the handler to write the row, it's not going > to be able to read the row. A simple INSERT, SELECT 4 statement > (including create and drop table) test will show this. > >> =A0 =A0 =A0 * It performed the operation successfully (SE used the >> =A0 =A0 =A0 =A0 parameters correctly, SE dependent). Should be tested by >> =A0 =A0 =A0 =A0 making the TestSE work like some SE. > > A test SE isn't going to bring much to the table. To be remotely useful, > it has to be amazingly non-trivial. It takes approximately two years to > integrate a non-trivial storage engine with the MySQL server so that > there aren't easily discoverable bugs by knowledgeable people. > > I think a test storage engine is a really bad idea. > > Use real engines. If the tests are engine independent, then you're > providing something engine developers have been wanting for a few years > now. An engine test suite has merit and real benefits now. > > Engines do have different features - so providing some method of input > to the test suite of these would be useful too. > > Durability tests also, kill the server (in all sorts of places), check > that data is there. > > > Success is also not what you want to test. The existing test suite does > an okay job of making sure that INSERT means that a row got written. > What we're really lacking (outside of NDB kernel at least) is testing of > failures. > > I've been meaning to take 15 minutes "spare time" and develop a > LD_PRELOAD that wraps all IO calls and on each one, forks with one > returning a temporary error, the other a temporary error and see how > badly things break. Anecdotal (and more.. like reading the source) > evidence shows that in some parts of the server instead of throwing > error messages we just loose data/ignore errors/harm puppies. > > >> =A0 =A0 =A0 * The actual flow of calls made from server->SE. We should l= ook >> =A0 =A0 =A0 =A0 into the feasibility of this in terms of comparing the s= erver >> =A0 =A0 =A0 =A0 traces. > > Comparing traces (if you mean debug traces) is not very useful. > > It's also only really valid if you're only running one thread. > > Tests that involve concurrency are currently hard/impossible to write > with mysql-test. Extra stuff that makes it easy is useful. See my SoC > project for last year (dynamic autogenerated and verifiable sql for > mysqlslap) for some ideas. Also look at the NDB Hugo classes and the > tests run in autotest (i can explain more, perhaps via phone). > >> I also understand it has been tested through its usage in storage >> engines and the distribution of tests in /mysql-test contain >> -Tests for particular SEs. >> -Common set of tests with test wrappers to change the SEs. > > only for a small number of tests. > > We've had to make a fair amount of changes to tests in drizzle because > the default engine is now InnoDB. Some tests do really "smart" things > like load thousands of rows with autocommit on (wrapping begin and > commit statements around the data load functions took minutes off the > test run time). > > A serious effort to fix all of this up would be very useful. > >> Though the organization of tests makes it a bit difficult to say we have >> comprehensive standalone engine suits or a complete common set for >> different SEs. I believe it needs some effort there too to have it more >> streamlined. > > We don't have a standalone engine test suite. > > We should have any engine test suite in the source tree, not separate - > or it will never be run. > > > hope this helps, > stewart > -- > Stewart Smith, Senior Software Engineer (MySQL Cluster) > Sun Microsystems Inc > Office: +61398640020 =A0Mob: +61438844332 =A0VoIP: 6616@stripped > --=20 Mark Callaghan mdcallag@stripped