RE: bzr commit into mysql-6.0-backup branch (hema:2685) WL#4227

[Hema Sridharan]

Hi Chuck,

I modified the existing test cases for WL#4227(backup_partitions.test and
backup_datatypes.test) in accordance to the suggestions of you and Bar and
made the following modifications:

1) The tests are shifted from backup_charsets suite to backup suite.
2) The test cases will be executed only for all storage engines (will not be
executed for all character set / collation combinations).
3) I have randomly used accented letters and character sets to perform
testing. This will ensure that backup stores identifiers and objects
properly in utf8 format and data is retreived during restore without
changing the client character set.

http://lists.mysql.com/commits/54170

There are 2 more test cases for this WL(backup_accented.test and
backup_special_characters.test). These tests fail sporadically because of
Bug#38363. I have attached the patch for these test cases in the bug report,
so that developer who is working on this will eventually push the tests case
once they are fixed.


Warm Regards,
Hema Sridharan
QA Developer
Sun Microsystems / Mysql Inc, www.sun.com
 www.mysql.com
Office: Austin TX 78728, USA
 
Are you MySQL certified?  www.mysql.com/certification
 

> 
> You've done a great job with this work. The comments below 
> have nothing to do with the quality of your work. 
> 
> >  2685 Hema Sridharan	2008-08-19
> >       WL#4227
> > modified:
> >   mysql-test/suite/backup/t/backup_functions.test
> 
> Hema: Not all of the tests passed. I didn't run the falcon 
> tests but 6 failed for various reasons on Windows. I think 
> you need to test this on Windows. 
> 
> mysql-test-run in default mode: *** Failing the test(s):
> backup_charsets.backup_triggers 
> backup_charsets.backup_triggers 
> backup_charsets.backup_triggers 
> backup_charsets.backup_triggers 
> backup_charsets.backup_triggers backup_charsets.backup_triggers
> 
> master.err: backup_charsets.backup_triggers: 080825 22:21:50 [ERROR]
> Restore: Could not restore trigger `bup_ts`.`trau`
> master.err: backup_charsets.backup_triggers: 080825 22:37:09 [ERROR]
> Restore: Could not restore trigger `bup_ts`.`capu_tri`
> master.err: backup_charsets.backup_triggers: 080825 22:56:16 [ERROR]
> Restore: Could not restore trigger `bup_ts`.`capu_tri`
> 
> I cannot approve this patch. The time it takes to run the 
> suite is excessive. There are over 1200 tests run! When I ran 
> the suite it took almost 3 hours to run. And that was without 
> the falcon storage engine!
> 
> I think using a brute force method to test every combination 
> of character sets and collations against three different 
> storage engines and five different basic test combinations is 
> overkill. We need to adopt a more agile perspective -- we 
> should be doing partitioning.
> 
> Partitioning in this case would be examining the character 
> sets and determining the boundaries of the features and 
> forming partitions on the result. For example (and example 
> only), we could simply partition on the size of characters:
> 
> 1-byte
> 2-byte
> 3-byte
> ...
> 
> We create a test run for each of these and that becomes our 
> basis for coverage of the capabilities of backup WRT 
> character sets. Running all of the character sets through all 
> of the engines through all of the
> (applicable) collations is not very efficient nor effective 
> as at least 3 of the tests failed on Windows under dubious 
> conditions (reminiscent of non-deterministic tests of the past).
> 
> If we include this suite in the pushbuild then every 
> pushbuild machine would slow down considerably. Given the 
> valgrind machine's process can be measured by the passing of 
> days, it is unacceptable to exasperate an intolerable 
> performance. It goes without saying that the performance of 
> this suite is not good enough to push to main. Even if we 
> enable it in the team tree, it will mean a pushbuild run of 
> several days. It's already bad enough at 26-30 hours!
> 
> We must avoid using brute force methods except where 
> warranted and this type of coverage for testing character 
> sets and collations in backup isn't warranted.
> 
> Chuck
> 
> 
> --
> MySQL Code Commits Mailing List
> For list archives: http://lists.mysql.com/commits
> To unsubscribe:    http://lists.mysql.com/commits?unsub=1
> 
>