From: John Embretsen
Date: January 9 2009 10:38am
Subject: Re: Downmerge from 6.0, and merge of falcon-team to falcon
List-Archive: http://lists.mysql.com/falcon/354
Message-Id: <49672942.1090006@Sun.COM>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7BIT

Hakan Kuecuekyilmaz wrote:
> On Fr, 2009-01-09 at 09:57 +0100, John Embretsen wrote:
>> Vladislav Vaintroub wrote:
>>> John, thanks for the update.
>>>
>>> don't you think it makes sense to finish the MWH project and remove the
>>> "falcon_team" tree hack from mysql-test-run.pl. The amount of red seen
>>> recently is the team tree is alarming (for me).
>> Short version: Yes, I agree.
>>
> 
> I don't agree at all. We need a mechanism to move failing or unstable
> tests in case of an upmerge. We cannot stop an upmerge just because one
> test out of ~250 is failing. And no: using a different disabled.def
> depending whether we are in main or Falcon trees is error prone.
> 
> Moreover, moving a test by disabling it ends up in > 20 disabled tests.
> Disabled tests are not run and end up in "out of sight out mind" thus
> leading to an ever increasing disabled.def list. We did that experiment
> around two years ago and it failed badly, very badly. In contrary the
> falcon_team test suite experiment succeeded. It took very long time -
> around 9 months - but we are down from 25 or so unstable tests to 3 - 4.
> I consider this as a success.

Right. Good point. However, the falcon_team suite itself has not been
running in Pushbuild during that period of test maintenance ("project
MWH"), so I don't think that is necessarily a prerequisite of success.

There is not much point in running a test suite with tests that fail
practically all the time, as with falcon_team today. So for us to
continue running falcon_team in Pushbuild, we must focus on fixing the
issues making the tests fail.

>> falcon_bug_34174
>>  - Failing (lock wait timeout instead of deadlock).
>>  - Hakan is looking into this one.
> 
> This fails from time to time only. I told Ann and Kevin about it. I
> think Kevin is working on it. We should talk about this failure on
> today's meeting.
> 
>> falcon_bug_36294-big
>>  - Times out with default MTR settings.
>>  - Not run in Pushbuild since it is --big-test.
>>
> And if run, it fails. This is
>     http://bugs.mysql.com/bug.php?id=36631
> 
> test written by Wlad and bug assigned to Chris.
> 
>> falcon_deadlock
>>  - Failing (lock wait timeout)
>>  - Svoj and Hakan were working in this area (e.g. Bug#34182) before the
>> holidays, but I'm not sure about the current status.
>>
> 
> Svoj fixed the falcon_deadlock issue already and is waiting for code
> review.

OK, thank you for providing details. Then it seems we have people
assigned to all issues, we even have some fixes in progress :)

>> Anyway...
>>
>> I will implement the following change as soon as possible unless someone
>> disagrees:
>>
>> Remove falcon_team "hack" in mysql-test-run.pl, meaning that the
>> falcon_team suite will no longer run by default in falcon branches in
>> Pushbuild.
> 
> I would agree on running falcon_team in mysql-6.0-falcon-team and not in
> mysql-6.0-falcon tree.

OK, then I think this would be a good compromise going forward. I
propose to disable the falcon_team suite in mysql-6.0-falcon suite but
keep it running in mysql-6.0-falcon-team as well as -ann, -kevin,
-chris, for now at least.

Patch available at http://lists.mysql.com/commits/62790

If we get annoyed or worried about the "redness" of
mysql-6.0-falcon-team due to failing tests in the falcon_team suite,
then the most correct solution is to fix the issues (whether they are
test issues or product issues) as soon as possible.

If an issue causing test failure has too low priority to be fixed any
time soon, I think it is a good time to disable the test and make a
prominent note of that in the bug report(s).


> Instead of complaining around which test belongs to where and how an
> unstable test should be temporarily disabled, we should focus on fixing
> those issues!

Of course... Yet we cannot avoid the fact that fixing some of these
issues takes time and dedication... What should we do in the meantime?


-- 
John