From: Olav Sandstaa
Date: January 21 2009 2:45pm
Subject: Problems with record visibility and how it is computed
List-Archive: http://lists.mysql.com/falcon/404
Message-Id: <49773501.4000306@sun.com>
MIME-Version: 1.0
Content-Type: text/plain; format=flowed; charset=ISO-8859-1
Content-Transfer-Encoding: 7BIT


Jim, Kevin, everybody,

I am working on a group of bugs related to transactions not returning 
the correct data every time. These issues are most likely related to our 
implementation of isolation levels and visibility of record versions. I 
have found the cause for one of these bugs but would like input on what 
is the best way to solve it.

The bug I have been analyzing is Bug #41357 "falcon.falcon_bug_34351_C 
fails with assertion IS_CONSISTENT_READ". The assert happens in 
Table::fetchForUpdate() due to the call to 
Transaction::getRelativeState() returns CommittedInvisible for a 
recordVersion when the current
transaction is running in isolation level TRANSACTION_WRITE_COMMITTED. 
This return value
should only happen if the isolation level is TRANSACTION_CONSISTENT_READ.

After having reproduced this assert several times with instrumentation 
in the source it
seems like this problems is caused by that the Transaction object that 
the recordVersion
object has a pointer to and which is used for computing the relative 
state between the two
transaction "changes state" during the execution of 
Transaction::getRelativeState().

Here is what normally happens when the CommittedInvisible state is 
reached by the following code path:

1. Transaction::getRelativeState() is called and the following code is run:

if (transaction->state == Committed)
{
   // Return CommittedVisible if the other trans has a lower TransId and
   // it was committed when we started.

   if (visible (transaction, transId, FOR_WRITING))
       return CommittedVisible;

   return CommittedInvisible;
}

when this code is run the transaction->state == Committed (and the if 
test is true).
This results in Transaction::visible() being called.

2. In the code for Transaction::visible() the following code is run:

// If the other transaction is not yet committed, the trans is not visible.

if (transaction->state != Committed)
   return false;

When the same test now is evaluated "transaction->state != Committed" 
this test too
succeed and returns that the transaction should not be visible.

So based on traces in the code and studying core files it seems like the
transaction->state variable changes value from being Committed to not 
being Committed
during the execution of these two methods. This leads to 
Transaction::visible() returning "false" which again leads to
Transaction::getRelativeState() returning CommittedInvisible (and the 
assert occurs).

The reason for the "mysterious" state change (at least in some of the 
crashes I have reproduced) is that the transaction object which the 
record version points too is deleted by another thread and thus the 
check for committed (or not)  in Transaction::visible() is now checking 
"free memory".

This situation seems very similar to an issue Kevin had (and still 
have?) with the scavenger checking which records that can be scavenged.

Questions:

-how do we either handle that Transaction object that is being pointed 
to by RecordVersion objects are deleted at "any time". Like the code for 
Transaction::getRelativeState and Transaction::visible() is now this can 
lead to wrong conclusions about which records are visible or not.

-or do we need to introduce some kind of locking so that we can safely 
use RecordVersion objects and the Transaction object they normally 
refers to?

-or can we do something about the order we clean up old transaction 
objects versus the record version objects to avoid that we have record 
object without a transaction object?

I do not know enough about this part of the code (yet) to propose a good 
solution so any suggestions would be appreciated.

Olav