List:Falcon Storage Engine« Previous MessageNext Message »
From:Olav Sandstaa Date:January 21 2009 2:45pm
Subject:Problems with record visibility and how it is computed
View as plain text  
Jim, Kevin, everybody,

I am working on a group of bugs related to transactions not returning 
the correct data every time. These issues are most likely related to our 
implementation of isolation levels and visibility of record versions. I 
have found the cause for one of these bugs but would like input on what 
is the best way to solve it.

The bug I have been analyzing is Bug #41357 "falcon.falcon_bug_34351_C 
fails with assertion IS_CONSISTENT_READ". The assert happens in 
Table::fetchForUpdate() due to the call to 
Transaction::getRelativeState() returns CommittedInvisible for a 
recordVersion when the current
transaction is running in isolation level TRANSACTION_WRITE_COMMITTED. 
This return value
should only happen if the isolation level is TRANSACTION_CONSISTENT_READ.

After having reproduced this assert several times with instrumentation 
in the source it
seems like this problems is caused by that the Transaction object that 
the recordVersion
object has a pointer to and which is used for computing the relative 
state between the two
transaction "changes state" during the execution of 
Transaction::getRelativeState().

Here is what normally happens when the CommittedInvisible state is 
reached by the following code path:

1. Transaction::getRelativeState() is called and the following code is run:

if (transaction->state == Committed)
{
   // Return CommittedVisible if the other trans has a lower TransId and
   // it was committed when we started.

   if (visible (transaction, transId, FOR_WRITING))
       return CommittedVisible;

   return CommittedInvisible;
}

when this code is run the transaction->state == Committed (and the if 
test is true).
This results in Transaction::visible() being called.

2. In the code for Transaction::visible() the following code is run:

// If the other transaction is not yet committed, the trans is not visible.

if (transaction->state != Committed)
   return false;

When the same test now is evaluated "transaction->state != Committed" 
this test too
succeed and returns that the transaction should not be visible.

So based on traces in the code and studying core files it seems like the
transaction->state variable changes value from being Committed to not 
being Committed
during the execution of these two methods. This leads to 
Transaction::visible() returning "false" which again leads to
Transaction::getRelativeState() returning CommittedInvisible (and the 
assert occurs).

The reason for the "mysterious" state change (at least in some of the 
crashes I have reproduced) is that the transaction object which the 
record version points too is deleted by another thread and thus the 
check for committed (or not)  in Transaction::visible() is now checking 
"free memory".

This situation seems very similar to an issue Kevin had (and still 
have?) with the scavenger checking which records that can be scavenged.

Questions:

-how do we either handle that Transaction object that is being pointed 
to by RecordVersion objects are deleted at "any time". Like the code for 
Transaction::getRelativeState and Transaction::visible() is now this can 
lead to wrong conclusions about which records are visible or not.

-or do we need to introduce some kind of locking so that we can safely 
use RecordVersion objects and the Transaction object they normally 
refers to?

-or can we do something about the order we clean up old transaction 
objects versus the record version objects to avoid that we have record 
object without a transaction object?

I do not know enough about this part of the code (yet) to propose a good 
solution so any suggestions would be appreciated.

Olav


Thread
Problems with record visibility and how it is computedOlav Sandstaa21 Jan
  • search for null values in indexed columnsVladislav Vaintroub21 Jan
    • Re: search for null values in indexed columnsLars-Erik Bjørk21 Jan
    • Re: search for null values in indexed columnsKevin Lewis21 Jan
      • RE: search for null values in indexed columnsVladislav Vaintroub21 Jan
        • Re: search for null values in indexed columnsKevin Lewis21 Jan
      • Re: search for null values in indexed columnsJames Day23 Jan
  • Re: Problems with record visibility and how it is computedOlav Sandstaa23 Jan
    • Re: Problems with record visibility and how it is computedJim Starkey23 Jan
      • Re: Problems with record visibility and how it is computedKevin Lewis28 Jan
        • Re: Problems with record visibility and how it is computedJim Starkey28 Jan
          • Re: Problems with record visibility and how it is computedKevin Lewis28 Jan
            • Cycle Locking (was Problems with record visibility and how it iscomputed)Jim Starkey28 Jan
            • Re: Problems with record visibility and how it is computedAnn W. Harrison28 Jan
              • Re: Problems with record visibility and how it is computedJim Starkey28 Jan
                • Re: Problems with record visibility and how it is computedAnn W. Harrison28 Jan
                  • Re: Problems with record visibility and how it is computedOlav Sandstaa26 Feb
                    • New Transaction State object (Was: Problems with record visibility andhow it is computed)Olav Sandstaa16 Mar
                      • Re: New Transaction State object (Was: Problems with record visibilityand how it is computed)Kevin Lewis16 Mar
                      • Re: New Transaction State object (Was: Problems with record visibilityand how it is computed)Jim Starkey16 Mar
                        • Re: New Transaction State object (Was: Problems with record visibilityand how it is computed)Olav Sandstaa19 Mar
                          • Re: New Transaction State object (Was: Problems with record visibilityand how it is computed)Jim Starkey19 Mar
                            • RE: New Transaction State object (Was: Problems with record visibilityand how it is computed)Vladislav Vaintroub20 Mar
                            • Re: New Transaction State object (Was: Problems with record visibilityand how it is computed)Olav Sandstaa20 Mar
                              • Re: New Transaction State object (Was: Problems with record visibilityand how it is computed)Jim Starkey20 Mar
          • Re: Problems with record visibility and how it is computedOlav Sandstaa28 Jan
            • Re: Problems with record visibility and how it is computedJim Starkey28 Jan
              • Another Idea for Transaction Lifetime ControlJim Starkey28 Jan
                • Re: Another Idea for Transaction Lifetime ControlJim Starkey29 Jan
              • RE: Problems with record visibility and how it is computedXuekun Hu4 Feb
          • Re: Problems with record visibility and how it is computedAnn W. Harrison28 Jan
  • Quick question on row countsKeith Murphy24 Jan
    • Re: Quick question on row countsJim Starkey25 Jan
      • Re: Quick question on row countsKeith Murphy25 Jan