List:Falcon Storage Engine« Previous MessageNext Message »
From:Olav Sandstaa Date:March 19 2009 10:07pm
Subject:Re: New Transaction State object (Was: Problems with record visibility
and how it is computed)
View as plain text  
Hi Jim,

Thanks for comments. I have gotten into the coding and have a running 
prototype that fixes some of the issues we have seen. See some comment 
inline:

Jim Starkey wrote:
> I think TransactionState will also need:
>
>    * Transaction id (for validation and sanity control)

Agree but mainly for saving space by being able to remove it from the 
RecordVersion object

>    * Transaction state

Initially I agree with you but was also considering if the commitId 
would be enough (commitId == 0 means active transaction, commitId != 0 
means committed - enough in most situations where we use the 
TransactionState object). The main argument for not moving it from the 
Transaction object to the TransactionState object is that this is used 
many places in the Transaction code, also after the transaction has 
committed. And that makes it a bit difficult to let the Transaction 
object decrement the use count on the TransactionState object when the 
transaction commits (and that is needed if we do not want to use 
interlocked instructions for the use count). We can of course duplicate 
the transaction state in both the Transaction object and the 
TransactionState object (but I do not like that kind of duplication).

>    * syncPending (so it can be waited on)

The main problem with including this is that it increases the size of 
the TransactionState object from about 12 bytes to about 150 bytes (at 
least with the currently enabled tracing code). How many of these can 
"extreme testcases" produce?

In the initial prototype I am working on I have moved the syncIsActive 
(which is hopefully the same as you refer to as syncPending) to the 
TransactionState object.
We are only waiting for active transactions so in all committed 
TransactionState object this syncObject will never be used. In the 
current implementation of Transaction::waitForTransaction() already has 
a shared lock on the active transaction list. So I thing it will be 
possible to ensure that we safely can use the syncObject also when it is 
part of the Transaction object. The advantage of this is that we do not 
need to ten-double the size of the TransactionState object.

The drawback is that we can not easily get rid of neither the current 
shared lock on the active transaction list or the active transaction 
list (which I see you are aiming for). (and yes, I have an idea for how 
to get rid of the currently used shared lock in waitForTransaction()).

>
> This makes TransactionState the primary operational object and 
> Transaction mostly for bookkeeping.  I think this also eliminates the 
> need for TransactionManager::activeTransactions.

I agree but that should likely be a next step goal and not part of this 
(which started as a bug fix for 41357).

Olav

>
> Olav Sandstaa wrote:
>> Subj: New TransactionState object
>>
>> Hi,
>>
>> In our Athens meeting I went through the alternatives for how to 
>> solve the issues with the "RecordVersion to Transaction object 
>> pointer". We agreed on that introducing a "transaction state object" 
>> (proposal 9 quoted below) seemed to be the best and least intrusive 
>> alternative.
>>
>> I have not looked into the actual existing code yet but here is an 
>> initial overview of how my current plan for the "transaction state 
>> object" is (it might changes as I look more at the existing code and 
>> use of it):
>>
>> A. New data structure:
>> ======================
>>
>>   A new class called TransactionState will be introduced. This will 
>> as a minimum store the following members:
>>
>>   class TransactionState
>>   {
>>     TransId commitId;
>>     volatile INTERLOCKED useCount;      }
>>
>>   It is also likely that we will move the transaction id to this
>>   class (could save us from storing this in each RecordVersion
>>   object). The transaction state of the transaction might also be
>>   moved to this object.
>>
>>   Changes to existing classes:
>>
>>   1. The Transaction object will have a pointer to a TransactionState
>>      object. When committing the transaction the commitId will be
>>      stored in the TransactionState object, not in the transaction
>>      object.
>>
>>   2. The RecordVersion object will have a pointer to a
>>      TransactionState object (replacing the existing pointer to the
>>      Transaction object) (if there is need for accessing the
>>      Transaction object from the RecordVersion object we will either
>>      keep the transaction pointer or have a transaction pointer from
>>      the TransactionState object back to its Transaction object)
>>
>>   Goal with this new TransactionState object is to let it "live" as
>>   long as there is RecordVersion objects refering to it (and let the
>>   "real" transaction object be "purged" at "any time").
>>
>> B. Controlling the "life span" of TransactionState objects
>> ==========================================================
>>
>>   We will use a reference counter in the Transaction state object for
>>   controlling when a Transaction state object can be deleted.
>>   Operations on this reference counter will be done by using
>>   interlocked operations (if would probably be possible to implement
>>   this without interlocked instruction but that would make the
>>   implementation a bit more complex and harder to maintain and the
>>   discussion in Athens concluded that the cost of using interlocked
>>   operations for this case should not be a performance problem).
>>
>>   The reference counter will be incremented the following places:
>>
>>    1. In the Transcation's constructor when the TransactionState
>>       object is created (we can also delay this until the first
>>       Record is attached to a Transaction (Transaction::addRecord())
>>       but it is simpler to just create it when we create the
>>       Transaction object)
>>    2. For every RecordVersion having a pointer to it (most likely in
>>       the RecordVersion's constructor - have not looked at the code
>>       yet)
>>
>>   The reference counter will be decremented the following places:
>>
>>    1. In the Transaction's destructor
>>    2. When the RecordVersion is deleted (most likely in the
>>       destructor)
>>
>>   There should be no objects using the TransactionState object
>>   without having incremented the reference counter (or accessing it
>>   indirectly via the a Transaction object or a RecordVersion object).
>>
>> C. Usage of the TransactionState object:
>> ========================================
>>
>>   The initial usage of the TransactionState objects will be as
>>   follows:
>>
>>    1. Each time when creating a transaction object, we create a
>>       transaction state object and a pointer to the transaction state
>>       object from the transaction object.
>>
>>    2. When we commit: the commitId is written to the transaction
>>       state object (not the transaction object). So the commit is the
>>       update to the transaction state object (not that it is possible
>>       that we also will store the state of the Transaction in the
>>       TransactionState object)
>>
>>    3. Each time we create a record version object it has a pointer to
>>       the transaction state object - and the reference count is
>>       incremented (if needed also a pointer to the
>>       transaction object (hopefully not, alternatively have a pointer
>>       from the TransactionState object back to the transaction)
>>
>>    4. When we need to compute visibility of records we only need to
>>       use the transaction state object. This will have at least the
>>       same life span as the record version objects pointing to it
>>       (thus the pointer is guaranteed to be valid)
>>
>>    5. When the scavenger removes a recordversion it decrement the
>>       reference counter in the transaction state object
>>
>>    6. When we purge old Transaction objects we decrement the referece
>>       counter in the Transaction State object
>>
>>    7. When the reference counter reaches 0 we delete the transaction
>>       state object (by "self destruction")
>>
>>
>> That is the current plan - and as most plans it might change :-)
>>
>> Please let me know if you have any feedback or suggestions.
>>
>> Olav
>>
>> Olav Sandstaa wrote:
>>> In addition to these I also have two new possible solutions:
>>>
>>> 9. (This is partly based on something Ann wrote: Make the 
>>> transasction pointer indirect combined with my original proposal 
>>> number 5: let the transaction objects live as long as there are 
>>> record version pointing to them): Introduce a new "Transaction state 
>>> object" that contains as a minimum the "commitId" but most likely 
>>> also the "transactionId" and "state":
>>>
>>>     1. Each time when creating a transaction object, we create a 
>>> transaction state object and a pointer to the transaction state 
>>> object from the transaction object.
>>>     2. When we commit: the commitId is written to the transaction 
>>> state object (not the transaction object). So the commit is the 
>>> update to the transaction state object.
>>>     3. Each time we create a record version object it has a pointer 
>>> to the transaction state object (and if needed also to the 
>>> transaction object)
>>>     4. Each time we add a record to a transaction (ie. update a 
>>> record) we increment a useCount in the transaction state object: 
>>> This should not need any extra locking (I think)
>>>     5. When we need to compute visibility of records we only need to 
>>> use the transaction state object. This will have at least the same 
>>> life span as the record version objects pointing to it (thus the 
>>> pointer is guaranteed to be valid)
>>>     6. When the scavenger removes a recordversion it decrement the 
>>> useCount in the transaction state object (I think this can be done 
>>> safely without locking given that we have one scavenger)
>>>     7. When the useCount reaches 0 we delete the transaction state 
>>> object.
>>>
>>>   Issue/question: is there any other places in the code that will 
>>> access this transaction state object than by accessing it from the 
>>> record version object?
>>>
>>>    I assume when normal transactions accesses record version objects 
>>> they are "locked" and this will implicitely also lock the 
>>> transaction state object when normal transactions access it?
>>
>>
>
>

Thread
Problems with record visibility and how it is computedOlav Sandstaa21 Jan
  • search for null values in indexed columnsVladislav Vaintroub21 Jan
    • Re: search for null values in indexed columnsLars-Erik Bjørk21 Jan
    • Re: search for null values in indexed columnsKevin Lewis21 Jan
      • RE: search for null values in indexed columnsVladislav Vaintroub21 Jan
        • Re: search for null values in indexed columnsKevin Lewis21 Jan
      • Re: search for null values in indexed columnsJames Day23 Jan
  • Re: Problems with record visibility and how it is computedOlav Sandstaa23 Jan
    • Re: Problems with record visibility and how it is computedJim Starkey23 Jan
      • Re: Problems with record visibility and how it is computedKevin Lewis28 Jan
        • Re: Problems with record visibility and how it is computedJim Starkey28 Jan
          • Re: Problems with record visibility and how it is computedKevin Lewis28 Jan
            • Cycle Locking (was Problems with record visibility and how it iscomputed)Jim Starkey28 Jan
            • Re: Problems with record visibility and how it is computedAnn W. Harrison28 Jan
              • Re: Problems with record visibility and how it is computedJim Starkey28 Jan
                • Re: Problems with record visibility and how it is computedAnn W. Harrison28 Jan
                  • Re: Problems with record visibility and how it is computedOlav Sandstaa26 Feb
                    • New Transaction State object (Was: Problems with record visibility andhow it is computed)Olav Sandstaa16 Mar
                      • Re: New Transaction State object (Was: Problems with record visibilityand how it is computed)Kevin Lewis16 Mar
                      • Re: New Transaction State object (Was: Problems with record visibilityand how it is computed)Jim Starkey16 Mar
                        • Re: New Transaction State object (Was: Problems with record visibilityand how it is computed)Olav Sandstaa19 Mar
                          • Re: New Transaction State object (Was: Problems with record visibilityand how it is computed)Jim Starkey19 Mar
                            • RE: New Transaction State object (Was: Problems with record visibilityand how it is computed)Vladislav Vaintroub20 Mar
                            • Re: New Transaction State object (Was: Problems with record visibilityand how it is computed)Olav Sandstaa20 Mar
                              • Re: New Transaction State object (Was: Problems with record visibilityand how it is computed)Jim Starkey20 Mar
          • Re: Problems with record visibility and how it is computedOlav Sandstaa28 Jan
            • Re: Problems with record visibility and how it is computedJim Starkey28 Jan
              • Another Idea for Transaction Lifetime ControlJim Starkey28 Jan
                • Re: Another Idea for Transaction Lifetime ControlJim Starkey29 Jan
              • RE: Problems with record visibility and how it is computedXuekun Hu4 Feb
          • Re: Problems with record visibility and how it is computedAnn W. Harrison28 Jan
  • Quick question on row countsKeith Murphy24 Jan
    • Re: Quick question on row countsJim Starkey25 Jan
      • Re: Quick question on row countsKeith Murphy25 Jan