List:Falcon Storage Engine« Previous MessageNext Message »
From:Olav Sandstaa Date:December 3 2008 10:17pm
Subject:Dependency manager and Transaction::commitRecords()
View as plain text  

Just sharing some information if someone is interested or feel for 
commenting - feedback is appreciated (but feel free to press the delete 
(or power)  button too):

One of the uses of the old (as well as the new) dependency manager is to 
determine when it is safe to call Transaction::commitRecords() (and in 
most cases also being able to delete/release the transaction object 
itself). When using the old dependency manager one of the requirement 
was that the dependency counter had reached zero before calling 

In the code using the old dependency manager 
Transaction::commitRecords() could be called in four/five different places:

1. Transaction::commit()

2 a+b). Transaction::releaseDependency() called either from 
Transaction::expungeTransaction() or Transaction::releaseDependencies().

3. Transaction::writeComplete() (callback from serial log/gopher thread)

4. TransactionManager::purgeTransaction() (clean-up of transaction by 
the scavenger).

In the initial patch for the new dependency manager 
Transaction::commitRecords is called only from one place: 
Transaction::purgeTransaction() (case 4 above), i.e. it is not called 
until the Scavenger comes around (about every 30 seconds) [1]. Case 2a+b 
has been eliminated by removing the old dependency manager code. The 
code for case 1 and 3 have been temporarily commented out with a comment 
that "this code will be fixed by the next patch" :-) . Still, with only 
case 4 in the code it seems that Falcon works both correctly and "nice" 
but it might not be optimal to delay all calls to 
Transaction::commitRecords() until the scavenger comes around.

So to get some data about which of the above cases that actually are in 
use I instrumented the code and ran the falcon test suite. The result is 
(number of calls to commitRecords()):

Transaction::commit:                                      1074
Transaction::releaseDependency:                10839
Transaction::writeComplete:                      179389
TransactionManager::purgeTransaction: 168713

Some more instrumentation revealed that for most Transactions 
commitRecords() was called twice. And further, in the table above *all* 
calls done by Transaction::commit() and 
TransactionManager::purgeTransactions() actually was a "second" call to 

This leaves me believe/suggest the following for the new dependency 
manager for how and when to call commitRecords() (and delete committed 
transaction objects):

* Case 1 above (in Transaction::commit()) is not necessary at all - 
remove it.

* Case 2a+b: is already gone since the old dependency manger code is gone

* Case 3: is very useful but in order to re-implement the "old 
dependency check" that just checks that the dependencies counter is 
zero, with the new dependency manager I would need to introduce a shared 
lock on the active transaction list to get the "startEvent"  (also now 
known as the transactionId) of the oldest active transaction in order to 
do the "new dependency check". This does not seem like the best thing to do.

  So instead (based on a suggestion from Jim) I think the best (and 
easiest) way to call Transaction::commitRecords() and delete old 
transaction objects will be to do something similar to what the new 
implementation of TransactionManager::purgeTransaction() does each time 
we commit a record when we anyway have the exclusive lock on both the 
active and committed lists. This code will call commitRecords() and 
release the transaction objects from the front of the committed 
transaction list as long as they  have a commitId (commitEvent) that is 
smaller than the transactionId (startEvent) of the oldest active 
transactions. The cost per transaction object should be very "similar" 
to the cost of doing the same in the scavenger.

* Case 4: TransactionManager::purgeTransaction() can probably be deleted 
(if I implement what suggested above).

One extra benefit of this that instead of having five code paths that 
lead to commitRecords() in the old implementation, the new 
implementation will be simpler with commitRecords() only called in a 
single place.

Everyone that is still reading and think they understand what I wrote: 
feel free to comment as I might have misunderstood or jumped to wrong 
conclusions. Everyone else: feel free to ask questions :-)

Thanks for listening! (and yes, I will update the worklog....)


[1] The "record number" of committed transaction ready for clean-up by 
the scavenger I have seen in one of my tests is 330.000 transactions :-) 
(but that will not be typical because in the initial patch also 
read-only transactions ends up in the committed list)

Dependency manager and Transaction::commitRecords()Olav Sandstaa3 Dec
  • Re: Dependency manager and Transaction::commitRecords()Kevin Lewis4 Dec
    • Re: Dependency manager and Transaction::commitRecords()Olav Sandstaa4 Dec
      • Re: Dependency manager and Transaction::commitRecords()Olav Sandstaa4 Dec
        • Re: Dependency manager and Transaction::commitRecords()Kevin Lewis4 Dec
          • Re: Dependency manager and Transaction::commitRecords()Olav Sandstaa5 Dec
            • Re: Dependency manager and Transaction::commitRecords()Olav Sandstaa8 Dec