From: Jim Starkey
Date: December 12 2008 3:20pm
Subject: Re: Lock Waits
List-Archive: http://lists.mysql.com/falcon/304
Message-Id: <49428144.6050107@nimbusdb.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Falcon uses a user mode SyncObject for thread synchronization.
SyncObject is portable, supports read/write locks, and handles
recursion.  It is also fair in the sense that locks are granted in the
order requested.  Instrumenting pthread mutex and rwlocks does nothing
for Falcon.

As for the callback during monitoring, there are at least three (no! 
four!) reasons
why it is desirable, if not necessary:

   1. A goal is to be unobtrusive as possible when monitoring is
      compiled in but turned off.  Adding memory for monitoring to the
      objects monitored bulks them up, changing the profile of memory
      usage.  Since Falcon allocates hundreds of thousand of SyncObjects
      (one per buffer in the page cache), this can amount to a
      significant amount of memory.  Allocating memory only for actual
      monitored objects mitigates this problem.
   2. It isn't enough to monitor usage.  You also have to be able to
      collect the data.  Without a central registry of monitored object,
      how are you going to collect the data for reporting?  In some
      cases, the monitored object themselves may have been deleted.
      Without that data, how can someone draw accurate conclusions?
   3. Potential deadlock analysis is a difficult but necessary process.
      Centralizing monitoring allows collection of data on overlapping
      locks, data that can reveal potential deadlocks.
   4. A central monitoring facility can track locks on a per thread 
basis.  Handling this in target objects is expensive in code, effort, 
debugging, and resource utilization.

I'm hope your message was in the spirit of on-going discussion rather
than the delivery of stone tablets memorializing decisions made.


Peter Gulutzan wrote:
> Hi Jim,
>
> Jim Starkey wrote:
>   
>> Waiting for the hotel shuttle at the Riga airport, Chris and Robin
>> raised the question of instrumenting lock waits in Falcon.  While it is
>> true that 99.9% of lock (SyncObject) requests are granted without
>> contention, it would be nice to know a great deal more about the 0.1%
>> that gum up the works.
>>
>>     
>
> I hope that "instrumenting lock waits" includes instrumenting mutexes
> rwlocks etc., in which case I think we've made relevant progress.
>
>   
>> What I think makes sense is a mechanism for a thread to register itself
>> before it actually waits on something with a corresponding call when it
>> wakes up after the look has been granted.  Since we would only register
>> waits, the overhead should be insignificant.  We would want to registers
>> waits on SyncObjects and transactions (we might even register the record
>> we're blocking on).
>>
>> A off-the-top-of-my-head API might be something like:
>>
>>    WaitMask    waitEvents;
>>    WaitToken   waitRegister(int waitType, void *waitObject);
>>    void             waitCompleted(WaitToken token);
>>
>> (by all means, change the names).
>>
>>     
>
> Sorry, we change more than the names and (rather than registering
> a routine to call) we have fixed code which either gets executed
> or skipped. The spec and the code now exist in mysql-6.0-perf tree.
>
> We surround code snippets with instrumentation before/after macros.
> Users can select from read-only in-memory tables and find
> * what the latest mutex/rwlock/etc. waits were for each thread,
>   and how long they took
> * summary of the above, by thread
> * other stuff but I won't repeat what's in WL#2360 specification.
>
> At the Riga meeting Chris liked our (Marc Alff's and my) description,
> and now the specification is 'done' unless Mikael rejects it.
>
>   
>> The global mask would indicate which classes of events are interesting
>> to the wait monitor; if zero, nobody would register anything.
>>     
>
> Yes.
>
>   
>> If
>> non-zero, specific wait types would be registered.  Timing and
>> accumulating would be the responsibility of the wait monitor, not the
>> mainline code.
>>     
>
> No. Flexibility = a bit more overhead, which we can ill afford.
> I posited the idea of a callback in discussions, there were no takers.
>
>   
>> The monitor could, if it understood a specific wait
>> type, get more specific information about the the blocking object
>> (SyncObject name, table name and record number for record, etc.).
>>
>>     
>
> Yes.
>
>   
>> Even without Robin pushing for external use, this might be a very useful
>> tool for internal analysis, particularly with Philip's pathological test
>> cases.
>>     
>
> Our big hope is the "external use" bit, especially via Enterprise Tools.
>
>   


-- 
Jim Starkey
President, NimbusDB, Inc.
978 526-1376