List:Falcon Storage Engine« Previous MessageNext Message »
From:Christopher Powers Date:January 15 2009 6:06pm
Subject:Re: More on Scavenge vs DDL
View as plain text  
Vladislav Vaintroub wrote:
> Ok, I see.
> 
> But, now unrelated to the patch, threads are thought to improve performance,
> not for sleeping and designs where thread does nothing else then sleep()
> most of the time does not feel aesthetically pleasing to me.
> And many other people also think that threads are for concurrency rather than
> simplifying
> programming tasks

I agree that running updateCardinalities() on the same thread as record 
scavenging would be ideal, but is it necessary?

I think this is a good design, given the requirements:

1) We must scavenge during memory-intensive DDL operations like CREATE 
INDEX.
    - This is a requirement.

2) updateCardinalities() requires a DDL lock, the Scavenger does not.
    - The reason to decouple these tasks.

3) We want to run updateCardinalities() with each scavenge.
    - Ideal, but only a nice-to-have.

4) We want updateCardinalities() to closely reflect the actual cardinality.
    - This is important. The frequency of load-based scavenges is driven 
by system activity. Running
      updateCardinalities() with the scavenge keeps the cardinality 
close to actual.

Loosely coupling scavenge and updateCardinalities() achieves these goals 
and eliminates a pathological dependency.

I also believe that the "loosely coupled" paradigm will serve us better 
down the road--more robust, less brittle.

Perhaps we could have a pool of Falcon utility threads with which a 
foreground task can signal one of several pre-registered tasks to run 
concurrently. This would be worth implementing, I think.

> Here for example, the guy titles his blog entry "Designing for high
> performance" and what he talks about? Reducing number of threads;)
>
> http://blogs.technet.com/winserverperformance/archive/2008/04/25/designing-applications-for-high-performance-part-1.aspx
> 

Good article. Thanks!

One point was to minimize cross-CPU interaction and minimize updates to 
shared data. I agree that such analysis should be a pre-requisite of 
assigning a concurrent task, but practically speaking, what does that 
mean in the Falcon context?

Adding one or two utility threads will have little or no overall impact 
in the context of the other Falcon system threads (gophers, io, etc), 
plus the tens, hundreds or even thousands of client connection threads.

This will be a great topic for discussion in March. Thanks for the feedback.


> 
>> -----Original Message-----
>> From: Christopher.Powers@stripped [mailto:Christopher.Powers@stripped]
>> Sent: Thursday, January 15, 2009 3:17 AM
>> To: Vladislav Vaintroub
>> Cc: 'Kevin Lewis'; 'FalconDev'
>> Subject: Re: More on Scavenge vs DDL
>>
>> Vladislav Vaintroub wrote:
>>> Chris, just out of curiosity - is it necessary to have a thread for
>> each
>>> single task? I might be mistaken, but do not  we have a timer or
>> similar for
>>> that, specifically for periodic tasks?
>> Yes, we have two timing threads, the Scheduler and the Ticker. The
>> Scheduler runs tasks at predefined fixed-length intervals. The Ticker
>> runs them every second.
>>
>> I initially proposed scheduling updateCardinalities() as a separate
>> task, like the Scavenger. Kevin suggested that it would be better to
>> signal updateCardinalities *from* the Scavenger, which is now more
>> responsive to cache activity.
>>
>> Cardinality has historically been updated at scavenge time because
>> that's a naturally good time to do it--both tasks process the entire
>> record tree.
>>
>> This approach has three benefits:
>>
>> 1) Scavenger and updateCardinalities() are independent (no syncSysDDL
>> nonesense).
>> 2) Cardinality will more closely reflect reality because it is invoked
>> at scavenge time.
>> 3) We retain the intent of the original design.
>>
>>
>>>> -----Original Message-----
>>>> From: Christopher.Powers@stripped [mailto:Christopher.Powers@stripped]
>>>> Sent: Thursday, January 15, 2009 2:04 AM
>>>> To: Kevin Lewis
>>>> Cc: FalconDev
>>>> Subject: More on Scavenge vs DDL
>>>>
>>>> I decoupled updateCardinalities() from the Scavenger and moved it on
>> to
>>>> its own thread. Now, each scavenge simply signals it and moves on,
>> no
>>>> waiting on syncSysDDL.
>>>>
>>>> BUT...
>>>>
>>>> Database::scavengeRecords() commits pending system transactions,
>> which
>>>> of course requires a lock syncSysDDL. In the old code, this was done
>>>> only inside of syncScavenge AND if the scavenge was not 'forced'.
>>>>
>>>> Load-based scavenges are the current equivalent of forced scavenges,
>>>> and
>>>> also should not commit pending system transactions.
>>>>
>>>>
>>>> // NEW CODE
>>>>
>>>> void Database::scavengeRecords(void)
>>>> {
>>>>     // Commit pending system transactions before proceeding
>>>>
>>>>     if (systemConnection->transaction)
>>>>        commitSystemTransaction();
>>>>
>>>>     Sync syncScavenger(&syncScavenge,
>>>> "Database::scavengeRecords(Scavenge)");
>>>>     syncScavenger.lock(Exclusive)
>>>> [...]
>>>>
>>>>
>>>> // OLD CODE
>>>>
>>>> void Database::retireRecords(bool forced)
>>>> {
>>>>      int cycle = scavengeCycle;
>>>>
>>>>      Sync syncScavenger(&syncScavenge,
>> "Database::retireRecords(1)");
>>>>      syncScavenger.lock(Exclusive);
>>>>
>>>>      if (forced && scavengeCycle > cycle)
>>>>         return;
>>>>
>>>>      // Commit pending system transactions before proceeding
>>>>
>>>>      if (!forced && systemConnection->transaction)
>>>>         commitSystemTransaction();
>>>>
>>>> --
>>>> Falcon Storage Engine Mailing List
>>>> For list archives: http://lists.mysql.com/falcon
>>>> To unsubscribe:    http://lists.mysql.com/falcon?unsub=1
>>>
>> --
>> Falcon Storage Engine Mailing List
>> For list archives: http://lists.mysql.com/falcon
>> To unsubscribe:    http://lists.mysql.com/falcon?unsub=1
> 
> 
Thread
More on Scavenge vs DDLChristopher Powers15 Jan
  • RE: More on Scavenge vs DDLVladislav Vaintroub15 Jan
    • Re: More on Scavenge vs DDLChristopher Powers15 Jan
      • RE: More on Scavenge vs DDLVladislav Vaintroub15 Jan
        • Re: More on Scavenge vs DDLChristopher Powers15 Jan
  • Re: More on Scavenge vs DDLChristopher Powers15 Jan