Vladislav Vaintroub wrote:
> Ok, I see.
>
> But, now unrelated to the patch, threads are thought to improve performance,
> not for sleeping and designs where thread does nothing else then sleep()
> most of the time does not feel aesthetically pleasing to me.
> And many other people also think that threads are for concurrency rather than
> simplifying
> programming tasks
I agree that running updateCardinalities() on the same thread as record
scavenging would be ideal, but is it necessary?
I think this is a good design, given the requirements:
1) We must scavenge during memory-intensive DDL operations like CREATE
INDEX.
- This is a requirement.
2) updateCardinalities() requires a DDL lock, the Scavenger does not.
- The reason to decouple these tasks.
3) We want to run updateCardinalities() with each scavenge.
- Ideal, but only a nice-to-have.
4) We want updateCardinalities() to closely reflect the actual cardinality.
- This is important. The frequency of load-based scavenges is driven
by system activity. Running
updateCardinalities() with the scavenge keeps the cardinality
close to actual.
Loosely coupling scavenge and updateCardinalities() achieves these goals
and eliminates a pathological dependency.
I also believe that the "loosely coupled" paradigm will serve us better
down the road--more robust, less brittle.
Perhaps we could have a pool of Falcon utility threads with which a
foreground task can signal one of several pre-registered tasks to run
concurrently. This would be worth implementing, I think.
> Here for example, the guy titles his blog entry "Designing for high
> performance" and what he talks about? Reducing number of threads;)
>
> http://blogs.technet.com/winserverperformance/archive/2008/04/25/designing-applications-for-high-performance-part-1.aspx
>
Good article. Thanks!
One point was to minimize cross-CPU interaction and minimize updates to
shared data. I agree that such analysis should be a pre-requisite of
assigning a concurrent task, but practically speaking, what does that
mean in the Falcon context?
Adding one or two utility threads will have little or no overall impact
in the context of the other Falcon system threads (gophers, io, etc),
plus the tens, hundreds or even thousands of client connection threads.
This will be a great topic for discussion in March. Thanks for the feedback.
>
>> -----Original Message-----
>> From: Christopher.Powers@stripped [mailto:Christopher.Powers@stripped]
>> Sent: Thursday, January 15, 2009 3:17 AM
>> To: Vladislav Vaintroub
>> Cc: 'Kevin Lewis'; 'FalconDev'
>> Subject: Re: More on Scavenge vs DDL
>>
>> Vladislav Vaintroub wrote:
>>> Chris, just out of curiosity - is it necessary to have a thread for
>> each
>>> single task? I might be mistaken, but do not we have a timer or
>> similar for
>>> that, specifically for periodic tasks?
>> Yes, we have two timing threads, the Scheduler and the Ticker. The
>> Scheduler runs tasks at predefined fixed-length intervals. The Ticker
>> runs them every second.
>>
>> I initially proposed scheduling updateCardinalities() as a separate
>> task, like the Scavenger. Kevin suggested that it would be better to
>> signal updateCardinalities *from* the Scavenger, which is now more
>> responsive to cache activity.
>>
>> Cardinality has historically been updated at scavenge time because
>> that's a naturally good time to do it--both tasks process the entire
>> record tree.
>>
>> This approach has three benefits:
>>
>> 1) Scavenger and updateCardinalities() are independent (no syncSysDDL
>> nonesense).
>> 2) Cardinality will more closely reflect reality because it is invoked
>> at scavenge time.
>> 3) We retain the intent of the original design.
>>
>>
>>>> -----Original Message-----
>>>> From: Christopher.Powers@stripped [mailto:Christopher.Powers@stripped]
>>>> Sent: Thursday, January 15, 2009 2:04 AM
>>>> To: Kevin Lewis
>>>> Cc: FalconDev
>>>> Subject: More on Scavenge vs DDL
>>>>
>>>> I decoupled updateCardinalities() from the Scavenger and moved it on
>> to
>>>> its own thread. Now, each scavenge simply signals it and moves on,
>> no
>>>> waiting on syncSysDDL.
>>>>
>>>> BUT...
>>>>
>>>> Database::scavengeRecords() commits pending system transactions,
>> which
>>>> of course requires a lock syncSysDDL. In the old code, this was done
>>>> only inside of syncScavenge AND if the scavenge was not 'forced'.
>>>>
>>>> Load-based scavenges are the current equivalent of forced scavenges,
>>>> and
>>>> also should not commit pending system transactions.
>>>>
>>>>
>>>> // NEW CODE
>>>>
>>>> void Database::scavengeRecords(void)
>>>> {
>>>> // Commit pending system transactions before proceeding
>>>>
>>>> if (systemConnection->transaction)
>>>> commitSystemTransaction();
>>>>
>>>> Sync syncScavenger(&syncScavenge,
>>>> "Database::scavengeRecords(Scavenge)");
>>>> syncScavenger.lock(Exclusive)
>>>> [...]
>>>>
>>>>
>>>> // OLD CODE
>>>>
>>>> void Database::retireRecords(bool forced)
>>>> {
>>>> int cycle = scavengeCycle;
>>>>
>>>> Sync syncScavenger(&syncScavenge,
>> "Database::retireRecords(1)");
>>>> syncScavenger.lock(Exclusive);
>>>>
>>>> if (forced && scavengeCycle > cycle)
>>>> return;
>>>>
>>>> // Commit pending system transactions before proceeding
>>>>
>>>> if (!forced && systemConnection->transaction)
>>>> commitSystemTransaction();
>>>>
>>>> --
>>>> Falcon Storage Engine Mailing List
>>>> For list archives: http://lists.mysql.com/falcon
>>>> To unsubscribe: http://lists.mysql.com/falcon?unsub=1
>>>
>> --
>> Falcon Storage Engine Mailing List
>> For list archives: http://lists.mysql.com/falcon
>> To unsubscribe: http://lists.mysql.com/falcon?unsub=1
>
>
