List:Internals« Previous MessageNext Message »
From:Jay Pipes Date:May 13 2009 12:09pm
Subject:Re: MySQL Reengineering Project
View as plain text  
Alex Esterkin wrote:
> Jay,
> 
> I don't think intra-statement parallelism implementation belongs in a
> database server microkernel.  However, this aspect should be discussed
> at the Drizzle-discuss forum.

You're absolutely correct, Alex.  I mistakenly wrote intra-statement 
when I described and meant to say *intra-Session*.

> In general, having worked on intra-statement parallelism architecture
> and implementation at Dataupia (in the Postgres server grid context),
> I find this objective to be unrealistic for MySQL.   For N shards you
> will need N*(N-1)/2 exchange nodes - asynchronous, with real time load
> balancing, capable of dealing with data skew, and a zillion more
> issues to solve, such as not being able to rely on the determinism of
> demand-driven execution.
> 
> Drizzle has a better shot at intra-statement parallelism - leveraging
> Gearman as opposed to doing it inside a Session.
> 
> Regards,
> 
> Alex Esterkin
> 
> On Tue, May 12, 2009 at 4:58 PM, Jay Pipes <Jay.Pipes@stripped> wrote:
>> Sergei Golubchik wrote:
>>
>>>>>> Without doing this, and using encapsulation so that a THD can
> have
>>>>>> multiple Statements, it will be very difficult to work on any
> future
>>>>>> parallelization efforts.
>>>>> You mean, a Statement can have multiple THDs, I suppose :)
>>>> No he means what he is saying. I don't know why you would want a
>>>> statement shared across multiple THD, but having a THD be able to handle
>>>> multiple statements means that you can do asynchronous queries within a
>>>> single connection.
>>> Ah, okay. I see.
>>>
>>> I thought that "parallelization", that Jay mentioned, means executing
>>> parts of a single statement in different threads - which, indeed, may
>>> need two THDs sharing the same Statement.
>> Yes, what Brian says...once the THD is distinguised from a pthread,
>> intra-statement parallelization is possible.  We've only just begun this
>> step in Drizzle (a Session is now no longer intricately linked to a pthread
>> in Drizzle) but there's clearly a ton more to do. :) I know Alex Esterkin
>> thinks there is no reason to do such as thing, but perhaps he just needs an
>> example :)
>>
>> Imagine a Session which sends a few long-running SELECTs in the same client
>> connection.  Currently, because the THD is linked to a single pthread in
>> MySQL, these SELECTs will not only block each other, but will be executed in
>> order.  What if there was no reason to do so?  The three Statements, if a
>> Session contained a vector of Statement objects, could parse and optimize
>> all three statements and decide to send two of them off to other scheduler
>> threads for execution, essentially parallelizing the operation (particularly
>> on a distributed node architecture...)
>>
>> Sure, like Alex mentioned, Drizzle's protocol supports parallel operations
>> (well, not really, it is non-blocking operations, but still...similar
>> concept).  But, separating the notion of a thread with a Statement means
>> that we can achieve similar theoretical points for parallel operations.  The
>> more options/points for parallel opportunities, the better we can scale, no?
>>
>> Cheers!
>>
>> Jay
>>
>> --
>> MySQL Internals Mailing List
>> For list archives: http://lists.mysql.com/internals
>> To unsubscribe:
>>  http://lists.mysql.com/internals?unsub=1
>>
>>

Thread
MySQL Reengineering ProjectManyi Lu - Sun Norway12 May
  • Re: MySQL Reengineering ProjectJay Pipes12 May
  • Re: MySQL Reengineering ProjectMARK CALLAGHAN12 May
    • Re: MySQL Reengineering ProjectMats Kindahl12 May
      • Re: MySQL Reengineering ProjectManyi Lu - Sun Norway12 May
        • Re: MySQL Reengineering ProjectMats Kindahl12 May
    • Re: MySQL Reengineering ProjectStewart Smith13 May
  • Re: MySQL Reengineering ProjectMikiya Okuno18 May
    • Re: MySQL Reengineering ProjectSergei Golubchik18 May
      • Re: MySQL Reengineering ProjectRoy Lyseng18 May
Re: MySQL Reengineering ProjectAlaric Snell-Pym12 May
  • Re: MySQL Reengineering ProjectMats Kindahl12 May
    • Re: MySQL Reengineering ProjectJay Pipes12 May
      • Re: MySQL Reengineering ProjectJay Pipes12 May
        • Re: MySQL Reengineering ProjectAlex Esterkin12 May
          • Re: MySQL Reengineering ProjectBrian Aker12 May
      • Re: MySQL Reengineering ProjectSergei Golubchik12 May
        • Re: MySQL Reengineering ProjectBrian Aker12 May
          • Re: MySQL Reengineering ProjectSergei Golubchik12 May
            • Re: MySQL Reengineering ProjectAlex Esterkin12 May
              • Re: MySQL Reengineering ProjectBrian Aker12 May
            • Re: MySQL Reengineering ProjectJay Pipes12 May
              • Re: MySQL Reengineering ProjectDavi Arnaut13 May
              • Re: MySQL Reengineering ProjectAlex Esterkin13 May
                • Re: MySQL Reengineering ProjectJay Pipes13 May
            • Re: MySQL Reengineering ProjectMichael Widenius14 May
          • Re: MySQL Reengineering ProjectJonas Oreland13 May
        • Re: MySQL Reengineering ProjectRoy Lyseng12 May
      • Re: MySQL Reengineering ProjectMats Kindahl12 May
    • Re: MySQL Reengineering ProjectAlaric Snell-Pym12 May
      • Re: MySQL Reengineering ProjectMats Kindahl12 May
      • Re: MySQL Reengineering ProjectMichael Widenius14 May
        • Re: MySQL Reengineering ProjectAlaric Snell-Pym15 May
          • Re: MySQL Reengineering ProjectKristian Nielsen15 May
            • Re: MySQL Reengineering ProjectBrian Aker15 May
            • Re: MySQL Reengineering ProjectAlaric Snell-Pym15 May
          • Re: MySQL Reengineering ProjectMichael Widenius6 Jun
Re: MySQL Reengineering ProjectMats Kindahl12 May
Re: MySQL Reengineering ProjectAlex Esterkin12 May
  • Re: MySQL Reengineering ProjectMasood Mortazavi13 May
    • Re: MySQL Reengineering ProjectBrian Aker13 May
    • Re: MySQL Reengineering ProjectAlex Esterkin13 May
      • Re: MySQL Reengineering ProjectBaron Schwartz13 May
        • Re: MySQL Reengineering ProjectStewart Smith13 May
          • Re: MySQL Reengineering ProjectBaron Schwartz13 May
    • Re: MySQL Reengineering ProjectAlex Esterkin13 May
    • Re: MySQL Reengineering ProjectStewart Smith13 May
  • Re: MySQL Reengineering ProjectMichael Widenius14 May