List:Internals« Previous MessageNext Message »
From:Alex Esterkin Date:May 13 2009 12:25am
Subject:Re: MySQL Reengineering Project
View as plain text  
Jay,

I don't think intra-statement parallelism implementation belongs in a
database server microkernel.  However, this aspect should be discussed
at the Drizzle-discuss forum.

In general, having worked on intra-statement parallelism architecture
and implementation at Dataupia (in the Postgres server grid context),
I find this objective to be unrealistic for MySQL.   For N shards you
will need N*(N-1)/2 exchange nodes - asynchronous, with real time load
balancing, capable of dealing with data skew, and a zillion more
issues to solve, such as not being able to rely on the determinism of
demand-driven execution.

Drizzle has a better shot at intra-statement parallelism - leveraging
Gearman as opposed to doing it inside a Session.

Regards,

Alex Esterkin

On Tue, May 12, 2009 at 4:58 PM, Jay Pipes <Jay.Pipes@stripped> wrote:
> Sergei Golubchik wrote:
>
>>>>> Without doing this, and using encapsulation so that a THD can have
>>>>> multiple Statements, it will be very difficult to work on any future
>>>>> parallelization efforts.
>>>>
>>>> You mean, a Statement can have multiple THDs, I suppose :)
>>>
>>> No he means what he is saying. I don't know why you would want a
>>> statement shared across multiple THD, but having a THD be able to handle
>>> multiple statements means that you can do asynchronous queries within a
>>> single connection.
>>
>> Ah, okay. I see.
>>
>> I thought that "parallelization", that Jay mentioned, means executing
>> parts of a single statement in different threads - which, indeed, may
>> need two THDs sharing the same Statement.
>
> Yes, what Brian says...once the THD is distinguised from a pthread,
> intra-statement parallelization is possible.  We've only just begun this
> step in Drizzle (a Session is now no longer intricately linked to a pthread
> in Drizzle) but there's clearly a ton more to do. :) I know Alex Esterkin
> thinks there is no reason to do such as thing, but perhaps he just needs an
> example :)
>
> Imagine a Session which sends a few long-running SELECTs in the same client
> connection.  Currently, because the THD is linked to a single pthread in
> MySQL, these SELECTs will not only block each other, but will be executed in
> order.  What if there was no reason to do so?  The three Statements, if a
> Session contained a vector of Statement objects, could parse and optimize
> all three statements and decide to send two of them off to other scheduler
> threads for execution, essentially parallelizing the operation (particularly
> on a distributed node architecture...)
>
> Sure, like Alex mentioned, Drizzle's protocol supports parallel operations
> (well, not really, it is non-blocking operations, but still...similar
> concept).  But, separating the notion of a thread with a Statement means
> that we can achieve similar theoretical points for parallel operations.  The
> more options/points for parallel opportunities, the better we can scale, no?
>
> Cheers!
>
> Jay
>
> --
> MySQL Internals Mailing List
> For list archives: http://lists.mysql.com/internals
> To unsubscribe:
>  http://lists.mysql.com/internals?unsub=1
>
>
Thread
MySQL Reengineering ProjectManyi Lu - Sun Norway12 May
  • Re: MySQL Reengineering ProjectJay Pipes12 May
  • Re: MySQL Reengineering ProjectMARK CALLAGHAN12 May
    • Re: MySQL Reengineering ProjectMats Kindahl12 May
      • Re: MySQL Reengineering ProjectManyi Lu - Sun Norway12 May
        • Re: MySQL Reengineering ProjectMats Kindahl12 May
    • Re: MySQL Reengineering ProjectStewart Smith13 May
  • Re: MySQL Reengineering ProjectMikiya Okuno18 May
    • Re: MySQL Reengineering ProjectSergei Golubchik18 May
      • Re: MySQL Reengineering ProjectRoy Lyseng18 May
Re: MySQL Reengineering ProjectAlaric Snell-Pym12 May
  • Re: MySQL Reengineering ProjectMats Kindahl12 May
    • Re: MySQL Reengineering ProjectJay Pipes12 May
      • Re: MySQL Reengineering ProjectJay Pipes12 May
        • Re: MySQL Reengineering ProjectAlex Esterkin12 May
          • Re: MySQL Reengineering ProjectBrian Aker12 May
      • Re: MySQL Reengineering ProjectSergei Golubchik12 May
        • Re: MySQL Reengineering ProjectBrian Aker12 May
          • Re: MySQL Reengineering ProjectSergei Golubchik12 May
            • Re: MySQL Reengineering ProjectAlex Esterkin12 May
              • Re: MySQL Reengineering ProjectBrian Aker12 May
            • Re: MySQL Reengineering ProjectJay Pipes12 May
              • Re: MySQL Reengineering ProjectDavi Arnaut13 May
              • Re: MySQL Reengineering ProjectAlex Esterkin13 May
                • Re: MySQL Reengineering ProjectJay Pipes13 May
            • Re: MySQL Reengineering ProjectMichael Widenius14 May
          • Re: MySQL Reengineering ProjectJonas Oreland13 May
        • Re: MySQL Reengineering ProjectRoy Lyseng12 May
      • Re: MySQL Reengineering ProjectMats Kindahl12 May
    • Re: MySQL Reengineering ProjectAlaric Snell-Pym12 May
      • Re: MySQL Reengineering ProjectMats Kindahl12 May
      • Re: MySQL Reengineering ProjectMichael Widenius14 May
        • Re: MySQL Reengineering ProjectAlaric Snell-Pym15 May
          • Re: MySQL Reengineering ProjectKristian Nielsen15 May
            • Re: MySQL Reengineering ProjectBrian Aker15 May
            • Re: MySQL Reengineering ProjectAlaric Snell-Pym15 May
          • Re: MySQL Reengineering ProjectMichael Widenius6 Jun
Re: MySQL Reengineering ProjectMats Kindahl12 May
Re: MySQL Reengineering ProjectAlex Esterkin12 May
  • Re: MySQL Reengineering ProjectMasood Mortazavi13 May
    • Re: MySQL Reengineering ProjectBrian Aker13 May
    • Re: MySQL Reengineering ProjectAlex Esterkin13 May
      • Re: MySQL Reengineering ProjectBaron Schwartz13 May
        • Re: MySQL Reengineering ProjectStewart Smith13 May
          • Re: MySQL Reengineering ProjectBaron Schwartz13 May
    • Re: MySQL Reengineering ProjectAlex Esterkin13 May
    • Re: MySQL Reengineering ProjectStewart Smith13 May
  • Re: MySQL Reengineering ProjectMichael Widenius14 May