Just to throw out (plug) an ongoing project:
From the site:
HiveDB is an open source framework for horizontally partitioning MySQL
systems. Building scalable and high performance MySQL-backed systems
requires a good deal of expertise in designing the system and
implementing the code. One of the main strategies for scaling MySQL is
by partitioning your data across many servers. While it is not difficult
to accomplish this, it is difficult to do it in such a way that the
system is easily maintained and transparent to the developer.
We've been working on HiveDB precisely to avoid the large amount of
(quite specialized) code in the application.
Naz Gassiep wrote:
> The problem with sharding I have is the large amount of code
> required in the app to make it work. IMHO the app should be agnostic to
> the underlying database system (by that I don't mean the DB in use such
> as MySQL or whatever or the schema, I mean the way the DB has been
> deployed) so that changes can be made to it without having to worry
> about impacting app code. This is one of my fundamental design imperatives.
> Then again, I'm not a regular MySQL user so I don't know what is and
> is not the norm in the MySQL world.
> - Naz.
> Evaldas Imbrasas wrote:
>> You certainly have a right to disagree, but pretty much every
>> scalability talk at the MySQL conference a few weeks ago was focused
>> on data partitioning and sharding. And those talks very given by folks
>> working for some of the most popular (top 100) websites in the world.
>> It certainly looks like data partitioning is the way to go in the
>> MySQL world at this point, probably at least until production-ready
>> and feature-full MySQL Cluster is out. And even then large percentage
>> of dotcom companies would use data partitioning instead since it can
>> be implemented on commodity hardware.
>> Once again, we're talking *really* big websites using MySQL (not
>> Oracle or SQL Server or whatever) here. Most websites won't ever need
>> to partition their production databases, and different RDMS might have
>> different approaches for scalability.
>> On 5/24/07, Naz Gassiep <naz@stripped> wrote:
>>> Data partitioning? Sorry, I disagree that partitioning a table into more
>>> and more servers is the way to scale properly. Perhaps putting
>>> databases' tables onto different servers with different hardware
>>> designed to meat different usage patterns is a good idea, but data
>>> partitioning was a very short lived idea in the world of databases and
>>> I'm glad that as an idea it is dying in practice.
high performance mysql consulting