List:General Discussion« Previous MessageNext Message »
From:Jeremy Cole Date:May 26 2007 5:00am
Subject:Re: Integrity on large sites
View as plain text  
Hi Naz,

Just to throw out (plug) an ongoing project:

 From the site:

HiveDB is an open source framework for horizontally partitioning MySQL 
systems. Building scalable and high performance MySQL-backed systems 
requires a good deal of expertise in designing the system and 
implementing the code. One of the main strategies for scaling MySQL is 
by partitioning your data across many servers. While it is not difficult 
to accomplish this, it is difficult to do it in such a way that the 
system is easily maintained and transparent to the developer.

We've been working on HiveDB precisely to avoid the large amount of 
(quite specialized) code in the application.



Naz Gassiep wrote:
> Wow.
>     The problem with sharding I have is the large amount of code
> required in the app to make it work. IMHO the app should be agnostic to
> the underlying database system (by that I don't mean the DB in use such
> as MySQL or whatever or the schema, I mean the way the DB has been
> deployed) so that changes can be made to it without having to worry
> about impacting app code. This is one of my fundamental design imperatives.
>     Then again, I'm not a regular MySQL user so I don't know what is and
> is not the norm in the MySQL world.
> - Naz.
> Evaldas Imbrasas wrote:
>> You certainly have a right to disagree, but pretty much every
>> scalability talk at the MySQL conference a few weeks ago was focused
>> on data partitioning and sharding. And those talks very given by folks
>> working for some of the most popular (top 100) websites in the world.
>> It certainly looks like data partitioning is the way to go in the
>> MySQL world at this point, probably at least until production-ready
>> and feature-full MySQL Cluster is out. And even then large percentage
>> of dotcom companies would use data partitioning instead since it can
>> be implemented on commodity hardware.
>> Once again, we're talking *really* big websites using MySQL (not
>> Oracle or SQL Server or whatever) here. Most websites won't ever need
>> to partition their production databases, and different RDMS might have
>> different approaches for scalability.
>> On 5/24/07, Naz Gassiep <naz@stripped> wrote:
>>> Data partitioning? Sorry, I disagree that partitioning a table into more
>>> and more servers is the way to scale properly. Perhaps putting
>>> databases' tables onto different servers with different hardware
>>> designed to meat different usage patterns is a good idea, but data
>>> partitioning was a very short lived idea in the world of databases and
>>> I'm glad that as an idea it is dying in practice.

high performance mysql consulting
Integrity on large sitesNaz Gassiep24 May
  • Re: Integrity on large sitesPeter Brawley24 May
  • Re: Integrity on large sitesMartijn Tonies24 May
  • Re: Integrity on large sitesPhilip Mather24 May
  • Re: Integrity on large sitesEvaldas Imbrasas24 May
    • Re: Integrity on large sitesNaz Gassiep25 May
      • Re: Integrity on large sitesB. Keith Murphy25 May
      • Re: Integrity on large sitesEvaldas Imbrasas25 May
        • Re: Integrity on large sitesNaz Gassiep25 May
          • Re: Integrity on large sitesB. Keith Murphy25 May
            • Re: Integrity on large sitesBarry Newton25 May
              • Re: Integrity on large sitesNaz Gassiep25 May
            • Re: Integrity on large sitesNaz Gassiep25 May
          • Re: Integrity on large sitesJeremy Cole26 May
  • Re: Integrity on large sitesMartijn Tonies25 May
RE: Integrity on large sitesRhys Campbell25 May