List:Replication« Previous MessageNext Message »
From:Marcus Bointon Date:July 29 2008 11:48am
Subject:Re: any advice on this sharding and replication setup?
View as plain text  
On 29 Jul 2008, at 07:36, Konstantin Rozinov wrote:

> 1.  It would seem that without creating another level of MySQL servers
> that would act as directories, the 1 MySQL server that would act as a
> directory in the current setup could become overwhelmed, because of
> the millions of users, comments, photos, etc that need to be looked up
> for each and every request?  Is there a solution to this problem?

I agree, I think that's also a good way of introducing a nasty failure  
point as well. Normally you'd use some kind of hashing function to  
split data across partitions, for example put users a-m on one server,  
n-z on another. That's a bit naive - you may end up with a big bias -  
but you get the general idea. The thing to bear in mind is that if  
your mechanism for partitioning is on the DB client end (i.e. in your  
scripts on each web server), it will generally be scalable, as the DB  
servers don't need to know about it, so you can keep adding them with  
minimal overhead. Aside from that, I'd be tempted to think up some  
scheme to keep the mapping in something faster than a DB, e.g.  
memcache, though of course you'd have to have some means of rebuilding  
the map in the event of cache expiry/loss. memcache can handle tens of  
thousands of hits per second, and it scales so nicely.

If you have shared files (e.g. your pictures), I'd strongly suggest  
you take a look at GlusterFS. It has a really nice mechanism for  
handling distributed file systems with redundancy - think networked  
RAID - that also scales more or less infinitely.

Marcus
-- 
Marcus Bointon
Synchromedia Limited: Creators of http://www.smartmessages.net/
UK resellers of info@hand CRM solutions
marcus@stripped | http://www.synchromedia.co.uk/



Attachment: [application/pkcs7-signature] smime.p7s
Thread
any advice on this sharding and replication setup?Konstantin Rozinov29 Jul
  • Re: any advice on this sharding and replication setup?Marcus Bointon29 Jul