List:Cluster« Previous MessageNext Message »
From:Mikael Ronstrošm Date:September 24 2010 7:17am
Subject:Re: Performance and Scalability and cluster
View as plain text  
Hi,

Schweiger, Jeff - ES skrev:
> Thanks for the response... One follow up question:
> 
>> You know, 60k queries/min is not that large for cluster although it
>> depends on the queries.  How is the load of your servers at 60k/s?  Can
>> you estimate what will be the first bottleneck?
> 
> I guess that is part of my problem...  I would like to get an idea how many queries
> cluster *can* handle.  At my current query rate, the load on my servers is very low.  I
> know performance is highly variable and is app dependant, hardware dependant and query
> type dependant.  For example, currently I am running on old Sun 2000's, and know I will
> get a boost if I go to a newer Intel multi core platform.  But I think I have tuned by app
> and cluster to perform well.
> 
> So given a 4 data node 2 mysqld cluster, can I expect 500,000 queries per second, can
> I expect more?  I would love to know what type of performance people have seen.
> 

Before you reach to 500kqueries/sec you will most likely hit bottlenecks
in the MySQL server code (250k queries per mysqld). Should you reach
this limit you can add more mysqld's to the config.

The limit on the data nodes is very dependent on the queries as
mentioned. So very hard to predict that number. I benchmarked
DBT2 a few years ago and reached about 350kqueries per second
but on a larger cluster, but this was on v6.3, so if you use
v7.x you should have about 4x more performance per data node.
So a few 100kqueries per second is achievable and for special
types of queries even millions per second (e.g. loads of
Primary Key access queries).

Rgrds Mikael

> Thanks.
> 
> Jeff Schweiger
> jeff.schweiger@stripped
> 44965 Aviation Dr. Suite 400
> Dulles, VA 20166
> Office: (703) 996-2941
> 
> -----Original Message-----
> From: Yves Trudeau [mailto:y.trudeau@stripped] 
> Sent: Thursday, September 23, 2010 3:24 PM
> To: Schweiger, Jeff - ES
> Cc: cluster@stripped
> Subject: Re: Performance and Scalability and cluster
> 
> Hi Jeff,
>   my answers are embedded.
> 
> On Thu, 2010-09-23 at 10:11 -0400, Schweiger, Jeff - ES wrote:
>> Hello,
>>
>> I have a problem that I am trying to wrap my head around, and I am
>> hoping someone in this group can help me.  We have been testing
>> cluster for a while now, and we have liked how cluster has performed
>> thus far.  However, before we move into deploying cluster, I would
>> like to know the answers to a couple of scalability questions.
>>
>> On our current system, we have 4 data nodes, and 2 mysqld nodes that
>> run about 60,000 queries a minute.  We expect our traffic to increase
>> significantly over the next couple of years, and I am look for some
>> guides, or any  source of information  that will help answer the
>> following questions.
>>
>> 1.  How can get a determination of how my cluster configuration is
>> performing.  Given my 4 data node, 2 mysqld nodes, 60,000 query a
>> minute load, how close am I to impacting performance?
> 
> You know, 60k queries/min is not that large for cluster although it
> depends on the queries.  How is the load of your servers at 60k/s?  Can
> you estimate what will be the first bottleneck? 
> 
>> 2.  Is there any benchmark data out there?
> 
> Cluster is extremely app dependent so your app is the best benchmark you
> have.
> 
>> 3.  How do I know when/how to add nodes to the cluster?  For example,
>> what is the deference between adding a mysqld node, and a data node,
>> and how do I know when it would be necessary to add either?
> 
> If you are out of memory space in the data nodes and/or CPU usage is
> high there, then it might be an indication that more data nodes are
> needed.  If you can design your applications and schema in a way that a
> business logic rule hit only one node group, then you'll have a very
> scalable application.
> 
> About the same for MySQL.  The rule of thumb is more MySQL nodes than
> data nodes although since MySQL can now use a pool of connections to the
> NDB cluster, it is less true. 
> 
>> 4.  Is there any tips/procedures used to measure performance of
>> cluster?
> 
> It depends what is important for you, throughput and/or latency.  Tuning
> for NDB and your application is critical there. 
> 
> Regards,
> 
> Yves
>> Thank for any help you can provide.
>>
>> Jeff Schweiger
>> jeff.schweiger@stripped
>> 44965 Aviation Dr. Suite 400
>> Dulles, VA 20166
>> Office: (703) 996-2941
>>
>> -----Original Message-----
>> From: Dan Frincu [mailto:dfrincu@stripped]
>> Sent: Monday, July 26, 2010 12:38 PM
>> To: Ricardo Soares Guimarães
>> Cc: cluster@stripped
>> Subject: Re: Converting MyISAM with 300GB to NDB.
>>
>> Try ndb_size.pl script on the 300GB database (it will take some time) to
>> get the settings necessary for NDB, however based on the amount of data
>> and the limited amount of RAM, unless you seriously upgrade your RAM and
>> add several servers in the equation, I don't see how this would be
>> achievable with this H/W configuration.
>>
>> You might want to check out this link
>> http://www.fromdual.com/mysql-cluster-memory-sizing
>>
>> Regards,
>> Dan.
>>
>> Ricardo Soares Guimarães wrote:
>>> Hello,
>>>
>>> I have one database with 300Gb of data, and want to try NDB.
>>>
>>> That´s possible?
>>>
>>> I tried several configurations, and keys, but seems that can´t import
>>> such data
>>>
>>> My server has (now) 16Gb of RAM, but soon will have 32.
>>>
>>> Anyone tried such configuration?
>>>
>>> I am using mysql-cluster 7.1.x with 64bits linux server.
>>>
>>> Thanks
>>>
>> --
>> Dan FRINCU
>> Systems Engineer
>> CCNA, RHCE
>> Streamwide Romania
>>
>>
>> This e-mail and any files transmitted with it may be proprietary and are intended
> solely for the use of the individual or entity to whom they are addressed. If you have
> received this e-mail in error please notify the sender.
>> Please note that any views or opinions presented in this e-mail are solely those
> of the author and do not necessarily represent those of ITT Corporation. The recipient
> should check this e-mail and any attachments for the presence of viruses. ITT accepts no
> liability for any damage caused by any virus transmitted by this e-mail.
>>
> 
> 
Thread
Converting MyISAM with 300GB to NDB.Ricardo Soares Guimarães26 Jul
  • Re: Converting MyISAM with 300GB to NDB.Dan Frincu26 Jul
    • Performance and Scalability and clusterJeff - ES Schweiger23 Sep
  • Re: Converting MyISAM with 300GB to NDB.Dan Frincu26 Jul
  • Re: Converting MyISAM with 300GB to NDB.Stewart Smith26 Jul
    • Re: Converting MyISAM with 300GB to NDB.Ricardo Soares Guimarães26 Jul
      • Re: Converting MyISAM with 300GB to NDB.Jason Brooke26 Jul
  • Re: Converting MyISAM with 300GB to NDB.David Ennis26 Jul
    • Re: Converting MyISAM with 300GB to NDB.Ricardo Soares Guimarães27 Jul
  • Re: Performance and Scalability and clusterYves Trudeau23 Sep
    • RE: Performance and Scalability and clusterJeff - ES Schweiger23 Sep
      • Re: Performance and Scalability and clusterAdam Dixon23 Sep
        • RE: Performance and Scalability and clusterAndrew Morgan24 Sep
          • High availability implementationramarovelo28 Sep
            • RE: High availability implementationAndrew Morgan28 Sep
              • Re: High availability implementationMat Keep28 Sep
              • Re: High availability implementationramarovelo28 Sep
      • Re: Performance and Scalability and clusterMikael Ronstrošm24 Sep