List:Cluster« Previous MessageNext Message »
From:Mikael Ronström Date:August 11 2004 9:38am
Subject:Re: gigabit performance...?
View as plain text  
Hi Luke and Magnus,

2004-08-11 kl. 10.24 skrev Magnus Svensson:

> Hi Luke,
>> I calculate the row size as 99 bytes.
>> mysql> select count(*) from gmargin;
>> +----------+
>> | count(*) |
>> +----------+
>> |  4355766 |
>> +----------+
>> 1 row in set (26.64 sec)
>> and, as I understand, this sent all 4,355,766 rows over the network 
>> to the mysql server, so 400,355,766 bytes, right? does this seem like 
>> a reasonable time? I'm not sure how much time it is taking the mysql 
>> server to count up all the rows...?
> Actually only the primary key of each row is fetched. The limiting
> factor here is that too few records are fetched at a time.  We are
> working on improving the fetch algorithm too allow for more records to
> be fetched in each "batch" thus utilizing the network bandwith better.

As Magnus mentioned an optimisation is worked on. Currently 16 records 
at a time is fetched per fragment and there are 2 fragments per node so 
32 primary keys are sent per batch which is
32 * //No of records
( 8 + //Assumed primary key size
4 + //Overhead per attribute
12 + //Overhead per record
20/4 //Overhead per message
) => 928 Bytes in practice 960 bytes (8 messages).

Sending 960 bytes at a time over TCP/IP doesn't give any good 
performance. So we are ensuring more records sent at a time to get at 
more like 32k-64k
per batch.

The calculations show that you transported 4,355,766 * 30 = 
130,672,980/26,64 = 4,905,141 bytes per second => ~39Mbit/sec which is 
very far from the
Gigabit. The optimisation we are doing should hopefully improve this 
considerably. Going from 16 to 64 increased throughput with 27,5% for a 

4,355,766/26,64 = 163,504 records per second.
Even with these optimisations it will still be difficult to get more 
than say 300k - 500k records per second through. This is limited by 
processing time.
But we'll see when the optimisations are ready what the measurements 

Rgrds Mikael
PS: I took a look at the web site you had found. Tampering with delays 
to RxInt and TxInt can most likely provide some benefits since it 
decreases the processor
usage for interrupt handling. Haven't tampered with these myself so I 
have no idea what is appropriate figures.

>> 400MB/26.6s = 15MB/s or 120 Mbit.
>> basically, we're interested in all the settings we can change to max 
>> out the performance on our cluster. is there any kind of optimization 
>> guide for the cluster?
> In order to help you I will need some more knowledge concerning the 
> kind
> of queries your application will be running, I assume you will not only
> do "select count(*)". :)
> MySQL Cluster is very good at primary key and unique index lookups,
> where a hash index is used to find the record. There are also ordered
> indexes available, which improves performance for a query a lot. So
> please give me some examples of queries that you intend to run, and 
> I'll
> advise how to optimize them.
> Best regards
> Magnus
> -- 
> Magnus Svensson, Software Engineer
> Office: +46 709 164 491
> -- 
> MySQL Cluster Mailing List
> For list archives:
> To unsubscribe:    
Mikael Ronström, Senior Software Architect


gigabit performance...?Luke H. Crouch6 Aug
  • Re: gigabit performance...?Mikael Ronström6 Aug
RE: gigabit performance...?Luke H. Crouch6 Aug
RE: gigabit performance...?Luke H. Crouch9 Aug
  • RE: gigabit performance...?Magnus Svensson10 Aug
RE: gigabit performance...?Luke H. Crouch10 Aug
  • RE: gigabit performance...?Magnus Svensson11 Aug
    • Re: gigabit performance...?Mikael Ronström11 Aug