MySQL Cluster contention when increasing amount connections

[Nam Luc Tran]

Hello,

I have set up a cluster with 4 servers (quad core opteron, 8Gb Ram,
Gigabit Ethernet, Fedora 11) hosting 4 datanodes and 1 SQL node + 1
separate management node using MySQL Cluster  7.0.6 Community
Edition. I have run several benchmarks on it.
When running the Hammerora benchmark  (adapting a version of the
TPC-C) my throughput values are increasing until ca. 50 users (~140000
tpm). From a certain amount of users (>50-60) the test starts with
increasing values of throughput then drops violently and ends
stagnating round poor values (~6000 tpm) till the end. Most of the
users report "Lock wait timeout exceeded; try restarting transaction"
error. When I run the same benchmark on 2 additional SQL nodes in the
cluster at the same time on the same DB, the problem does not happen
as the global throughput is higher (3*~70000 tpm), so this let me
think that this is not a network or load problem between the nodes.

The same happens with the Sysbench oltp benchmark. Starting from ~50
running threads, I get poor response time and transactions/min values.

So where can the problem come from? Is the maximum amount of
connections to the SQL node reached? 50 seems quite little. I already
set those values to my.cnf on the SQL node:
engine_condition_pushdown=1
max_connections=1000
ndb-cluster-connection-pool=2

And here is an overview of my config.ini:
[begin]
[ndbd default]
NoOfReplicas=2   
DataMemory=5400M    
IndexMemory=675M 

#LockPagesInMainMemory=1 
RedoBuffer=32M
NoOfFragmentLogFiles=768     
MaxNoOfConcurrentOperations=500000  
MaxNoOfLocalOperations=700000
MaxNoOfConcurrentTransactions=10000
MemReportFrequency=30
TransactionDeadlockDetectionTimeout=10000

BackupMaxWriteSize=1M
BackupDataBufferSize=16M
BackupLogBufferSize=4M
BackupMemory=20M

[tcp default]
#portnumber=2202  

[ndb_mgmd]
hostname=192.168.0.42       
datadir=/var/lib/mysql-cluster  
ArbitrationRank=1

[ndbd]
hostname=192.168.0.35          
datadir=/usr/local/mysql/data   

[ndbd]
hostname=192.168.0.38         
datadir=/usr/local/mysql/data   

[ndbd]
hostname=192.168.0.37
datadir=/usr/local/mysql/data

[ndbd]
hostname=192.168.0.34
datadir=/usr/local/mysql/data
[mysqld]
[...]
[end]

Thanks for your help

Tran Nam-Luc