From: Mika Raento
Date: September 12 2007 7:04am
Subject: NDB performance/throughput analysis tools?
List-Archive: http://lists.mysql.com/ndb-connectors/161
Message-Id: <46E78F79.8070209@iki.fi>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hiya guys

I've been trying to build a message-routing framework on top of NDB. The 
basic idea is to have queues of messages, which get picked by worker 
processes, which transform and route the message, inserting them to new 
queues.

My design has been to have several workers for the same queue for 
reliability and HA. Each worker would do something like:

* scan queue for items (with no locks)
* attempt to lock some of the items
* for each successful lock:
   * process item (includes reading and writing some other data in NDB)
   * delete from queue

Now each worker can proceed in parallel, excluding others for the same 
message through a lock in NDB with automatic failover within a lock 
timeout. All transactions are async.

I'm only getting something like 500 messages/second throughput (similar 
results on running both ndbd and worker on one box or each on its own 
box over 100M ethernet). 500/sec is not cutting it for our needs.

The problem is that I'm a bit stuck: I can't find a good way to figure 
out what the bottlenecks are and if I can do anything about them. I 
can't find references to relative cost of different operations in NDB, 
profiling tools for the data node or for NDBAPI application programs. 
I've done basic tuning for batch parameters to sendPollNdb, the number 
of items to handle in parallel and simplifying the processing task and 
got up from an initial 100 req/sec to the 500, but I'm making very 
little further process.

I'm CPU bound on a single box, split pretty evenly between my 
application program and the two ndbd(s). On several boxes I'm seeing 
quite low CPU usage on ndbd.

Any pointers (e.g., profiling NDB internal operations)?

	Mika