List:NDB Connectors« Previous MessageNext Message »
From:Monty Taylor Date:September 13 2007 11:21pm
Subject:Re: NDB performance/throughput analysis tools?
View as plain text  
Hey Mika,

You're right, I would expect cluster to handle more than this.

I'd recommend running oprofile and see what's going on. Kristian has a
good walkthrough of using for MySQL on his blog:
http://mysqldump.azundris.com/archives/61-Using-oprofile.html
You'll need to build cluster with symbols...

Also, you could grab the latest version of get-ndb-status from
https://launchpad.net/get-ndb-status/ which is an MGM API program that
will spit out some stats for you on how many types of what operations
are happening - although I would think you would already know that.

How many items do you have in your queue? The index scan there might be
sucking some... I'm interested to know if you can find anything.

Monty

Mika Raento wrote:
> Hiya guys
> 
> I've been trying to build a message-routing framework on top of NDB. The
> basic idea is to have queues of messages, which get picked by worker
> processes, which transform and route the message, inserting them to new
> queues.
> 
> My design has been to have several workers for the same queue for
> reliability and HA. Each worker would do something like:
> 
> * scan queue for items (with no locks)
> * attempt to lock some of the items
> * for each successful lock:
>   * process item (includes reading and writing some other data in NDB)
>   * delete from queue
> 
> Now each worker can proceed in parallel, excluding others for the same
> message through a lock in NDB with automatic failover within a lock
> timeout. All transactions are async.
> 
> I'm only getting something like 500 messages/second throughput (similar
> results on running both ndbd and worker on one box or each on its own
> box over 100M ethernet). 500/sec is not cutting it for our needs.
> 
> The problem is that I'm a bit stuck: I can't find a good way to figure
> out what the bottlenecks are and if I can do anything about them. I
> can't find references to relative cost of different operations in NDB,
> profiling tools for the data node or for NDBAPI application programs.
> I've done basic tuning for batch parameters to sendPollNdb, the number
> of items to handle in parallel and simplifying the processing task and
> got up from an initial 100 req/sec to the 500, but I'm making very
> little further process.
> 
> I'm CPU bound on a single box, split pretty evenly between my
> application program and the two ndbd(s). On several boxes I'm seeing
> quite low CPU usage on ndbd.
> 
> Any pointers (e.g., profiling NDB internal operations)?
> 
>     Mika
> 


-- 
Monty Taylor
Senior Consultant
MySQL Inc., www.mysql.com
Get More with MySQL!  www.mysql.com/consulting
Thread
NDB performance/throughput analysis tools?Mika Raento12 Sep
  • Re: NDB performance/throughput analysis tools?Monty Taylor14 Sep