List:Cluster« Previous MessageNext Message »
From:Raymond Peachey Date:October 7 2013 10:56pm
Subject:Data retrieval causes query times to spike
View as plain text  
Hello,

Running MySQL Cluster 7.2 with 2 API (MYSQLD)/MGM nodes, 4 Data nodes (64
GB RAM).

We are running into an issue when a job that runs every few minutes is
grabbing changed data (determined by triggers) and dumping it out to disk.
During this process, we often hit a 5 second timeout that we have
configured for MySQL Cluster, causing a MYSQL HAS GONE AWAY error.

Raising the timeout is not really an option we're looking for currently. We
have 60 API connections split between our two MYSQLD nodes. This job is
currently only running against one of the two nodes, and only that node is
experiencing the issue.

My thoughts:

* I feel this is possibly due to saturation of the link between the MYSQLD
node and the data nodes PER API node connection. Does this make sense? If
so, how do I fix this? Possibly more importantly, how do I measure this
bandwidth? (I assume I should be looking at the bytes sent/received in the
ndbinfo.Transporters table?)
* I thought this might have to do something with the
[Total]SendBufferMemory and ReceiveBufferMemory, but in the
ndbinfo.Transporters table I have not seen the overload_count increase, so
perhaps I am mistaken?
* SendBufferMemory and ReceiveBufferMemory are set within the [tcp default]
section as 32M and 8M respectively.

Additionally, this is compounded by the fact that this data dumping process
(no locks involved) is only on a small subset of the tables at a time. Any
thoughts, or possible direction into where I should look to troubleshoot
this?

Thank you,

Raymond

Thread
Data retrieval causes query times to spikeRaymond Peachey7 Oct
  • Re: Data retrieval causes query times to spikeMagnus BlĂ„udd9 Oct
    • Re: Data retrieval causes query times to spikeRaymond Peachey9 Oct
      • Re: Data retrieval causes query times to spikeWagner Bianchi9 Oct
        • Re: Data retrieval causes query times to spikeRaymond Peachey10 Oct
          • Re: Data retrieval causes query times to spikeHendrik Woltersdorf10 Oct
            • Re: Data retrieval causes query times to spikeRaymond Peachey10 Oct
              • Re: Data retrieval causes query times to spikeJohan Andersson10 Oct
                • Re: Data retrieval causes query times to spikeRaymond Peachey10 Oct
                  • Re: Data retrieval causes query times to spikeJohan Andersson10 Oct
                    • Re: Data retrieval causes query times to spikeRaymond Peachey14 Oct
                      • Re: Data retrieval causes query times to spikeHendrik Woltersdorf15 Oct
                        • Re: Data retrieval causes query times to spikeRaymond Peachey15 Oct
                          • Re: Data retrieval causes query times to spikeJohan Andersson15 Oct
                            • Re: Data retrieval causes query times to spikeRaymond Peachey16 Oct
                              • Re: Data retrieval causes query times to spikeRaymond Peachey21 Oct