List:Cluster« Previous MessageNext Message »
From:Jeff Sturm Date:November 19 2010 5:40pm
Subject:RE: MySQL CLuster Data Nodes Lose Connection
View as plain text  
When this happened to us, I found that the memory size of mysqld had
grown to the point where it began to swap.  Swapping introduced enough
latency to cause the API node to disconnect from the cluster.

I have no idea if that's your problem, just a tip.  Putting an upper
bound on the memory size of mysqld was maddeningly difficult--it didn't
seem to obey the tunable settings we gave it.  If I started over with
cluster again I'd give the API nodes a *lot* more RAM (we had only 8GB
on each).

-Jeff

> -----Original Message-----
> From: artfire-kyle [mailto:kyleerror@stripped]
> Sent: Friday, November 19, 2010 11:21 AM
> To: cluster@stripped
> Subject: RE: MySQL CLuster Data Nodes Lose Connection
> 
> 
> First thank you for your response.
> 
> I apologize as the title is a little misleading. It is the API
(MySQLD) nodes that are losing
> connection.
> I have 4 servers total in the cluster. 2 x Management nodes and then
the data and
> mysqld run on the other two.
> 
> So yes the connection between mysql console and mysqld are what loses
the
> connection (i use "mysql -u root -p -h x.x.x.214" to connect to
console), but only from
> remote servers. If i access mysqld on API node 1 via API node
> 2 they establish fine. It is only servers from outside the cluster who
try to access
> mysqld that lose the connection (which are the servers that need to
connect via PHP).
> 
> I do have a firewall which is outside the cluster (and the rest of my
web
> servers) and is the main entry point for all traffic. I have all four
of the servers open
> on all ports (just until i can isolate this issue), and everything is
done with NAT.
> 
> When the remote connection is lost, I do not see any new entries in
either the
> management log file or the mysqld logging.
> 
> Here is the last entries in the mysqld log from one of the mysqld
nodes.
> 
> 090812 22:26:52 [Note] Event Scheduler: Loaded 0 events
> 090812 22:26:53 [Note] NDB Binlog: DISCOVER TABLE Event:
> REPL$mysql/ndb_schema
> 090812 22:26:53 [Note] NDB Binlog: logging ./mysql/ndb_schema
> (UPDATED,USE_WRITE)
> 090812 22:26:53 [Note] NDB Binlog: DISCOVER TABLE Event:
> REPL$mysql/ndb_apply_status
> 090812 22:26:53 [Note] NDB Binlog: logging ./mysql/ndb_apply_status
> (UPDATED,USE_WRITE)
> 2009-08-12 22:26:53 [NdbApi] INFO     -- Flushing incomplete GCI:s <
33737/6
> 2009-08-12 22:26:53 [NdbApi] INFO     -- Flushing incomplete GCI:s <
33737/6
> 090812 22:26:53 [Note] NDB Binlog: starting log at epoch 33737/6
> 090812 22:26:53 [Note] NDB Binlog: ndb tables writable
> 090812 22:26:53 [Note] mysqld: ready for connections.
> Version: '5.1.39-ndb-7.0.9-1ubuntu7'  socket:
'/var/run/mysqld/mysqld.sock'
> port: 3306  (Ubuntu)
> 090812 22:27:08 [Note] NDB Binlog: Node: 3, subscribe from node 5,
Subscriber
> bitmask 020
> 090812 22:27:08 [Note] NDB Binlog: Node: 4, subscribe from node 5,
Subscriber
> bitmask 020
> 
> 
> I'm not sure how to procede, everything works great until the remote
access is lost to
> mysqld.
> 
> Thanks again.
> --
> View this message in context:
http://mysql-cluster.1050023.n4.nabble.com/MySQL-
> CLuster-Data-Nodes-Lose-Connection-tp3049544p3050611.html
> Sent from the MySQL - Cluster mailing list archive at Nabble.com.
> 
> --
> MySQL Cluster Mailing List
> For list archives: http://lists.mysql.com/cluster
> To unsubscribe:
http://lists.mysql.com/cluster?unsub=1
> 


Thread
MySQL CLuster Data Nodes Lose Connectionartfire-kyle18 Nov
  • RE: MySQL CLuster Data Nodes Lose ConnectionAndrew Morgan19 Nov
    • RE: MySQL CLuster Data Nodes Lose Connectionartfire-kyle19 Nov
      • RE: MySQL CLuster Data Nodes Lose ConnectionAndrew Morgan19 Nov
      • RE: MySQL CLuster Data Nodes Lose ConnectionJeff Sturm19 Nov
        • RE: MySQL CLuster Data Nodes Lose Connectionartfire-kyle22 Nov
          • RE: MySQL CLuster Data Nodes Lose ConnectionJeff Sturm22 Nov
            • RE: MySQL CLuster Data Nodes Lose ConnectionBradley Watkins22 Nov
          • RE: MySQL CLuster Data Nodes Lose ConnectionMatthew Boehm22 Nov