Hi,
my cluster (7.0.8a) was running a few weeks without any problems or error messages. But
today, it just did a complete restart of all NDB-nodes.
The send buffer is large enough, it was only "full" due to these problem.
Here is the log output:
2009-11-10 10:02:54 [MgmtSrvr] WARNING -- Node 3: Node 6 missed heartbeat 2
2009-11-10 10:02:54 [MgmtSrvr] WARNING -- Node 3: Node 7 missed heartbeat 2
2009-11-10 10:02:54 [MgmtSrvr] WARNING -- Node 3: Node 8 missed heartbeat 2
2009-11-10 10:02:54 [MgmtSrvr] WARNING -- Node 3: Node 9 missed heartbeat 2
2009-11-10 10:02:54 [MgmtSrvr] WARNING -- Node 3: Node 11 missed heartbeat 2
2009-11-10 10:02:54 [MgmtSrvr] WARNING -- Node 3: Node 28 missed heartbeat 2
2009-11-10 10:02:57 [MgmtSrvr] WARNING -- Node 2: Detected GCP stop(1)...sending kill to
[SignalCounter: m_count=2 0000000000000028]
2009-11-10 10:02:58 [MgmtSrvr] WARNING -- Node 3: Node 6 missed heartbeat 2
2009-11-10 10:02:58 [MgmtSrvr] WARNING -- Node 3: Node 7 missed heartbeat 2
2009-11-10 10:02:58 [MgmtSrvr] WARNING -- Node 3: Node 10 missed heartbeat 2
2009-11-10 10:02:58 [MgmtSrvr] WARNING -- Node 3: Node 11 missed heartbeat 2
2009-11-10 10:02:58 [MgmtSrvr] WARNING -- Node 3: Node 28 missed heartbeat 2
2009-11-10 10:03:03 [MgmtSrvr] WARNING -- Node 2: Transporter to node 5 reported error
0x16: The send buffer was full, but sleeping for
a while solved
2009-11-10 10:03:03 [MgmtSrvr] WARNING -- Node 2: Transporter to node 5 reported error
0x16: The send buffer was full, but sleeping for
a while solved
2009-11-10 10:03:05 [MgmtSrvr] WARNING -- Node 3: Node 6 missed heartbeat 2
2009-11-10 10:03:05 [MgmtSrvr] WARNING -- Node 3: Node 7 missed heartbeat 2
2009-11-10 10:03:05 [MgmtSrvr] WARNING -- Node 3: Node 11 missed heartbeat 2
2009-11-10 10:03:05 [MgmtSrvr] WARNING -- Node 3: Node 28 missed heartbeat 2
2009-11-10 10:03:07 [MgmtSrvr] ALERT -- Node 2: Forced node shutdown completed. Caused
by error 2341: 'Internal program error (failed
ndbrequire)(Internal error, programming error or missing error message, please report a
bug). Temporary error, restart node'.
Any suggestions? Is this fixed on 7.0.9?
Thanks and regards,
Robert
