Hi guys,
Anyone could help me check what problem happened in my mysql cluster(ndb-7.0.7 for Linux
x86_64).
The following is some information from cluster.log:
... ...
2009-10-05 19:47:24 [MgmSrvr] INFO -- Node 5: Data usage is
75%(156835 32K pages of total 208000)
2009-10-05 19:48:19 [MgmSrvr] INFO -- Node 3: Data usage is
75%(156674 32K pages of total 208000)
2009-10-05 19:48:26 [MgmSrvr] INFO -- Node 2: Data usage is
75%(157183 32K pages of total 208000)
2009-10-05 19:48:40 [MgmSrvr] INFO -- Node 4: Data usage is
75%(157113 32K pages of total 208000)
2009-10-05 19:49:23 [MgmSrvr] INFO -- Node 4: Index usage is
77%(101564 8K pages of total 131104)
2009-10-05 19:49:46 [MgmSrvr] INFO -- Node 3: Index usage is
77%(101625 8K pages of total 131104)
2009-10-05 19:49:53 [MgmSrvr] INFO -- Node 5: Index usage is
77%(101646 8K pages of total 131104)
2009-10-05 19:50:05 [MgmSrvr] INFO -- Node 2: Index usage is
77%(101652 8K pages of total 131104)
2009-10-05 19:51:46 [MgmSrvr] ALERT -- Node 2: Forced node shutdown
completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal
error, programming error or missing error message, please report a bug). Temporary error,
restart node'.
2009-10-05 19:51:46 [MgmSrvr] INFO -- Node 3: Communication to
Node 2 closed
2009-10-05 19:51:46 [MgmSrvr] ALERT -- Node 4: Node 2 Disconnected
2009-10-05 19:51:46 [MgmSrvr] INFO -- Node 4: Communication to
Node 2 closed
2009-10-05 19:51:46 [MgmSrvr] INFO -- Node 5: Communication to
Node 2 closed
2009-10-05 19:51:46 [MgmSrvr] ALERT -- Node 1: Node 2 Disconnected
2009-10-05 19:51:46 [MgmSrvr] ALERT -- Node 3: Arbitration check won -
node group majority
2009-10-05 19:51:46 [MgmSrvr] INFO -- Node 3: President restarts
arbitration thread [state=6]
2009-10-05 19:51:46 [MgmSrvr] ALERT -- Node 3: Node 2 Disconnected
2009-10-05 19:51:46 [MgmSrvr] ALERT -- Node 5: Node 2 Disconnected
2009-10-05 19:51:47 [MgmSrvr] ALERT -- Node 1: Node 2 Disconnected
2009-10-05 19:51:47 [MgmSrvr] WARNING -- Node 5: Transporter to node 3 reported
error 0x16: The send buffer was full, but sleeping for a while solved
2009-10-05 19:51:47 [MgmSrvr] WARNING -- Node 4: Transporter to node 3 reported
error 0x16: The send buffer was full, but sleeping for a while solved
2009-10-05 19:51:47 [MgmSrvr] WARNING -- Node 5: Transporter to node 3 reported
error 0x16: The send buffer was full, but sleeping for a while solved
2009-10-05 19:51:47 [MgmSrvr] WARNING -- Node 4: Transporter to node 3 reported
error 0x16: The send buffer was full, but sleeping for a while solved
2009-10-05 19:51:47 [MgmSrvr] WARNING -- Node 4: Transporter to node 3 reported
error 0x16: The send buffer was full, but sleeping for a while solved
2009-10-05 19:51:47 [MgmSrvr] WARNING -- Node 5: Transporter to node 3 reported
error 0x16: The send buffer was full, but sleeping for a while solved
... ...
2009-10-05 19:52:46 [MgmSrvr] WARNING -- Node 4: Failure handling of node 2 has not
completed in 1 min. - state = 3
2009-10-05 19:52:46 [MgmSrvr] INFO -- Node 5: Data usage is
76%(158439 32K pages of total 208000)
2009-10-05 19:52:47 [MgmSrvr] WARNING -- Node 3: Failure handling of node 2 has not
completed in 1 min. - state = 3
2009-10-05 19:52:48 [MgmSrvr] WARNING -- Node 5: Failure handling of node 2 has not
completed in 1 min. - state = 3
2009-10-05 19:53:28 [MgmSrvr] INFO -- Node 3: Data usage is
76%(158391 32K pages of total 208000)
2009-10-05 19:53:40 [MgmSrvr] INFO -- Node 4: Data usage is
76%(158807 32K pages of total 208000)
2009-10-05 19:53:46 [MgmSrvr] WARNING -- Node 4: Failure handling of node 2 has not
completed in 2 min. - state = 3
2009-10-05 19:53:49 [MgmSrvr] ALERT -- Node 3: Forced node shutdown
completed. Caused by error 2303: 'System error, node killed during node restart by other
node(Internal error, programming error or missing error message, please report a bug).
Temporary error, restart node'.
I have some issues for the above info:
1)Who can tell me what the 2341 error is? this error is very boring!!! I can not
take any useful information from this error.
2)why was send buffer full after node2 crashed?
node2 and node3 are node group 0; node4 and node5 are node group 1.
Why was send buffer full between node4/5 and node3? there shouldn't be data sync.
totalsendbuffermemory is 256M , reservedsendbuffermemory is 128M.
Are they too small???
3) the error 2303 on node3 is GCP Stop error. Why will GCP stop ??
Thank you very much!!!
*^_^*
___________________________________________________________
好玩贺卡等你发，邮箱贺卡全新上线！
http://card.mail.cn.yahoo.com/
| Thread |
|---|
| • send buffer was full-----TCP transporter | raid fifa | 10 Oct |
