When trying to restart my cluster today, it got stuck in a loop,
crashing and restarting. This has happened twice now, both in the same
way, so I think I know how to duplicate it (will outline that below) -
but here's the mgm output. A few notes before I go on ... I'm using an
interface to ndb_mgm that I wrote, so things might look a bit different
than they do at the normal prompt. The main thing this allows me to do
is run commands and shell scripts from the same console where the mgm
logs go; I can still send cmds to the mgm process too. And it's all
color coded :)
The log is very long ... this shows the start of the problem ... I was
running concurrent inserts on 3 API nodes and restarted one of the DB
nodes intentionally, to see how the cluster would handle it. So that's
where this starts ... but after a little while, the other 3 nodes fail
as well, and then it gets stuck in the loop, and never recovers.
Let me know if any trace files would be helpful.
Devananda
Neopets, Inc
Jul 20 18:26:31 images111 NDB[10570]: [MgmSrvr] Node 12: Local
checkpoint 51 started. Keep GCI = 190222 oldest restorable GCI = 190224
Jul 20 18:26:53 images111 NDB[10570]: [MgmSrvr] Node 12: Local
checkpoint 52 started. Keep GCI = 190232 oldest restorable GCI = 190237
Jul 20 18:27:25 images111 NDB[10570]: [MgmSrvr] Node 12: Local
checkpoint 53 started. Keep GCI = 190242 oldest restorable GCI = 190249
Jul 20 18:28:19 images111 NDB[10570]: [MgmSrvr] Node 12: Local
checkpoint 54 started. Keep GCI = 190257 oldest restorable GCI = 190269
! all status
-- NDB Cluster -- Management Client --
Connecting to Management Server: cluster11:28000
Node 12: Started (Version 3.5.0)
Node 13: Started (Version 3.5.0)
Node 14: Started (Version 3.5.0)
Node 15: Started (Version 3.5.0)
! 12 restart
-- NDB Cluster -- Management Client --
Connecting to Management Server: cluster11:28000
Executing RESTART on node 12.
Database node 12 is being restarted.
Jul 20 18:29:09 images111 NDB[10570]: [MgmSrvr] Node 12: Node shutdown
initiated
Jul 20 18:29:18 images111 NDB[10570]: [MgmSrvr] Node 63: Node 12
Disconnected
Jul 20 18:29:18 images111 NDB[10570]: [MgmSrvr] Lost connection to node 12
Node 12 lost
Jul 20 18:29:18 images111 NDB[10570]: [MgmSrvr] Node 14: Arbitration
check won - node group majority
Jul 20 18:29:18 images111 NDB[10570]: [MgmSrvr] Node 14: President
restarts arbitration thread [state=6]
Jul 20 18:29:18 images111 NDB[10570]: [MgmSrvr] Node 14: GCP Take over
started
Jul 20 18:29:19 images111 NDB[10570]: [MgmSrvr] Node 14: GCP Take over
completed
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14: LCP Take over
started
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14:
ParticipatingDIH = 000000000000e000
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14:
ParticipatingLQH = 000000000000e000
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14:
m_LCP_COMPLETE_REP_Counter_DIH = [SignalCounter: m_count=0 0000000000000000]
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14:
m_LCP_COMPLETE_REP_Counter_LQH = [SignalCounter: m_count=3 000000000000e000]
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14:
m_LAST_LCP_FRAG_ORD = [SignalCounter: m_count=3 000000000000e000]
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14:
m_LCP_COMPLETE_REP_From_Master_Received = 0
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14: LCP Take over
completed (state = 5)
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14:
ParticipatingDIH = 000000000000e000
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14:
ParticipatingLQH = 000000000000e000
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14:
m_LCP_COMPLETE_REP_Counter_DIH = [SignalCounter: m_count=3 000000000000e000]
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14:
m_LCP_COMPLETE_REP_Counter_LQH = [SignalCounter: m_count=3 000000000000e000]
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14:
m_LAST_LCP_FRAG_ORD = [SignalCounter: m_count=3 000000000000e000]
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14:
m_LCP_COMPLETE_REP_From_Master_Received = 0
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14: Node 13:
started: table=18 fragment=3 replica=71
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14: Node 14:
started: table=18 fragment=2 replica=69
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14: Node 15:
started: table=18 fragment=2 replica=68
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14: Table 0:
TabCopyStatus: 0 TabUpdateStatus: 0 TabLcpStatus: 3
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14: Fragment 0:
noLcpReplicas==0 0(on 13)=54(Idle)
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14: Fragment 1:
noLcpReplicas==0 0(on 14)=54(Idle) 1(on 15)=54(Idle)
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14: Table 2:
TabCopyStatus: 0 TabUpdateStatus: 0 TabLcpStatus: 3
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14: Fragment 0:
noLcpReplicas==0 0(on 14)=54(Idle) 1(on 15)=54(Idle)
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14: Fragment 1:
noLcpReplicas==0 0(on 13)=54(Idle)
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14: Table 4:
TabCopyStatus: 0 TabUpdateStatus: 0 TabLcpStatus: 3
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14: Fragment 0:
noLcpReplicas==0 0(on 13)=54(Idle)
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14: Fragment 1:
noLcpReplicas==0 0(on 14)=54(Idle) 1(on 15)=54(Idle)
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14: Fragment 2:
noLcpReplicas==0 0(on 13)=54(Idle)
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14: Fragment 3:
noLcpReplicas==0 0(on 15)=54(Idle) 1(on 14)=54(Idle)
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14: Table 6:
TabCopyStatus: 0 TabUpdateStatus: 0 TabLcpStatus: 3
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14: Fragment 0:
noLcpReplicas==0 0(on 13)=0(Idle)
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14: Fragment 1:
noLcpReplicas==0 0(on 14)=0(Idle) 1(on 15)=0(Idle)
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14: Fragment 2:
noLcpReplicas==0 0(on 13)=0(Idle)
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14: Fragment 3:
noLcpReplicas==0 0(on 15)=0(Idle) 1(on 14)=0(Idle)
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14: Table 8:
TabCopyStatus: 0 TabUpdateStatus: 0 TabLcpStatus: 3
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14: Fragment 0:
noLcpReplicas==0 0(on 13)=0(Idle)
Jul 20 18:29:32 images111 NDB[10570]: [MgmSrvr] Node 14: Fragment 1:
noLcpReplicas==0 0(on 14)=0(Idle) 1(on 15)=0(Idle)
Jul 20 18:29:34 images111 NDB[10570]: [MgmSrvr] Node 14: Local
checkpoint 55 started. Keep GCI = 190282 oldest restorable GCI = 190298
Jul 20 18:29:36 images111 NDB[10570]: [MgmSrvr] Node 12: Start phase 1
completed
Node 12: phase 1
Jul 20 18:29:36 images111 NDB[10570]: [MgmSrvr] Node 12: Start phase 2
completed (node restart)
Node 12: phase 2
Jul 20 18:29:36 images111 NDB[10570]: [MgmSrvr] Node 12: Receive
arbitrator node 63 [ticket=2d1d0001df4ada83]
Jul 20 18:29:51 images111 NDB[10570]: [MgmSrvr] Node 12: Start phase 3
completed (node restart)
Node 12: phase 3
Jul 20 18:29:52 images111 NDB[10570]: [MgmSrvr] Node 12: Start phase 4
completed (node restart)
Node 12: phase 4
! show
-- NDB Cluster -- Management Client --
Connecting to Management Server: cluster11:28000
Cluster Configuration
---------------------
4 NDB Node(s)
DB node: 12 (Version: 3.5.0)
DB node: 13 (Version: 3.5.0)
DB node: 14 (Version: 3.5.0)
DB node: 15 (Version: 3.5.0)
5 API Node(s) ##### note, this
appears to be a bug, since all 5 API nodes *are* connected
API node: 21 (not connected)
API node: 22 (not connected)
API node: 23 (not connected)
API node: 24 (not connected)
API node: 25 (not connected)
1 MGM Node(s)
MGM node: 63 (Version: 3.5.0)
Jul 20 18:31:19 images111 NDB[10570]: [MgmSrvr] Node 12: DICT: index 6
activated
Jul 20 18:31:19 images111 NDB[10570]: [MgmSrvr] Node 12: DICT: index 8
activated
Jul 20 18:31:19 images111 NDB[10570]: [MgmSrvr] Node 12: DICT: index 10
activated
Jul 20 18:31:19 images111 NDB[10570]: [MgmSrvr] Node 12: DICT: index 12
activated
Jul 20 18:31:19 images111 NDB[10570]: [MgmSrvr] Node 12: DICT: index 14
activated
Jul 20 18:31:20 images111 NDB[10570]: [MgmSrvr] Node 12: DICT: index 16
activated
Jul 20 18:31:20 images111 NDB[10570]: [MgmSrvr] Node 12: DICT: index 20
activated
Jul 20 18:31:20 images111 NDB[10570]: [MgmSrvr] Node 12: DICT: index 22
activated
Jul 20 18:31:20 images111 NDB[10570]: [MgmSrvr] Node 14: Local
checkpoint 56 started. Keep GCI = 190316 oldest restorable GCI = 190338
Jul 20 18:32:21 images111 NDB[10570]: [MgmSrvr] Node 15: Data usage
increased to 83%(21299 32K pages of total 25600)
Jul 20 18:32:21 images111 NDB[10570]: [MgmSrvr] Node 14: Data usage
increased to 83%(21299 32K pages of total 25600)
Jul 20 18:32:23 images111 NDB[10570]: [MgmSrvr] Node 13: Data usage
increased to 83%(21275 32K pages of total 25600)
Jul 20 18:33:46 images111 NDB[10570]: [MgmSrvr] Node 14: Local
checkpoint 57 started. Keep GCI = 190364 oldest restorable GCI = 190394
! all status
-- NDB Cluster -- Management Client --
Connecting to Management Server: cluster11:28000
Node 12: Starting (Start phase 5) (Version 3.5.0)
Node 13: Started (Version 3.5.0)
Node 14: Started (Version 3.5.0)
Node 15: Started (Version 3.5.0)
Jul 20 18:34:16 images111 NDB[10570]: [MgmSrvr] Node 15: Data usage
increased to 91%(23414 32K pages of total 25600)
Jul 20 18:34:17 images111 NDB[10570]: [MgmSrvr] Node 14: Data usage
increased to 91%(23414 32K pages of total 25600)
Jul 20 18:34:23 images111 NDB[10570]: [MgmSrvr] Node 13: Data usage
increased to 91%(23361 32K pages of total 25600)
Jul 20 18:34:39 images111 NDB[10570]: [MgmSrvr] Node 13: Node 15 missed
heartbeat 2
Jul 20 18:34:43 images111 NDB[10570]: [MgmSrvr] Node 15: Node 14 missed
heartbeat 2
Jul 20 18:34:44 images111 NDB[10570]: [MgmSrvr] Node 15: Node 14 missed
heartbeat 3
Jul 20 18:35:00 images111 NDB[10570]: [MgmSrvr] Node 12: Node 13 missed
heartbeat 2
! all status
-- NDB Cluster -- Management Client --
Connecting to Management Server: cluster11:28000
Node 12: Starting (Start phase 5) (Version 3.5.0)
Node 13: Started (Version 3.5.0)
Node 14: Started (Version 3.5.0)
Node 15: Started (Version 3.5.0)
Jul 20 18:35:01 images111 NDB[10570]: [MgmSrvr] Node 12: Node 13 missed
heartbeat 3
Jul 20 18:35:03 images111 NDB[10570]: [MgmSrvr] Node 12: Node 13 missed
heartbeat 4
Jul 20 18:35:03 images111 NDB[10570]: [MgmSrvr] Node 12: Node 13
declared dead due to missed heartbeat
Jul 20 18:35:03 images111 NDB[10581]: [MgmSrvr] Node 63: Node 13
Disconnected
Jul 20 18:35:03 images111 NDB[10581]: [MgmSrvr] Lost connection to node 13
Node 13 lost
Jul 20 18:35:04 images111 NDB[10570]: [MgmSrvr] Node 63: Node 14
Disconnected
Jul 20 18:35:04 images111 NDB[10570]: [MgmSrvr] Lost connection to node 14
Node 14 lost
Jul 20 18:35:04 images111 NDB[10570]: [MgmSrvr] Node 15: Possible bug in
Dbdih::execBLOCK_COMMIT_ORD c_blockCommit = 1 c_blockCommitNo = 3
sig->failNo =
Jul 20 18:35:04 images111 NDB[10570]: [MgmSrvr] Node 12: Possible bug in
Dbdih::execBLOCK_COMMIT_ORD c_blockCommit = 1 c_blockCommitNo = 3
sig->failNo =
Jul 20 18:35:04 images111 NDB[10570]: [MgmSrvr] Node 15: Network
partitioning - arbitration required
Jul 20 18:35:04 images111 NDB[10570]: [MgmSrvr] Node 15: President
restarts arbitration thread [state=7]
Jul 20 18:35:04 images111 NDB[10570]: [MgmSrvr] Node 63: Node 15
Disconnected
Jul 20 18:35:04 images111 NDB[10570]: [MgmSrvr] Lost connection to node 15
Node 15 lost
Jul 20 18:35:05 images111 NDB[10570]: [MgmSrvr] Node 63: Node 12
Disconnected
Jul 20 18:35:05 images111 NDB[10570]: [MgmSrvr] Lost connection to node 12
Node 12 lost
Number of processes running now: 0
040720 18:36:59 mysqld restarted
Jul 20 18:35:24 images111 NDB[10570]: [MgmSrvr] Node 12: Start phase 1
completed
Node 12: phase 1
Number of processes running now: 0
040720 18:37:30 mysqld restarted
Number of processes running now: 0
040720 18:37:16 mysqld restarted
Jul 20 18:35:26 images111 NDB[10570]: [MgmSrvr] Node 13: CM_REGCONF
president = 12, own Node = 13, our dynamic id = 3
Jul 20 18:35:26 images111 NDB[10570]: [MgmSrvr] Node 13: Start phase 1
completed
Node 13: phase 1
Jul 20 18:35:27 images111 NDB[10570]: [MgmSrvr] Node 15: CM_REGCONF
president = 12, own Node = 15, our dynamic id = 4
Jul 20 18:35:27 images111 NDB[10570]: [MgmSrvr] Node 15: Start phase 1
completed
Node 15: phase 1
Jul 20 18:35:27 images111 NDB[10570]: [MgmSrvr] Node 13: Start phase 2
completed (system restart)
Node 13: phase 2
Jul 20 18:35:27 images111 NDB[10570]: [MgmSrvr] Node 12: Start phase 2
completed (system restart)
Node 12: phase 2
Jul 20 18:35:27 images111 NDB[10570]: [MgmSrvr] Node 15: Start phase 2
completed (system restart)
Node 15: phase 2
Jul 20 18:35:27 images111 NDB[10570]: [MgmSrvr] Node 14: Start phase 2
completed (system restart)
Node 14: phase 2
Jul 20 18:35:42 images111 NDB[10570]: [MgmSrvr] Node 15: Start phase 3
completed (system restart)
Node 15: phase 3
Jul 20 18:35:42 images111 NDB[10570]: [MgmSrvr] Node 12: Start phase 3
completed (system restart)
Node 12: phase 3
Jul 20 18:35:42 images111 NDB[10570]: [MgmSrvr] Node 14: Start phase 3
completed (system restart)
Node 14: phase 3
Jul 20 18:35:42 images111 NDB[10570]: [MgmSrvr] Node 13: Start phase 3
completed (system restart)
Node 13: phase 3
Jul 20 18:35:48 images111 NDB[10570]: [MgmSrvr] Node 12: DICT: index 6
activated done
Jul 20 18:35:48 images111 NDB[10570]: [MgmSrvr] Node 12: DICT: index 8
activated done
Jul 20 18:35:48 images111 NDB[10570]: [MgmSrvr] Node 12: DICT: index 10
activated done
Jul 20 18:35:48 images111 NDB[10570]: [MgmSrvr] Node 12: DICT: index 12
activated done
Jul 20 18:35:48 images111 NDB[10570]: [MgmSrvr] Node 12: DICT: index 14
activated done
Jul 20 18:35:48 images111 NDB[10570]: [MgmSrvr] Node 12: DICT: index 16
activated done
Jul 20 18:35:48 images111 NDB[10570]: [MgmSrvr] Node 12: DICT: index 20
activated done
Jul 20 18:35:48 images111 NDB[10570]: [MgmSrvr] Node 12: DICT: index 22
activated done
040720 18:38:10 mysqld ended
040720 18:38:41 mysqld ended
040720 18:38:27 mysqld ended
! show
-- NDB Cluster -- Management Client --
Connecting to Management Server: cluster11:28000
Cluster Configuration
---------------------
4 NDB Node(s)
DB node: 12 (Version: 3.5.0)
DB node: 13 (Version: 3.5.0)
DB node: 14 (Version: 3.5.0)
DB node: 15 (Version: 3.5.0)
5 API Node(s)
API node: 21 (not connected)
API node: 22 (not connected)
API node: 23 (not connected)
API node: 24 (not connected)
API node: 25 (not connected)
1 MGM Node(s)
MGM node: 63 (Version: 3.5.0)
! all status
-- NDB Cluster -- Management Client --
Connecting to Management Server: cluster11:28000
Node 12: Starting (Start phase 4) (Version 3.5.0)
Node 13: Starting (Start phase 4) (Version 3.5.0)
Node 14: Starting (Start phase 4) (Version 3.5.0)
Node 15: Starting (Start phase 4) (Version 3.5.0)
Jul 20 18:38:08 images111 NDB[10570]: [MgmSrvr] Node 12: Start phase 4
completed (system restart)
Node 12: phase 4
Jul 20 18:38:08 images111 NDB[10570]: [MgmSrvr] Node 14: Start phase 4
completed (system restart)
Node 14: phase 4
Jul 20 18:38:08 images111 NDB[10570]: [MgmSrvr] Node 15: Start phase 4
completed (system restart)
Node 15: phase 4
Jul 20 18:38:08 images111 NDB[10570]: [MgmSrvr] Node 13: Start phase 4
completed (system restart)
Node 13: phase 4
Jul 20 18:38:09 images111 NDB[10570]: [MgmSrvr] Node 12: Local
checkpoint 57 started. Keep GCI = 190364 oldest restorable GCI = 190394
Jul 20 18:41:09 images111 NDB[10570]: [MgmSrvr] Node 12: Start phase 5
completed (system restart)
Node 12: phase 5
Jul 20 18:41:09 images111 NDB[10570]: [MgmSrvr] Node 12: Start phase 6
completed (system restart)
Node 12: phase 6
Jul 20 18:41:09 images111 NDB[10570]: [MgmSrvr] Node 12: President
restarts arbitration thread [state=1]
Jul 20 18:41:09 images111 NDB[10570]: [MgmSrvr] Node 12: Start phase 7
completed (system restart)
Node 12: phase 7
Jul 20 18:41:09 images111 NDB[10570]: [MgmSrvr] Node 15: Start phase 5
completed (system restart)
Node 15: phase 5
Jul 20 18:41:09 images111 NDB[10570]: [MgmSrvr] Node 14: Start phase 5
completed (system restart)
Node 14: phase 5
Jul 20 18:41:09 images111 NDB[10570]: [MgmSrvr] Node 13: Start phase 5
completed (system restart)
Node 13: phase 5
Jul 20 18:41:09 images111 NDB[10570]: [MgmSrvr] Node 15: Start phase 6
completed (system restart)
Node 15: phase 6
Jul 20 18:41:09 images111 NDB[10570]: [MgmSrvr] Node 15: Start phase 7
completed (system restart)
Node 15: phase 7
Jul 20 18:41:09 images111 NDB[10570]: [MgmSrvr] Node 14: Start phase 6
completed (system restart)
Node 14: phase 6
Jul 20 18:41:09 images111 NDB[10570]: [MgmSrvr] Node 14: Start phase 7
completed (system restart)
Node 14: phase 7
Jul 20 18:41:09 images111 NDB[10570]: [MgmSrvr] Node 13: Start phase 6
completed (system restart)
Node 13: phase 6
Jul 20 18:41:09 images111 NDB[10570]: [MgmSrvr] Node 13: Start phase 7
completed (system restart)
Node 13: phase 7
Jul 20 18:41:11 images111 NDB[10570]: [MgmSrvr] Node 13: Data usage
increased to 88%(22528 32K pages of total 25600)
Jul 20 18:41:11 images111 NDB[10570]: [MgmSrvr] Node 14: Data usage
increased to 88%(22583 32K pages of total 25600)
Jul 20 18:41:11 images111 NDB[10570]: [MgmSrvr] Node 15: Data usage
increased to 88%(22581 32K pages of total 25600)
Jul 20 18:41:13 images111 NDB[10570]: [MgmSrvr] Node 12: DICT: index 6
rebuild done
Jul 20 18:41:14 images111 NDB[10570]: [MgmSrvr] Node 63: Node 12
Disconnected
Jul 20 18:41:14 images111 NDB[10570]: [MgmSrvr] Lost connection to node 12
Node 12 lost
Jul 20 18:41:14 images111 NDB[10570]: [MgmSrvr] Node 63: Node 14
Disconnected
Jul 20 18:41:14 images111 NDB[10570]: [MgmSrvr] Lost connection to node 14
Node 14 lost
Jul 20 18:41:14 images111 NDB[10570]: [MgmSrvr] Node 13: Possible bug in
Dbdih::execBLOCK_COMMIT_ORD c_blockCommit = 1 c_blockCommitNo = 2
sig->failNo =
Jul 20 18:41:14 images111 NDB[10570]: [MgmSrvr] Node 15: Possible bug in
Dbdih::execBLOCK_COMMIT_ORD c_blockCommit = 1 c_blockCommitNo = 2
sig->failNo =
Jul 20 18:41:14 images111 NDB[10570]: [MgmSrvr] Node 63: Node 13
Disconnected
Jul 20 18:41:14 images111 NDB[10570]: [MgmSrvr] Lost connection to node 13
Node 13 lost
Jul 20 18:41:15 images111 NDB[10570]: [MgmSrvr] Node 63: Node 15
Disconnected
Jul 20 18:41:15 images111 NDB[10570]: [MgmSrvr] Lost connection to node 15
