List:Cluster« Previous MessageNext Message »
From:Serhat Rıfat Demircan Date:August 20 2015 10:46pm
Subject:Mysql Cluster Node failure caused abort of transaction Errors
View as plain text  
Hi,

I have an mysql cluster which have 4 api nodes, 2 management nodes and 4
data nodes. Today, I was having problems while trying to connect database
and all queries was hanging at "Opening tables" state. After inspecting
logs i have found these erros on logs.

Api Node:

2015-08-20 19:44:14 15540 [Note] NDB Schema dist: Data node: 5 failed,
subscriber bitmask 00
2015-08-20 19:44:14 15540 [Note] NDB Schema dist: Data node: 6 failed,
subscriber bitmask 00
2015-08-20 19:44:14 15540 [Note] NDB Schema dist: Data node: 7 failed,
subscriber bitmask 00
2015-08-20 19:44:14 15540 [Note] NDB Schema dist: Data node: 8 failed,
subscriber bitmask 00
2015-08-20 19:44:14 15540 [Note] NDB Schema dist: cluster failure at epoch
3313124/17.
2015-08-20 19:44:14 15540 [Note] NDB Binlog: ndb tables initially read only
on reconnect.
2015-08-20 19:44:14 15540 [ERROR] /opt/mysql/server-5.6/bin/mysqld: Got
temporary error 4028 'Node failure caused abort of transaction' from
NDBCLUSTER
2015-08-20 19:44:14 15540 [ERROR] /opt/mysql/server-5.6/bin/mysqld: Sort
aborted: Got temporary error 4028 'Node failure caused abort of
transaction' from NDBCLUSTER
2015-08-20 19:44:14 15540 [ERROR] Got error 4010 when reading table
'./database_name/table'
2015-08-20 19:44:14 15540 [Note] NDB Binlog: cluster failure for
./database_name/table_name at epoch 3313124/17.

mysql> show processlists;

Id User Host db Command Time State Info
1 system user NULL Daemon 1497 Waiting for ndbcluster to start NULL

Data Node:

2015-08-20 19:44:14 [ndbd] ERROR -- c_gcp_list.seize() failed: gci:
14229759227592721 nodes:
0000000000000000000000000000040000000000000000000000000000001a00
2015-08-20 19:44:14 [ndbd] WARNING -- ACK wo/ gcp record (gci: 3313124/17)
ref: 0fa2000b from: 0fa2000b
2015-08-20 19:44:14 [ndbd] WARNING -- ACK wo/ gcp record (gci: 3313124/17)
ref: 0fa2000c from: 0fa2000c
2015-08-20 19:44:14 [ndbd] WARNING -- ACK wo/ gcp record (gci: 3313124/17)
ref: 0fa2008a from: 0fa2008a

After restarting api node, database has started to work normally. What
might cause "Node failure caused abort of transaction" error?

Thread
Mysql Cluster Node failure caused abort of transaction ErrorsSerhat Rıfat Demircan20 Aug
  • Re: Mysql Cluster Node failure caused abort of transaction ErrorsHartmut Holzgraefe21 Aug
    • Re: Mysql Cluster Node failure caused abort of transaction ErrorsSerhat Rıfat Demircan21 Aug
      • Re: Mysql Cluster Node failure caused abort of transaction ErrorsSerhat Rıfat Demircan24 Aug