List:Cluster« Previous MessageNext Message »
From:Anjitha Date:September 26 2008 2:32am
Subject:Re: MySQL Cluster Crashing Frequently
View as plain text  
Hi Jeff

Thank you very much for the reply. We are trying that solution now . We are 
trying to move NDB alone to a different network :).

Mean while i would like to confirm one thing - The configuration we are 
using have no issues ,right?


Thanking you once gain for the reply

Regards
Anjitha
Software Engineer
Mystiq Mobile


----- Original Message ----- 
From: "Jeff Sturm" <jeff.sturm@stripped>
To: "Anjitha" <anjitha@stripped>; <cluster@stripped>
Sent: Thursday, September 25, 2008 8:02 PM
Subject: RE: MySQL Cluster Crashing Frequently


So each node simultaneously lost its connection to the other, shutting
itself down?

That normally would not happen unless the intervening network were to
fail.  Make sure your network is good.  It's not a bad idea to dedicate
some network resources to NDB.  If there are only two NDB nodes, you
might even directly attach them with an Ethernet crossover cable.

The following page from the manual gives an example of NDB network
configuration with a crossover:

http://dev.mysql.com/doc/refman/5.1/en/mysql-cluster-tcp-definition-dire
ct.html

Jeff

> -----Original Message-----
> From: Anjitha [mailto:anjitha@stripped]
> Sent: Thursday, September 25, 2008 2:26 AM
> To: Jeff Sturm; cluster@stripped
> Subject: Re: MySQL Cluster Crashing Frequently
>
> Hi Jeff
>
> Thank you for the reply . I am attaching the logs on the
> management node here by
>
> 2008-09-25 09:38:48 [MgmSrvr] ALERT    -- Node 1: Node 2 Disconnected
> 2008-09-25 09:38:48 [MgmSrvr] ALERT    -- Node 1: Node 3 Disconnected
> 2008-09-25 09:38:48 [MgmSrvr] ALERT    -- Node 1: Node 2 Disconnected
> 2008-09-25 09:38:48 [MgmSrvr] ALERT    -- Node 1: Node 3 Disconnected
> 2008-09-25 09:38:50 [MgmSrvr] INFO     -- Node 1: Node 3 Connected
> 2008-09-25 09:38:51 [MgmSrvr] INFO     -- Node 1: Node 2 Connected
> 2008-09-25 09:38:51 [MgmSrvr] INFO     -- Node 2: Node 4: API
> mysql-5.1.23
> ndb-6.2.15
> 2008-09-25 09:38:51 [MgmSrvr] INFO     -- Node 2: Node 7: API
> mysql-5.1.23
> ndb-6.2.15
> 2008-09-25 09:38:51 [MgmSrvr] INFO     -- Node 2: Node 5 Connected
> 2008-09-25 09:38:51 [MgmSrvr] INFO     -- Node 2: Node 5: API
> mysql-5.1.23
> ndb-6.2.15
> 2008-09-25 09:38:51 [MgmSrvr] ALERT    -- Node 1: Node 3 Disconnected
> 2008-09-25 09:38:51 [MgmSrvr] ALERT    -- Node 1: Node 2 Disconnected
> 2008-09-25 09:38:51 [MgmSrvr] ALERT    -- Node 3: Forced node shutdown
> completed. Caused by error 2305: 'Node lost connection to
> other nodes and can not form a unpartitioned cluster, please
> investigate if there are
> error(s) on other node(s)(Arbitration error). Temporary
> error, restart node'.
> 2008-09-25 09:38:51 [MgmSrvr] INFO     -- Mgmt server state: nodeid 3
> reserved for ip 192.168.1.43, m_reserved_nodes
> 0000000000000000000000000000000000000000000000000000000000003ffa.
> 2008-09-25 09:38:51 [MgmSrvr] ALERT    -- Node 2: Forced node shutdown
> completed. Caused by error 2305: 'Node lost connection to
> other nodes and can not form a unpartitioned cluster, please
> investigate if there are
> error(s) on other node(s)(Arbitration error). Temporary
> error, restart node'.
> 2008-09-25 09:38:51 [MgmSrvr] INFO     -- Mgmt server state: nodeid 2
> reserved for ip 192.168.1.40, m_reserved_nodes
> 0000000000000000000000000000000000000000000000000000000000003ffe.
> 2008-09-25 09:38:52 [MgmSrvr] INFO     -- Node 1: Node 3 Connected
> 2008-09-25 09:38:52 [MgmSrvr] INFO     -- Node 1: Node 2 Connected
> 2008-09-25 09:38:53 [MgmSrvr] INFO     -- Mgmt server state:
> nodeid 3 freed,
> m_reserved_nodes
> 0000000000000000000000000000000000000000000000000000000000003ff6.
> 2008-09-25 09:38:54 [MgmSrvr] INFO     -- Mgmt server state:
> nodeid 2 freed,
> m_reserved_nodes
> 0000000000000000000000000000000000000000000000000000000000003ff2.
> 2008-09-25 09:38:55 [MgmSrvr] INFO     -- Node 3: CM_REGCONF
> president = 2,
> own Node = 3, our dynamic id = 2
> 2008-09-25 09:38:55 [MgmSrvr] INFO     -- Node 2: Node 3: API
> mysql-5.1.23
> ndb-6.2.15
> 2008-09-25 09:38:55 [MgmSrvr] INFO     -- Node 2: Start phase
> 1 completed
> 2008-09-25 09:38:55 [MgmSrvr] INFO     -- Node 3: Node 2: API
> mysql-5.1.23
> ndb-6.2.15
> 2008-09-25 09:38:55 [MgmSrvr] INFO     -- Node 3: Start phase
> 1 completed
> 2008-09-25 09:38:56 [MgmSrvr] INFO     -- Node 2: Start phase
> 2 completed
> (system restart)
> 2008-09-25 09:38:56 [MgmSrvr] INFO     -- Node 3: Start phase
> 2 completed
> (system restart)
> 2008-09-25 09:38:56 [MgmSrvr] INFO     -- Node 2: Start phase
> 3 completed
> (system restart)
> 2008-09-25 09:38:56 [MgmSrvr] INFO     -- Node 3: Start phase
> 3 completed
> (system restart)
>
> Thanking all in advance,
>
> Regards
> Anjitha
> Software Engineer
> Mystiq Mobile
>
>
> ----- Original Message ----- 
>
>
>
> From: "Jeff Sturm" <jeff.sturm@stripped>
> To: "Anjitha" <anjitha@stripped>; <cluster@stripped>
> Sent: Thursday, September 25, 2008 11:59 AM
> Subject: RE: MySQL Cluster Crashing Frequently
>
>
> Anjitha,
>
> Check the logs on the management node for this time
> period--those may have additional information.
>
> Jeff
>
> > -----Original Message-----
> > From: Anjitha [mailto:anjitha@stripped]
> > Sent: Wednesday, September 24, 2008 11:25 PM
> > To: cluster@stripped
> > Subject: MySQL Cluster Crashing Frequently
> >
> >
> >
> >   Hii
> >
> >   I am facing some problem with MySQL cluster . Please help me .
> >   We have a MySQL Cluster setup with 2 servers acting as
> > data+sql nodes and 1 server acting as a management node. The
> > ndbd process keeps crashing regularly and the error log says:
> >
> >    Time: Thursday 25 September 2008 - 09:38:51
> >
> >   Status: Temporary error, restart node
> >
> >   Message: Node lost connection to other nodes and can not form a
> > unpartitioned cluster, please investigate if there are
> > error(s) on other node(s) (Arbitration error)
> >
> >   Error: 2305
> >
> >   Error data: Arbitrator decided to shutdown this node
> >
> >   Error object: QMGR (Line: 5024) 0x00000008
> >
> >   Program: /usr/local/mysql/bin/ndbd
> >
> >   Pid: 5064
> >
> >   Trace: /datastore/mysql-cluster/data/ndb_2_trace.log.10
> >
> >   Version: mysql-5.1.23 ndb-6.2.15
> >
> >   ***EOM***
> >
> >
> >
> >   Config:
> >
> >   [TCP DEFAULT]
> >
> >   #SendBufferMemory=2M
> >
> >   #ReceiveBufferMemory=2M
> >
> >
> >
> >   [NDB_MGMD DEFAULT]
> >
> >   PortNumber=1186
> >
> >   Datadir=/datastorage/mysql-cluster
> >
> >
> >
> >   [NDB_MGMD]
> >
> >   Hostname=192.168.1.47
> >
> >
> >
> >   [NDBD DEFAULT]
> >
> >   NoOfReplicas=2
> >
> >   Datadir=/datastore/mysql-cluster/data
> >
> >   DataMemory=1024M
> >
> >   IndexMemory=128M
> >
> >   #LockPagesInMainMemory=1
> >
> >   StopOnError=N
> >
> >
> >
> >   MaxNoOfConcurrentOperations=100000
> >
> >   TransactionDeadlockDetectionTimeout=10000
> >
> >   HeartbeatIntervalDbDb=3000
> >
> >   HeartbeatIntervalDbApi=3000
> >
> >
> >
> >
> >
> >   StringMemory=25
> >
> >   MaxNoOfTables=4096
> >
> >   MaxNoOfOrderedIndexes=2048
> >
> >   MaxNoOfUniqueHashIndexes=512
> >
> >   MaxNoOfAttributes=24576
> >
> >   DiskCheckpointSpeedInRestart=100M
> >
> >   FragmentLogFileSize=256M
> >
> >   NoOfFragmentLogFiles=6
> >
> >   RedoBuffer=32M
> >
> >
> >
> >   #TimeBetweenLocalCheckpoints=20
> >
> >   #TimeBetweenGlobalCheckpoints=1000
> >
> >   #TimeBetweenEpochs=100
> >
> >
> >
> >   ### Disk data related
> >
> >   DiskPageBufferMemory=64M
> >
> >   SharedGlobalMemory=256M
> >
> >   [NDBD]
> >
> >   Hostname=192.168.1.40
> >
> >
> >
> >   [NDBD]
> >
> >   Hostname=192.168.1.43
> >
> >
> >
> >   [MYSQLD]
> >
> >   Hostname=192.168.1.40
> >
> >   [MYSQLD]
> >
> >   Hostname=192.168.1.40
> >
> >   [MYSQLD]
> >
> >   Hostname=192.168.1.40
> >
> >   [MYSQLD]
> >
> >   Hostname=192.168.1.40
> >
> >   [MYSQLD]
> >
> >   Hostname=192.168.1.40
> >
> >
> >
> >   [MYSQLD]
> >
> >   Hostname=192.168.1.43
> >
> >   [MYSQLD]
> >
> >   Hostname=192.168.1.43
> >
> >   [MYSQLD]
> >
> >   Hostname=192.168.1.43
> >
> >   [MYSQLD]
> >
> >   Hostname=192.168.1.43
> >
> >   [MYSQLD]
> >
> >   Hostname=192.168.1.43
> >
> >
> >
> >
> >
> >   Regards
> >
> >   Anjitha
> >
> >   Software Engineer
> >
> >   Mystiq Mobile
> >
>
>
> 

Thread
MySQL Cluster Crashing FrequentlyAnjitha25 Sep
  • RE: MySQL Cluster Crashing FrequentlyJeff Sturm25 Sep
    • Re: MySQL Cluster Crashing FrequentlyAnjitha25 Sep
      • RE: MySQL Cluster Crashing FrequentlyJeff Sturm25 Sep
        • Re: MySQL Cluster Crashing FrequentlyAnjitha26 Sep
  • Re: MySQL Cluster Crashing FrequentlyJohan Andersson25 Sep
    • Re: MySQL Cluster Crashing FrequentlyAnjitha25 Sep