List:Cluster« Previous MessageNext Message »
From:Manoj Burande Date:September 10 2009 11:05am
Subject:Re: Unable to configuring multiple Management nodes.
View as plain text  
Hi Geert,

Thanks for your reply.

Again here with some new issues.

I am very cleared and understood with the all points which you had written
in your previous mail. The provided points are very valuable for me.

As you written in your previous mail I have tried first to get 2 data
nodes up with 1 ndb_mgmd and it was successfully implemented at my end.
But my goal is to configure 2 mgmt servers with 1 data node, 1 mysql node
on each host. Also the data node of each server should has data
replication of each other.

If one mgmt servers will goes down completely (including 1 data node of
itself, 1 mgmt node and 1 mysql node) due to system crash or due to any
hardware failure then other mgmgt server should responsible to serve the
clients with same databases or data(Because of replication).

In addition to the above as I had written in my previous mail, I have two
physical load balancer servers configured with HA Proxy(Except two MGMT
hosts). Through that it will manage the incoming connections to the MySQL
server for both hosts or servers(MGMT). The load balancers will keep
balanced connections to the both servers. In short both the MySQL servers
should able to accept the MySQL connections at the same time.

I have tried and successfully implemented the same scenario at my end with
the help of your blog
"http://blog.some-abstract-type.com/2009/08/how-to-start-mysql-cluster-70-with-2.html"(So
much Thanks for this.....). Both the servers are able to communicate to
each other. Even data is also getting replicate between the two data
nodes. The problem is that, if I shutdown the one mgmt server(#ndb_mgm -e
shutdown) then another mgmt server  automatically get shutdown or not
getting any response from it. As per my goal if one mgmt server will goes
down then another server should responsible to respond MySQL clients but
this is not possible to me with my configuration.

Here are the "config.ini" and "my.cnf" on both the servers:
===========================================================

"my.cnf" On host 192.168.1.107:
-------------------------------
[mysqld]
ndbcluster
#connectstring: primary,secondary management nodes
ndb-connectstring=192.168.1.107,192.168.1.125

[mysql_cluster]
ndb-connectstring=nodeid=1,192.168.1.107,192.168.1.125


"my.cnf" On host 192.168.1.125:
-------------------------------
[mysqld]
ndbcluster
#connectstring: primary,secondary management nodes
ndb-connectstring=192.168.1.107,192.168.1.125

[mysql_cluster]
ndb-connectstring=nodeid=1,192.168.1.107,192.168.1.125


"config.ini" On host 192.168.1.107:
===================================
[NDBD DEFAULT]
NoOfReplicas=2    # Number of replicas
DataMemory=3500M    # How much memory to allocate for data storage
IndexMemory=1000M   # How much memory to allocate for index storage
datadir=/usr/local/mysql/data   # Directory for this data node's datafiles
#MaxNoOfConcurrentOperations=1048576
#MaxNoOfConcurrentTransactions= 1048576
#MaxNoOfLocalOperations=1048576
#MaxNoOfConcurrentIndexOperations=16384
#MaxNoOfConcurrentScans=500
#LockPagesInMainMemory=Y
# TCP/IP options:
[TCP DEFAULT]
portnumber=2202   # This the default; however, you can use any

# Management process options:
[NDB_MGMD DEFAULT]
datadir=/var/lib/mysql-cluster  # Directory for MGM node logfiles

[NDB_MGMD]
id=1
hostname=192.168.1.107           # Hostname or IP address of MGM node

[NDB_MGMD]
id=2
hostname=192.168.1.125           # Hostname or IP address of MGM node

# Options for data node "A":
[NDBD]
id=3                             # (one [NDBD] section per data node)
hostname=192.168.1.107           # Hostname or IP address

# Options for data node "B":
[NDBD]
id=4
hostname=192.168.1.125          # Hostname or IP address

# SQL node options:
[MYSQLD]
id=5
hostname=192.168.1.107
[MYSQLD]
id=6
hostname=192.168.1.125
[MYSQLD]
[MYSQLD]


"config.ini" On host 192.168.1.125:
===================================
[mysqld]
ndbcluster
#connectstring: primary,secondary management nodes
ndb-connectstring=192.168.1.107,192.168.1.125

[mysql_cluster]
ndb-connectstring=nodeid=1,192.168.1.107,192.168.1.125


"config.ini" On host 192.168.1.107:
===================================
[NDBD DEFAULT]
NoOfReplicas=2    # Number of replicas
DataMemory=3500M    # How much memory to allocate for data storage
IndexMemory=1000M   # How much memory to allocate for index storage
datadir=/usr/local/mysql/data   # Directory for this data node's datafiles
#MaxNoOfConcurrentOperations=1048576
#MaxNoOfConcurrentTransactions= 1048576
#MaxNoOfLocalOperations=1048576
#MaxNoOfConcurrentIndexOperations=16384
#MaxNoOfConcurrentScans=500
#LockPagesInMainMemory=Y
# TCP/IP options:
[TCP DEFAULT]
portnumber=2202   # This the default; however, you can use any

# Management process options:
[NDB_MGMD DEFAULT]
datadir=/var/lib/mysql-cluster  # Directory for MGM node logfiles

[NDB_MGMD]
id=1
hostname=192.168.1.107           # Hostname or IP address of MGM node

[NDB_MGMD]
id=2
hostname=192.168.1.125           # Hostname or IP address of MGM node

# Options for data node "A":
[NDBD]
id=3                             # (one [NDBD] section per data node)
hostname=192.168.1.107           # Hostname or IP address

# Options for data node "B":
[NDBD]
id=4
hostname=192.168.1.125          # Hostname or IP address

# SQL node options:
[MYSQLD]
id=5
hostname=192.168.1.107
[MYSQLD]
id=6
hostname=192.168.1.125
[MYSQLD]
[MYSQLD]


Steps carried out on host 192.168.1.107:
----------------------------------------
1] [root@ns mysql-cluster]# ndb_mgmd -f config.ini
--configdir=/var/lib/mysql-cluster/ --initial
2009-09-10 16:09:25 [MgmSrvr] INFO     -- NDB Cluster Management Server.
mysql-5.1.34 ndb-7.0.6
2009-09-10 16:09:25 [MgmSrvr] INFO     -- Loaded config from
'/var/lib/mysql-cluster//ndb_1_config.bin.1'

2] [root@ns mysql-cluster]# ndbd -c 192.168.1.107
2009-09-10 16:10:30 [ndbd] INFO     -- Configuration fetched from
'192.168.1.107:1186', generation: 1

3] [root@ns mysql-cluster]# mysqld_safe --ndbcluster
--ndb-connectstring=192.168.1.107 &
[1] 4950
[root@ns mysql-cluster]# 090910 16:10:59 mysqld_safe Logging to
'/var/lib/mysql/ns.err'.
090910 16:10:59 mysqld_safe Starting mysqld daemon with databases from
/var/lib/mysql

4] [root@ns mysql-cluster]# ndb_mgm -e show
Connected to Management Server at: 192.168.1.107:1186
Cluster Configuration
---------------------
[ndbd(NDB)]	2 node(s)
id=3	@192.168.1.107  (mysql-5.1.34 ndb-7.0.6, starting, Nodegroup: 0)
id=4 (not connected, accepting connect from 192.168.1.125)

[ndb_mgmd(MGM)]	2 node(s)
id=1	@192.168.1.107  (mysql-5.1.34 ndb-7.0.6)
id=2 (not connected, accepting connect from 192.168.1.125)

[mysqld(API)]	4 node(s)
id=5 (not connected, accepting connect from 192.168.1.107)
id=6 (not connected, accepting connect from any host)
id=7 (not connected, accepting connect from any host)
id=8 (not connected, accepting connect from any host)


Commands carried out on host 192.168.1.125:
-------------------------------------------

1] [root@ns mysql-cluster]# ndb_mgmd -c 192.168.1.107 -ndb-nodeid=2
--configdir=/var/lib/mysql-cluster/

2] ndbd -c 192.168.1.125

3] mysqld_safe --ndbcluster --ndb-connectstring=192.168.1.125 &

[root@test mysql-cluster]# ndb_mgm -e show
Connected to Management Server at: 192.168.1.107:1186
Cluster Configuration
---------------------
[ndbd(NDB)]	2 node(s)
id=3	@192.168.1.107  (mysql-5.1.34 ndb-7.0.6,Nodegroup: 0, Master)
id=4    @192.168.1.125  (mysql-5.1.34 ndb-7.0.6,Nodegroup: 0)

[ndb_mgmd(MGM)]	2 node(s)
id=1	@192.168.1.107  (mysql-5.1.34 ndb-7.0.6)
id=2    @192.168.1.125  (mysql-5.1.34 ndb-7.0.6)

[mysqld(API)]	4 node(s)
id=5 @192.168.1.107  (mysql-5.1.34 ndb-7.0.6)
id=6 @192.168.1.125  (mysql-5.1.34 ndb-7.0.6)
id=7 (not connected, accepting connect from any host)
id=8 (not connected, accepting connect from any host)

It's all ok up to this time...ok...but

When I stop or shut down the one MGMT node machine or
server(host192.168.1.107) then other MGMT server(host:192.168.1.125)
should responsible to respond the clients but this is not happening..:-(

It shows me the error as,
-------------------------
[root@test mysql-cluster]# ndb_mgm -e show
Connected to Management Server at: 192.168.1.107:1186
Connected to Management Server at: 192.168.1.125:1186
Could not get status
*  1006: Illegal reply from server
*        Probably disconnected


So can you please tell me if I did anything wrong configuration or any
wrong step, so I can correct it.

Please reply as soon as possible.

Waiting for your reply.






> Hello Manoj,
>
> Please recheck how MySQL Cluster actually works before continuing!
>
> On Sep 7, 2009, at 14:27, Manoj Burande wrote:
> ..
>> On host: 192.168.1.107:
>> =======================
>> 1] MGMT Node
>> 2] DATA Node
>> 3] MYSQL[API] Node
>>
>> On host: 192.168.1.125:
>> =======================
>> 1] MGMT Node
>> 2] DATA Node
>> 3] MYSQL[API] Node
>
> If this is a test setup, then OK.
> BUT!
> In production, you should have at least 3 machines:
> - 2 machines running each ndbd's: the Data nodes
> - 1 machine running ndb_mgmd (Management node) and mysqld (SQL Node)
>
> You can add a 4th machine which will run mysqld and other ndb_mgmd.
>
>> Both the servers are placed behind "load balancer" to manage incoming
>> connections. So the incoming connections or requests to the server
>> will be
>> managed(distribute) by load balancer to control traffic on the server.
>
> If you want to use a 'load balancer', you should only use it to load-
> balance the SQL Nodes, the mysqld processes.
>
> ..
>> "config.ini" On host: 192.168.1.107:
>> ====================================
>
> And this is wrong. If you put config.ini on both machines which are
> running ndb_mgmd, then they should be _exactly_ the same.
>
>>
>> [NDBD DEFAULT]
>> NoOfReplicas=2    # Number of replicas
>> DataMemory=3500M    # How much memory to allocate for data storage
>> IndexMemory=1000M   # How much memory to allocate for index storage
>> datadir=/usr/local/mysql/data   # Directory for this data node's
>> datafiles
> ..
>> [NDB_MGMD]
>> hostname=192.168.1.107           # Hostname or IP address of MGM node
>
> You need *2* NDB_MGMD sections, because you have 2 management nodes.
>
>>
>> # Options for data node "A":
>> [NDBD]
>> id=3                             # (one [NDBD] section per data node)
>> hostname=192.168.1.107           # Hostname or IP address
>>
>> # Options for data node "B":
>> [NDBD]
>> id=4
>> hostname=192.168.1.125          # Hostname or IP address
>>
>> # SQL node options:
>> [MYSQLD]
>> id=5
>> hostname=192.168.1.107
>
> This not so important to get it running, but you _should_ have a few
> slots for API or MYSQLD open. So add a few (4?) empty [API] slots at
> the end.
>
>> "config.ini" On host: 192.168.1.125:
>> ====================================
>
> If you use MySQL Cluster 7.0, then 192.168.1.125 does not need a
> config.ini. It will fetch the configuration from the 1st ndb_mgmd if
> you give it the connect-string and its node-id.
>
>> To create the replication between two data nodes. Means if one
>> server goes
>> down or crashes then automatically the same data should be served by
>> another one live server.
>
> You do not need to setup replication between data nodes, you already
> did: NoOfReplicas=2. This way if you have 2 data nodes, each will have
> a replica of the other's data.
>
> ..
>> Connected to Management Server at: 192.168.1.125:1186
>> Cluster Configuration
>> ---------------------
>> [ndbd(NDB)]	2 node(s)
>> id=3 (not connected, accepting connect from 192.168.1.107)
>> id=4	@192.168.1.125  (mysql-5.1.34 ndb-7.0.6, Nodegroup: 0, Master)
>>
>> [ndb_mgmd(MGM)]	1 node(s)
>> id=1	@192.168.1.125  (mysql-5.1.34 ndb-7.0.6)
>>
>> [mysqld(API)]	1 node(s)
>> id=5	@192.168.1.107  (mysql-5.1.34 ndb-7.0.6)
>>
>>
>> The above resulted out put shows that the both data nodes from each
>> server
>> are not able to communicate with each other. So please tell me the
>> proper
>> configuration or the proper steps to implement my scenario perfectly.
>
> No, they are not, because you actually created 2 MySQL Cluster's.
>
> Please check again our manual on dev.mysql.com to check how MySQL
> Cluster works.
> Or check my blog post, and leave comments if something wasn't clear!
>  
> http://blog.some-abstract-type.com/2009/08/how-to-start-mysql-cluster-70-with-2.html
>
> Try first to get 2 data nodes up with 1 ndb_mgmd (one [NDB_MGMD]
> section), just to start with.
>
> config.ini
>
> [NDBD DEFAULT]
> NoOfReplicas=2
> DataMemory=3500M
> IndexMemory=1000M
> datadir=/usr/local/mysql/data
> [NDB_MGMD DEFAULT]
> datadir=/var/lib/mysql-cluster
> [NDB_MGMD]
> hostname=192.168.1.107
> id=1
> [NDBD]
> id=3
> hostname=192.168.1.107
> [NDBD]
> id=4
> hostname=192.168.1.125
> [MYSQLD]
> id=5
> hostname=192.168.1.107
> [API]
> [API]
>
> On machine 192.168.1.107 launch
>   shell> ndb_mgmd -f config.ini --configdir /path/to/cache/dir
>   shell> ndbd -c 192.168.1.107
> On machine 192.168.1.125
>   shell> ndbd -c 192.168.1.107
>
> Cheers,
>
> Geert
>
>
> --
> Geert Vanderkelen, MySQL Senior Support Engineer
> Sun Microsystems GmbH     Sonnenallee 1, D-85551 Kirchheim-Heimstetten
>
> Geschaeftsfuehrer: Thomas Schröder, Wolfgang Engels, Wolf Frenkel
> Vorsitz d. Aufs.rat.: Martin Haering                    HRB MUC 161028
>
>


-- 
Manoj M. Burande,
Artificial Machines Pvt Ltd,
System Administrator.


Thread
Re: Unable to configuring multiple Management nodes.Geert Vanderkelen31 Aug
  • Re: Unable to configuring multiple Management nodes.Manoj Burande31 Aug
Re: Unable to configuring multiple Management nodes.Geert Vanderkelen31 Aug
  • Re: Unable to configuring multiple Management nodes.Geert Vanderkelen31 Aug
    • Re: Unable to configuring multiple Management nodes.Johan Andersson31 Aug
      • Re: Unable to configuring multiple Management nodes.Manoj Burande1 Sep
Re: Unable to configuring multiple Management nodes.Geert Vanderkelen1 Sep
  • Re: Unable to configuring multiple Management nodes.Hendrik Woltersdorf1 Sep
  • Re: Unable to configuring multiple Management nodes.Manoj Burande7 Sep
Re: Unable to configuring multiple Management nodes.Geert Vanderkelen7 Sep
  • Re: Unable to configuring multiple Management nodes.Manoj Burande10 Sep