List:Cluster« Previous MessageNext Message »
From:Subodh Nair Date:June 24 2011 2:54pm
Subject:RE: MySQL cluster- migrating from WIndows to Linux
View as plain text  
Hello Geert,


I am tying to upgrade my mysql cluster to ver mysql-5.1.56 ndb-7.1.13. I
have tried to install the windows as well as the Linux ver. In both the
cases I have problem bringing up both my data nodes together. In windows,
when I start the ndb (data node) I get an error where the memory usage by
the ndb process goes up very high and the server hangs saying that not
enough virtual memory available.
Both my data nodes have 8 GB of RAM and 8 GB of virtual memory allocated.

I have a five node setup,

1 mgmt node,
2 data nodes and 2 SQL nodes.

I tried the same version of mysql mysql-5.1.56 ndb-7.1.13 on Redhat 5.1 OS.
Here also I got the same errors.

The config.ini is this :

[ndbd default]
noofreplicas=2
DataMemory=4096M
IndexMemory=512M
#MaxNoOfConcurrentOperations=5M
#MaxNoOfLocalOperations=9M
#MaxNoOfAttributes=10000
#MaxNoOfOrderedIndexes=256
TotalSendBufferMemory=16M
#HeartbeatIntervalDbDb=60000
#HeartbeatIntervalDbApi=60000
diskless=1
#NoOfFragmentLogFiles=32
Stoponerror=False
TimeBetweenGlobalCheckPoints=3000
TimeBetweenLocalCheckPoints=5
#RedoBuffer=64M
#DiskSyncSize=30M

[ndb_mgmd]
hostname=10.90.90.35
DataDir=/var/lib/mysql-cluster/data_ars

[ndbd]
hostname=10.90.90.28
DataDir=/var/lib/mysql-cluster/data_ars
MaxNoOfLocalOperations=6M

[ndbd]
hostname=10.90.90.29
DataDir=/var/lib/mysql-cluster/data_ars
MaxNoOfLocalOperations=6M

[mysqld]
hostname=10.90.90.33

[mysqld]
hostname=10.90.90.38

[mysqld]
hostname=10.90.90.39



The error log on the data node is attached. Could you please let me know if
this a bug with this version? I have also tried the previous 2 versions
mysql-5.1.56 ndb-7.1.8 and mysql-5.1.56 ndb-7.1.9a. I had the same problems
in both these versions when I installed them in Linux.





Best Regards,
 
Subodh
 
ARS | Software Engineering Pvt Ltd – Subodh Nair, Manager, TechSupport
131-135, Nila, Technopark, Trivandrum-695581; Tel: +91 0471 2527155, Ext
112; TechSupport: 9995007988 
Web: www.arssoftware.com

-----Original Message-----
From: Geert.Vanderkelen@stripped [mailto:Geert.Vanderkelen@stripped] 
Sent: Wednesday, June 16, 2010 7:07 PM
To: Subodh Nair
Cc: Matt.Montgomery@stripped; cluster@stripped
Subject: Re: MySQL cluster- migrating from WIndows to Linux


On Jun 16, 2010, at 14:54 , Subodh Nair wrote:
..
> Sure it is Windows platform. But the handle leak is caused due to the
mysqld
> process. Is this version of mysql cluster known to have this bug.

No idea, I never use the MS Windows-thing :)

> As can be seen from the log I had included earlier, the handle count
> increases after every write or read operation.

Chatting with developers, it would be good if you could open a bug report
for this using:
  http://bugs.mysql.com

Cheers,

Geert

-- 
Geert Vanderkelen, MySQL Senior Support Engineer
Sun Microsystems GmbH     Sonnenallee 1, D-85551 Kirchheim-Heimstetten

Geschaeftsfuehrer: Jürgen Kunz                          HRB MUC 161028







Current byte-offset of file-pointer is: 1566                      


Time: Thursday 23 June 2011 - 18:47:30
Status: Temporary error, restart node
Message: WatchDog terminate, internal error or massive overload on the machine running
this node (Internal error, programming error or missing error message, please report a
bug)
Error: 6050
Error data: Allocating memory
Error object: WatchDog.cpp
Program: ndbd
Pid: 7184
Version: mysql-5.1.56 ndb-7.1.13
Trace: /var/lib/mysql-cluster/data_ars/ndb_2_trace.log.1
***EOM***
                                                 
Time: Thursday 23 June 2011 - 18:48:15
Status: Temporary error, restart node
Message: WatchDog terminate, internal error or massive overload on the machine running
this node (Internal error, programming error or missing error message, please report a
bug)
Error: 6050
Error data: Allocating memory
Error object: WatchDog.cpp
Program: ndbd
Pid: 7192
Version: mysql-5.1.56 ndb-7.1.13
Trace: /var/lib/mysql-cluster/data_ars/ndb_2_trace.log.2
***EOM***
                                                 
Time: Thursday 23 June 2011 - 18:49:03
Status: Temporary error, restart node
Message: WatchDog terminate, internal error or massive overload on the machine running
this node (Internal error, programming error or missing error message, please report a
bug)
Error: 6050
Error data: Allocating memory
Error object: WatchDog.cpp
Program: ndbd
Pid: 7198
Version: mysql-5.1.56 ndb-7.1.13
Trace: /var/lib/mysql-cluster/data_ars/ndb_2_trace.log.3
***EOM***
                                                 

2011-06-23 18:46:59 [ndbd] INFO     -- Angel pid: 7183 started child: 7184
2011-06-23 18:47:00 [ndbd] INFO     -- Configuration fetched from '10.90.90.35:1186',
generation: 1
NDBMT: non-mt
2011-06-23 18:47:00 [ndbd] INFO     -- NDB Cluster -- DB node 2
2011-06-23 18:47:00 [ndbd] INFO     -- mysql-5.1.56 ndb-7.1.13 --
2011-06-23 18:47:00 [ndbd] INFO     -- numa_set_interleave_mask(numa_all_nodes) : OK
2011-06-23 18:47:00 [ndbd] INFO     -- Ndbd_mem_manager::init(1) min: 4714Mb initial:
4734Mb
Adding 36Mb to ZONE_LO (1,1151)
Instantiating DBSPJ instanceNo=0
2011-06-23 18:47:01 [ndbd] INFO     -- Start initiated (mysql-5.1.56 ndb-7.1.13)
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Adding 4699Mb to ZONE_LO (1152,150344)
WOPool::init(61, 9)
RWPool::init(22, 14)
2011-06-23 18:47:05 [ndbd] INFO     -- timerHandlingLab now: 98650867 sent: 98650398
diff: 469
2011-06-23 18:47:05 [ndbd] INFO     -- timerHandlingLab now: 98651555 sent: 98650867
diff: 688
2011-06-23 18:47:06 [ndbd] INFO     -- timerHandlingLab now: 98651859 sent: 98651555
diff: 304
2011-06-23 18:47:06 [ndbd] INFO     -- Watchdog: User time: 94  System time: 403
2011-06-23 18:47:06 [ndbd] WARNING  -- Watchdog: Warning overslept 299 ms, expected 100
ms.
2011-06-23 18:47:07 [ndbd] INFO     -- Watchdog: User time: 98  System time: 500
2011-06-23 18:47:07 [ndbd] WARNING  -- Watchdog: Warning overslept 206 ms, expected 100
ms.
2011-06-23 18:47:12 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Allocating memory
elapsed=6042
2011-06-23 18:47:12 [ndbd] INFO     -- Watchdog: User time: 102  System time: 568
2011-06-23 18:47:18 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Allocating memory
elapsed=12054
2011-06-23 18:47:18 [ndbd] INFO     -- Watchdog: User time: 102  System time: 572
2011-06-23 18:47:24 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Allocating memory
elapsed=18070
2011-06-23 18:47:24 [ndbd] INFO     -- Watchdog: User time: 102  System time: 576
2011-06-23 18:47:37 [ndbd] INFO     -- Angel detected startup failure, count: 1
2011-06-23 18:47:37 [ndbd] ALERT    -- Node 2: Forced node shutdown completed,
restarting. Occured during startphase 0. Initiated by signal 11.
2011-06-23 18:47:38 [ndbd] INFO     -- Ndb has terminated (pid 7184) restarting
2011-06-23 18:47:38 [ndbd] INFO     -- Angel reconnected to '10.90.90.35:1186'
2011-06-23 18:47:39 [ndbd] INFO     -- Angel reallocated nodeid: 2
2011-06-23 18:47:39 [ndbd] INFO     -- Angel pid: 7183 started child: 7192
2011-06-23 18:47:41 [ndbd] INFO     -- Configuration fetched from '10.90.90.35:1186',
generation: 1
NDBMT: non-mt
2011-06-23 18:47:41 [ndbd] INFO     -- NDB Cluster -- DB node 2
2011-06-23 18:47:41 [ndbd] INFO     -- mysql-5.1.56 ndb-7.1.13 --
2011-06-23 18:47:42 [ndbd] INFO     -- numa_set_interleave_mask(numa_all_nodes) : OK
2011-06-23 18:47:42 [ndbd] INFO     -- Ndbd_mem_manager::init(1) min: 4714Mb initial:
4734Mb
Adding 36Mb to ZONE_LO (1,1151)
Instantiating DBSPJ instanceNo=0
2011-06-23 18:47:47 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Job Handling
elapsed=99
2011-06-23 18:47:47 [ndbd] INFO     -- Watchdog: User time: 2  System time: 36
2011-06-23 18:47:47 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Job Handling
elapsed=200
2011-06-23 18:47:47 [ndbd] INFO     -- Watchdog: User time: 2  System time: 36
2011-06-23 18:47:48 [ndbd] INFO     -- timerHandlingLab now: 98693657 sent: 98693342
diff: 315
2011-06-23 18:47:48 [ndbd] INFO     -- Start initiated (mysql-5.1.56 ndb-7.1.13)
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
2011-06-23 18:47:48 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Polling for
Receive elapsed=100
2011-06-23 18:47:48 [ndbd] INFO     -- Watchdog: User time: 7  System time: 78
2011-06-23 18:47:48 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Polling for
Receive elapsed=199
2011-06-23 18:47:48 [ndbd] INFO     -- Watchdog: User time: 8  System time: 91
2011-06-23 18:47:48 [ndbd] INFO     -- timerHandlingLab now: 98694181 sent: 98693658
diff: 523
2011-06-23 18:47:49 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Polling for
Receive elapsed=100
2011-06-23 18:47:49 [ndbd] INFO     -- Watchdog: User time: 13  System time: 145
2011-06-23 18:47:49 [ndbd] INFO     -- timerHandlingLab now: 98694875 sent: 98694641
diff: 234
2011-06-23 18:47:50 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Job Handling
elapsed=100
2011-06-23 18:47:50 [ndbd] INFO     -- Watchdog: User time: 24  System time: 285
2011-06-23 18:47:50 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Job Handling
elapsed=200
2011-06-23 18:47:50 [ndbd] INFO     -- Watchdog: User time: 24  System time: 294
2011-06-23 18:47:50 [ndbd] INFO     -- timerHandlingLab now: 98696499 sent: 98696184
diff: 315
Adding 4699Mb to ZONE_LO (1152,150344)
2011-06-23 18:47:51 [ndbd] INFO     -- Watchdog: User time: 30  System time: 370
2011-06-23 18:47:51 [ndbd] WARNING  -- Watchdog: Warning overslept 216 ms, expected 100
ms.
WOPool::init(61, 9)
RWPool::init(22, 14)
2011-06-23 18:47:52 [ndbd] WARNING  -- timerHandlingLab now: 98698109 sent: 98697037
diff: 1072
2011-06-23 18:47:52 [ndbd] INFO     -- timerHandlingLab now: 98698391 sent: 98698109
diff: 282
2011-06-23 18:47:56 [ndbd] INFO     -- Watchdog: User time: 108  System time: 607
2011-06-23 18:47:56 [ndbd] WARNING  -- Watchdog: Warning overslept 211 ms, expected 100
ms.
2011-06-23 18:47:58 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Allocating memory
elapsed=6059
2011-06-23 18:47:58 [ndbd] INFO     -- Watchdog: User time: 109  System time: 641
2011-06-23 18:48:04 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Allocating memory
elapsed=12077
2011-06-23 18:48:04 [ndbd] INFO     -- Watchdog: User time: 110  System time: 647
2011-06-23 18:48:10 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Allocating memory
elapsed=18081
2011-06-23 18:48:10 [ndbd] INFO     -- Watchdog: User time: 110  System time: 659
2011-06-23 18:48:20 [ndbd] INFO     -- Angel detected startup failure, count: 2
2011-06-23 18:48:20 [ndbd] ALERT    -- Node 2: Forced node shutdown completed,
restarting. Occured during startphase 0. Initiated by signal 11.
2011-06-23 18:48:20 [ndbd] INFO     -- Ndb has terminated (pid 7192) restarting
2011-06-23 18:48:20 [ndbd] INFO     -- Angel reconnected to '10.90.90.35:1186'
2011-06-23 18:48:20 [ndbd] INFO     -- Angel reallocated nodeid: 2
2011-06-23 18:48:21 [ndbd] INFO     -- Angel pid: 7183 started child: 7198
2011-06-23 18:48:21 [ndbd] INFO     -- Configuration fetched from '10.90.90.35:1186',
generation: 1
NDBMT: non-mt
2011-06-23 18:48:21 [ndbd] INFO     -- NDB Cluster -- DB node 2
2011-06-23 18:48:21 [ndbd] INFO     -- mysql-5.1.56 ndb-7.1.13 --
2011-06-23 18:48:24 [ndbd] INFO     -- numa_set_interleave_mask(numa_all_nodes) : OK
2011-06-23 18:48:24 [ndbd] INFO     -- Ndbd_mem_manager::init(1) min: 4714Mb initial:
4734Mb
Adding 36Mb to ZONE_LO (1,1151)
Instantiating DBSPJ instanceNo=0
2011-06-23 18:48:25 [ndbd] INFO     -- Start initiated (mysql-5.1.56 ndb-7.1.13)
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
Adding 4699Mb to ZONE_LO (1152,150344)
2011-06-23 18:48:29 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Job Handling
elapsed=100
2011-06-23 18:48:29 [ndbd] INFO     -- Watchdog: User time: 24  System time: 316
2011-06-23 18:48:29 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Job Handling
elapsed=200
2011-06-23 18:48:29 [ndbd] INFO     -- Watchdog: User time: 24  System time: 316
2011-06-23 18:48:29 [ndbd] INFO     -- timerHandlingLab now: 98734867 sent: 98734561
diff: 306
WOPool::init(61, 9)
RWPool::init(22, 14)
2011-06-23 18:48:29 [ndbd] INFO     -- timerHandlingLab now: 98735389 sent: 98734868
diff: 521
2011-06-23 18:48:30 [ndbd] INFO     -- timerHandlingLab now: 98736103 sent: 98735389
diff: 714
2011-06-23 18:48:30 [ndbd] WARNING  -- Time moved forward with 1713 ms
2011-06-23 18:48:30 [ndbd] INFO     -- timerHandlingLab now: 98736395 sent: 98736103
diff: 292
2011-06-23 18:48:42 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Allocating memory
elapsed=6005
2011-06-23 18:48:58 [ndbd] INFO     -- Watchdog: User time: 205  System time: 652
2011-06-23 18:48:58 [ndbd] INFO     -- Watchdog: User time: 205  System time: 656
2011-06-23 18:48:59 [ndbd] WARNING  -- Watchdog: Warning overslept 17875 ms, expected 100
ms.
2011-06-23 18:48:59 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Allocating memory
elapsed=23880
2011-06-23 18:48:59 [ndbd] INFO     -- Watchdog: User time: 205  System time: 656
2011-06-23 18:49:05 [ndbd] ALERT    -- Angel detected too many startup failures(3), not
restarting again
2011-06-23 18:49:06 [ndbd] ALERT    -- Node 2: Forced node shutdown completed. Occured
during startphase 0. Initiated by signal 11.

Thread
[Fwd: RE: MySQL cluster- migrating from WIndows to Linux]Matthew Montgomery25 Mar
  • RE: MySQL cluster- migrating from WIndows to LinuxMatthew Montgomery25 Mar
    • RE: MySQL cluster- migrating from WIndows to LinuxSubodh Nair16 Jun
      • Re: MySQL cluster- migrating from WIndows to LinuxGeert Vanderkelen16 Jun
        • RE: MySQL cluster- migrating from WIndows to LinuxSubodh Nair16 Jun
          • Re: MySQL cluster- migrating from WIndows to LinuxGeert Vanderkelen16 Jun
            • RE: MySQL cluster- migrating from WIndows to LinuxSubodh Nair16 Jun
              • Re: MySQL cluster- migrating from WIndows to LinuxGeert Vanderkelen16 Jun
                • RE: MySQL cluster- migrating from WIndows to LinuxSubodh Nair24 Jun
                  • Re: MySQL cluster- migrating from WIndows to LinuxGeert Vanderkelen24 Jun
                    • RE: MySQL cluster- migrating from WIndows to LinuxSubodh Nair25 Jun
  • RE: MySQL cluster- migrating from WIndows to LinuxSubodh Nair29 Mar
    • RE: MySQL cluster- migrating from WIndows to LinuxStewart Smith29 Mar
      • RE: MySQL cluster- migrating from WIndows to LinuxSubodh Nair29 Mar
        • RE: MySQL cluster- migrating from WIndows to LinuxAndrew Morgan29 Mar
          • RE: MySQL cluster- migrating from WIndows to LinuxSubodh Nair31 Mar
            • Re: MySQL cluster- migrating from WIndows to LinuxGeert Vanderkelen31 Mar
              • RE: MySQL cluster- migrating from WIndows to LinuxSubodh Nair31 Mar