List:Cluster« Previous MessageNext Message »
From:He Xinghua Date:April 23 2012 7:11am
Subject:problem about the cluster with load big file
View as plain text  
hi experts of cluster,
I have a problem about the cluster with load big file.
I use 7 host to setup cluster,  4 data nodes, 2 sql nodes. 1 mgm node.
the data nodes hardware:  12 core CPU,  64G mem, 
the sql nodes hardware:  12 core CPU, 64G mem
the mgm node hard ware:  2 core cpu  8G mem 


I  load 7G file which have 100 million rows, and have 7 columns which type is int or vachar(50)
the table has two index.
I load the file to cluster cost 4 hours,  but  just cost 20 min using MySIAM.
can you give me some advise for solving the problem?
config.ini of mgm node as follows:
[NDBD DEFAULT] 
NoOfReplicas=1 
DataMemory=5000M 
IndexMemory=200M 
DataDir=/app/ndb/data 
BackupDataDir=/app/ndb/data/backup 
LockPagesInMainMemory=1 

TimeBetweenLocalCheckpoints=20 
TimeBetweenGlobalCheckpoints=1000 
TimeBetweenEpochs=100 
TimeBetweenWatchdogCheckInitial=60000 

StringMemory=20 
MaxNoOfTables=1024 
MaxNoOfOrderedIndexes=2048 
MaxNoOfUniqueHashIndexes=512 
MaxNoOfAttributes=20480 
MaxNoOfTriggers=10240 

DiskCheckpointSpeedInRestart=100M 
FragmentLogFileSize=256M 
NoOfFragmentLogFiles=16 
RedoBuffer=64M 
MaxNoOfConcurrentOperations=500000 
TransactionInactiveTimeout=50000 
MaxNoOfExecutionThreads=8 
BatchSizePerLocalScan=512 
SharedGlobalMemory=20M 
DiskPageBufferMemory=80M 

[COMPUTER] 
Id=1 
HostName=ip1 
...........
[COMPUTER] 
Id=5 
HostName=ip5 

[NDBD] 
NodeId=1 
ExecuteOnComputer=1 
..........
[NDBD] 
NodeId=3 
ExecuteOnComputer=3 

[NDBD] 
NodeId=32 
ExecuteOnComputer=4 
.......
### Management node 
[NDB_MGMD] 
Id=33 
ExecuteOnComputer=5 

[MYSQLD] 
[MYSQLD]






Thanks & Best Regards!
Helios
2012/04/23



网易Lofter,专注兴趣,分享创作!
Thread
problem about the cluster with load big fileHe Xinghua23 Apr