List:Cluster« Previous MessageNext Message »
From:Ph.D. Joseph E. Sacco Date:August 3 2004 1:14pm
Subject:Re: Building a cluster with a slow and a fast machine...
View as plain text  
Magnus,

Thanks for taking the time to reply.

The four node configuration with this hardware is just an experiment.  I
wanted to explore the consequences of adding a slow machine with minimal
RAM to a cluster. The following are DB parameter settings used in the
experiment:

        [DB DEFAULT]
        NoOfReplicas: 2
        DataMemory: 40M
        IndexMemory: 12M
        HeartbeatIntervalDbDb: 5000
        TimeBetweenWatchDogCheck: 30000


You are correct about the paging. The lack of RAM on the slow machine
resulted in swapd using about 240MB of VM.  The paging has consequences
since it makes the slow machine even slower. In was necessary to
increase both HeartbeatIntervalDbDb and TimeBetweenWatchDogCheck just to
get the cluster to come up.  Once up,  the cluster is stable.


Lesson learned:
	
	 MySQL cluster requires fast systems with several GB's of RAM.


-Joseph

=======================================================================
On Tue, 2004-08-03 at 04:40, Magnus Svensson wrote:
> On Thu, 2004-07-29 at 17:34, Joseph E. Sacco, Ph.D. wrote:
> > Systems:
> > * PowerMac with dual   G4 533MHz CPU's, 1GB RAM, SCSI drives
> > * Powermac with single G3 240MHz CPU, 264MB RAM, SCSI drives
> > * Yellow Dog Linux-3.0.1 [Redhat clone]
> > * mysql-4.1 from BK tree 28July04
> > 
> > ====================================================================
> > Question:
> > 
> > What can be done to improve cluster startup when clustering two machines
> > with drastically different resource/performance characteristics?
> 
> 
> > 
> > ====================================================================
> > 
> > I am running a two replica, four node cluster on two PowerMacs. One
> > machine is considerably faster than the other:
> > 
> > [Slow G3 machine]
> > # cat /proc/cpuinfo
> > processor       : 0
> > cpu             : 740/750
> > revision        : 2.2 (pvr 0008 0202)
> > bogomips        : 478.41
> > machine         : PowerMac,NuBus
> > motherboard     : PDM MacRISC
> > detected as     : 0 (<NULL>)
> > pmac flags      : 00000000
> > memory          : 264MB
> 
> Hi,
> 
> I believe memory is your main problem here. Running a four node cluster,
> where 2 ndb nodes is running with only 264MB is just too little. I guess
> the problems you see are related to swapping?
> 
> If you want to continúe running with this setup, I would recomend using
> a two node cluster, and thus only one node would have to run on each
> machine, or why not run all nodes on the fast machine? You can try out
> the HA features by stopping and starting the individual processes just
> as you would if they were on two different machines.
> 
> Also you will have to decrease memory consumtion of the nodes by
> decreasing the settings of DataMemory, IndexMemory and
> MacNoOfConcurrentOperations in config.ini
> 
> Also keep in mind that the load will be evenly spread on all machines in
> the cluster, so the slow machine will impact performance on the whole
> cluster.
> 
> 
> Best regards
> Magnus
> 
> > pmac-generation : NuBus
> > 
> > 
> > [Fast G4 machine]
> > # cat /proc/cpuinfo
> > processor       : 0
> > cpu             : 7410, altivec supported
> > temperature     : 58-60 C (uncalibrated)
> > clock           : 533MHz
> > revision        : 17.3 (pvr 800c 1103)
> > bogomips        : 1064.96
> >  
> > processor       : 1
> > cpu             : 7410, altivec supported
> > temperature     : 34-36 C (uncalibrated)
> > clock           : 533MHz
> > revision        : 17.3 (pvr 800c 1103)
> > bogomips        : 1064.96
> >  
> > total bogomips  : 2129.92
> > machine         : PowerMac3,4
> > motherboard     : PowerMac3,4 MacRISC2 MacRISC Power Macintosh
> > board revision  : 00000000
> > detected as     : 69 (PowerMac G4 Silver)
> > pmac flags      : 00000000
> > L2 cache        : 1024K unified
> > memory          : 1024MB
> > pmac-generation : NewWorld
> > 
> > It is often difficult to get the DB nodes on the slower machine to
> > start. It typically takes several attempts to get the two nodes on the
> > slow machine up and running. Once the cluster is established it is
> > stable.
> > 
> > One of the things I have noticed is the DB node startup process on the
> > slow machine temporarily spawns a *large* number of ndbd processes.
> > During the startup it is not uncommon to see 20 -> 30 ndbd processes
> > started, which effectively overload the slow machine. Once all the nodes
> > in the cluster are up and running the number of ndbd processes drops to
> > two per DB node.  
> > 
> > What's going on??? Are things timing out so the "Angel" processes
> > continually fire up other ndbd processes?  If so, what can be done?
> > 
> > -Joseph
> > 
> > 
> > -- 
> > Joseph E. Sacco, Ph.D. <jsacco[at]earthlink[dot]net>
-- 
Joseph E. Sacco, Ph.D. <jsacco[at]earthlink[dot]net>

Thread
Building a cluster with a slow and a fast machine...Ph.D. Joseph E. Sacco29 Jul
  • Re: Building a cluster with a slow and a fast machine...Magnus Svensson3 Aug
    • Re: Building a cluster with a slow and a fast machine...Ph.D. Joseph E. Sacco3 Aug