From: Luke H. Crouch Date: August 4 2004 3:54pm Subject: RE: memory overhead question List-Archive: http://lists.mysql.com/cluster/267 Message-Id: <075C170BC7219F439DD3FE8C880648C1AB0A76@lakota.red-man.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable 50? or 50,000?! for our DB settings now, we have: DataMemory: 1000M IndexMemory: 350M MaxNoOfConcurrentOperations: 65536 and we have 2G of memory on the machines. now when we run the load of = inserts, it seems to do better, BUT... I think our problem last time may have happened because we only had 80M = allocated to DataMemory, and when it started loading, it just started = loading a bunch into swap, and the swap memory was causing the = problems...does this sound reasonable? it looks like the ndbd is still dipping into swap memory...? not nearly = as much as before (400M now vs. 2G before), but as I understand it, ndbd = using swap memory can cause problems? is there a way to prevent ndbd = from using any swap memory at all? disabling swap? we're planning on getting some more memory (about 4G in each = machine)...will that take care of it, or will swap always be used to = some extent? thanks, -L > -----Original Message----- > From: Mikael Ronstr=F6m [mailto:mikael@stripped] > Sent: Wednesday, August 04, 2004 8:35 AM > To: Crouch, Luke H. > Cc: cluster@stripped > Subject: Re: memory overhead question >=20 >=20 > Hi Luke, > Due to a bug that I am currently fixing you need to have twice the=20 > number of operation records as the maximum records involved in a=20 > transaction. > I am not exactly sure of how big your insert transactions are. I have=20 > seen 50.000 records per transaction in another mysqldump usage which=20 > would > then require around 100.000. From what I have seen, each insert is=20 > about 1 MByte in size when using extended_insert, dependent on your=20 > record > size this should at least not be bigger than 50.000. >=20 > Rgrds Mikael >=20 > 2004-08-04 kl. 15.16 skrev Crouch, Luke H.: >=20 > > thanks Mikael...that's very helpful. we had a very high number for=20 > > MaxNoOfConcurrentOperations, which was probably another thing using=20 > > lots of our memory. what would be the suggested number of=20 > concurrent=20 > > operations? right now, we're just trying to load our large table (4=20 > > million records, dumped with --extended-inserts)... > > > > thanks again, > > -L > > > >> -----Original Message----- > >> From: Mikael Ronstr=F6m [mailto:mikael@stripped] > >> Sent: Wednesday, August 04, 2004 5:39 AM > >> To: Crouch, Luke H. > >> Cc: cluster@stripped > >> Subject: Re: memory overhead question > >> > >> > >> Hi Luke, > >> Here are some experiments I performed on my machine. > >> > >> PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE > >> VSIZE > >> 465 ndbd 0.0% 0:02.35 22 99 155 230M > >> 3.25M 232M > >> 301M > >> 464 ndbd 0.0% 0:00.00 1 9 27 720K > >> 3.24M 188K > >> 48.7M > >> 462 ndb_mgmd 0.0% 0:00.19 11 46 49 6.10M > >> 1.30M 1.20M > >> 41.0M > >> > >> This is a printout from a quick-start of a 1-node MySQL=20 > Cluster on my > >> PowerBook running > >> Mac OS X 10.3. So about 300M of virtual address space is used for a > >> standard configured > >> ndbd process. > >> > >> PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE > >> VSIZE > >> 486 ndbd 0.0% 0:03.11 22 99 155 367M > >> 3.25M 368M > >> 437M > >> > >> This printout was achieved when IndexMemory was set to 50M and > >> DataMemory to 190M. > >> Memory increased as much as IndexMemory and DataMemory increased. > >> > >> PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE > >> VSIZE > >> 496 ndbd 0.8% 0:03.15 20 95 149 405M+ > >> 3.25M 406M+ > >> 476M > >> > >> And changing MaxNoOfConcurrentOperations to 65536 adds=20 > another 39M of > >> memory. > >> > >> Optimisations of MySQL Cluster both in terms of memory usage and > >> improved performance > >> of mysql clients using the storage engine is on the TODO > >> list. At first > >> however we focus on > >> ensuring that MySQL Cluster is fully integrated with MySQL. > >> > >> Rgrds Mikael > >> > >> > >> 2004-08-03 kl. 17.23 skrev Crouch, Luke H.: > >> > >>> when we start our cluster up with default settings (DataMemory: > >>> 80000k, IndexMemory: 24000k), and check the memory usage on our > >>> different nodes, it shows ndbd size as 2395M! and the > >> machine is using > >>> 2G memory, and 1G of swap! > >>> > >>> are these numbers overhead with only 80M of memory > >> allocated to data? > >>> how much memory is required other than the memory=20 > dedicated to data > >>> via DataMemory setting? > >>> > >>> thanks, > >>> -L > >>> > >>> --=20 > >>> MySQL Cluster Mailing List > >>> For list archives: http://lists.mysql.com/cluster > >>> To unsubscribe: > >>> http://lists.mysql.com/cluster?unsub=3Dmikael@stripped > >>> > >>> > >>> > >> Mikael Ronstr=F6m, Senior Software Architect > >> MySQL AB, www.mysql.com > >> > >> Clustering: > >> http://www.infoworld.com/article/04/04/14/HNmysqlcluster_1.html > >> > >> http://www.eweek.com/article2/0,1759,1567546,00.asp > >> > >> > >> > > > > > Mikael Ronstr=F6m, Senior Software Architect > MySQL AB, www.mysql.com >=20 > Clustering: > http://www.infoworld.com/article/04/04/14/HNmysqlcluster_1.html >=20 > http://www.eweek.com/article2/0,1759,1567546,00.asp >=20 >=20 >=20