50? or 50,000?!
for our DB settings now, we have:
DataMemory: 1000M
IndexMemory: 350M
MaxNoOfConcurrentOperations: 65536
and we have 2G of memory on the machines. now when we run the load of inserts, it seems to
do better, BUT...
I think our problem last time may have happened because we only had 80M allocated to
DataMemory, and when it started loading, it just started loading a bunch into swap, and
the swap memory was causing the problems...does this sound reasonable?
it looks like the ndbd is still dipping into swap memory...? not nearly as much as before
(400M now vs. 2G before), but as I understand it, ndbd using swap memory can cause
problems? is there a way to prevent ndbd from using any swap memory at all? disabling
swap?
we're planning on getting some more memory (about 4G in each machine)...will that take
care of it, or will swap always be used to some extent?
thanks,
-L
> -----Original Message-----
> From: Mikael Ronström [mailto:mikael@stripped]
> Sent: Wednesday, August 04, 2004 8:35 AM
> To: Crouch, Luke H.
> Cc: cluster@stripped
> Subject: Re: memory overhead question
>
>
> Hi Luke,
> Due to a bug that I am currently fixing you need to have twice the
> number of operation records as the maximum records involved in a
> transaction.
> I am not exactly sure of how big your insert transactions are. I have
> seen 50.000 records per transaction in another mysqldump usage which
> would
> then require around 100.000. From what I have seen, each insert is
> about 1 MByte in size when using extended_insert, dependent on your
> record
> size this should at least not be bigger than 50.000.
>
> Rgrds Mikael
>
> 2004-08-04 kl. 15.16 skrev Crouch, Luke H.:
>
> > thanks Mikael...that's very helpful. we had a very high number for
> > MaxNoOfConcurrentOperations, which was probably another thing using
> > lots of our memory. what would be the suggested number of
> concurrent
> > operations? right now, we're just trying to load our large table (4
> > million records, dumped with --extended-inserts)...
> >
> > thanks again,
> > -L
> >
> >> -----Original Message-----
> >> From: Mikael Ronström [mailto:mikael@stripped]
> >> Sent: Wednesday, August 04, 2004 5:39 AM
> >> To: Crouch, Luke H.
> >> Cc: cluster@stripped
> >> Subject: Re: memory overhead question
> >>
> >>
> >> Hi Luke,
> >> Here are some experiments I performed on my machine.
> >>
> >> PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE
> >> VSIZE
> >> 465 ndbd 0.0% 0:02.35 22 99 155 230M
> >> 3.25M 232M
> >> 301M
> >> 464 ndbd 0.0% 0:00.00 1 9 27 720K
> >> 3.24M 188K
> >> 48.7M
> >> 462 ndb_mgmd 0.0% 0:00.19 11 46 49 6.10M
> >> 1.30M 1.20M
> >> 41.0M
> >>
> >> This is a printout from a quick-start of a 1-node MySQL
> Cluster on my
> >> PowerBook running
> >> Mac OS X 10.3. So about 300M of virtual address space is used for a
> >> standard configured
> >> ndbd process.
> >>
> >> PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE
> >> VSIZE
> >> 486 ndbd 0.0% 0:03.11 22 99 155 367M
> >> 3.25M 368M
> >> 437M
> >>
> >> This printout was achieved when IndexMemory was set to 50M and
> >> DataMemory to 190M.
> >> Memory increased as much as IndexMemory and DataMemory increased.
> >>
> >> PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE
> >> VSIZE
> >> 496 ndbd 0.8% 0:03.15 20 95 149 405M+
> >> 3.25M 406M+
> >> 476M
> >>
> >> And changing MaxNoOfConcurrentOperations to 65536 adds
> another 39M of
> >> memory.
> >>
> >> Optimisations of MySQL Cluster both in terms of memory usage and
> >> improved performance
> >> of mysql clients using the storage engine is on the TODO
> >> list. At first
> >> however we focus on
> >> ensuring that MySQL Cluster is fully integrated with MySQL.
> >>
> >> Rgrds Mikael
> >>
> >>
> >> 2004-08-03 kl. 17.23 skrev Crouch, Luke H.:
> >>
> >>> when we start our cluster up with default settings (DataMemory:
> >>> 80000k, IndexMemory: 24000k), and check the memory usage on our
> >>> different nodes, it shows ndbd size as 2395M! and the
> >> machine is using
> >>> 2G memory, and 1G of swap!
> >>>
> >>> are these numbers overhead with only 80M of memory
> >> allocated to data?
> >>> how much memory is required other than the memory
> dedicated to data
> >>> via DataMemory setting?
> >>>
> >>> thanks,
> >>> -L
> >>>
> >>> --
> >>> MySQL Cluster Mailing List
> >>> For list archives: http://lists.mysql.com/cluster
> >>> To unsubscribe:
> >>> http://lists.mysql.com/cluster?unsub=1
> >>>
> >>>
> >>>
> >> Mikael Ronström, Senior Software Architect
> >> MySQL AB, www.mysql.com
> >>
> >> Clustering:
> >> http://www.infoworld.com/article/04/04/14/HNmysqlcluster_1.html
> >>
> >> http://www.eweek.com/article2/0,1759,1567546,00.asp
> >>
> >>
> >>
> >
> >
> Mikael Ronström, Senior Software Architect
> MySQL AB, www.mysql.com
>
> Clustering:
> http://www.infoworld.com/article/04/04/14/HNmysqlcluster_1.html
>
> http://www.eweek.com/article2/0,1759,1567546,00.asp
>
>
>
