List:Cluster« Previous MessageNext Message »
From:Luke H. Crouch Date:July 30 2004 3:36pm
Subject:RE: table full error?
View as plain text  
the rest of the memory usage is like so...

Swap: 2048276k av,    3712k used, 2044564k free                   11280k cached

so, there is 2 GB free swap space memory. does ndbd assume memory is continuous? we are
using Red Hat 9 Enterprise with default memory settings. how can we check or change the
memory fragmentation?

thanks,
-L

> -----Original Message-----
> From: Mikael Ronström [mailto:mikael@stripped]
> Sent: Friday, July 30, 2004 10:24 AM
> To: Crouch, Luke H.
> Cc: cluster@stripped
> Subject: Re: table full error?
> 
> 
> Hi Luke,
> No, I am pretty clueless on this one. Only possibility I can think of 
> is that the OS has fragmented its memory.
> Second possibility is lack of swap memory?? It doesn't sound as a 
> problem with MySQL Cluster.
> 
> Rgrds Mikael
> 
> 2004-07-30 kl. 16.25 skrev Crouch, Luke H.:
> 
> > I don't get the Ndb kernel stuck message, but I do still 
> get an error 
> > trying to start up...
> >
> > 2004-07-30 04:19:52 [NDB] INFO     -- Angel pid: 2097 ndb pid: 2098
> > 2004-07-30 04:19:52 [NDB] INFO     -- NDB Cluster -- DB node 5
> > 2004-07-30 04:19:52 [NDB] INFO     -- Version 3.5.0 (beta) --
> > 2004-07-30 04:20:01 [NDB] INFO     -- Start initiated 
> (version 3.5.0)
> > Error handler shutting down system
> > Error handler shutdown completed - exiting
> >
> > error.log:
> >
> > Date/Time: Friday 30 July 2004 - 04:20:16
> > Type of error: error
> > Message: Memory allocation failure
> > Fault ID: 2327
> > Problem data: DBTUP could not allocate memory for Page
> > Object of reference: Requested: 32768x16000 = 524288000 bytes
> > ProgramName: NDB Kernel
> > ProcessID: 2098
> > TraceFile: NDB_TraceFile_13.trace
> >
> > that's after I changed the DataMemory to be 500M. and when 
> I check the 
> > memory on the boxes, I see...
> >
> > Mem:  2061516k av,   55052k used, 2006464k free,       0k shrd,    
> > 7324k buff
> >                      21728k actv,    1936k in_d,       8k in_c
> >
> > that's 2 GB free memory! and it's having problems 
> allocating just 500M 
> > ?
> >
> > is there a portion of the trace file you'd like to see for 
> this, or do 
> > you have an idea of what is happening now?
> >
> > thanks again for all the help,
> > -L
> >
> >> -----Original Message-----
> >> From: Mikael Ronström [mailto:mikael@stripped]
> >> Sent: Friday, July 30, 2004 9:09 AM
> >> To: Crouch, Luke H.
> >> Cc: cluster@stripped
> >> Subject: Re: table full error?
> >>
> >>
> >> Hi Luke,
> >>
> >> There is an outstanding bug report I filed on this problem. The
> >> workaround is to
> >> increase Watch Dog Timeout.
> >> TimeBetweenWatchDogCheck: 30000
> >> as an example.
> >>
> >> 2004-07-30 kl. 15.41 skrev Crouch, Luke H.:
> >>
> >>> since we have 4 machines with 2GB memory in each, I went ahead and
> >>> set...
> >>>
> >>> DataMemory: 1600M
> >>> IndexMemory: 48M
> >>>
> >>
> >> Would recommend more balance between DataMemory and 
> IndexMemory, also
> >> providing some
> >> memory for other processes running on the machine (if I remember
> >> correctly you also run a
> >> mysqld on the machine.
> >>
> >> So something like:
> >> DataMemory: 800M
> >> IndexMemory: 250M
> >>
> >>> in the DB DEFAULT section. but when I try to start up 
> now, I get...
> >>>
> >>>
> >>> [root@rh-mysql-1 2-db2]# ndbd -i
> >>> 2004-07-30 03:21:33 [NDB] INFO     -- Angel pid: 2904 ndb 
> pid: 2906
> >>> 2004-07-30 03:21:33 [NDB] INFO     -- NDB Cluster -- DB node 2
> >>> 2004-07-30 03:21:33 [NDB] INFO     -- Version 3.5.0 (beta) --
> >>> 2004-07-30 03:21:33 [NDB] INFO     -- Start initiated
> >> (version 3.5.0)
> >>> Ndb kernel is stuck in: Job Handling
> >>> Error handler shutting down system
> >>> Error handler shutdown completed - exiting
> >>>
> >>> another question regarding the memory requirement here...if
> >> we have 4
> >>> nodes, and 2 replicas, and only 1 node on each machine, then each
> >>> machine will only need .5*DBSIZE of memory, correct? and for this
> >>> particular table, if it is, say, 400MB, then each machine would
> >>> actually only be using around 200MB for this particular
> >> table, right?
> >>>
> >>
> >> Right.
> >>
> >> Rgrds Mikael
> >>
> >>> thanks,
> >>> -L
> >>>
> >>>> -----Original Message-----
> >>>> From: Mikael Ronström [mailto:mikael@stripped]
> >>>> Sent: Thursday, July 29, 2004 5:29 PM
> >>>> To: Crouch, Luke H.
> >>>> Cc: cluster@stripped
> >>>> Subject: Re: table full error?
> >>>>
> >>>>
> >>>> Hi Luke,
> >>>>
> >>>> 2004-07-30 kl. 00.07 skrev Crouch, Luke H.:
> >>>>
> >>>>> Mikael,
> >>>>>
> >>>>> the table has 4 million records, each about 100 bytes, and
> >>>> no primary
> >>>>> key (doh!)
> >>>>>
> >>>>
> >>>> If no primary key exists then an 8 byte primary key will be
> >>>> generated.
> >>>> 100 bytes and default DataMemory setting should not
> >>>> provide space for more than around 800k records so 996k
> >>>> records is even
> >>>> more than that. IndexMemory default setting with
> >>>> 8 byte PK should also provide space for around 800k 
> records. If you
> >>>> have a 4 node cluster then it can handle twice the amount
> >>>> of data.
> >>>>
> >>>>> could that be part of our problem? this doesn't seem like
> >>>> an unusually
> >>>>> large table...do you think the default values would be able
> >>>> to handle
> >>>>> a table this size? could it be something else other than
> >>>> these memory
> >>>>> settings?
> >>>>>
> >>>>
> >>>> You need to increase default settings. Default settings is
> >> mostly for
> >>>> easy start on any machine, also a small machine. Sounds like you
> >>>> need to set DataMemory to something like 500M and IndexMemory to
> >>>> something like 150M. Check the manual at
> >>>> 
> http://dev.mysql.com/doc/mysql/en/MySQL_Cluster_DB_Definition.html
> >>>> and the entries for DataMemory and IndexMemory for more info
> >>>> on how to
> >>>> calculate the proper parameters.
> >>>>
> >>>> Also I am not exactly sure what your error code was. I
> >> can't think of
> >>>> any other table full variant than what here has been 
> discussed but
> >>>> providing the error message will clear the doubts away.
> >>>>
> >>>> Rgrds Mikael
> >>>>
> >>>>> thanks,
> >>>>> -L
> >>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Mikael Ronström [mailto:mikael@stripped]
> >>>>>> Sent: Thursday, July 29, 2004 4:47 PM
> >>>>>> To: Crouch, Luke H.
> >>>>>> Cc: Vinay Joosery; cluster@stripped
> >>>>>> Subject: Re: table full error?
> >>>>>>
> >>>>>>
> >>>>>> Hi Luke,
> >>>>>> I am referring to the MySQL Manual available on
> >>>>>> http://dev.mysql.com/doc/mysql/en/index.html
> >>>>>> where chapter 17 is dedicated to MySQL Cluster. This page
> >>>> is updated
> >>>>>> immediately as changes are done in the manual.
> >>>>>>
> >>>>>> To set a parameter to 300 MByte you write 300M as an
> example.
> >>>>>>
> >>>>>> The cluster administration guide is a useful resource
> >>>> still but it is
> >>>>>> not updated so the
> >>>>>> manual contains the "latest and the greatest".
> >>>>>>
> >>>>>> Rgrds Mikael
> >>>>>>
> >>>>>> 2004-07-29 kl. 23.30 skrev Crouch, Luke H.:
> >>>>>>
> >>>>>>> Mikael,
> >>>>>>>
> >>>>>>> what manual are you referencing? the mysql cluster
> >> administration
> >>>>>>> guide that I have has the parameters NoOfIndexPages and
> >>>>>> NoOfDataPages
> >>>>>>> for DB definitions, with a unit of 8KB pages. what unit
> is the
> >>>>>>> DataMemory and IndexMemory parameters expressed in? kB?
> >>>>>>>
> >>>>>>> is there a kind of 'live' manual available online?
> where
> >>>> can we get
> >>>>>>> the most recent manual each day?
> >>>>>>>
> >>>>>>> thanks,
> >>>>>>> -L
> >>>>>>>
> >>>>>>>> -----Original Message-----
> >>>>>>>> From: Mikael Ronström
> [mailto:mikael@stripped]
> >>>>>>>> Sent: Thursday, July 29, 2004 3:58 PM
> >>>>>>>> To: Crouch, Luke H.
> >>>>>>>> Cc: Vinay Joosery; cluster@stripped
> >>>>>>>> Subject: Re: table full error?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Hi Luke,
> >>>>>>>> Most likely DataMemory and IndexMemory from your
> short
> >>>> description.
> >>>>>>>> Check the manual for more details on how
> >>>>>>>> to set it. Default is DataMemory size of 80000
> kBytes sounds
> >>>>>>>> reasonable
> >>>>>>>> if your records are about 60-70 bytes.
> >>>>>>>> IndexMemory is by default set to 24000 kBytes so it
> is
> >>>>>>>> reasonable that
> >>>>>>>> this parameter is your problem if you
> >>>>>>>> have a 4 byte primary key.
> >>>>>>>>
> >>>>>>>> Rgrds Mikael
> >>>>>>>>
> >>>>>>>> 2004-07-29 kl. 22.36 skrev Crouch, Luke H.:
> >>>>>>>>
> >>>>>>>>> we are trying to load up an NDB table thru one
> of the API
> >>>>>> nodes. we
> >>>>>>>>> have a file (generated by a mysqldump) that
> inserts 4
> >>>>>>>> million records,
> >>>>>>>>> but when we run it, it stops at about 996k and
> says
> >>>> table is full!
> >>>>>>>>> where is the setting that controls this?
> >>>>>>>>>
> >>>>>>>>> thanks,
> >>>>>>>>> -L
> >>>>>>>>>
> >>>>>>>
> >>>>>>> -- 
> >>>>>>> MySQL Cluster Mailing List
> >>>>>>> For list archives: http://lists.mysql.com/cluster
> >>>>>>> To unsubscribe:
> >>>>>>> http://lists.mysql.com/cluster?unsub=1
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>> Mikael Ronström, Senior Software Architect
> >>>>>> MySQL AB, www.mysql.com
> >>>>>>
> >>>>>> Clustering:
> >>>>>>
> http://www.infoworld.com/article/04/04/14/HNmysqlcluster_1.html
> >>>>>>
> >>>>>> http://www.eweek.com/article2/0,1759,1567546,00.asp
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>> -- 
> >>>>> MySQL Cluster Mailing List
> >>>>> For list archives: http://lists.mysql.com/cluster
> >>>>> To unsubscribe:
> >>>>> http://lists.mysql.com/cluster?unsub=1
> >>>>>
> >>>>>
> >>>>>
> >>>> Mikael Ronström, Senior Software Architect
> >>>> MySQL AB, www.mysql.com
> >>>>
> >>>> Clustering:
> >>>> http://www.infoworld.com/article/04/04/14/HNmysqlcluster_1.html
> >>>>
> >>>> http://www.eweek.com/article2/0,1759,1567546,00.asp
> >>>>
> >>>>
> >>>>
> >>>>
> >> Mikael Ronström, Senior Software Architect
> >> MySQL AB, www.mysql.com
> >>
> >> Clustering:
> >> http://www.infoworld.com/article/04/04/14/HNmysqlcluster_1.html
> >>
> >> http://www.eweek.com/article2/0,1759,1567546,00.asp
> >>
> >>
> >>
> >>
> Mikael Ronström, Senior Software Architect
> MySQL AB, www.mysql.com
> 
> Clustering:
> http://www.infoworld.com/article/04/04/14/HNmysqlcluster_1.html
> 
> http://www.eweek.com/article2/0,1759,1567546,00.asp
> 
> 
> 
Thread
table full error?Luke H. Crouch29 Jul
  • Re: table full error?Mikael Ronström29 Jul
RE: table full error?Luke H. Crouch29 Jul
  • Re: table full error?Mikael Ronström29 Jul
RE: table full error?Luke H. Crouch30 Jul
  • Re: table full error?Mikael Ronström30 Jul
RE: table full error?Luke H. Crouch30 Jul
  • Re: table full error?Mikael Ronström30 Jul
RE: table full error?Luke H. Crouch30 Jul
  • Re: table full error?Mikael Ronström30 Jul
RE: table full error?Luke H. Crouch30 Jul
  • Re: table full error?pekka31 Jul
RE: table full error?Luke H. Crouch31 Jul
  • Re: table full error?pekka1 Aug
    • NDB documentationSundeep Narravula1 Aug
      • Re: NDB documentationpekka1 Aug
RE: table full error?Luke H. Crouch2 Aug
  • Re: table full error?pekka2 Aug