Hi Matthijs!
Matthijs van der Klip wrote:
> On Fri, 29 Jul 2005, Joerg Bruehe wrote:
>
>>>Now the only question that remains is why the Active memory goes close to
>>>zero when exiting fillmem and is not when ending a compile run. I asked
>>
>>Again IMHO, it shows an error in memory management.
>
>
> I do not know if it's an error or not. I do agree with you that the memory
Well, I understood your description as "after several compile runs or a
mysql run, 'active memory' is not returned". This is what looks to me
like being an error.
> management in Linux 2.6 does not seem to be ideal. I even found the
> following comment in the malloc manpage:
>
> 'By default, Linux follows an optimistic memory allocation strategy. This
> means that when malloc() returns non-NULL there is no guarantee that
> the memory really is available. [[...]]
> See also the kernel Documentation directory, files
> vm/overcommit-accounting and sysctl/vm.txt.'
This "memory overcommitment" is the other way around: Processes get more
memory requests granted (in total) than they can use afterwards.
AIUI, this is implemented by a delayed allocation in paging space: A
page that got added to the process' address space (like by a "malloc()")
is not immediately assigned a location in the paging space, only when
it got modified in RAM and/or then needs to be written out to the paging
space for the first time.
As a result, the allocation succeeds, but some process gets killed when
the paging space cannot take such an additional page. To the affected
process, this looks like a crash.
>
>
> What I don't understand is why I seem to be one of few suffering from this
> problem. MySQL on Linux 2.6 (combined with a massive amount of RAM) is
> hardly an uncommon configuration nowadays.
>
> Secondly it seems two parties (MySQL and Fedora) are pointing to each
> other right now. Let me quote:
>
> On Fri, 29 Jul 2005, Rick Stevens wrote:
>
>>Well, malloc() will fail if you request a chunk of memory and there
>>isn't a SINGLE chunk available of that size. So if memory gets fragged,
>>there isn't a single 7GB chunk available and malloc() will fail.
>>fillmem allocates in smaller chunks, then releases it all so the
>>memory defragger can clean things up.
>>
>>Ideally, that's what mysql should do. Or start off at some huge
>>size and keep trying progressively smaller chunks until it gets some,
>>e.g. try 8GB. If that fails, try 6GB, then 4, then 2, you get the
>>idea. It could then link those together and manage them.
>>
>>I'm not surprised that it fails. You're asking a single application to
>>grab 7/8 of your RAM--and all in one chunk--regardless of what else has
>>been run before it. On a pristine system (e.g. right after a boot),
>>it may work. After that...
>
>
> It sounds kind of reasonable if explained like this. Now, which method
> (allocating all in one single malloc call or allocating multiple smaller
> blocks) is considered good programming practice? And would this be
> something InnoDB would be likely to change? (A long shot, I guess)
This is thin ice for me, but still:
I am a bit surprised that the Linux kernel management will only allocate
memory if a single chunk of sufficient size is available. My
understanding was that in a paging system this is not necessary.
If this is (becoming) standard Linux policy, it might be necessary to
demand memory piecewise. One drawback of this approach is increased
bookeeping, if it ever needs to be released.
I have no idea how the developers view this issue - you might open a
change request if you consider this Linux kernel policy definite.
You wrote that if a mysql server start fails, you can run "fillmem", and
after its exit the memory will be available. I am not sure whether
Rick's explanation addresses this issue as well - it might be the
"memory defragger" he refers to. If not, the once used chunks might
still be considered "active".
Regards,
Jörg
--
Joerg Bruehe, Senior Production Engineer
MySQL AB, www.mysql.com
