On Fri, 29 Jul 2005, Jigal van Hemert wrote:
> I do not know exactly which speedup optimizations might be taken in
> Fedora Core 4 (as mentioned in your first posting) in general, or in a
> 64 bit version specifically, so I am speculating:
> A running MySQL server as configured by you, with 7 GB buffer pool, will
> occupy substantial amounts of RAM, probably backed in the "swap area"
> (even though this is really a paging area). When the process terminates,
> all its resources need to be freed, including flushing files, closing
> file descriptors, and releasing these 7 GB. This may take some time.
> Consider that there are file systems that delay writes in order to
> optimize disk I/O and to favor reads on which other processes might be
> waiting. I suspect that similar strategies might be used on the page device.
> IOW: I doubt that the removal of a process from "ps" output implies that
> all its resources have already been freed, and are available.
> I admit that the Linux kernel should detect such a situation and delay
> the new request (rather than reject it) as the scarce resources are just
> getting available, but maybe this is not (yet) done?
I am a colleague of Jigal van Hemert with which you had this discussion
earlier. I subscribed to the MySQL list to clarify the situation, as I'm
the one actually experiencing the problems.
I'd like to start with the following:
This implies I'm not the only one strugling with 'Active' memory on a 2.6
kernel. Interesting detail: the problem report is issued by a MySQL
developer named Ingo Strüwing, maybe you know him? Either way, I already
contacted him to share my experiences.
Furthermore I have started a thread on the Fedora mailing list about this,
as it seems to be related to somewhat whacky memory management.
Now back to the problem, what I've found out is basically the following:
- When doing a malloc call it appears the requested amount of memory is
tested against the total amount of memory minus the amount of 'Active'
(according to /proc/meminfo) memory. So when 6GB of Active memory has
piled up on my system after a couple of compiles, the largest block of
memory allocatable through malloc seems to be roughly 8GB-6GB=2GB. This
is why the single malloc call for 7GB from InnoDB fails.
- Interestingly enough it is perfectly possible to allocate multiple 2GB
blocks in above situation. This can be done almost without limit,
because the memory is not actually in use yet, it is only allocated. I
have been able to allocate up to 12GB (did not try any higher) this way.
As longs as the single malloc calls request blocks which fall within the
Total - Active equation, this will succeed. I tested this by modifying
the 'fillmem' utility from the 'memtest' package:
- Even more interesting is the fact that 'fillmem' is in fact able to
reclaim the Active memory. If I instruct fillmem to allocate (and
actually use it by filling it with random values) near to 8GB of RAM, it
does so with success and in the end the total amount of Active memory is
near zero. After this I can restart MySQL again. This is a temporary
Now the only question that remains is why the Active memory goes close to
zero when exiting fillmem and is not when ending a compile run. I asked
this question on the Fedora list to find out if this is a normal situation
or if there could be a memory leak somewhere in the compiler, linker etc
chain. In the meanwhile I can use the mentioned workaround, but it's still
a bit weird situation.
> Have you ever tried to wait a bit after a failing restart and then
> attempt it again, rather than rebooting?
We have waited up to 48 hours, but alas the Active memory never
Thanks for your time,
Matthijs van der Klip