Hi Jigal, all;
Jigal van Hemert wrote:
> Hi Joerg,
> From: "Joerg Bruehe"
>>Jigal van Hemert wrote:
>>>050726 14:13:12 mysqld started
>>>050726 14:13:12 InnoDB: Error: cannot allocate 7340048384 bytes of
>>>InnoDB: memory with malloc! Total allocated memory
>>>InnoDB: by InnoDB 78086952 bytes. Operating system errno: 12
>>On my machine (Linux: SuSE 9.1), I have this line in
>> #define ENOMEM 12 /* Out of memory */
> And perror 12 also produces a similar error description.
>>So it looks like some address space (paging area?) was not yet free when
>>the restart was attempted. Maybe the MySQL server had not yet fully
> MySQL server was terminated; at least it didn't show up in the output of the
I do not know exactly which speedup optimizations might be taken in
Fedora Core 4 (as mentioned in your first posting) in general, or in a
64 bit version specifically, so I am speculating:
A running MySQL server as configured by you, with 7 GB buffer pool, will
occupy substantial amounts of RAM, probably backed in the "swap area"
(even though this is really a paging area). When the process terminates,
all its resources need to be freed, including flushing files, closing
file descriptors, and releasing these 7 GB. This may take some time.
Consider that there are file systems that delay writes in order to
optimize disk I/O and to favor reads on which other processes might be
waiting. I suspect that similar strategies might be used on the page device.
IOW: I doubt that the removal of a process from "ps" output implies that
all its resources have already been freed, and are available.
I admit that the Linux kernel should detect such a situation and delay
the new request (rather than reject it) as the scarce resources are just
getting available, but maybe this is not (yet) done?
> It doesn't happen all the time; the server was running for a few days now.
> We have never encountered such a situation on a 32-bit machine yet. You
> could simply terminate MySQL and start it immediately.
Well, on a 32 bit machine the areas are smaller, so freeing them should
> Can memory get fragmented in some way after it is allocated?
AFAIK, this should not happen since Linux is a paging system, not swapping.
Of course I can imagine (RAM or paging space) management strategies that
try to keep areas continuous, to allow larger I/O transfers, but IMHO
these should not be taken so absolute that they delay operation.
All in all, I suspect that with your growing storage sizes you need
growing amounts of time to release them.
Even though hardware gets faster, resource consumption manages to grow
at at least the same rate ;-)
Have you ever tried to wait a bit after a failing restart and then
attempt it again, rather than rebooting?
Sorry I can not give a more concrete help,
Joerg Bruehe, Senior Production Engineer
MySQL AB, www.mysql.com