List:Internals« Previous MessageNext Message »
From:Antony Curtis Date:June 29 2009 7:03pm
Subject:Re: About fixing
"memory exaustion when performing large multi-statement queries"
View as plain text  
 
On Monday, June 29, 2009, at 09:02AM, "Davi Arnaut" <Davi.Arnaut@stripped> wrote:
>On 6/29/09 12:54 PM, Antony T Curtis wrote:
>>
>> On 29 Jun 2009, at 08:47, Antony T Curtis wrote:
>>
>>> Hi,
>>>
>>> A quick solution is to simply mark blocks as free. This does not
>>> return the memory to the operating system so it is safe from
>>> interference from other threads. You can then alloc a block again and
>>> use memmove to copy in the query text again.... so something like:
>>>
>>>
>>> const char * old_query = thd->query;
>>> size_t length = thd->query_length + 1 + thd->db_length + 1 +
>>> QUERY_CACHE_FLAGS_SIZE;
>>> free_root(thd->mem_root, MYF(MY_MARK_BLOCKS_FREE));
>>> if ((thd->query= (char*) thd->alloc(length)) != old_query)
>>> memmove(thd->query, old_query, length);
>>>
>>
>> Oh, one little detail I forgot, you'd want to adjust the lexer pointers
>> in the case where the ptr of thd->query has moved.
>>
>
>Also, THD::query is visible to other threads. Updating what THD::query 
>points to is a bit messy...

The cleaner option would be to teach MEM_ROOT how to do mark/sweep - which would also help
reduce memory consumption in other areas of the server.

A 'mark' would be performed by examining all the memory blocks which the MEM_ROOT reports
as in use and taking note of each blocks fill level.
'sweep' would be accomplished by moving memory blocks which are not on the 'mark' list
onto the free list and restoring the fill levels of known blocks.

Both mark and sweep operations would be quite fast as they won't require any inter-thread
communication or kernel calls. You may argue that the 'mark' option would need to
allocate memory but you can just grab memory from the MEM_ROOT which would conveniently
be disposed upon sweep.

In some areas of the server where a temporary MEM_ROOT is allocated just to be freed soon
after can be replaced by using mark/sweep.

Just my 2¢

Regards,
Antony

>-- 
>MySQL Internals Mailing List
>For list archives: http://lists.mysql.com/internals
>To unsubscribe:    http://lists.mysql.com/internals?unsub=1
>
>
>
Thread
About fixing"memory exaustion when performing large multi-statement queries"Guilhem Bichot29 Jun
  • Re: About fixing"memory exaustion when performing large multi-statement queries"Antony T Curtis29 Jun
    • Re: About fixing"memory exaustion when performing large multi-statement queries"Antony T Curtis29 Jun
      • Re: About fixing"memory exaustion when performing large multi-statement queries"Davi Arnaut29 Jun
        • Re: About fixing"memory exaustion when performing large multi-statement queries"Antony Curtis29 Jun