List:Internals« Previous MessageNext Message »
From:Davi Arnaut Date:October 17 2008 6:44pm
Subject:Re: memory allocation architecture in MySQL
View as plain text  
Hi Mark,

On 10/17/08 2:43 PM, MARK CALLAGHAN wrote:
> Is the memory management model documented for MySQL? I want to know
> what APIs exist to allocate memory in the context of a user
> connection. Are there heaps for global, connection duration and
> statement duration allocation? And if there are, how are they
> accessed?

Quoting one of MySQL Internals document:

".. the memory root dedicated to execution of a statement will be 
denoted as the runtime memory root of the statement. Because allocations 
are done indirectly via THD::mem_root, THD::mem_root at any given moment 
of time can point either to the permanent or to the runtime memory root 
of the statement. Consequently, THD::mem_root and THD::free_list can be 
denoted as 'currently active arena' of THD."

This comment also sheds some light:

/**
   This memory root is used for two purposes:
   - for conventional queries, to allocate structures stored in main_lex
   during parsing, and allocate runtime data (execution plan, etc.)
   during execution.
   - for prepared queries, only to allocate runtime data. The parsed
   tree itself is reused between executions and thus is stored elsewhere.
*/
MEM_ROOT main_mem_root;

Also, take a look in the relationship between THD, Statement and 
Query_arena classes.

> I am curious because of a bug in MySQL 5.0 --
> http://bugs.mysql.com/bug.php?id=40097. A per-THD variable references
> stack-allocated memory used by a String instance.

Good catch.

> I found this, but it doesn't describe what/where MEM_ROOT instances
> exist in the context of a connection (THD).
> http://forge.mysql.com/wiki/MySQL_memory_handling_and_memory_handling_in_Falcon
>
> This states there is a connection duration memory heap -- use sql_alloc()
> http://forge.mysql.com/wiki/MysysLibraryAndAlgorithms

Not quite 'connection', it can be a runtime or permanent one.

> There appears to be a lot of allocation that does not use the
> per-connection MEM_ROOT. String (from sql/sql_string.{cc,h}) uses
> my_malloc and my_realloc to get memory. my_malloc and my_realloc don't
> use per-THD heaps like sql_alloc() and get memory directly from
> malloc(). Is there a goal to make all allocation done in the context
> of a user query allocated via the per-connection MEM_ROOT? It seems
> like a bad idea to allow memory references between objects from
> different allocation arenas -- things allocated from malloc()
> referencing things allocated from sql_alloc() and vice-versa.

I agree with the "not a good idea" part :)

> Finally, String provides 'operator new' that uses MEM_ROOT to allocate
> the String object from that MEM_ROOT. But if that String instance ever
> needs to reallocate memory for the contained string, it will get it
> from malloc() and not use the MEM_ROOT. Is there an unstated
> assumption that all uses of this won't need to allocate more memory?

The idea is that the String object is allocated from a memory root, but 
the data (string of characters) can is stored in a passed buffer or in 
memory allocated from malloc.

Regards,

-- Davi Arnaut
Thread
memory allocation architecture in MySQLMARK CALLAGHAN17 Oct
  • Re: memory allocation architecture in MySQLMARK CALLAGHAN17 Oct
  • Re: memory allocation architecture in MySQLDavi Arnaut17 Oct
    • Re: memory allocation architecture in MySQLMichael Widenius31 Oct
      • Re: memory allocation architecture in MySQLMARK CALLAGHAN31 Oct
        • Re: memory allocation architecture in MySQLMichael Widenius31 Oct
        • Re: memory allocation architecture in MySQLMichael Widenius31 Oct
          • Re: memory allocation architecture in MySQLMARK CALLAGHAN30 Apr
            • Re: memory allocation architecture in MySQLDavi Arnaut30 Apr
              • Re: memory allocation architecture in MySQLBrian Aker30 Apr
              • Re: memory allocation architecture in MySQLSergei Golubchik30 Apr
                • Re: memory allocation architecture in MySQLDavi Arnaut30 Apr