List:Cluster« Previous MessageNext Message »
From:David Ennis Date:July 26 2010 9:43pm
Subject:Re: Converting MyISAM with 300GB to NDB.
View as plain text  
Hi.

Even if you use disk based fields to get around the memory issue, 
remember that all indexed fields are always stored in memory. 

So, if all of the index data is more than the available memory, its 
simply impossible.

Even if you were to say that there  were no indexed fields (which would 
just be odd for that much data or any amount of data for that matter), 
MySQL will still create an index - using memory regardless.. (8 Bytes)

Pluse, all text and blob fields are not stored completely on disk.. The 
first 256 bytes fore EVERY FIELD of EVERY RECORD are also stored in 
memory.. The remainder is on disk..

So, a record of 10 text/blob fields with no indexes(silly) all stored to 
disk will still use a minimum of 2,568 bytes of memory (8 + (256 *10))

Add your indexes and the number goes up..

So, if you are sure that you have enough memory to store only indexed 
fields in memory and the rest to disk, then try the following:

mysqldump the table definitions
mysqldump the data

alter the table definitions to match what is needed for NDB and each 
field as stored to disk.
import the table defs into a new database
import the data (try just a few records first  to make sure)


A second note on memory..
You say that you have one server with 16 gig of memory..  Well, NDB is 
meant to run on a cluster of servers.  Although it is possible to run 
multiple instances on a single machine (killing redndancy and 
preformance), you dont really have as much memory as you think as there 
are all sorts of buffers, etc as well


-david




Ricardo Soares Guimarães wrote:
> Hello,
>
> I have one database with 300Gb of data, and want to try NDB.
>
> That´s possible?
>
> I tried several configurations, and keys, but seems that can´t import 
> such data
>
> My server has (now) 16Gb of RAM, but soon will have 32.
>
> Anyone tried such configuration?
>
> I am using mysql-cluster 7.1.x with 64bits linux server.
>
> Thanks
>

Thread
Converting MyISAM with 300GB to NDB.Ricardo Soares Guimarães26 Jul
  • Re: Converting MyISAM with 300GB to NDB.Dan Frincu26 Jul
    • Performance and Scalability and clusterJeff - ES Schweiger23 Sep
  • Re: Converting MyISAM with 300GB to NDB.Dan Frincu26 Jul
  • Re: Converting MyISAM with 300GB to NDB.Stewart Smith26 Jul
    • Re: Converting MyISAM with 300GB to NDB.Ricardo Soares Guimarães26 Jul
      • Re: Converting MyISAM with 300GB to NDB.Jason Brooke26 Jul
  • Re: Converting MyISAM with 300GB to NDB.David Ennis26 Jul
    • Re: Converting MyISAM with 300GB to NDB.Ricardo Soares Guimarães27 Jul
  • Re: Performance and Scalability and clusterYves Trudeau23 Sep
    • RE: Performance and Scalability and clusterJeff - ES Schweiger23 Sep
      • Re: Performance and Scalability and clusterAdam Dixon23 Sep
        • RE: Performance and Scalability and clusterAndrew Morgan24 Sep
          • High availability implementationramarovelo28 Sep
            • RE: High availability implementationAndrew Morgan28 Sep
              • Re: High availability implementationMat Keep28 Sep
              • Re: High availability implementationramarovelo28 Sep
      • Re: Performance and Scalability and clusterMikael Ronstrošm24 Sep