On Wed, Apr 2, 2014 at 10:56 PM, shawn l.green <shawn.l.green@stripped> wrote:
> Hello Harry,
> You appear to face the same resource dilemmas that all storage engines face:
> there will be more data stored on disk than you can possibly buffer in
> memory. There have been many strategies designed to keep only the 'most
> important' or 'most recently used' elements in memory. Please look to them
> for inspiration.
> To go along with trying to keep a 'warm' cache of data for faster retrieval,
> you need to decide how to manage scans of large quantities of data (such as
> your SELECT * ... query) and if you want them to replace the warm buffer
> you have been cultivating since startup. This may mean that you skip trying
> to buffer any scan that uses more than X percent of your table (you get to
> pick what the X is) to keep your cache nice and toasty.
> And you may have multiple cache types, hot, warm, cold, temporary, ...
> depending on how you want to manage them. Again, this is a hugely broad
> topic with many different algorithms used in both commercial and open source
> products. There is no one 'correct' answer for all situations.
> Then you get to the subject of index management, file storage formats, row
> identification, and how to locate a row on disk. That's an entirely
> different beast. Of course, if you store nothing on disk then those design
> questions become irrelevant.
Thanks very much, I would take your above suggestions into consideration.