List:Community« Previous MessageNext Message »
From:harryxiyou Date:April 3 2014 12:41am
Subject:Re: How to get large items from a table
View as plain text  
On Wed, Apr 2, 2014 at 10:56 PM, shawn l.green <shawn.l.green@stripped> wrote:
> Hello Harry,

Hi Shawn,

> You appear to face the same resource dilemmas that all storage engines face:
> there will be more data stored on disk than you can possibly buffer in
> memory. There have been many strategies designed to keep only the 'most
> important' or 'most recently used' elements in memory. Please look to them
> for inspiration.
>
> To go along with trying to keep a 'warm' cache of data for faster retrieval,
> you need to decide how to manage scans of large quantities of data (such as
> your SELECT * ...  query) and if you want them to replace the warm buffer
> you have been cultivating since startup.  This may mean that you skip trying
> to buffer any scan that uses more than X percent of your table (you get to
> pick what the X is) to keep your cache nice and toasty.
>
> And you may have multiple cache types, hot, warm, cold, temporary, ...
> depending on how you want to manage them.  Again, this is a hugely broad
> topic with many different algorithms used in both commercial and open source
> products. There is no one 'correct' answer for all situations.
>
> Then you get to the subject of index management, file storage formats, row
> identification, and how to locate a row on disk.  That's an entirely
> different beast. Of course, if you store nothing on disk then those design
> questions become irrelevant.
>

Thanks very much, I would take your above suggestions into consideration.


Harry.
Thread
Re: How to get large items from a tableharryxiyou2 Apr 2014