List:MySQL++« Previous MessageNext Message »
From:F.N. Bruggeman Date:March 23 2011 4:15pm
Subject:Re: StoreQueryResults destructor runs without end...
View as plain text  
I'd like to add something to that:
"Use" in combination with processing the raw results
(UseQueryResult.fetch_raw_row) resulted in a major performance improvement
for our project, as the std conversions (streams) used in mysqlpp row are
(too) slow.

With "Use" and the mysqlpp row it took roughly 14 seconds to process a 400k
row result, whereas using itoa and family on the raw rows only took 2
seconds (release compile w/ VC10)

You do lose a lot of the features mysqlpp provides of course, but I figured
this might be useful to other people trying to handle large results better
(this particular project's database is roughly 30GB in size, with some
tables containing over 10M rows).

As for the multiplier - we ended up creating our own list, vector, string
etc due to std's poor performance and ended up using Fibonacci because our
#1 concern is heap fragmentation.


Kind regards,

Freddy


-----Oorspronkelijk bericht-----

Van: Dino Korah [mailto:dino.korah@stripped] 
Verzonden: woensdag 23 maart 2011 15:42
Aan: plusplus@stripped
Onderwerp: Re: List etiquette - please use good email subjects :)

"Use" is always better than "Store" method for big result sets.
So if this change makes no difference to rest of your application, this 
must be the one that you try first.

You can always profile and debug your "Store" based solution, but it 
will be of academic value (very likely).

Bests,
Dino


On 23/03/11 14:21, Paul Dalach wrote:
> Hi all,
> Thanks for any responses.
>
> The max memory a typical query requires under 100mb...well within my
> 10gb memory limit.
>
> I have tried both debug and release libraries and get the same issue
> with both.
>
> I am trying now to profile the application to find out precisely where
> the hangup is...any suggestions would be welcome.
>
> Any other thoughts?  Should I switch to use instead of query?
>
> Paul
>
>
> On 3/23/2011 7:26 AM, Tomalak Geret'kal wrote:
>> On 23/03/2011 12:22, Joel Fielder wrote:
>>> Hi all,
>>>
>>> This isn't aimed at anyone in particular, but the past
>>> couple of days have had topics turn into a discussion about
>>> other stuff.
>>>
>>> It would really help if posters would add OT to the subject,
>>> or specify a new subject if this happens.
>>>
>>> I call this rule "try before you buy" - we all have lots of
>>> email and good email subjects really make a difference to
>>> whether I read or delete :)
>>>
>>> Anybody got any comments or thoughts? Or am I being high
>>> maintenance?
>>>
>>> Joel
>>>
>> No, I think you're absolutely right.
>>
>> The most recent thread in particular has now gone away from the scope
>> of StoreQueryResults.
>>
>> I'm hesitant to comment more on the matter seeing as my suggestion to
>> add a subject line prefix to list mails was shot down so vehemently... :P
>>
>> Tom
>>
>

Thread
StoreQueryResults destructor runs without end...Paul Dalach22 Mar
Re: StoreQueryResults destructor runs without end...Dino Korah22 Mar
  • Re: StoreQueryResults destructor runs without end...Paul Dalach23 Mar
    • Re: StoreQueryResults destructor runs without end...Warren Young23 Mar
      • Re: StoreQueryResults destructor runs without end...Warren Young23 Mar
        • Re: StoreQueryResults destructor runs without end...Tomalak Geret'kal23 Mar
          • Re: StoreQueryResults destructor runs without end...Mickael Wolff23 Mar
            • Re: StoreQueryResults destructor runs without end...Tomalak Geret'kal23 Mar
              • Re: StoreQueryResults destructor runs without end...Mickael Wolff23 Mar
                • Re: StoreQueryResults destructor runs without end...Tomalak Geret'kal23 Mar
          • Re: StoreQueryResults destructor runs without end...Warren Young23 Mar
            • Re: StoreQueryResults destructor runs without end...Tomalak Geret'kal23 Mar
              • List etiquette - please use good email subjects :)Joel Fielder23 Mar
                • Re: List etiquette - please use good email subjects :)Tomalak Geret'kal23 Mar
                  • Re: List etiquette - please use good email subjects :)Paul Dalach23 Mar
                    • Re: List etiquette - please use good email subjects :)Dino Korah23 Mar
                      • Re: StoreQueryResults destructor runs without end...Paul Dalach23 Mar
                      • Re: StoreQueryResults destructor runs without end...F.N. Bruggeman23 Mar
                    • Re: List etiquette - please use good email subjects :)Joel Fielder23 Mar
              • Re: growth behavior of std::vectorWarren Young23 Mar
                • Re: growth behavior of std::vectorTomalak Geret'kal23 Mar
                  • Re: growth behavior of std::vectorJonathan Wakely23 Mar
                    • Re: (OT) growth behavior of std::vectorTomalak Geret'kal23 Mar
                      • Re: (OT) growth behavior of std::vectorJonathan Wakely23 Mar
                • Re: (OT) growth behavior of std::vectorTomalak Geret'kal23 Mar
            • Re: StoreQueryResults destructor runs without end...Tomalak Geret'kal23 Mar
              • re: (OT) growth behaviour of std::vector [was Re: StoreQueryResultsdestructor runs without end...]Tomalak Geret'kal23 Mar
        • Re: StoreQueryResults destructor runs without end...Warren Young23 Mar
Re: StoreQueryResults destructor runs without end...Dino Korah23 Mar