List:General Discussion« Previous MessageNext Message »
From:Martin Date:February 22 2005 3:41am
Subject:Re: Odd rounding errors with 4.1 [Duh, me!]
View as plain text  
Yeah, I checked the math after another responder.  Either I was smoking 
crack at work today (a state likely induced by too much/too little 
caffeine) or I mis-noted my numbers.

I'm going to check again when I get back there tomorrow.

But thanks :)

Martin

Dan Nelson wrote:
> In the last episode (Feb 21), Martin said:
> 
>>Version: Using 4.1.10 on WinXP pro, currently interacting with it
>>using the Query Browser for testing.
>>
>>I have a table set up that contains a column of DECIMAL(15,12) --
>>financial data, where the precision is highly important.  I am
>>building up a series of SQL statements, and I noticed that when doing
>>SUM() on this decimal column, I get a strange rounding error (see
>>below), and was hoping that someone out there can help me with this.
>>
>>My recent test involved the following three values from the column:
>>2.4950
>>2.5950
>>2.7700
>>(Chosen, for this example, as they are precise at few decimal places).
>>
>>When I use a SUM() on these I get: 7.860
>>
>>If I switch the column over to a FLOAT, then the SUM() becomes 
>>7.8599998950958
>>
>>Using Excel to test the numbers, or hand-calculating, I get:
>>7.8550.
> 
> 
> You hand calculations (and Excel) are wrong, apparently.  Here's my
> hand-calculation:
> 
>    1 11                  1
> 
>    2.495               5.090
>  + 2.595             + 2.770
>  =======             =======
>    5.090               7.860
> 
> , which matches MySQL's results.
> 
Thread
Odd rounding errors with 4.1Martin22 Feb
  • Re: Odd rounding errors with 4.1Dan Nelson22 Feb
    • Re: Odd rounding errors with 4.1 [Duh, me!]Martin22 Feb
Re: Odd rounding errors with 4.1Martin22 Feb
  • RE: Odd rounding errors with 4.1Gordon23 Feb