List:General Discussion« Previous MessageNext Message »
From:Dan Nelson Date:February 22 2005 3:22am
Subject:Re: Odd rounding errors with 4.1
View as plain text  
In the last episode (Feb 21), Martin said:
> Version: Using 4.1.10 on WinXP pro, currently interacting with it
> using the Query Browser for testing.
> 
> I have a table set up that contains a column of DECIMAL(15,12) --
> financial data, where the precision is highly important.  I am
> building up a series of SQL statements, and I noticed that when doing
> SUM() on this decimal column, I get a strange rounding error (see
> below), and was hoping that someone out there can help me with this.
> 
> My recent test involved the following three values from the column:
> 2.4950
> 2.5950
> 2.7700
> (Chosen, for this example, as they are precise at few decimal places).
> 
> When I use a SUM() on these I get: 7.860
> 
> If I switch the column over to a FLOAT, then the SUM() becomes 
> 7.8599998950958
> 
> Using Excel to test the numbers, or hand-calculating, I get:
> 7.8550.

You hand calculations (and Excel) are wrong, apparently.  Here's my
hand-calculation:

   1 11                  1

   2.495               5.090
 + 2.595             + 2.770
 =======             =======
   5.090               7.860

, which matches MySQL's results.

-- 
	Dan Nelson
	dnelson@stripped
Thread
Odd rounding errors with 4.1Martin22 Feb
  • Re: Odd rounding errors with 4.1Dan Nelson22 Feb
    • Re: Odd rounding errors with 4.1 [Duh, me!]Martin22 Feb
Re: Odd rounding errors with 4.1Martin22 Feb
  • RE: Odd rounding errors with 4.1Gordon23 Feb