List:General Discussion« Previous MessageNext Message »
From:Martin Date:February 22 2005 1:19am
Subject:Odd rounding errors with 4.1
View as plain text  
Version: Using 4.1.10 on WinXP pro, currently interacting with it using 
the Query Browser for testing.

I have a table set up that contains a column of DECIMAL(15,12) -- 
financial data, where the precision is highly important.  I am building 
up a series of SQL statements, and I noticed that when doing SUM() on 
this decimal column, I get a strange rounding error (see below), and was 
hoping that someone out there can help me with this.

My recent test involved the following three values from the column:
2.4950
2.5950
2.7700
(Chosen, for this example, as they are precise at few decimal places).

When I use a SUM() on these I get: 7.860

If I switch the column over to a FLOAT, then the SUM() becomes 
7.8599998950958

Using Excel to test the numbers, or hand-calculating, I get:
7.8550.

Shouldn't the SUM() remain with the precision of the DECIMAL type and 
not try to round to 2 decimal places?

Anyway, any help is appreciated.

Martin
Thread
Odd rounding errors with 4.1Martin22 Feb
  • Re: Odd rounding errors with 4.1Dan Nelson22 Feb
    • Re: Odd rounding errors with 4.1 [Duh, me!]Martin22 Feb
Re: Odd rounding errors with 4.1Martin22 Feb
  • RE: Odd rounding errors with 4.1Gordon23 Feb