List:General Discussion« Previous MessageNext Message »
From:Michael Satterwhite Date:October 26 2010 2:49pm
Subject:Re: Query precision issue
View as plain text  
This is a problem that is as old as computers. Floating point decimal data 
does not translate directly into binary. Whole numbers are (usually) fine, but 
fractions bring out this problem. I remember writing comparisons where I 
wanted a value of "10.203" correct to 3 decimal places as 

	if abs(x -10.203) < 0.00001



On Monday, October 25, 2010 12:15:49 pm Sairam Krishnamurthy wrote:
> All,
> 
> I have simple query like 'select * from table1 where
> column1=-107.6898780000'. This returns an empty set. But there is data
> corresponding to this value of column.
> 
> When I looked more into it, it seems like a precision issue. The value
> for column1 is -107.689878.
> 
> More interesting is that the following query fetches the row,
> 
> 'select * from table1 where column1=-107.689878000'
> 
> Note that there are only three trailing zeros in the second query while
> there were four in the first.
> 
> Can somebody help me to find out the problem? I can very well truncate
> the trailing zeros when querying, but I am interested in finding why an
> additional trailing zero returns an empty set.
Thread
Query precision issueSairam Krishnamurthy25 Oct
  • Re: Query precision issueCarlos Proal25 Oct
  • Re: Query precision issueDan Nelson25 Oct
    • Re: Query precision issueSairam Krishnamurthy25 Oct
  • Re: Query precision issueJoerg Bruehe25 Oct
  • Re: Query precision issueMichael Satterwhite26 Oct