List:General Discussion« Previous MessageNext Message »
From:Doug Thompson Date:April 13 2003 5:05am
Subject:Re: rounding/truncation of decimal fields
View as plain text  

FP number inaccuracy is an old and well-worn topic and the answers haven't changed through
all the resurrections.

The conversion from string literal can and probably will result in different values from a
numeric entry that has the same representation on your screen.  This matter affects all
software, not just dbms's and certainly not only MySQL.  Format your numeric display to
maximum decimal places and you should see the small variations that are the result of the
conversion and display processes, similar to the variations of the values in the
explanation in the manual.

Inaccuracy is the nature of the beast as far as floating point numbers are concerned. 
There really is nothing more to be said.

Doug

Yes.

This is an old and well-worn topic and the answers haven't changed through all the
resurrections.

The conversion from string literal can and probably will result in different values from a
numeric entry that has the same representation on your screen.  This matter affects all
software, not just dbms's and certainly not only MySQL.

Inaccuracy is the nature of the beast as far as floating point numbers are concerned. 
There really is nothing more to be said.

Doug

"I now inform you that you are too far from reality." - Mohammed Saeed al-Sahaf


On Sun, 13 Apr 2003 00:00:25 -0400 (EDT), Matt Loschert wrote:

>On Sat, 12 Apr 2003, Doug Thompson wrote:
>
>> On Sat, 12 Apr 2003 10:41:01 -0400 (EDT), Matt Loschert wrote:
>>
>> >Interesting.  I had never seen this page of the manual before.  However,
>> >you appear to be correct.  I don't see an applicable reference there
>> >either.
>> >
>> >I think I am going to re-post my message to the bugs list since no one has
>> >shown me yet that this is definitely expected behavior.
>> >
>>
>> Huh?!?
>>
>> Here are the first two sentences from Appendix A.5.7.
>>
>> <quote>
>> floating-point numbers cause confusion sometimes, because these numbers
>> are not stored as exact values inside computer architecture. What one
>> can see on the screen usually is not the exact value of the number.
>>
>> Field types FLOAT, DOUBLE and DECIMAL are such.
>> <end quote>
>>
>> Seems to me that's rather explicit that what you are seeing is expected
> behavior.
>
>That may be the case, but I would assume (again, I guess incorrectly) that
>on a given architecture, the following two values would be treated
>identically.  Is quoting really supposed to change the representation of
>the number?
>
>CREATE TABLE test1 (field1 decimal(12,2));
>INSERT INTO test1 VALUES ('0.005');
>INSERT INTO test1 VALUES ( 0.005 );
>SELECT * FROM test1;
>+--------+
>| field1 |
>+--------+
>|   0.00 |
>|   0.01 |
>+--------+
>2 rows in set (0.02 sec)
>
>- Matt
>
>--
>Matt Loschert - Software Engineer       | email: loschert@stripped    |
>ServInt Internet Services               | web:   http://www.servint.net/ |
>McLean, Virginia USA                    | phone: (703) 847-1381          |
>
>-- 
>MySQL General Mailing List
>For list archives: http://lists.mysql.com/mysql
>To unsubscribe:    http://lists.mysql.com/mysql?unsub=1
>


Thread
rounding/truncation of decimal fieldsMatt Loschert10 Apr
  • Re: rounding/truncation of decimal fieldsgerald_clark11 Apr
RE: rounding/truncation of decimal fieldsClifton R [CC] Jones11 Apr
  • Re: rounding/truncation of decimal fieldsgerald_clark11 Apr
    • Re: rounding/truncation of decimal fieldsMatt Loschert11 Apr
      • Re: rounding/truncation of decimal fieldsDoug Thompson11 Apr
        • Re: rounding/truncation of decimal fieldsMatt Loschert12 Apr
          • Re: rounding/truncation of decimal fieldsDoug Thompson12 Apr
            • More on enumMartin Gainty12 Apr
            • Re: rounding/truncation of decimal fieldsMatt Loschert13 Apr
              • Re: rounding/truncation of decimal fieldsDoug Thompson13 Apr
              • Re: rounding/truncation of decimal fieldsDan Nelson13 Apr
                • Re: rounding/truncation of decimal fieldsMatt Loschert14 Apr
Re: rounding/truncation of decimal fieldsMoon79611 Apr