List:General Discussion« Previous MessageNext Message »
From:Matt Loschert Date:April 14 2003 3:19pm
Subject:Re: rounding/truncation of decimal fields
View as plain text  
On Sun, 13 Apr 2003, Dan Nelson wrote:

> In the last episode (Apr 13), Matt Loschert said:
> > That may be the case, but I would assume (again, I guess incorrectly)
> > that on a given architecture, the following two values would be
> > treated identically.  Is quoting really supposed to change the
> > representation of the number?
>
> Most definitely.
>
> > CREATE TABLE test1 (field1 decimal(12,2));
> > INSERT INTO test1 VALUES ('0.005');
> > INSERT INTO test1 VALUES ( 0.005 );
> > SELECT * FROM test1;
> > +--------+
> > | field1 |
> > +--------+
> > |   0.00 |
> > |   0.01 |
> > +--------+
> > 2 rows in set (0.02 sec)
>
> The first is inserting a string value, which is probably just copied
> directly into the DECIMAL field (with truncation).  The second is
> inserting a floating-point value, which has to have been parsed from
> the original ASCII SQL statement into a float, then converted back into
> ASCII to store into the DECIMAL field.
>
> --
> 	Dan Nelson
> 	dnelson@stripped

Okay, light bulb time.  I get it.

I was assuming that everything was going through a string to float
conversion, and that therefore, everything would see the same float
idiosyncrasies.  However, since the decimal type uses string-based storage
and manipulation, the string form that I was inserting probably never went
through atof() conversion (or something similar).

Thank you to everyone for the information.

- Matt

--
Matt Loschert - Software Engineer       | email: loschert@stripped    |
ServInt Internet Services               | web:   http://www.servint.net/ |
McLean, Virginia USA                    | phone: (703) 847-1381          |
Thread
rounding/truncation of decimal fieldsMatt Loschert10 Apr
  • Re: rounding/truncation of decimal fieldsgerald_clark11 Apr
RE: rounding/truncation of decimal fieldsClifton R [CC] Jones11 Apr
  • Re: rounding/truncation of decimal fieldsgerald_clark11 Apr
    • Re: rounding/truncation of decimal fieldsMatt Loschert11 Apr
      • Re: rounding/truncation of decimal fieldsDoug Thompson11 Apr
        • Re: rounding/truncation of decimal fieldsMatt Loschert12 Apr
          • Re: rounding/truncation of decimal fieldsDoug Thompson12 Apr
            • More on enumMartin Gainty12 Apr
            • Re: rounding/truncation of decimal fieldsMatt Loschert13 Apr
              • Re: rounding/truncation of decimal fieldsDoug Thompson13 Apr
              • Re: rounding/truncation of decimal fieldsDan Nelson13 Apr
                • Re: rounding/truncation of decimal fieldsMatt Loschert14 Apr
Re: rounding/truncation of decimal fieldsMoon79611 Apr