List: | General Discussion | « Previous MessageNext Message » | |

From: | Matt Loschert | Date: | April 14 2003 3:19pm |

Subject: | Re: rounding/truncation of decimal fields | ||

View as plain text |

On Sun, 13 Apr 2003, Dan Nelson wrote: > In the last episode (Apr 13), Matt Loschert said: > > That may be the case, but I would assume (again, I guess incorrectly) > > that on a given architecture, the following two values would be > > treated identically. Is quoting really supposed to change the > > representation of the number? > > Most definitely. > > > CREATE TABLE test1 (field1 decimal(12,2)); > > INSERT INTO test1 VALUES ('0.005'); > > INSERT INTO test1 VALUES ( 0.005 ); > > SELECT * FROM test1; > > +--------+ > > | field1 | > > +--------+ > > | 0.00 | > > | 0.01 | > > +--------+ > > 2 rows in set (0.02 sec) > > The first is inserting a string value, which is probably just copied > directly into the DECIMAL field (with truncation). The second is > inserting a floating-point value, which has to have been parsed from > the original ASCII SQL statement into a float, then converted back into > ASCII to store into the DECIMAL field. > > -- > Dan Nelson > dnelson@strippedOkay, light bulb time. I get it. I was assuming that everything was going through a string to float conversion, and that therefore, everything would see the same float idiosyncrasies. However, since the decimal type uses string-based storage and manipulation, the string form that I was inserting probably never went through atof() conversion (or something similar). Thank you to everyone for the information. - Matt -- Matt Loschert - Software Engineer | email: loschert@stripped| ServInt Internet Services | web: http://www.servint.net/ | McLean, Virginia USA | phone: (703) 847-1381 |

Thread | ||
---|---|---|

• rounding/truncation of decimal fields | Matt Loschert | 10 Apr |

• Re: rounding/truncation of decimal fields | gerald_clark | 11 Apr |

• RE: rounding/truncation of decimal fields | Clifton R [CC] Jones | 11 Apr |

• Re: rounding/truncation of decimal fields | gerald_clark | 11 Apr |

• Re: rounding/truncation of decimal fields | Matt Loschert | 11 Apr |

• Re: rounding/truncation of decimal fields | Doug Thompson | 11 Apr |

• Re: rounding/truncation of decimal fields | Matt Loschert | 12 Apr |

• Re: rounding/truncation of decimal fields | Doug Thompson | 12 Apr |

• More on enum | Martin Gainty | 12 Apr |

• Re: rounding/truncation of decimal fields | Matt Loschert | 13 Apr |

• Re: rounding/truncation of decimal fields | Doug Thompson | 13 Apr |

• Re: rounding/truncation of decimal fields | Dan Nelson | 13 Apr |

• Re: rounding/truncation of decimal fields | Matt Loschert | 14 Apr |

• Re: rounding/truncation of decimal fields | Moon796 | 11 Apr |