Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>That's the usual problem of binary floating point types: <code>0.005</code> isn't exactly representable by a <code>double</code>, so the conversion from the "decimal literal" yields the most similar number representable by a <code>double</code>.</p> <p>There's no solution if you want to use <code>double</code> or other finite, binary floating point types, since <code>0.005</code> has a finite form in decimal, but <a href="http://www.wolframalpha.com/input/?i=0.005%20in%20binary" rel="nofollow">is periodic in binary</a> (and the mantissa of any FP type is limited).</p> <p>To see how is this possible, consider the fraction 1/3: in base 3 it has a finite representation, while in base 10 it's periodic:</p> <p>(1/3)<sub>10</sub> = 0.33333...<sub>10</sub> = 0.1<sub>3</sub></p> <p>now, any "regular" floating point type has only some limited space for the mantissa (i.e. the significant digits to store), so every periodic number has to be truncated (actually rounded) at some point; this results in the loss of precision you see.</p> <p>Notice that several "arbitrary precision libraries" are available to deal with this kind of problems and do math with precision limited only by the available memory (e.g. <a href="http://gmplib.org/" rel="nofollow">GMP</a>), but you should consider if the additional complexity/loss in performance is actually justified by the gain in precision.</p> <p>Another solution (commonly used when doing calculations that involve money) is to use fixed-precision types, that store the amount as an integer "implicitly scaled" by some decimal factor (the concept is that you store cents as integers instead of dollars as <code>float</code> or whatever); this avoids any surprise deriving from the decimal to binary conversion, since all integers are exactly representable with a limited number of digits in any base.</p>
    singulars
    1. This table or related slice is empty.
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. This table or related slice is empty.
    2. VO
      singulars
      1. This table or related slice is empty.
    3. VO
      singulars
      1. This table or related slice is empty.
    1. This table or related slice is empty.
 

Querying!

 
Guidance

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload