StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POc++ floating point precision loss: 3015/0.00025298219406977296
text
Body
copied!<p>The problem.</p> <p>Microsoft Visual C++ 2005 compiler, 32bit windows xp sp3, amd 64 x2 cpu.</p> <p>Code:</p> <pre><code>double a = 3015.0; double b = 0.00025298219406977296; //*((unsigned __int64*)(&a)) == 0x40a78e0000000000 //*((unsigned __int64*)(&b)) == 0x3f30945640000000 double f = a/b;//3015/0.00025298219406977296; </code></pre> <p>the result of calculation (i.e. "f") is 11917835.000000000 (<em>((unsigned __int64</em>)(&f)) == 0x4166bb4160000000) although it should be 11917834.814763514 (i.e. <em>((unsigned __int64</em>)(&f)) == 0x4166bb415a128aef).<br> I.e. fractional part is lost.<br> Unfortunately, I need fractional part to be correct.</p> <p>Questions:<br> 1) Why does this happen?<br> 2) How can I fix the problem?</p> <p>Additional info:<br> 0) The result is taken <em>directly</em> from "watch" window (it wasn't printed, and I didn't forget to set printing precision). I also provided hex dump of floating point variable, so I'm absolutely sure about calculation result.<br> 1) The disassembly of f = a/b is: </p> <pre><code>fld qword ptr [a] fdiv qword ptr [b] fstp qword ptr [f] </code></pre> <p>2) f = 3015/0.00025298219406977296; yields correct result (f == 11917834.814763514 , <em>((unsigned __int64</em>)(&f)) == 0x4166bb415a128aef ), but it looks like in this case result is simply calculated during compile-time:</p> <pre><code>fld qword ptr [__real@4166bb415a128aef (828EA0h)] fstp qword ptr [f] </code></pre> <p>So, how can I fix this problem? </p> <p>P.S. I've found a temporary workaround (i need only fractional part of division, so I simply use f = fmod(a/b)/b at the moment), but I still would like to know how to fix this problem properly - double precision is supposed to be 16 decimal digits, so such calculation isn't supposed to cause problems.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload