Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <blockquote> <p>we get exactly 0.012</p> </blockquote> <p>No you don't. Neither <code>float</code> nor <code>double</code> can represent 3/250 exactly. What you do get is a value that is rendered by the string formatter <code>Double.ToString()</code> as <code>"0.012"</code>. But this happens because the formatter doesn't display the exact value.</p> <p>Going through <code>decimal</code> is causing rounding. It is likely much faster (not to mention easier to understand) to just use <code>Math.Round</code> with the rounding parameters you want. If what you care about is the number of significant digits, see:</p> <ul> <li><a href="https://stackoverflow.com/questions/374316/round-a-double-to-x-significant-figures">Round a double to x significant figures</a></li> </ul> <hr> <p>For what it's worth, <code>0.012f</code> (which means the 32-bit IEEE-754 value nearest to 0.012) is exactly</p> <pre><code>0x3C449BA6 </code></pre> <p>or</p> <pre><code>0.012000000104308128 </code></pre> <p>and this is <em>exactly</em> representable as a <code>System.Decimal</code>. But <code>Convert.ToDecimal(0.012f)</code> won't give you that exact value -- per <a href="http://msdn.microsoft.com/en-us/library/he38a8ca.aspx" rel="nofollow noreferrer">the documentation there is a rounding step</a>.</p> <blockquote> <p>The <code>Decimal</code> value returned by this method contains a maximum of seven significant digits. If the value parameter contains more than seven significant digits, it is rounded using rounding to nearest. </p> </blockquote>
    singulars
    1. This table or related slice is empty.
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. This table or related slice is empty.
    2. VO
      singulars
      1. This table or related slice is empty.
    3. VO
      singulars
      1. This table or related slice is empty.
 

Querying!

 
Guidance

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload