Databases
StackOverflow2013
Postsdbo
POHow to simulate Single precision rounding with Doubles?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHow to simulate Single precision rounding with Doubles?
    primarykey
    data
    text
    <p>i had a problem where i was trying to reconstruct the the formula used in an existing system, a fairly simple formula of one input and one output:</p> <pre><code>y = f(x) </code></pre> <p>After a lot of puzzling, <a href="http://sacredduty.net/2012/09/14/avoidance-diminishing-returns-in-mop-followup/" rel="nofollow noreferrer">we</a> managed to figure out the formula that fit our observed data points:</p> <p><img src="https://i.stack.imgur.com/Mt2R3.png" alt="enter image description here"></p> <p>And as you can see our theoretical model fit observed data very well:</p> <p><img src="https://i.stack.imgur.com/56kcQ.png" alt="enter image description here"></p> <p>Except when we plot residual errors (i.e. <code>y = f(x) - actualY</code>), we see some lines appear in the residuals:</p> <p><img src="https://i.stack.imgur.com/i4kpc.png" alt="enter image description here"></p> <p>It was obvious that these lines were the result of applying some intermediate rounding in our formula, but it was not obvious <em>where</em>. Eventually it was realized that the <strong>original</strong> system (the one we're trying to reverse engineer) is storing values in an intermediate <strong><code>Decimal</code></strong> data type:</p> <ul> <li>with <strong>8-bit precision</strong> of the fraction</li> <li>using the <strong>0.5 round-up</strong> rounding model:</li> </ul> <p>We could <em>simulate</em> this 8-bit precision in the fraction by:</p> <pre><code>multiply by 128 (i.e. 2^8) apply the round divide by 128 (i.e. 2^8) </code></pre> <p>Changing our equation above into:</p> <p><img src="https://i.stack.imgur.com/Dgr5e.png" alt="enter image description here"></p> <p>This reduces the residual errors <em>significantly</em>:</p> <p><img src="https://i.stack.imgur.com/eBTp0.png" alt="enter image description here"></p> <p>Now, all of that above has no relevance to my question except:</p> <ol> <li>To show that simulating the numerical representation in the computer can help the model</li> <li>To get people's attention with pretty pictures and colors </li> <li>Silence critics who <a href="https://stackoverflow.com/questions/11972829/how-to-call-use-net-rijndaelmanaged-from-native-com">would</a> <a href="https://stackoverflow.com/questions/12323303/how-to-inherit-from-control-rather-than-usercontrol">refuse</a> to <a href="https://stackoverflow.com/questions/2224220/win32-how-to-make-drop-shadow-honor-non-rectangular-layered-window">contribute</a> until i <a href="https://stackoverflow.com/questions/7111618/win32-how-to-validate-credentials-against-active-directory">explain</a> <em>why</em> i'm asking my question</li> </ol> <hr> <p>Now i want to simulate <strong><code>Single Precision</code></strong> floating point numbers, inside a programming language (and Excel) which use <strong><code>Double Precision</code></strong> floating point numbers. i want to do this because <em>i <strong>think</em></strong> it is what's needed.</p> <p>In the above example i <strong>thought</strong> the original system was using a <em><code>Decimal data type with fixed 8-bit fractional precision using 0.5 round-up rules</code></em>. i then had to find a way to simulate that computation model with <code>Double</code> math. Now i <em>think</em> the original system is using <code>Single</code> precision math, that i want to simulate using <code>Double</code>.</p> <blockquote> <p>How do i simulate single-precision rounding using doubles?</p> </blockquote> <p>In my current model, i once again have residuals that fall into the regular linear patterns - that are a tell-tale sign of rounding:</p> <p><img src="https://i.stack.imgur.com/51rm1.png" alt="enter image description here"></p> <p>The problem is that the error becomes larger, and only visible, as my input variables become larger. i realized this is likely caused by the fact that all floating point numbers are normalized into <a href="http://en.wikipedia.org/wiki/IEEE_floating_point" rel="nofollow noreferrer">IEEE 754</a> "scientific notation".</p> <p>And even if i'm wrong, i still want to try it.</p> <p>And even if i don't want to trying it, i'm still asking the question</p> <blockquote> <p>How do i simulate <code>Single</code> precision rounding using <code>Doubles</code>?</p> </blockquote> <hr> <p>It seems to me i could still apply the concept of <em>"rounding after 8 fractional bits"</em> (although 24 bits for <code>Single</code> precision floating point), as long as i can first <em>"normalize"</em> the value. e.g.</p> <pre><code>1234567898.76543 </code></pre> <p>needs to be converted into (something similar to):</p> <pre><code>1.23456789876543 E-09 </code></pre> <p>Then i could apply my "round to the 24th bit" (i.e. 2^24 = 16,777,216)</p> <pre><code>floor(1.23456789876543E-09 * 16777216 + 0.5) / 16777216; </code></pre> <p>The problem, then, is what combination of <code>sign</code>, <code>abs</code>, <code>ln</code>, <code>exp</code> (or other functions) can i possible apply so that i can "normalize" my value, round it to the n-th binary place, then "denormalize" it?</p> <p><strong>Note</strong>: i realize IEEE representation keeps a binary <code>1</code> as the most significant bit. i might not need to duplicate that behavior in order to get correct results. So it's not a deal-breaker, nor is it cause to suggest that the entire approach is a failure.</p> <h1>See also</h1> <ul> <li><a href="https://stackoverflow.com/questions/8130688/how-to-emulate-single-precision-float-operations-in-php">How to emulate single precision float operations in PHP?</a></li> <li><a href="http://www.mathworks.cn/matlabcentral/answers/35085-simulate-single-precision-arithmetic-in-matlab" rel="nofollow noreferrer">Simulate single precision arithmetic in Matlab?</a></li> <li><a href="http://en.wikipedia.org/wiki/IEEE_floating_point" rel="nofollow noreferrer">IEEE 754 Floating Point</a></li> </ul>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USIan Boyd
    plurals
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTLinked
    3. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POHow to simulate Single precision rounding with Doubles?
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POHow to simulate Single precision rounding with Doubles?
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POHow to simulate Single precision rounding with Doubles?
      1. This table or related slice is empty.
      1. VTUpMod
    1. COAre you looking for generic answers or will language-specific answers do?
      singulars
      1. POHow to simulate Single precision rounding with Doubles?
      1. USNeil
    2. CO@Neil Ideally it will work in Excel and in Lua. As a bonus it will work in C# Double, Dephi Real, and Javascript number.
      singulars
      1. POHow to simulate Single precision rounding with Doubles?
      1. USIan Boyd
    3. COI'm not smart enough to develop a working solution but couldn't you solve this by treating the double as a binary value and set the training (53-23) bits of the mantissa to zero? Rounding would probably be more complicated, but doable.
      singulars
      1. POHow to simulate Single precision rounding with Doubles?
      1. USusr
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload