StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POIncorrect division results
text
Body
copied!<p>I've got a time calculator that has worked reasonably well for a number of years. One thing that always bothered me, though, was that if one used fractional seconds, the results would fall victim to floating-point "errors". So, I recently switched to using <a href="https://github.com/dtrebbien/BigDecimal.js" rel="nofollow">this BigDecimal library</a>.</p> <p>Now, I'm getting math errors. Here's a simplified test case from a bug report I received today: <code>27436 / 30418</code> is returning <code>1</code> instead of the expected <code>0.9019659412190151</code>.</p> <p>To illustrate my problem, here's a Javascript console session in Chrome:</p> <pre><code>> first = 27436 27436 > second = 30418 30418 > first / second 0.9019659412190151 // expected result, but using JS numbers, not BigDecimals > firstB = new BigDecimal(first.toString()) o > secondB = new BigDecimal(second.toString()) o > firstB / secondB 0.9019659412190151 // this is a JS number, not a BigDecimal, so it's susceptible to the problems associated with floating-point. > firstB.divide(secondB) o // BigDecimal object > firstB.divide(secondB).toString() "1" // huh? Why 1? > firstB.divideInteger(secondB).toString() "0" </code></pre> <p>As you can see, the <code>divide()</code> method isn't producing the result I expect. What do I need to do differently?</p> <h1>Update</h1> <p>Here are some more details, in response to the comments.</p> <p>First, several people suggested that using BigDecimal was overkill. That might be, but I think that more details are necessary before making that decision. This app is a <a href="http://www.scottseverance.us/html/time_calculator.htm" rel="nofollow">time calculator</a>, so there are a couple of things that pushed me to switch to BigDecimal. First, because this is a calculator, it's important to show the user a correct answer. If the user enters <code>0.1 s + 0.2 s</code>, they expect the answer to be <code>0.3 s</code>, not the answer that Javascript will show them (<code>0.30000000000000004</code>).</p> <p>I don't really want to limit the precision beyond what I can use in JS so that I can use integers, because I don't know the maximum precision my users need. Most never use fractional seconds, I think, but judging from the email I've received, some do. I'm currently storing all times internally as seconds.</p> <p>Someone suggested that I store the numbers as exact fractions. Unfortunately, I don't know what that means. Perhaps it's because I don't know too much about math. I don't know enough to roll my own math library; that's why I'm using BigDecimal. It's been around for a long time, so I'm hesitant to say that my problem is due to a bug in BigDecimal. I suspect rather that it's a bug in the way I'm using it.</p> <p>Finally, I'm not wedded to BigDecimal specifically. I'm open to other suggestions, provided that I can use them with my deficient math skills.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload