StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>I wrote <a href="http://benchmarkjs.com" rel="noreferrer">Benchmark.js</a>, which <a href="http://jsperf.com/browse" rel="noreferrer">jsPerf</a> uses.</p> <ol> <li><p>"<code>ops/sec</code>" stands for operations per second. That is how many times a test is projected to execute in a second.</p></li> <li><p>A test is repeatedly executed until it reaches the minimum time needed to get a <a href="http://spiff.rit.edu/classes/phys273/uncert/uncert.html" rel="noreferrer">percentage uncertainty</a> for the measurement of less than or equal to <code>1%</code>. The number of iterations will vary depending on the resolution of the environment’s timer and how many times a test can execute in the minimum run time. We collect completed test runs for <code>5</code> seconds <em>(configurable)</em>, or at least <code>5</code> runs <em>(also configurable)</em>, and then perform statistical analysis on the sample. So, a test may be repeated <code>100,000</code> times in <code>50 ms</code> <em>(the minimum run time for most environments)</em>, and then repeated <code>100</code> times more <em>(<code>5</code> seconds)</em>. A larger sample size <em>(in this example, <code>100</code>)</em>, leads to a smaller <a href="http://en.wikipedia.org/wiki/Margin_of_error" rel="noreferrer">margin of error</a>.</p></li> <li><p>We base the decision of which test is faster on more than just ops/sec by also accounting for margin of error. For example, a test with a lower ops/sec but higher margin of error may be statistically <em>indistinguishable</em> from a test with higher ops/sec and lower margin of error.</p> <p>We used a <a href="http://en.wikipedia.org/wiki/Welch%27s_t_test" rel="noreferrer">welch t-test</a>, similar to what <a href="http://www.webkit.org/perf/sunspider/sunspider.html" rel="noreferrer">SunSpider</a> uses, but switched to an <a href="https://github.com/mathiasbynens/Benchmark.js/blob/d4e874f2c0956d11bd1187870c2655eb3a3ab692/benchmark.js#L1175-1194" rel="noreferrer">unpaired 2-sample t-test</a> for equal variance <em>(the variance is extremely small)</em> because the welch t-test had problems comparing lower ops/sec and higher ops/sec with small variances which caused the <a href="http://en.wikipedia.org/wiki/Degrees_of_freedom_(statistics)" rel="noreferrer">degrees of freedom</a> to be computed as less than <code>1</code>. We also add a <code>5.5%</code> allowance on tests with similar ops/sec because real world testing showed that identical tests can swing ~<code>5%</code> from test to re-test. T-tests are used to check that differences between tests are <a href="http://en.wikipedia.org/wiki/Statistical_significance" rel="noreferrer">statistically significant</a>.</p></li> </ol>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload