StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p><strong>Edit: modified, faster version</strong></p> <p><strong>Notes:</strong> This is actually LESS concise, but should run significantly faster. To run on an iterator you call the static methodBulkCalcRunner.runBulkCalc(Iterator,Calc) or specify a number of threads. Clean, fairly concise, and probably the fastest solution you can get. </p> <h2>Reasons this is faster:</h2> <ul> <li>Results are collected in a thread-local HashMap -- no synchronization required for collecting them. Otherwise, synchronization would be required for storing each result. This improves scaling per-thread, and provides better locality of reference (your HashMaps can live entirely in L2 cache for each processor, no need to communicate).</li> <li>HashMap used in place of less efficient Map collections</li> <li><p>Errors are bundled up into a collection for later handling. With a thread pool, every Exception requires a thread to die and be recreated</p> <p>interface BigThing { int getId(); String getString(); }</p> <pre><code>class Calc { // somewhat expensive computation double calc(BigThing bigThing) { Random r = new Random(bigThing.getString().hashCode()); double d = 0; for (int i = 0; i < 100000; i++) { d += r.nextDouble(); } return d; } } static class BulkCalcRunner implements Runnable { Calc calc; CountDownLatch latch; Iterator<BigThing> it; Collection<Throwable> errors; Map<Integer,Double> results; public BulkCalcRunner (Calc calc, Iterator<BigThing> it, CountDownLatch latch, Map<Integer,Double> results, Collection<Throwable> errors) { this.calc = calc; this.latch = latch; this.errors = errors; this.results = results; } public void run() { ArrayList<Throwable> errorLocal = new ArrayList<Throwable>(); HashMap<Integer,Double> resultsLocal = new HashMap<Integer,Double>(); while (true) { BigThing thing = null; try { synchronized (it) { if (it.hasNext()) { thing = it.next(); } } } catch (Exception e) { //prevents iterator errors from causing endless loop thing = null; } //finished when first null BigThing encountered if (thing == null) { synchronized (errors) { errors.addAll(errorLocal); } synchronized(results) { results.putAll(resultsLocal); } latch.countDown(); break; } try { resultsLocal.put(thing.getId(), calc.calc(thing)); } catch (Exception e) { errorLocal.add(e); } } } public static Map<Integer,Double> runBulkCalc(Iterator<BigThing> iterator, Calc calculation, int numThreads) { final ConcurrentHashMap<Integer, Double> results = new ConcurrentHashMap<Integer, Double>(); final ArrayList<Throwable> errors = new ArrayList<Throwable>(); final CountDownLatch latch = new CountDownLatch(numThreads); //start up the worker threads for (int i = 0; i < numThreads; i++) { new Thread(new BulkCalcRunner(calculation,iterator,latch, results, errors)).start(); } try { //Latch waits for all the worker threads to check in as "done" latch.await(); } catch (InterruptedException ex) { // swallowing is better than spitting it out... } //finally, propagate errors! for (Throwable th : errors) { throw Throwables.propagate(th.getCause()); } return results; } public static Map<Integer,Double> runBulkCalc(Iterator<BigThing> iterator, Calc calculation) { return runBulkCalc(iterator,calculation,Runtime.getRuntime().availableProcessors()); } } </code></pre></li> </ul>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload