Databases
StackOverflow2013
Postsdbo
POparallel computation for an Iterator of elements in Java
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POparallel computation for an Iterator of elements in Java
    primarykey
    data
    text
    <p>I've had the same need a few times now and wanted to get other thoughts on the right way to structure a solution. The need is to perform some operation on many elements on many threads without needing to have all elements in memory at once, just the ones under computation. As in, <a href="http://google-collections.googlecode.com/svn/trunk/javadoc/com/google/common/collect/Iterables.html#partition(java.lang.Iterable,%20int)" rel="nofollow noreferrer">Iterables.partition</a> is insufficient because it brings all elements into memory up front.</p> <p>Expressing it in code, I want to write a BulkCalc2 that does the same thing as BulkCalc1, just in parallel. Below is sample code that illustrates my best attempt. I'm not satisfied because it's big and ugly, but it does seem to accomplish my goals of keeping threads highly utilized until the work is done, <a href="http://guava-libraries.googlecode.com/svn/trunk/javadoc/com/google/common/base/Throwables.html#propagate(java.lang.Throwable)" rel="nofollow noreferrer">propagating</a> any exceptions during computation, and not having more than <em>numThreads</em> instances of BigThing necessarily in memory at once.</p> <p>I'll accept the answer which meets the stated goals in the most concise way, whether it's a way to improve my BulkCalc2 or a completely different solution.</p> <pre><code>interface BigThing { int getId(); String getString(); } class Calc { // somewhat expensive computation double calc(BigThing bigThing) { Random r = new Random(bigThing.getString().hashCode()); double d = 0; for (int i = 0; i &lt; 100000; i++) { d += r.nextDouble(); } return d; } } class BulkCalc1 { final Calc calc; public BulkCalc1(Calc calc) { this.calc = calc; } public TreeMap&lt;Integer, Double&gt; calc(Iterator&lt;BigThing&gt; in) { TreeMap&lt;Integer, Double&gt; results = Maps.newTreeMap(); while (in.hasNext()) { BigThing o = in.next(); results.put(o.getId(), calc.calc(o)); } return results; } } class SafeIterator&lt;T&gt; { final Iterator&lt;T&gt; in; SafeIterator(Iterator&lt;T&gt; in) { this.in = in; } synchronized T nextOrNull() { if (in.hasNext()) { return in.next(); } return null; } } class BulkCalc2 { final Calc calc; final int numThreads; public BulkCalc2(Calc calc, int numThreads) { this.calc = calc; this.numThreads = numThreads; } public TreeMap&lt;Integer, Double&gt; calc(Iterator&lt;BigThing&gt; in) { ExecutorService e = Executors.newFixedThreadPool(numThreads); List&lt;Future&lt;?&gt;&gt; futures = Lists.newLinkedList(); final Map&lt;Integer, Double&gt; results = new MapMaker().concurrencyLevel(numThreads).makeMap(); final SafeIterator&lt;BigThing&gt; it = new SafeIterator&lt;BigThing&gt;(in); for (int i = 0; i &lt; numThreads; i++) { futures.add(e.submit(new Runnable() { @Override public void run() { while (true) { BigThing o = it.nextOrNull(); if (o == null) { return; } results.put(o.getId(), calc.calc(o)); } } })); } e.shutdown(); for (Future&lt;?&gt; future : futures) { try { future.get(); } catch (InterruptedException ex) { // swallowing is OK } catch (ExecutionException ex) { throw Throwables.propagate(ex.getCause()); } } return new TreeMap&lt;Integer, Double&gt;(results); } } </code></pre>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USBrian Harris
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POparallel computation for an Iterator of elements in Java
      1. USmaximdim
      1. VTFavorite
    2. VO
      singulars
      1. POparallel computation for an Iterator of elements in Java
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POparallel computation for an Iterator of elements in Java
      1. USfastcodejava
      1. VTFavorite
    1. COI'll tell you one thing: TreeMap is the slowest of the Java collections in most uses, and should be replaced with one of the others **whenever** possible. If I can find time I'll try to untangle this -- it's complex and I've had similar problems, so you have my sympathies. Good parallel coding is hard, even in Java.
      singulars
      1. POparallel computation for an Iterator of elements in Java
      1. USBobMcGee
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload