StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>The issue that you raise was the motivation for the foreach nesting operator, '%:%'. If the body of the inner loop takes a substantial amount of compute time, you're pretty safe using:</p> <pre><code>foreach(i = 1:I) %:% foreach(j = 1:J) %dopar% { # Do stuff } </code></pre> <p>This "unrolls" the nested loops, resulting in (I * J) tasks that can all be executed in parallel.</p> <p>If the body of the inner loop doesn't take much time, the solution is more difficult. The standard solution is to parallelize the outer loop, but that could still result in either many small tasks (when I is large and J is small) or a few large tasks (when I is small and J is large).</p> <p>My favorite solution is to use the nesting operator with task chunking. Here's a complete example using the doMPI backend:</p> <pre><code>library(doMPI) cl <- startMPIcluster() registerDoMPI(cl) I <- 100; J <- 2 opt <- list(chunkSize=10) foreach(i = 1:I, .combine='cbind', .options.mpi=opt) %:% foreach(j = 1:J, .combine='c') %dopar% { (i * j) } closeCluster(cl) </code></pre> <p>This results in 20 "task chunks", each consisting of 10 computations of the loop body. If you want to have a single task chunk for each worker, you can compute the chunk size as:</p> <pre><code>cs <- ceiling((I * J) / getDoParWorkers()) opt <- list(chunkSize=cs) </code></pre> <p>Unfortunately, not all parallel backends support task chunking. Also, doMPI doesn't support Windows.</p> <p>For more information on this topic, see my vignette "Nesting Foreach Loops" in the foreach package:</p> <pre><code>library(foreach) vignette('nesting') </code></pre>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload