Databases
StackOverflow2013
Postsdbo
POWhy is mclappy slower than apply in this case?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POWhy is mclappy slower than apply in this case?
    primarykey
    data
    text
    <p>i'm pretty confused. I want to speed up my algorithm by using <strong>mclapply:parallel</strong>, but when I compare time efficiency, apply still wins.</p> <p>I'm smoothing log2ratio data by <strong>rq.fit.fnb:quantreg</strong> which is called by my function quantsm and I'm wrapping my data into matrix/list for <strong>apply/lapply(mclapply)</strong> usage.</p> <p>I adjist my data like this:</p> <pre><code>q = matrix(data, ncol=N) # wrapping into matrix (using N = 2, 4, 6 or 8) ql = as.list(as.data.frame(q)) # making list </code></pre> <p>And time comparing:</p> <pre><code>apply=system.time(apply(q, 1, FUN=quantsm, 0.50, 2)) lapply=system.time(lapply(ql, FUN=quantsm, 0.50, 2)) mc2lapply=system.time(mclapply(ql, FUN=quantsm, 0.50, 2, mc.cores=2)) mc4lapply=system.time(mclapply(ql, FUN=quantsm, 0.50, 2, mc.cores=4)) mc6lapply=system.time(mclapply(ql, FUN=quantsm, 0.50, 2, mc.cores=6)) mc8lapply=system.time(mclapply(ql, FUN=quantsm, 0.50, 2, mc.cores=8)) timing=rbind(apply,lapply,mc2lapply,mc4lapply,mc6lapply,mc8lapply) </code></pre> <p>Function <strong>quantsm</strong>:</p> <pre><code>quantsm &lt;- function (y, p = 0.5, lambda) { # Quantile smoothing # Input: response y, quantile level p (0&lt;p&lt;1), smoothing parmeter lambda # Result: quantile curve # Augment the data for the difference penalty m &lt;- length(y) E &lt;- diag(m); Dmat &lt;- diff(E); X &lt;- rbind(E, lambda * Dmat) u &lt;- c(y, rep(0, m - 1)) # Call quantile regression q &lt;- rq.fit.fnb(X, u, tau = p) q } </code></pre> <p>Function <strong>rq.fit.fnb</strong> (quantreg library):</p> <pre><code>rq.fit.fnb &lt;- function (x, y, tau = 0.5, beta = 0.99995, eps = 1e-06) { n &lt;- length(y) p &lt;- ncol(x) if (n != nrow(x)) stop("x and y don't match n") if (tau &lt; eps || tau &gt; 1 - eps) stop("No parametric Frisch-Newton method. Set tau in (0,1)") rhs &lt;- (1 - tau) * apply(x, 2, sum) d &lt;- rep(1, n) u &lt;- rep(1, n) wn &lt;- rep(0, 10 * n) wn[1:n] &lt;- (1 - tau) z &lt;- .Fortran("rqfnb", as.integer(n), as.integer(p), a = as.double(t(as.matrix(x))), c = as.double(-y), rhs = as.double(rhs), d = as.double(d), as.double(u), beta = as.double(beta), eps = as.double(eps), wn = as.double(wn), wp = double((p + 3) * p), it.count = integer(3), info = integer(1), PACKAGE = "quantreg") coefficients &lt;- -z$wp[1:p] names(coefficients) &lt;- dimnames(x)[[2]] residuals &lt;- y - x %*% coefficients list(coefficients = coefficients, tau = tau, residuals = residuals) } </code></pre> <p>For data vector of length 2000 i get:</p> <p>(value = elapsed time in sec; columns = different number of columns of smoothed matrix/list)</p> <pre><code> 2cols 4cols 6cols 8cols apply 0.178 0.096 0.069 0.056 lapply 16.555 4.299 1.785 0.972 mc2lapply 11.192 2.089 0.927 0.545 mc4lapply 10.649 1.326 0.694 0.396 mc6lapply 11.271 1.384 0.528 0.320 mc8lapply 10.133 1.390 0.560 0.260 </code></pre> <p>For data of length 4000 i get:</p> <pre><code> 2cols 4cols 6cols 8cols apply 0.351 0.187 0.137 0.110 lapply 189.339 32.654 14.544 8.674 mc2lapply 186.047 20.791 7.261 4.231 mc4lapply 185.382 30.286 5.767 2.397 mc6lapply 184.048 30.170 8.059 2.865 mc8lapply 182.611 37.617 7.408 2.842 </code></pre> <p>Why is apply so much more efficient than mclapply? Maybe I'm just doing some usual beginner mistake.</p> <p>Thank you for your reactions.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USuser1991825
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. COShouldn't your `apply` call use `MARGIN = 2`, not `1`?
      singulars
      1. POWhy is mclappy slower than apply in this case?
      1. USflodel
    2. COOf course! Thank you both (@flodel and @SteveWeston). I really overlooked that.
      singulars
      1. POWhy is mclappy slower than apply in this case?
      1. USuser1991825
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload