Databases
StackOverflow2013
Postsdbo
POwhat parallel algorithms exist in R, working on large data
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POwhat parallel algorithms exist in R, working on large data
    primarykey
    data
    text
    <p>I'm trying to find out which statistical/data mining algorithms in R or R packages at CRAN/github/R-Forge exist that can handle large datasets either in parallel on 1 server or sequentially without running into out-of-memory issues or which work on several machines at once. This in order to evaluate if I can easily port them to work with ff/ffbase like ffbase::bigglm.ffdf.</p> <p>I would like to split these up into 3 parts:</p> <ol> <li><p>Algorithms that update or work on parameter estimates in parallel</p> <ul> <li><p>Buckshot (<a href="https://github.com/lianos/buckshot" rel="nofollow">https://github.com/lianos/buckshot</a>)</p></li> <li><p>lm.fit @ Programming For Big Data (<a href="https://github.com/RBigData" rel="nofollow">https://github.com/RBigData</a>)</p></li> </ul></li> <li><p>Algorithms that work sequentially (get data in R but only use 1 process and only 1 process updates the parameters)</p> <ul> <li><p>bigglm (<a href="http://cran.r-project.org/web/packages/biglm/index.html" rel="nofollow">http://cran.r-project.org/web/packages/biglm/index.html</a>)</p></li> <li><p>Compound Poisson linear models (<a href="http://cran.r-project.org/web/packages/cplm/index.html" rel="nofollow">http://cran.r-project.org/web/packages/cplm/index.html</a>)</p></li> <li><p>Kmeans @ biganalytics (<a href="http://cran.r-project.org/web/packages/biganalytics/index.html" rel="nofollow">http://cran.r-project.org/web/packages/biganalytics/index.html</a>)</p></li> </ul></li> <li><p>Work on part of the data</p> <ul> <li>Distributed text processing (<a href="http://www.jstatsoft.org/v51/i05/paper" rel="nofollow">http://www.jstatsoft.org/v51/i05/paper</a>)</li> </ul></li> </ol> <p>And I would like to exclude simple parallelisation like optimising over a hyperparameter by e.g. crossvalidating. Any other pointers to these kind of models/optimisers or algorithms? Maybe Bayesian? Maybe a package called RGraphlab (http://graphlab.org/)?</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USjwijffels
    1. USjwijffels
    plurals
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTLinked
    3. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POwhat parallel algorithms exist in R, working on large data
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POwhat parallel algorithms exist in R, working on large data
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POwhat parallel algorithms exist in R, working on large data
      1. This table or related slice is empty.
      1. VTUpMod
    1. COnobody ever got fired for using hadoop
      singulars
      1. POwhat parallel algorithms exist in R, working on large data
      1. USlynks
    2. COnot sure how monetdb handles multiple processors, but it certainly works fast on big data and is worth a look :) -- http://usgsd.blogspot.com/2012/11/why-and-how-to-install-monetdb-with-r.html
      singulars
      1. POwhat parallel algorithms exist in R, working on large data
      1. USAnthony Damico
    3. COThx but I am more looking for an algorithm, not a data storage.
      singulars
      1. POwhat parallel algorithms exist in R, working on large data
      1. USjwijffels
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload