StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>Have a look at the "Large memory and out-of-memory data" subsection of the <a href="http://cran.r-project.org/web/views/HighPerformanceComputing.html">high performance computing task view</a> on CRAN. <a href="http://cran.r-project.org/web/packages/bigmemory/index.html">bigmemory</a> and <a href="http://cran.r-project.org/web/packages/ff/index.html">ff</a> are two popular packages. For bigmemory (and the related <a href="http://cran.r-project.org/web/packages/biganalytics/index.html">biganalytics</a>, and <a href="http://cran.r-project.org/web/packages/bigtabulate/index.html">bigtabulate</a>), the <a href="http://www.bigmemory.org/">bigmemory website</a> has a few very good presentations, vignettes, and overviews from Jay Emerson. For ff, I recommend reading Adler Oehlschlägel and colleagues' excellent slide presentations on the <a href="http://ff.r-forge.r-project.org/">ff website</a>. </p> <p>Also, consider storing data in a database and reading in smaller batches for analysis. There are likely any number of approaches to consider. To get started, consdier looking through some of the examples in the <a href="http://cran.r-project.org/web/packages/biglm/index.html">biglm</a> package, as well as <a href="http://faculty.washington.edu/tlumley/tutorials/user-biglm.pdf">this presentation</a> from Thomas Lumley.</p> <p>And do investigate the other packages on the high-performance computing task view and mentioned in the other answers. The packages I mention above are simply the ones I've happened to have more experience with.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload