StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>Very true: <code>data.table</code> hasn't got to on-disk tables yet. In the meantime some options are :</p> <ul> <li><p>Don't exit R. Leave it running on a server and use svSocket to <code>evalServer()</code> to it, as the video on the <code>data.table</code> homepage demonstrates. Or the other similar options you mentioned.</p></li> <li><p>Use a database for persistency such as SQL or any other noSQL database.</p></li> <li><p>If you have large delimited files then some people have recently reported that <code>fread()</code> appears (much) faster than <code>load()</code>. But experiment with <code>compress=FALSE</code>. Also, we've just pushed <code>fwrite</code> to the most current development version (1.9.7, use <code>devtools::install_github("Rdatatable/data.table")</code> to install), which has some <a href="https://stackoverflow.com/a/36465497/3576984">reported write times</a> on par with native <code>save</code>.</p></li> <li><p>Packages <code>ff</code>, <code>bigmemory</code> and <code>sqldf</code>, too. See the HPC Task View, the "Large memory and out-of-memory data" section.</p></li> </ul> <p>In enterprises where <code>data.table</code> is being used, my guess is that it is mostly being fed with data from some other persistent database, currently. Those enterprises probably :</p> <ul> <li>use 64bit with say 16GB, 64GB or 128GB of RAM. RAM is cheap these days. (But I realise this doesn't address persistency.)</li> </ul> <p>The internals have been written with on-disk tables in mind. But don't hold your breath!</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload