StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POTricks to avoid duplication of memory allocation when returning data.table or data.frame in R?
text
Body
copied!<p>I have created a function that is called to read in and then return a data.table:</p> <pre><code>read.in.data <- function(filename) { library(data.table) data.holder<-read.table(filename, skip=1) return(data.table(data.holder)) } </code></pre> <p>I have noticed from observing my RAM as the function processes that R seems to process this in 2 steps (or at least this is my best guess for what's going on). For example, when I load a 1.5 GB file (15 columns with a total of 136 characters per row), R seems to 1) read in the data and use 1.5 GB of RAM, and then 2) use another 1.5 GB of RAM for the return.</p> <p>Are there some tricks to creating a function to create a data.table (or data.frame for that matter) and return the data.table without requiring duplication in memory? Or must I do all processing for the data.table within the function where the table is created?</p> <p>Observations: If I run this code twice in a row, the memory is not cleared; since I only have 8 GB of RAM, the function fails. If I skip the step of storing the "read.table" in a variable (as shown below), I don't get any benefit. I wouldn't want to do this any way, since I'd like to have the ability to clean up the data.table before returning it. A fix to my problem would also enable me to process larger files without running out of memory.</p> <pre><code>short.read.trk <- function(fntrk) { library(data.table) return(data.table(read.table(fntrk, skip=1))) } </code></pre>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload