Databases
StackOverflow2013
Postsdbo
POWhat a good way to get in-memory cache with data.table
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POWhat a good way to get in-memory cache with data.table
    primarykey
    data
    text
    <p>Let's say I have a 4GB dataset on a server with 32 GB. </p> <p>I can read all of that into R, make a data.table global variable and have all of my functions use that global as a kind of in-memory data-base. However, when I exit R and restart, I have to read that from disk again. Even with smart disk cacheing strategies (save/load or R.cache) I have 10 seconds delay or so getting that data in. Copying that data takes about 4 seconds. </p> <p>Is there a good way to cache this in memory that survives the exit of an R session?</p> <p>A couple of things comes to mind, RServe, redis/Rredis, Memcache, multicore ... Shiny-Server and Rstudio-Server also seem to have ways of solving this problem.</p> <p>But then again, it seems to me that perhaps data.table could provide this functionality since it appears to move data outside of R's memory block anyway. That would be ideal in that it wouldn't require any data copying, restructuring etc. </p> <p>Update:</p> <p>I ran some more detailed tests and I agree with the comment below that I probably don't have much to complain about. </p> <p>But here are some numbers that others might find useful. I have a 32GB server. I created a data.table of 4GB size. According to gc() and also looking at top, it appeared to use about 15GB peak memory and that includes making one copy of the data. That's pretty good I think.</p> <p>I wrote to disk with save(), deleted the object and used load() to remake it. This took 17 seconds and 10 seconds respectively. </p> <p>I did the same with the R.cache package and this was actually slower. 23 and 14 seconds. </p> <p>However both of those reload times are quite fast. The load() method gave me 357 MB/s transfer rate. By comparison, a copy took 4.6 seconds. This is a virtual server. Not sure what kind of storage it has or how much that read speed is influenced by the cache. </p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USDave31415
    1. USDave31415
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POWhat a good way to get in-memory cache with data.table
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POWhat a good way to get in-memory cache with data.table
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POWhat a good way to get in-memory cache with data.table
      1. USMatt Dowle
      1. VTFavorite
    1. COdata.table does not provide this functionality, anyway 10 seconds for a 5GB load look very good to me, plus it is a one shot so can you tell us why it is a problem ?
      singulars
      1. POWhat a good way to get in-memory cache with data.table
      1. USstatquant
    2. COThat 10s may not be accurate. Let me actually run it.
      singulars
      1. POWhat a good way to get in-memory cache with data.table
      1. USDave31415
    3. COYou could also potentially look into using `ff` or `bigmemory`.
      singulars
      1. POWhat a good way to get in-memory cache with data.table
      1. USKevin Ushey
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload