Databases
StackOverflow2013
Postsdbo
POModeling a very big data set (1.8 Million rows x 270 Columns) in R
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POModeling a very big data set (1.8 Million rows x 270 Columns) in R
    primarykey
    data
    text
    <p>I am working on a <em>Windows 8</em> OS with a <em>RAM of 8 GB</em> . I have a data.frame of <strong>1.8 million rows x 270 columns</strong> on which I have to perform a glm. (logit/any other classification)</p> <p>I've tried using ff and bigglm packages for handling the data.</p> <p>But I am still facing a problem with the error "<code>Error: cannot allocate vector of size 81.5 Gb</code>". So, I decreased the number of rows to 10 and tried the steps for bigglm on an object of class ffdf. However the error still is persisting.</p> <p>Can any one suggest me the solution of this problem of building a classification model with these many rows and columns?</p> <p><code>**EDITS**</code>:</p> <p>I am <strong>not</strong> using any other program when I am running the code. The RAM on the system is 60% free before I run the code and that is because of the R program. When I terminate R, the RAM 80% free.</p> <p>I am adding <strong>some of the columns</strong> which I am working with now as suggested by the commenters for reproduction. <strong>OPEN_FLG is the DV</strong> and others are IDVs</p> <pre><code>str(x[1:10,]) 'data.frame': 10 obs. of 270 variables: $ OPEN_FLG : Factor w/ 2 levels "N","Y": 1 1 1 1 1 1 1 1 1 1 $ new_list_id : Factor w/ 9 levels "0","3","5","6",..: 1 1 1 1 1 1 1 1 1 1 $ new_mailing_id : Factor w/ 85 levels "1398","1407",..: 1 1 1 1 1 1 1 1 1 1 $ NUM_OF_ADULTS_IN_HHLD : num 3 2 6 3 3 3 3 6 4 4 $ NUMBER_OF_CHLDRN_18_OR_LESS: Factor w/ 9 levels "","0","1","2",..: 2 2 4 7 3 5 3 4 2 5 $ OCCUP_DETAIL : Factor w/ 49 levels "","00","01","02",..: 2 2 2 2 2 2 2 21 2 2 $ OCCUP_MIX_PCT : num 0 0 0 0 0 0 0 0 0 0 $ PCT_CHLDRN : int 28 37 32 23 36 18 40 22 45 21 $ PCT_DEROG_TRADES : num 41.9 38 62.8 2.9 16.9 ... $ PCT_HOUSEHOLDS_BLACK : int 6 71 2 1 0 4 3 61 0 13 $ PCT_OWNER_OCCUPIED : int 91 66 63 38 86 16 79 19 93 22 $ PCT_RENTER_OCCUPIED : int 8 34 36 61 14 83 20 80 7 77 $ PCT_TRADES_NOT_DEROG : num 53.7 55 22.2 92.3 75.9 ... $ PCT_WHITE : int 69 28 94 84 96 79 91 29 97 79 $ POSTAL_CD : Factor w/ 104568 levels "010011203","010011630",..: 23789 45173 32818 6260 88326 29954 28846 28998 52062 47577 $ PRES_OF_CHLDRN_0_3 : Factor w/ 4 levels "","N","U","Y": 2 2 3 4 2 4 2 4 2 4 $ PRES_OF_CHLDRN_10_12 : Factor w/ 4 levels "","N","U","Y": 2 2 4 3 3 2 3 2 2 3 [list output truncated] </code></pre> <p>And this is the <strong>example</strong> of code which I am using.</p> <pre><code>require(biglm) mymodel &lt;- bigglm(OPEN_FLG ~ new_list_id+NUM_OF_ADULTS_IN_HHLD+OCCUP_MIX_PCT, data = x) require(ff) x$id &lt;- ffseq_len(nrow(x)) xex &lt;- expand.ffgrid(x$id, ff(1:100)) colnames(xex) &lt;- c("id","explosion.nr") xex &lt;- merge(xex, x, by.x="id", by.y="id", all.x=TRUE, all.y=FALSE) mymodel &lt;- bigglm(OPEN_FLG ~ new_list_id+NUM_OF_ADULTS_IN_HHLD+OCCUP_MIX_PCT, data = xex) </code></pre> <p><strong>The problem is both times I get the same error "<code>Error: cannot allocate vector of size 81.5 Gb</code>".</strong></p> <hr> <p>Please let me know if this is enough or should I include anymore details about the problem.</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USSrikanth Gorthy
    1. USSrikanth Gorthy
    plurals
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTLinked
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POModeling a very big data set (1.8 Million rows x 270 Columns) in R
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POModeling a very big data set (1.8 Million rows x 270 Columns) in R
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POModeling a very big data set (1.8 Million rows x 270 Columns) in R
      1. USAndy
      1. VTFavorite
    1. COYou have taken some of the steps that I would recommend. So, to get more specific feedback you need to create a reproducible example which shows us the problem you have.
      singulars
      1. POModeling a very big data set (1.8 Million rows x 270 Columns) in R
      1. USPaul Hiemstra
    2. COHow is your memory looking? Are any other applications using a lot of memory? Can you remove any unused objects from your R workspace and use `gc()` to clear up space?
      singulars
      1. POModeling a very big data set (1.8 Million rows x 270 Columns) in R
      1. USBill Beesley
    3. CODo show the code and some data (str(yourdata[1:10,]) will do) to get a good feedback so that we can see what you are doing and correct any mistakes you inadvertedly make.
      singulars
      1. POModeling a very big data set (1.8 Million rows x 270 Columns) in R
      1. USjwijffels
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload