StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POff data storage running big analysis
primarykey
Id
18685803
data
AcceptedAnswerId
0
AnswerCount
0
ClosedDate
CommentCount
8
CommunityOwnedDate
CreationDate
2013-09-08T16:09:54.093
FavoriteCount
0
LastActivityDate
2013-09-10T16:31:28.010
LastEditDate
2017-05-23T12:06:18.553
LastEditorUserId
-1
OwnerUserId
909645
ParentId
0
PostTypeId
1
Score
1
ViewCount
435
LastEditorDisplayName
text
Body
I've spent hours reading for using ff package and couldn't get a handle on this topic yet. Basically, I'd like to run a analysis on a big data and save the results/statistics from the analysis. I modified the example code written in ff package using biglm on my data set. <a href="http://cran.r-project.org/web/packages/ff/ff.pdf" rel="nofollow noreferrer">http://cran.r-project.org/web/packages/ff/ff.pdf</a> The problem is very similar to this one <a href="https://stackoverflow.com/questions/17295423/modeling-a-very-big-data-set-1-8-million-rows-x-270-columns-in-r?rq=1">Modeling a very big data set (1.8 Million rows x 270 Columns) in R</a> Here's my code below <pre><code>library(ff) library(ffbase) library(doSNOW) registerDoSNOW(makeCluster(4, type = "SOCK")) memory.limit(size=32000) setwd('Z:/data') wd <- getwd() data.path <- file.path(wd,'ffdb') data.path.train <- file.path(data.path,'train') ff.train <- read.table.ffdf(file='train.tsv', sep='\t') save.ffdf(ff.train, dir=data.path.train) library(biglm) # Here I'm implementing biglm model on ffdf data # Vi represents the column names form <- V27 ~ V3 + V4 + V5 + V6 + V7 + V8 + V9 + V10 + V11 + V12 + V13 + V14 + V15 ff.biglm <- for (i in chunk(ff.train, by=500)){ if (i[1]==1){ message("first chunk is: ", i[[1]],":",i[[2]]) biglmfit <- biglm(form, data=ff.train[i,,drop=FALSE]) }else{ message("next chunk is: ", i[[1]],":",i[[2]]) biglmfit <- update(biglmfit, ff.train[i,,drop=FALSE]) } } </code></pre> When the above code is ran, it gives the following error message: <blockquote> first chunk is: 1:494 Error: cannot allocate vector of size 19.4 Gb In addition: There were 50 or more warnings (use warnings() to see the first 50) </blockquote> Is this error message in regards to the size of biglmfit cannot be fitting to memory? Any work around to save biglmfit into ffdf data type? Or for that matter, is there any ways to store analysis statistics into ffdf type in chunk? Thank you. EDIT: <pre><code>vmode(ff.train) V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 "integer" "integer" "integer" "integer" "integer" "integer" "integer" "integer" "integer" "integer" V11 V12 V13 V14 V15 V16 V17 V18 V19 V20 "integer" "integer" "integer" "integer" "integer" "integer" "integer" "integer" "integer" "integer" V21 V22 V23 V24 V25 V26 V27 "integer" "integer" "integer" "integer" "integer" "integer" "integer" </code></pre>
Tags
<r><bigdata>
Title
ff data storage running big analysis
singulars
PostAcceptedAnswerId
1. This table or related slice is empty.
PostParentId
1. This table or related slice is empty.
PostTypePostTypeId
1. PTQuestion
UserLastEditorUserId
1. USCommunity
UserOwnerUserId
1. USElegantFellow
plurals
PostLinksPostIdRelatedPostId
1. PL
 singulars
 LinkTypeLinkTypeId
 LTLinked
PostLinksRelatedPostIdPostId
1. This table or related slice is empty.
PostsAcceptedAnswerId
1. This table or related slice is empty.
PostsParentIdCreationDate
1. This table or related slice is empty.
VotesPostIdCreationDate
1. This table or related slice is empty.
CommentsPostId

Querying!

Guidance

A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.