Databases
StackOverflow2013
Postsdbo
POPerformance problems when converting timestamped row data
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POPerformance problems when converting timestamped row data
    primarykey
    data
    text
    <p>I've written a function that takes a data.frame which represent intervals of data which occur across a 1 minute timeframe. The purpose of the function is to take these 1 minute intervals and convert them into higher intervals. Example, 1 minute becomes 5 minute, 60 minute etc...The data set itself has the potential to have gaps in the data i.e. jumps in time so it must accommodate for these bad data occurrences. I've written the following code which appears to work but the performance is absolutely terrible on large data sets.</p> <p>I'm hoping that someone could provide some suggestions on how I might be able to speed this up. See below.</p> <pre><code>compressMinute = function(interval, DAT) { #Grab all data which begins at the same interval length retSet = NULL intervalFilter = which(DAT$time$min %% interval == 0) barSet = NULL for (x in intervalFilter) { barEndTime = DAT$time[x] + 60*interval barIntervals = DAT[x,] x = x+1 while(x &lt;= nrow(DAT) &amp; DAT[x,"time"] &lt; barEndTime) { barIntervals = rbind(barIntervals,DAT[x,]) x = x + 1 } bar = data.frame(date=barIntervals[1,"date"],time=barIntervals[1,"time"],open=barIntervals[1,"open"],high=max(barIntervals[1:nrow(barIntervals),"high"]), low=min(barIntervals[1:nrow(barIntervals),"low"]),close=tail(barIntervals,1)$close,volume=sum(barIntervals[1:nrow(barIntervals),"volume"])) if (is.null(barSet)) { barSet = bar } else { barSet = rbind(barSet, bar) } } return(barSet) } </code></pre> <p>EDIT:</p> <p>Below is a row of my data. Each row represents a 1 minute interval, I am trying to convert this into arbitrary buckets which are the aggregates of these 1 minute intervals, i.e. 5 minutes, 15 minutes, 60 minutes, 240 minutes, etc...</p> <pre><code>date time open high low close volume 2005-09-06 2005-09-06 16:33:00 1297.25 1297.50 1297.25 1297.25 98 </code></pre>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USJoshua Ulrich
    1. USDave
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POPerformance problems when converting timestamped row data
      1. USrichallensf
      1. VTFavorite
    2. VO
      singulars
      1. POPerformance problems when converting timestamped row data
      1. USIterator
      1. VTFavorite
    3. VO
      singulars
      1. POPerformance problems when converting timestamped row data
      1. This table or related slice is empty.
      1. VTUpMod
    1. COno idea what you are really trying to do above, but the line `barIntervals = rbind(barIntervals,DAT[x,])` is one bottleneck. You are growing the barIntervals object every iteration, which forces R to make a copy of it...if you preallocate barIntervals it will work much better. Google `R preallocation` to read more. Also, does the function `na.approx()` from package `zoo` do what you want? Check out this question: http://stackoverflow.com/questions/7188807/interpolate-na-values
      singulars
      1. POPerformance problems when converting timestamped row data
      1. USChase
    2. CODiscovered XTS. HUGE difference.
      singulars
      1. POPerformance problems when converting timestamped row data
      1. USDave
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload