Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>This is a really good question. I'm glad to see some R folks weighing in. I too think R is the right tool for the job, although it's my main hammer so everything looks a bit like a nail to me. </p> <p>There are a handful of R concepts needed to tackle this challenge. As I see it, you need the following (references in parens) :</p> <ol> <li>Import data into R. (<a href="http://cran.r-project.org/doc/manuals/R-data.html" rel="nofollow noreferrer">R Import Export Guide</a>)</li> <li>Get the Data into an appropriate time series structure. (<a href="http://cran.r-project.org/web/packages/xts/vignettes/xts.pdf" rel="nofollow noreferrer">XTS Vignette PDF</a>)</li> <li>A little bit of plotting. (<a href="http://www.statmethods.net/graphs/index.html" rel="nofollow noreferrer">Quick-R intro to graphics</a>) </li> </ol> <p>Here's example code using 2mm points. If you notice, I don't illustrate plotting all 2mm points. It's slow and not that informative. But this should give you some ideas on getting started. Feel free to come back with more specific questions if you do decide to jump down the R rabbit hole! </p> <pre><code>require( xts ) require( lubridate ) ## set up some example data dataLength &lt;- 2e6 startTime &lt;- ymd_hms("2011-02-01-13-29-53") fistFullOfSeconds &lt;- 1:dataLength date &lt;- startTime + fistFullOfSeconds DD1 &lt;- rnorm( dataLength ) DD2 &lt;- DD1 + rnorm(dataLength, 0, .1 ) DD3 &lt;- rnorm( dataLength, 10, 2) DD4 &lt;- rnorm( dataLength ) myXts &lt;- xts(matrix( c( DD1, DD2, DD3, DD4 ), ncol=4 ), date) ## now all the data are in the myXts object so let's do some ## summarizing and visualization ## grabbing just a single day from the data ## converted to data.frame to illustrate default data frame plotting oneDay &lt;- data.frame( myXts["2011-02-02"] ) plot( oneDay ) </code></pre> <p>The relationship between DD1 and DD2 kinda jumps out<br> <img src="https://i.stack.imgur.com/nlpva.png" alt="enter image description here"></p> <pre><code>boxplot( oneDay ) </code></pre> <p>Boxplot is the piechart of statistical graphics. The plot you love to hate. Might as well <a href="http://imgs.xkcd.com/comics/boyfriend.png" rel="nofollow noreferrer">link to this</a> while we're here. <img src="https://i.stack.imgur.com/A6bAw.png" alt="enter image description here"></p> <pre><code>## look at the max value of each variable every minute par(mfrow=c(4,1)) ## partitions the graph window ep &lt;- endpoints(myXts,'minutes') plot(period.apply(myXts[,1],INDEX=ep,FUN=max)) plot(period.apply(myXts[,2],INDEX=ep,FUN=max)) plot(period.apply(myXts[,3],INDEX=ep,FUN=max)) plot(period.apply(myXts[,4],INDEX=ep,FUN=max)) </code></pre> <p>Even at one minute resolution I'm not sure this is informative. Should probably subset. <img src="https://i.stack.imgur.com/3Hb02.png" alt="enter image description here"></p>
    singulars
    1. This table or related slice is empty.
    1. POFastest & most flexible way to chart over 2 million rows of flat file data?
      singulars
      1. PTQuestion
    1. PTAnswer
    1. This table or related slice is empty.
    1. USJD Long
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POFastest & most flexible way to chart over 2 million rows of flat file data?
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. COnice answer JD, good illustration of some of the add-on packages that make these things easier than base R functionality.
      singulars
      1. PO
      1. USGavin Simpson
    2. COAgreed. But do play with a) 'alpha blending' to allow over-plotting (and not all graphics devices support it), b) the awesome hexbin package or, as JD did c) sub-sampling. Once every minute is still too fine-grained for the way the data is set up (with a second interval).
      singulars
      1. PO
      1. USDirk Eddelbuettel
    3. CO@Dirk re: every minute - indeed. I edited my answer to work on hourly aggregates after initially doing it at the minute time step, which was just a mess.
      singulars
      1. PO
      1. USGavin Simpson
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload