Note that there are some explanatory texts on larger screens.

plurals
  1. POFastest & most flexible way to chart over 2 million rows of flat file data?
    primarykey
    data
    text
    <p>I'm collecting some system data in a flatfile, which has this format: </p> <pre><code>YYYY-MM-DD-HH24:MI:SS DD1 DD2 DD3 DD4 </code></pre> <p>Where DD1-DD4 are four items of data. An example of the file is this: </p> <pre><code>2011-02-01-13:29:53 16 8 7 68 2011-02-01-13:29:58 13 8 6 110 2011-02-01-13:30:03 26 25 1 109 2011-02-01-13:30:08 13 12 1 31 2011-02-01-13:30:14 192 170 22 34 2011-02-01-13:30:19 16 16 0 10 2011-02-01-13:30:24 137 61 76 9 2011-02-01-13:30:29 452 167 286 42 2011-02-01-13:30:34 471 177 295 11 2011-02-01-13:30:39 502 192 309 10 </code></pre> <p>The file is over 2 million rows, with data points every five seconds.</p> <p>I need to chart this data to be able to derive meaning from it.</p> <p><strong>What I've tried</strong> </p> <p>At the moment I've tried gnuplot and rrdtool with a variety of unix tools (awk, sed, etc). Both of these work, but seem to require a lot of chopping and recutting the data each time I want to view it in a different way. My gut feel is that rrdtool is the right way to go, but at the moment I'm struggling to get the data into it fast enough, partly because I have to convert my timestamp into Unix epoch. My understanding is also that if I decide I want a new granularity of aggregation I have to rebuild the rrd (which makes sense for realtime collection, but not retrospective loads like this). These things make me think maybe I'm using the wrong tool.</p> <p>The collection of the data to flat file is fixed - for example I cannot pipe the collection directly into rrdtool.</p> <p><strong>My Question</strong></p> <p>I would like people's opinions on the best way to do charting. I have these requirements:</p> <ol> <li>It should be as fast as possible to create a graph (not just render, but set up to render too)</li> <li>It should be as flexible as possible - I need to muck around with the graphs to work out the best granularity for the data (5 seconds is probably too granular)</li> <li>It should be able to aggregate (MAX/AVG/etc) where necessary</li> <li>It should be repeatable and new data files as they come in</li> <li>Ideally I want to be able to overlay DD1 vs DD2, or DD1 last week with DD1 this week</li> <li>Unix or Windows, don't care. Prefer *nix though :-)</li> </ol> <p>Any suggestions? </p>
    singulars
    1. This table or related slice is empty.
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
 

Querying!

 
Guidance

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload