StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POLoading and analyzing massive amounts of data
text
Body
copied!<p>So for some research work, I need to analyze a ton of raw movement data (currently almost a gig of data, and growing) and spit out quantitative information and plots.</p> <p>I wrote most of it using Groovy (with JFreeChart for charting) and when performance became an issue, I rewrote the core parts in Java.</p> <p>The problem is that analysis and plotting takes about a minute, whereas loading all of the data takes about 5-10 minutes. As you can imagine, this gets really annoying when I want to make small changes to plots and see the output.</p> <p>I have a couple ideas on fixing this:</p> <ol> <li><p>Load all of the data into a SQLite database.<br> Pros: It'll be fast. I'll be able to run SQL to get aggregate data if I need to.</p> <p>Cons: I have to write all that code. Also, for some of the plots, I need access to each point of data, so loading a couple hundred thousand files, some parts may still be slow.</p></li> <li><p>Java RMI to return the object. All the data gets loaded into one root object, which, when serialized, is about 200 megs. I'm not sure how long it would take to transfer a 200meg object through RMI. (same client).</p> <p>I'd have to run the server and load all the data but that's not a big deal.</p> <p>Major pro: this should take the least amount of time to write</p></li> <li><p>Run a server that loads the data and executes a groovy script on command within the server vm. Overall, this seems like the best idea (for implementation time vs performance as well as other long term benefits)</p></li> </ol> <p>What I'd like to know is have other people tackled this problem?</p> <p>Post-analysis (3/29/2011): A couple months after writing this question, I ended up having to learn <strong>R</strong> to run some statistics. Using <strong>R</strong> was far, far easier and faster for data analysis and aggregation than what I was doing. </p> <p>Eventually, I ended up using Java to run preliminary aggregation, and then ran everything else in <strong>R</strong>. <strong>R</strong> was also much easier to make beautiful charts than using JFreeChart.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload