StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>Manipulating large amounts of data in an unknown format is...challenging for a generic program. Your best bet is probably similar to what you're doing already. Use a custom program to reformat your results into something easier to handle (<em>backend</em>), and a visualisation program of your choice to let you view and play around with the data (<em>frontend</em>).</p> <p><strong>Backend</strong></p> <p>For your problem I'd suggest a relational database (e.g.<a href="http://www.mysql.com/%20MySQL" rel="nofollow">Mysql</a>). Has a longer setup time than other options, but if this is an ongoing problem it should be worthwhile, as it allows you to easily pull fields of interest. <code>SELECT AVG(Speedup) FROM results WHERE Parameter1="column" AND Parameter2="cyclic"</code> for example. You'll then still need a simple script to insert your data in the first place, and then to pull the results of interest in a useful format you can stick into your viewer. Or if you so desire you can just run queries directly against the db.</p> <p>Alternatively, what I usually use is just Python or Perl. Read in your data files, strip the data you don't want, rearrange into the desire structure, and write out to some standard format your frontend and use. Replace Python/Perl with the language of your choice.</p> <p><strong>Frontend</strong></p> <p>Personally, I almost always use Excel. The backend does most of the heavy lifting, so I get a csv file with the results I care about all nicely ordered already. Excel then lets me play around with the data, doing stuff like taking averages, plotting, reordering, etc fairly simply.</p> <p>Other tools I use to display stuff which are probably not useful for you, but included for completeness include:</p> <p><a href="http://www.cs.waikato.ac.nz/ml/weka/" rel="nofollow">Weka</a> - Mostly machine learning targetted, but provides tools for searching for trends or correlations. Useful to play around with data looking for things of interest.</p> <p>Python/IDL/etc - For when I need data that can't be represented by a spreadsheet. These programs can, in addition to doing the backend's job of extracting and bulk manipulations, generate difference images, complicated graphs, or whatever else I need. </p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload