StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p><strong>Separate concerns:</strong></p> <p>Apart from pondering on the technology to use for storing data (DBMS, CSV, or maybe one of the <a href="http://mariovalle.name/sdm/scientific-data-management.html" rel="nofollow">specific formats for scientific data</a>), note that you have <strong>three very different kinds of data</strong> to manage: </p> <ul> <li><p><em>Simulation scenario configurations:</em> these are (typically) rather small, but they need to be simple to edit, simple to re-use, and should allow to reproduce a simulation run. Here, <strong>text or code files</strong> seem to be a good choice (these should also be version-controlled).</p></li> <li><p><em>Raw simulation data</em>: this is where you should be really careful if you are concerned with simulation performance, because writing 3 GB of data during a run can take a huge amount of time if implemented badly. One way to proceed would be to <strong>use existing file formats for this purpose</strong> (see below) and see if they work for you. If not, you can still use a DBMS. Also, it is usually a good idea to include a description of the scenario that generated the data (or at least a reference), as this helps you managing the results.</p></li> <li><p><em>Data for post-processing:</em> how to store this mostly <strong>depends on the post-processing tools</strong>. For example, if you already have a class structure for your visualization application, you could define a file format that makes it easy to read in the required data.</p></li> </ul> <p><strong>Look for existing solutions:</strong></p> <p>The problem you face (<em>How to manage simulation data?</em>) is fundamental and there are many potential solutions, each coming with certain trade-offs. As you are working in network simulation, <strong>check out what capabilities other tools used in your community provide</strong>. It could be that their developers ran into problems you are not even anticipating yet (regarding reproducibility etc.), and already found a good solution. For example, you could check out how <strong><a href="http://omnetpp.org" rel="nofollow">OMNeT++</a></strong> is handling simulation output: the simulation configurations are defined in a separate file, results are written to <code>vec</code> and <code>sca</code> files (depending on their nature). As far as I understood your problems with hierarchical data, this is supported as well (vectors get unique IDs and are associated with an attribute of some model entity). Additional tools already work with these file formats, e.g. to <a href="http://www.omnetpp.org/doc/omnetpp/manual/usman.html#sec411" rel="nofollow">convert</a> them to other formats like CSV/MATLAB files, so you could even think of creating files in the same format (documented <a href="http://www.omnetpp.org/doc/omnetpp/manual/usman.html#sec542" rel="nofollow">here</a>) and to <strong>use existing tools/converters for post-processing.</strong></p> <p>Many other simulation tools will have similar features, so take a look at what would work best for you.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload