Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>Matt Sheppard's answer is great (mod up), but I would take account these factors when thinking about a spindle:</p> <ol> <li>Structure : does it obviously break into pieces, or are you making tradeoffs?</li> <li>Usage : how will the data be analyzed/retrieved/grokked?</li> <li>Lifetime : how long is the data useful?</li> <li>Size : how much data is there?</li> </ol> <p>One particular advantage of CSV files over RDBMSes is that they can be easy to condense and move around to practically any other machine. We do large data transfers, and everything's simple enough we just use one big CSV file, and easy to script using tools like rsync. To reduce repetition on big CSV files, you could use something like <a href="http://yaml.org" rel="noreferrer">YAML</a>. I'm not sure I'd store anything like JSON or XML, unless you had significant relationship requirements.</p> <p>As far as not-mentioned alternatives, don't discount <a href="http://hadoop.apache.org/core/" rel="noreferrer">Hadoop</a>, which is an open source implementation of MapReduce. This should work well if you have a TON of loosely structured data that needs to be analyzed, and you want to be in a scenario where you can just add 10 more machines to handle data processing.</p> <p>For example, I started trying to analyze performance that was essentially all timing numbers of different functions logged across around 20 machines. After trying to stick everything in a RDBMS, I realized that I really don't need to query the data again once I've aggregated it. And, it's only useful in it's aggregated format to me. So, I keep the log files around, compressed, and then leave the aggregated data in a DB.</p> <p><em>Note</em> I'm more used to thinking with "big" sizes.</p>
    singulars
    1. This table or related slice is empty.
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. This table or related slice is empty.
    2. VO
      singulars
      1. This table or related slice is empty.
    3. VO
      singulars
      1. This table or related slice is empty.
 

Querying!

 
Guidance

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload