Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <h2>Our Solution</h2> <p>We do similar things on a large scale. If you're worried about scalability then keeping this code inside a Rails app going off of a basic postgres database is not your best choice. </p> <p>If you're going to be adding a bunch of metrics like this and if you're going to be counting words and diffs in the words by user, you should consider starting up a stream processing or batch processing platform. These solutions are not trivial, but worth it if you're going to need scale.</p> <p>Our solution uses twitter storm (<a href="http://storm-project.net" rel="nofollow">http://storm-project.net</a>) with the data counters in Mongo. In fact, their example is a word count application. Redis, as you've asked about isn't a bad choice, actually. I disagree with @jokklan because redis can implement counter storage with next-to-no effort.</p> <p>We do select the data out of a SQL database, so to start, postgres isn't a bad choice, but that will probably be the first thing you rip out when you start to really scale this thing.</p> <p>We also have forked storm deploy to help bring up storm servers more reliably. <a href="https://github.com/korrelate/storm-deploy" rel="nofollow">https://github.com/korrelate/storm-deploy</a></p> <h2>Other Options</h2> <p>Obviously, though, there are a bunch of different platforms to choose. </p> <ol> <li>You can use Hadoop MapReduce (<a href="http://hadoop.apache.org/docs/stable/mapred_tutorial.html" rel="nofollow">http://hadoop.apache.org/docs/stable/mapred_tutorial.html</a>)</li> <li>Pig which we use for other stuff through Mortar Data (<a href="http://www.mortardata.com" rel="nofollow">http://www.mortardata.com</a>)</li> <li>Amazon EMR which would allow you to do basic MapReduce or Pig jobs but this is more of an platform choice, not a framework and implementation choice</li> <li><p>Run some background jobs to compile this information using Sidekiq (<a href="https://github.com/mperham/sidekiq" rel="nofollow">https://github.com/mperham/sidekiq</a>) or Resque (not really recommended given sidekiq's advancements) or Iron Worker which runs as a service (<a href="http://www.iron.io/worker" rel="nofollow">http://www.iron.io/worker</a>)</p> <p>Here's a good article on some of the choices I've mentioned and probably some others (<a href="http://highlyscalable.wordpress.com/2013/08/20/in-stream-big-data-processing/" rel="nofollow">http://highlyscalable.wordpress.com/2013/08/20/in-stream-big-data-processing/</a>).</p></li> </ol> <h2>Recommendation</h2> <p>I can't honestly give you a good recommendation without more information about what sort of scale you're talking about. Given that, I might be able to help narrow down your choices a little better. How many users? Are you serious about giving all that granularity (that's fine if you are, just help determines scale)? Are there other things you'll want to do besides counting and diff'ing? </p>
    singulars
    1. This table or related slice is empty.
    1. POSmartly Tracking Words Written in a Rails App
      singulars
      1. PTQuestion
    1. PTAnswer
    1. This table or related slice is empty.
    1. USWattsInABox
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. COThis answer was extremely helpful and I wish I could have split the bounty between both you and jokklan, as you both gave me some new information and perspectives. I've voted this answer up and I hope others do as well.
      singulars
      1. PO
      1. USChris Fritz
    2. COThere is several of these projects i haven't heard about (twitter storm, or Hadoop MapReduce, etc.) but yes there is definitely a lot of options. Seems to me that @WattsInABox knows more about this topic then i do! But in the end it's all about finding what fits your needs. Check these projects out, and see what your project needs the most :)! I would recommend something simple like Sidekiq to start out with (you definitely need some kind of background worker!), and wait with complex solutions to you need them, but again i haven't tried these out before, and it's really all up to you :)!
      singulars
      1. PO
      1. USjokklan
    3. COI agree with @jokklan. The decision entirely depends on scale (current and future) and how many features you're planning on adding to this. If you want something that you plan to build extensively on top of, it may be worth making the jump to more complex tech early.
      singulars
      1. PO
      1. USWattsInABox
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload