Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>In order to summarize your 10-minute log files by the hour, you could round down the timestamp of each logfile to the nearest hour in the map function and group the results by hours in the reduce function.</p> <p>Here is a little dummy example that illustrates this from the mongo shell:</p> <ol> <li><p>Create 100 log files, each 10 minutes apart and containing a random number between 0-10, and insert them in the <code>logs</code> collection in the database:</p> <pre><code>for (var i = 0; i &lt; 100; i++) { d = new ISODate(); d.setMinutes(d.getMinutes() + i*10); r = Math.floor(Math.random()*11) db.logs.insert({timestamp: d, number: r}) } </code></pre> <p>To check what the <code>logs</code> collection looks like, send a query like <code>db.logs.find().limit(3).pretty()</code>, which results in:</p> <pre><code>{ "_id" : ObjectId("50455a3570537f9433c1efb2"), "timestamp" : ISODate("2012-09-04T01:32:37.370Z"), "number" : 2 } { "_id" : ObjectId("50455a3570537f9433c1efb3"), "timestamp" : ISODate("2012-09-04T01:42:37.370Z"), "number" : 3 } { "_id" : ObjectId("50455a3570537f9433c1efb4"), "timestamp" : ISODate("2012-09-04T01:52:37.370Z"), "number" : 8 } </code></pre></li> <li><p>Define a map function (in this example called <code>mapf</code>) that rounds the timestamps to the nearest hour (rounded down), which is used for the emit key. The emit value is the number for that log file.</p> <pre><code>mapf = function () { // round down to nearest hour d = this.timestamp; d.setMinutes(0); d.setSeconds(0); d.setMilliseconds(0); emit(d, this.number); } </code></pre></li> <li><p>Define a reduce function, that sums over all the emitted values (i.e. numbers). </p> <pre><code>reducef = function (key, values) { var sum = 0; for (var v in values) { sum += values[v]; } return sum; } </code></pre></li> <li><p>Now execute map/reduce on the logs collection. The <code>out</code> parameter here specifies that we want to write the results to the <code>hourly_logs</code> collection and merge existing documents with new results. This ensures that log files submitted later (e.g. after a server failure or other delay) will be included in the results once they appear in the logs.</p> <pre><code>db.logs.mapReduce(mapf, reducef, {out: { merge : "hourly_logs" }}) </code></pre></li> <li><p>Lastly, to see the results, you can query a simple find on <code>hourly_logs</code>:</p> <pre><code>db.hourly_logs.find() { "_id" : ISODate("2012-09-04T02:00:00Z"), "value" : 33 } { "_id" : ISODate("2012-09-04T03:00:00Z"), "value" : 31 } { "_id" : ISODate("2012-09-04T04:00:00Z"), "value" : 21 } { "_id" : ISODate("2012-09-04T05:00:00Z"), "value" : 40 } { "_id" : ISODate("2012-09-04T06:00:00Z"), "value" : 26 } { "_id" : ISODate("2012-09-04T07:00:00Z"), "value" : 26 } { "_id" : ISODate("2012-09-04T08:00:00Z"), "value" : 25 } { "_id" : ISODate("2012-09-04T09:00:00Z"), "value" : 46 } { "_id" : ISODate("2012-09-04T10:00:00Z"), "value" : 27 } { "_id" : ISODate("2012-09-04T11:00:00Z"), "value" : 42 } { "_id" : ISODate("2012-09-04T12:00:00Z"), "value" : 43 } { "_id" : ISODate("2012-09-04T13:00:00Z"), "value" : 35 } { "_id" : ISODate("2012-09-04T14:00:00Z"), "value" : 22 } { "_id" : ISODate("2012-09-04T15:00:00Z"), "value" : 34 } { "_id" : ISODate("2012-09-04T16:00:00Z"), "value" : 18 } { "_id" : ISODate("2012-09-04T01:00:00Z"), "value" : 13 } { "_id" : ISODate("2012-09-04T17:00:00Z"), "value" : 25 } { "_id" : ISODate("2012-09-04T18:00:00Z"), "value" : 7 } </code></pre></li> </ol> <p>The result is an hourly summary of your 10-minute logs, with the _id field containing the start of the hour and the value field the sum of the random numbers. In your case, you may have different aggregation operators; modify the reduce functions according to your needs.</p> <p>As Sammaye mentioned in the comment, you could automate the map/reduce call with a cron job entry to run every hour.</p> <p>If you don't want to process the full logs collection every time, you can run incremental updates by limiting the documents to hourly time windows like so:</p> <pre><code>var q = { $and: [ {timestamp: {$gte: new Date(2012, 8, 4, 12, 0, 0) }}, {timestamp: {$lt: new Date(2012, 8, 4, 13, 0, 0) }} ] } db.logs.mapReduce(mapf, reducef, {query: q, out: { merge : "hourly_logs" }}) </code></pre> <p>This would only include log files between the hours of 12 and 13. Note that the month value in the Date() object starts at 0 (8=September). Because of the <code>merge</code> option, it is safe to run the m/r on already processed log files.</p>
    singulars
    1. This table or related slice is empty.
    1. POIncremental MapReduce with time interval in mongoDB
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USThomas
    1. USThomas
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. This table or related slice is empty.
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload