StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POKeep history of field changes from another collection in a new collection in mongodb
text
Body
copied!<p>here's my problem: </p> <p>I have a collection with ~900k entries, it's about 6Gb,this will be updated daily, at least for a while. By updated I mean, dumped and imported from a json file, which we'll get every day (this may be just temp solution). </p> <p>The problem that I'm dealing with at the moment is keeping a history on 4,5 fields in a separate collection, that will be updated before/after each "update" of the main collection.</p> <p>What I did, was to get all the records and loop through the results updating history collection for each entry, but running ~900k update queries is time/resource consuming and I don't think is a good practice. </p> <p>The structure that I have in mind is something like this: </p> <pre><code>{unique_id: 'unique value from main collection', history: [ {field1: value1, field2: value2, field3: 'value3', date_updated: '2013-12-03' }, {...} ] } </code></pre> <p>I tried mapreduce too, with output to collection, which was a lot faster, the problem with mapreduce is that I can't really keep the history because none of the options for "action" param doesn't do what I need, each time it runs, it overwrites the entries, what I need it to do is something like an upsert. </p> <p>If anyone has some ideas about how can I do this I'll gladly put them to test, I ran out of ideas, and I'm not a guru in mongodb either. </p> <p>I think the import / update solution is not good either, I'm looking for a solution to that too, but first to get the history done. </p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload