Databases
StackOverflow2013
Postsdbo
POHow to reduce number of documents to be sync from a mongo DB
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHow to reduce number of documents to be sync from a mongo DB
    primarykey
    data
    text
    <p>In my current project, I am using two databases.</p> <ol> <li>A MongoDB instance gathering data from different data providers (abt 15M documents)</li> <li>Another (relational) database instance holding only the data which is needed for the application, i.e. a subset of the data in the MongoDB instance. (abt 5M rows)</li> </ol> <p>As part of the synchronisation process, I need to regularly check for new entries in the MongoDB depending on data in the relational DB.</p> <p>Let's say, this is about songs and artists, a document in the MongoDB might look like this:</p> <pre><code>{_id:1,artists:["Simon","Garfunkel"],"name":"El Condor Pasa"} </code></pre> <p>Part of the sync process is to import/update all songs from those artists that already exist in the relational DB, which are currently about 1M artists.</p> <p>So how do I retrieve all songs of 1M named artists from MongoDB for import?</p> <p>My first thought (and try) was to over all artists and query all songs for each artist (of course, there's an index on the "artists" field). But this takes several minutes for each batch of 1.000 artists, which would make this process a long runner.</p> <p>My second thought was to write all existing artists to a separate mongoDB collection and have a super query which only retrieves songs of artists that are stored in there. But so far I have not been able retrieve data based on two collections. Is this a good use case for map/reduce? If yes, can someone pls. give me a hint on how to achieve this? (I am not completely new to NoSQL, but sort of a newbie when it comes to map/reduce.) Or is this idea just crazy and I have to stick with a process that's running for several days?</p> <p>Thanks in advance for any hints.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USpeterp
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. CORetrieving documents from Mongo should be fairly fast. Is it reading 1000 artists data taking a lot of time or is it updating the relational DB that is consuming a lot of time ?
      singulars
      1. POHow to reduce number of documents to be sync from a mongo DB
      1. USuser1163459
    2. COThese test runs did not write anything to the relational DB. I queried batches of 1.000 from the relational DB (which takes a few seconds, mainly because I am currently retrieving the complete entity, not only the names). After that, I query MongoDB 1.000 times (once for each artist) to retrieve their songs. I think it is the mass of single queries taking the time, which is why I would prefer to have this done server-side (i.e. MongoDB-side)
      singulars
      1. POHow to reduce number of documents to be sync from a mongo DB
      1. USpeterp
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload