Databases
StackOverflow2013
Postsdbo
PONodeJS async queue too fast (Slowing down async queue method)
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PONodeJS async queue too fast (Slowing down async queue method)
    primarykey
    data
    text
    <p>I have an HTTP Get request and I want to parse the response and save it to my database.</p> <p>If i call crawl(i) independentely i get good results. But i have to call crawl() from 1 to 2000. I get good results but some responses seem to get lost and some responses are duplicates. I don't think I understand how to call thousands of asynchronous functions. I am using the <a href="https://github.com/caolan/async" rel="nofollow">async module</a> queue function but so far I am still missing some data and still have some duplicates. What am I doing wrong here? Thanks for your help.</p> <p><a href="http://magicseaweed.com/syndicate/rss/index.php?id=363&amp;unit=uk" rel="nofollow">What i am crawling </a></p> <p>My node functions :</p> <pre><code> function getOptions(i) { return { host: 'magicseaweed.com', path: '/syndicate/rss/index.php?id='+i+'&amp;unit=uk', method: 'GET' } }; function crawl(i){ var req = http.request(getOptions(i), function(res) { res.on('data', function (body) { parseLocation(body); }); }); req.end(); } function parseLocation(body){ parser.parseString(body, function(err, result) { if(result &amp;&amp; typeof result.rss != 'undefined') { var locationTitle = result.rss.channel[0].title; var locationString = result.rss.channel[0].item[0].link[0]; var location = new Location({ id: locationString.split('/')[2], name: locationTitle }); location.save(); } }); } N = 2 //# of simultaneous tasks var q = async.queue(function (task, callback) { crawl(task.url); callback(); }, N); q.drain = function() { console.log('Crawling done.'); } for(var i = 0; i &lt; 100; i++){ q.push({url: 'http://magicseaweed.com/syndicate/rss/index.php?id='+i+'&amp;unit=uk'}); } </code></pre> <p>[EDIT] WELL, after a lot of testing it seems that the service I am crawling cannot handle so many request that fast. Because when I do each requests sequentially, I can get all the good responses.</p> <p>Is there a way to SLOW DOWN ASYNC queue method?</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USWilliam Fortin
    1. USWilliam Fortin
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. PONodeJS async queue too fast (Slowing down async queue method)
      1. USWilliam Fortin
      1. VTFavorite
    1. COIt seems that a lot of requests are failing now... How can I insure request don't fail?
      singulars
      1. PONodeJS async queue too fast (Slowing down async queue method)
      1. USWilliam Fortin
    2. CODid you try using setTimeOut to delay the `callback()` call in your queue worker function? You can slow down the execution of tasks in the queue that way.
      singulars
      1. PONodeJS async queue too fast (Slowing down async queue method)
      1. USmax
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload