Databases
StackOverflow2013
Postsdbo
POZeroMQ with node.js pipeline sink stops receiving messages after a while
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POZeroMQ with node.js pipeline sink stops receiving messages after a while
    primarykey
    data
    text
    <p>I've been trying to set up a ventilator / worker / sink pattern in order to crawl pages, but I never got past the testing phase. The one particularity of my setup is that the sink lives in the same process as the ventilator. All nodes use ipc:// transport. For the moment only test messages are exchanged. the ventilator sends tasks, workers receive them and wait then send a confirmation to the sink.</p> <p><strong>Symptoms</strong>: After some time (generally less than 5 minutes) the sink stops receiving confirmation messages even though the ventilator keeps on sending tasks and workers keep on receiving them and sending confirmations messages.</p> <p>I know that confirmations are sent because if I restart my sink, it gets all the missing messages on startup.</p> <p>I thought ZeroMQ dealt with auto-reconnect.</p> <p><em>ventilator/sink</em> </p> <pre><code>var push = zmq.socket('push'); var sink = zmq.socket('pull'); var pi = 0; setInterval(function() { push.send(['ping', pi++], zmq.ZMQ_SNDMORE); push.send('end'); }, 2000); push.bind('ipc://crawl.ipc'); sink.bind('ipc://crawl-sink.ipc'); sink.on('message', function() { var args = [].slice.apply(arguments).map(function(e) {return e.toString()}); console.log('got message', args.join(' ')); }); </code></pre> <p><em>worker.js</em></p> <pre><code>var pull = zmq.socket('pull'); var sink = zmq.socket('push'); sink.connect(opt.sink); pull.connect(opt.push); pull.on('message', function() { var args = [].slice.apply(arguments).map(function(e) {return e.toString()}); console.log('got job ', args.join(' ')); setTimeout(function() { console.log('job done ', args.join(' ')); sink.send(['job done', args.join(' ')]); }, Math.random() * 5 * 1000); }); </code></pre> <p><strong>EDIT</strong> I tried moving the sink to another process and it seems to work. However I would really like it to live in the same process and I observed similar behaviour when dealing with more than one zmq socket per process, regardless of the pattern used</p> <p><strong>EDIT</strong> I'm using this module <a href="https://github.com/JustinTulloss/zeromq.node">https://github.com/JustinTulloss/zeromq.node</a></p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USFloby
    1. USFloby
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POZeroMQ with node.js pipeline sink stops receiving messages after a while
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POZeroMQ with node.js pipeline sink stops receiving messages after a while
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POZeroMQ with node.js pipeline sink stops receiving messages after a while
      1. USFloby
      1. VTFavorite
    1. COHave you tried other forms of transport?
      singulars
      1. POZeroMQ with node.js pipeline sink stops receiving messages after a while
      1. USrobertklep
    2. COyes, mainly tcp, and the problem remains.
      singulars
      1. POZeroMQ with node.js pipeline sink stops receiving messages after a while
      1. USFloby
    3. COI've been facing a similar issue, but with TCP. The push socket is a part of a c# application though, and I am trying to receive the data in a pull socket in node.js. It works perfectly, but it randomly stops working/listening after some time. I am not able to pin point the time when it stops working too.
      singulars
      1. POZeroMQ with node.js pipeline sink stops receiving messages after a while
      1. USMunim
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload