Databases
StackOverflow2013
Postsdbo
POSeeking Guidance on troubleshooting node memory issue
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POSeeking Guidance on troubleshooting node memory issue
    primarykey
    data
    text
    <p>I'm trying to implement a message queue using node, socket.io, and redis. I am attempting to follow the reliable queue pattern outlined <a href="http://redis.io/commands/rpoplpush" rel="nofollow">Here</a>. I am trying to read a logfile (60M in size) in line-by-line (will be changing this later) and pump the lines into the queue for processing later. However, I am running into a memory allocation issue. I'm not sure how to troubleshoot this and would like some guidance on where to start. I can't tell if the issue is in reading the file, or in the redis client. I have been able to add messages to the queue one by one like this :</p> <pre><code>socket.emit('message', 'some sort of log line here'); </code></pre> <p>Therefore I know the listener is working, but when I run the socketClient.js file It will spin out for a bit and then ultimately fail with the following generic error message:</p> <p>FATAL ERROR: JS Allocation failed - process out of memory</p> <p>Is there some error handling, or profiling I can add to get more information on where this is failing?</p> <p>Here is the code:</p> <p>socketListener.js</p> <pre><code>var util = require("util"), redis = require("redis"), io = require('socket.io').listen(8088) client = redis.createClient("7777", "localhost"); util.log("Established connection to redis"); io.sockets.on('connection', function(socket) { util.log("socket connection established for socket : " + socket); socket.on('message', function (data) { util.log("received the following data : "); util.log(JSON.stringify(data, 0, 3)); client.on("error", function(err) { util.log("Error " + err); }); try { // reliable queue pattern implementation util.log("queuing up the data in the list"); client.rpush('logList', data); client.brpoplpush('logList', 'dequeueList', 10); } catch (err) { util.log("An error occurred : "); util.log(JSON.stringify(err, 0, 3)); } }); socket.on('disconnect', function() {}); }); </code></pre> <p>socketClient.js</p> <pre><code>var io = require("socket.io-client"); var socket = io.connect('http://localhost:8088'); var redis = require('redis'); var util = require('util'); var fs = require('fs'); var readline = require('readline'); socket.on('connect', function() { client = redis.createClient("7777", "localhost"); var rd = readline.createInterface({ input: fs.createReadStream('someLogFile.log'), terminal: false }); rd.on('line', function(line) { util.log("reading line " + line); socket.emit('message', line); }); client.lrange('dequeueList', 0, -1, function(err, results) { if (err) { util.log(err); } else { var multi = client.multi(); for (var i=0; i &lt; results.length; i++) { util.log('got : ' + results[i]); multi.hgetall(results[i]); } multi.exec(function(err, logs) { util.log("executing the multi commands"); util.log(JSON.stringify(logs, 0 ,3)); }) }; }); }) </code></pre> <p>Thank you in advance for the help!</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USjanarde
    1. USjanarde
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. COdid you tried node-inspector ? https://github.com/node-inspector/node-inspector
      singulars
      1. POSeeking Guidance on troubleshooting node memory issue
      1. USCybermaxs
    2. CO@Cybermaxs-Betclic I did not think about this tool for some reason. I have used it before when debugging a web api but didn't think of it for this situation. Thanks!
      singulars
      1. POSeeking Guidance on troubleshooting node memory issue
      1. USjanarde
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload