StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>It turned out that our problem was caused by having our own custom <code>Resque.before_fork</code> and <code>Resque.after_fork</code> handlers.</p> <p>NewRelic's RPM gem will automatically set up hooks with <code>Resque.before_fork</code> and <code>Resque.after_fork</code> to establish a communication channel for the workers. As a limitation of Resque, it runs only the last assigned block/Proc to the before_fork and after_fork hooks. So, if you have your own custom before_fork/after_fork hooks, you *must * set up the agent's communication channel by hand, e.g. in a config/initializers/custom_resque.rb file:</p> <pre><code>Resque.before_fork do |job| NewRelic::Agent.register_report_channel(job.object_id) # extra custom stuff here end Resque.after_fork do |job| NewRelic::Agent.after_fork(:report_to_channel => job.object_id) # extra custom stuff here end </code></pre> <p>This code is directly taken from the RPM gem's file <code>gems/newrelic_rpm-3.5.0/lib/new_relic/agent/instrumentation/resque.rb</code></p> <p><strong>RPM bug update 12/27/2012:</strong> After deploying the technique above, we found that the RPM gem leaks file handles when used in forked mode (e.g. Resque). We observed error messages of the kind <code>ActiveRecord::StatementInvalid: ArgumentError: too large fdsets: SET client_min_messages TO ''</code>. After a lot of digging we found that these are caused when ActiveRecord tries to open a database connection and can't because the number of file descriptors is exhausted. New Relic confirmed that there is a bug in the agent when sampling the explain plan. This occurs when lots of Resque jobs run that connect to the DB.</p> <p><strong>Bug update 1/28/2013:</strong> After much head scratching we found out that this bug was caused by an unsupported interaction with the <a href="http://rubygems.org/gems/resque-lonely_job" rel="nofollow">resque-lonely_job</a> gem which uses Resque's <code>before_perform</code> hook that may stop a Resque job with a <code>Resque::Job::DontPerform</code> exception. The RPM client doesn't clean up properly in this situation and leaks file descriptors. New Relic has been informed and is working on a fix.</p> <p><strong>Bug update 4/10/2013:</strong> This has been fixed. We're using 3.6.0.78 and it handles this case. No more file descriptor leaks! Thank you New Relic.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload