Databases
StackOverflow2013
Postsdbo
POResque worker logging slows down Rails responsiveness
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POResque worker logging slows down Rails responsiveness
    primarykey
    data
    text
    <p>I have a Resque job which pulls a csv list of data off of a remote server and then runs through the +40k entries to add any new items to an existing database table. The job is running fine however it severely slows down the response time of any subsequent requests to the server. In the console which I've launched 'bundle exec rails server', I see not print statements though the job is running. However once I hit my rails server (via a page referesh), I see multiple SELECT / INSERT statements roll by before the server responds. The SELECT/INSERT statements are clearly generated by my Resque job but oddly they wait to print to the console unit I hit the server through the browser. </p> <p>It sure feels like I'm doing something wrong or not following the 'rails way'. Advice?</p> <p><strong>Here is the code in my Resque job which does the SELECT/INSERTS</strong></p> <pre><code># data is an array of hashes formed from parsing csv input. Max size is 1000 ActiveRecord::Base.transaction do data.each do |h| MyModel.find_or_create_by_X_and_Y( h[:x], h[:y], h ) end end </code></pre> <p><strong>Software Stack</strong></p> <ul> <li>Rails 3.2.0</li> <li>postgresql 9.1</li> <li>Resque 1.20.0</li> </ul> <p><strong>EDIT</strong></p> <p>I've finally take the time to debug this a bit more. Even a very simple worker, like below, slows down the next server response. In the console where I've launched the rail sever process I see that the delay occurs b/c stdout from the worker is being printed only after I ping the server.</p> <pre><code> def perform() s = Time.now 0.upto( 90000 ) do |i| Rails.logger.debug i * i end e = Time.now Rails.logger.info "Start: #{s} ---- End #{e}" Rails.logger.info "Total Time: #{e - s }" end </code></pre> <p>I can get the rails server back to its normal responsiveness again if I suppress stdout when I launch rails but it doesn't seem like that should be necessary... <code>bundle exec rails server &gt; /dev/nul</code></p> <p>Any input on a better way to solve this issue?</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USDavid J.
    1. USbons
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POResque worker logging slows down Rails responsiveness
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POResque worker logging slows down Rails responsiveness
      1. USDavid J.
      1. VTFavorite
    1. COI removed the PostgreSQL tags because they aren't key to the question -- as your example shows, you can recreate without a database.
      singulars
      1. POResque worker logging slows down Rails responsiveness
      1. USDavid J.
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload