StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p><a href="http://api.rubyonrails.org/classes/ActiveRecord/Batches.html">The documentation</a> says that find_each and find_in_batches don't retain sort order and limit because:</p> <ul> <li>Sorting ASC on the PK is used to make the batch ordering work.</li> <li>Limit is used to control the batch sizes.</li> </ul> <p>You could write your own version of this function like @rorra did. But you can get into trouble when mutating the objects. If for example you sort by created_at and save the object it might come up again in one of the next batches. Similarly you might skip objects because the order of results has changed when executing the query to get the next batch. Only use that solution with read only objects.</p> <p>Now my primary concern was that I didn't want to load 30000+ objects into memory at once. My concern was not the execution time of the query itself. Therefore I used a solution that executes the original query but only caches the ID's. It then divides the array of ID's into chunks and queries/creates the objects per chunk. This way you can safely mutate the objects because the sort order is kept in memory.</p> <p>Here is a minimal example similar to what I did:</p> <pre><code>batch_size = 512 ids = Thing.order('created_at DESC').pluck(:id) # Replace .order(:created_at) with your own scope ids.each_slice(batch_size) do |chunk| Thing.find(chunk, :order => "field(id, #{chunk.join(',')})").each do |thing| # Do things with thing end end </code></pre> <p>The trade-offs to this solution are:</p> <ul> <li>The complete query is executed to get the ID's</li> <li>An array of all the ID's is kept in memory</li> <li>Uses the MySQL specific FIELD() function</li> </ul> <p>Hope this helps!</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload