Databases
StackOverflow2013
Postsdbo
POFinding the cause of a memory leak in Ruby
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POFinding the cause of a memory leak in Ruby
    text
    copied!<p>I've discovered a memory leak in my Rails code - that is to say, I've found <em>what</em> code leaks but not <em>why</em> it leaks. I've reduced it down to a testcase that doesn't require Rails:</p> <pre><code>require 'csspool' require 'ruby-mass' def report puts 'Memory ' + `ps ax -o pid,rss | grep -E "^[[:space:]]*#{$$}"`.strip.split.map(&amp;:to_i)[1].to_s + 'KB' Mass.print end report # note I do not store the return value here CSSPool::CSS::Document.parse(File.new('/home/jason/big.css')) ObjectSpace.garbage_collect sleep 1 report </code></pre> <p><a href="https://github.com/archan937/ruby-mass">ruby-mass</a> supposedly lets me see all the objects in memory. <a href="https://github.com/JasonBarnabe/csspool">CSSPool</a> is a CSS parser based on <a href="https://github.com/tenderlove/racc">racc</a>. /home/jason/big.css is <a href="https://gist.github.com/JasonBarnabe/7799067">a 1.5MB CSS file</a>.</p> <p>This outputs:</p> <pre><code>Memory 9264KB ================================================== Objects within [] namespace ================================================== String: 7261 RubyVM::InstructionSequence: 1151 Array: 562 Class: 313 Regexp: 181 Proc: 111 Encoding: 99 Gem::StubSpecification: 66 Gem::StubSpecification::StubLine: 60 Gem::Version: 60 Module: 31 Hash: 29 Gem::Requirement: 25 RubyVM::Env: 11 Gem::Specification: 8 Float: 7 Gem::Dependency: 7 Range: 4 Bignum: 3 IO: 3 Mutex: 3 Time: 3 Object: 2 ARGF.class: 1 Binding: 1 Complex: 1 Data: 1 Gem::PathSupport: 1 IOError: 1 MatchData: 1 Monitor: 1 NoMemoryError: 1 Process::Status: 1 Random: 1 RubyVM: 1 SystemStackError: 1 Thread: 1 ThreadGroup: 1 fatal: 1 ================================================== Memory 258860KB ================================================== Objects within [] namespace ================================================== String: 7456 RubyVM::InstructionSequence: 1151 Array: 564 Class: 313 Regexp: 181 Proc: 113 Encoding: 99 Gem::StubSpecification: 66 Gem::StubSpecification::StubLine: 60 Gem::Version: 60 Module: 31 Hash: 30 Gem::Requirement: 25 RubyVM::Env: 13 Gem::Specification: 8 Float: 7 Gem::Dependency: 7 Range: 4 Bignum: 3 IO: 3 Mutex: 3 Time: 3 Object: 2 ARGF.class: 1 Binding: 1 Complex: 1 Data: 1 Gem::PathSupport: 1 IOError: 1 MatchData: 1 Monitor: 1 NoMemoryError: 1 Process::Status: 1 Random: 1 RubyVM: 1 SystemStackError: 1 Thread: 1 ThreadGroup: 1 fatal: 1 ================================================== </code></pre> <p>You can see the memory going <strong>way</strong> up. Some of the counters go up, but no objects specific to CSSPool are present. I used ruby-mass's "index" method to inspect the objects that have references like so:</p> <pre><code>Mass.index.each do |k,v| v.each do |id| refs = Mass.references(Mass[id]) puts refs if !refs.empty? end end </code></pre> <p>But again, this doesn't give me anything related to CSSPool, just gem info and such.</p> <p>I've also tried outputting "GC.stat"...</p> <pre><code>puts GC.stat CSSPool::CSS::Document.parse(File.new('/home/jason/big.css')) ObjectSpace.garbage_collect sleep 1 puts GC.stat </code></pre> <p>Result:</p> <pre><code>{:count=&gt;4, :heap_used=&gt;126, :heap_length=&gt;138, :heap_increment=&gt;12, :heap_live_num=&gt;50924, :heap_free_num=&gt;24595, :heap_final_num=&gt;0, :total_allocated_object=&gt;86030, :total_freed_object=&gt;35106} {:count=&gt;16, :heap_used=&gt;6039, :heap_length=&gt;12933, :heap_increment=&gt;3841, :heap_live_num=&gt;13369, :heap_free_num=&gt;2443302, :heap_final_num=&gt;0, :total_allocated_object=&gt;3771675, :total_freed_object=&gt;3758306} </code></pre> <p>As I understand it, if an object is not referenced and garbage collection happens, then that object should be cleared from memory. But that doesn't seem to be what's happening here.</p> <p>I've also read about C-level memory leaks, and since CSSPool uses Racc which uses C code, I think this is a possibility. I've run my code through Valgrind:</p> <pre><code>valgrind --partial-loads-ok=yes --undef-value-errors=no --leak-check=full --fullpath-after= ruby leak.rb 2&gt; valgrind.txt </code></pre> <p>Results are <a href="https://gist.github.com/JasonBarnabe/7862509">here</a>. I'm not sure if this confirms a C-level leak, as I've also read that Ruby does things with memory that Valgrind doesn't understand.</p> <p>Versions used:</p> <ul> <li>Ruby 2.0.0-p247 (this is what my Rails app runs)</li> <li>Ruby 1.9.3-p392-ref (for testing with ruby-mass)</li> <li>ruby-mass 0.1.3</li> <li>CSSPool 4.0.0 from <a href="https://github.com/JasonBarnabe/csspool/releases">here</a></li> <li>CentOS 6.4 and Ubuntu 13.10</li> </ul>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload