Databases
StackOverflow2013
Postsdbo
POHow much memory should be assigned to Java on Tomcat, using Solr with 20+ M items in the index?
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHow much memory should be assigned to Java on Tomcat, using Solr with 20+ M items in the index?
    text
    copied!<p>I'm looking for some advice on memory allocation to Java on Tomcat 7, running Solr :-)</p> <p>Here are the facts:</p> <ul> <li>Running <strong>Solr 1.4.1</strong> with approx <strong>20 Million items</strong> of different sizes in the index.</li> <li>The index, when trimmed, is approx <strong>40-45 GB</strong> large.</li> <li>Approx <strong>40000-60000 articles</strong> are added to the index every <strong>month</strong>. Articles vary in size, from a couple of kBs to 15-20 kBs.</li> <li>Approx <strong>5000-8000 images</strong> are added to the index every <strong>month</strong>. (Actually not the images themselves but some meta data).</li> </ul> <p>Solr config specifics:</p> <ul> <li>mergeFactor 10 </li> <li>ramBufferSizeMB 128 MB</li> <li>maxMergeMB 1024.0 MB</li> </ul> <p>When doing a full re-index (on a Win 2007 server with Intel Xeon CPU (4 cores) + 6 GB memory), indexing diary entries takes approx 3,5 hrs and indexing images takes approx 4,5 hrs.</p> <p>However, we've upgraded the <strong>index server to 12 GB memory</strong> and assigned Java <strong>6144 MB for the maximum memory pool</strong>. <strong>Initial memory pool: 1024 MB</strong>.</p> <p>When restarting Apache, it takes approx 5-10 minutes for Tomcat to reach ~ 5 GB of memory usage. Another hour or two and it hits the 6 GB barrier and stays there.</p> <p>When we had a 2 GB barrier using 32 bit Apache, we got an OutOfMemoryError: Java Heap Space after 1-1,5 weeks. 3 GB added up to maybe 2-3 weeks.</p> <p>Right now we've run Solr on 64 bit Apache and Java with 6 GB assigned for a couple of days. So far no problems.</p> <p>Given above facts, what would you do to trim the Solr index and tweak memory usage?</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload