Databases
StackOverflow2013
Postsdbo
POSolr (JVM) peak every hour
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POSolr (JVM) peak every hour
    primarykey
    data
    text
    <p>SOLVED</p> <p>In our case the problem was that for the SuggestRequestHandler (requestHandler name="/suggest") now facelimit has been set: 10 Also there has been several requests for each single suggest request made by the application. Why this led to a (just) hourly peak is quite not clear yet...</p> <p>Thank you all for tips and help - I appreciated!</p> <p>Every full hour (12:00, 13:00, 14:00, ..., 20:00, 21:00, 22:00, 23:00) our Solr/Java process has a peak - which means the Java process where Solr is running increases 3x times CPU usage and the response time take - which usually takes msecs to respond, up to 9 seconds. Always for 2 - 3 minutes and only if there is traffic on our site (there is a php application which calls Java). Crond was completely disabled but still the problem on every full hour. And basically I think we tried almost every GC and memory combination (or maybe not?)</p> <p>Someone any idea why this happens - here some details:</p> <ul> <li>System: 32 GB RAM, 24 Core (mostly shared with php-fpm, but even isolated just solr as test same problem)</li> <li>Solr version 3.6 (on Jetty - temporarily also Glassfish)</li> <li>OS: RHEL 5.7</li> <li>Multicore setup (4 indexes with each 2 cores)</li> </ul> <p>Used Handlers (solrconfig.xml):</p> <pre><code>&lt;requestHandler name="standard" class="solr.SearchHandler" default="true"&gt; &lt;requestHandler name="dismax" class="solr.SearchHandler" &gt; &lt;requestHandler name="/suggest" class="solr.SearchHandler"&gt; &lt;requestHandler name="/update" class="solr.XmlUpdateRequestHandler" /&gt; &lt;requestHandler name="/analysis/document" class="solr.DocumentAnalysisRequestHandler" /&gt; &lt;requestHandler name="/analysis/field" class="solr.FieldAnalysisRequestHandler" /&gt; &lt;requestHandler name="/admin/" class="org.apache.solr.handler.admin.AdminHandlers" /&gt; &lt;requestHandler name="/admin/ping" class="PingRequestHandler"&gt; &lt;requestHandler name="/debug/dump" class="solr.DumpRequestHandler" &gt; &lt;requestHandler name="/replication" class="solr.ReplicationHandler" &gt; </code></pre> <p>(also tested without replication and ping)</p> <p>Used filters:</p> <pre><code>&lt;filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true" /&gt; &lt;filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" &lt;filter class="solr.LowerCaseFilterFactory"/&gt; &lt;filter class="solr.PortugueseMinimalStemFilterFactory"/&gt; &lt;filter class="solr.ISOLatin1AccentFilterFactory"/&gt; &lt;filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true"/&gt; &lt;filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" &lt;filter class="solr.LowerCaseFilterFactory"/&gt; &lt;filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="false"/&gt; &lt;filter class="solr.PortugueseMinimalStemFilterFactory"/&gt; &lt;filter class="solr.LowerCaseFilterFactory" /&gt; &lt;filter class="solr.EdgeNGramFilterFactory" maxGramSize="30" minGramSize="1"/&gt; &lt;filter class="solr.ASCIIFoldingFilterFactory"/&gt; &lt;filter class="solr.LowerCaseFilterFactory" /&gt; </code></pre> <p>Index size: ~100 MB (actually even a bit less)</p> <p>Current Java Options:</p> <pre><code>JAVA_OPTS="-Xmx4096m -Xms4096m -XX:+UseGCOverheadLimit -XX:+UseConcMarkSweepGC -XX:+UseTLAB -XX:MaxPermSize=128m -XX:+DisableExplicitGC -Dsun.rmi.dgc.server.gcInterval=300000 -Dsun.rmi.dgc.client.gcInterval=300000 -XX:NewRatio=1 -Xloggc:/shop/logs/live/solr/gc.log -verbose:gc -XX:+PrintGCDateStamps" </code></pre> <p>The same options but with 1024, 2048, 8192 and 12 GB didn't help at all.</p> <p>Other try:</p> <pre><code>JAVA_OPTS="-server -Xmx2048m -XX:MaxPermSize=128m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+UseTLAB -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:CMSIncrementalDutyCycleMin=0 -XX:CMSIncrementalDutyCycle=10 -XX:MaxTenuringThreshold=0 -XX:SurvivorRatio=256 -XX:CMSInitiatingOccupancyFraction=60 -XX:+DisableExplicitGC" </code></pre> <p>Other try:</p> <pre><code>JAVA_OPTS="-Xmx2048m -Xms2048m -XX:+UseGCOverheadLimit -XX:+UseConcMarkSweepGC -XX:+UseTLAB -XX:MaxPermSize=128m -XX:+DisableExplicitGC -Djava.util.logging.config.file=/opt/solr-jetty/etc/jetty-logging.properties" </code></pre> <p>Here a excerpt of the gc.log (of such a full hour problem):</p> <pre><code>2013-03-03T19:59:04.157-0300: 8087.754: [GC 3433559K-&gt;1788819K(3914560K), 0.0358190 secs] 2013-03-03T19:59:12.031-0300: 8095.628: [GC 3437075K-&gt;1792088K(3914560K), 0.0365830 secs] 2013-03-03T19:59:22.419-0300: 8106.016: [GC 3440344K-&gt;1803266K(3914560K), 0.0422040 secs] 2013-03-03T19:59:29.044-0300: 8112.641: [GC 3451522K-&gt;1815743K(3914560K), 0.0439870 secs] 2013-03-03T19:59:37.002-0300: 8120.599: [GC 3463999K-&gt;1821601K(3914560K), 0.0378990 secs] 2013-03-03T19:59:45.468-0300: 8129.065: [GC 3469857K-&gt;1822911K(3914560K), 0.0386720 secs] 2013-03-03T19:59:53.750-0300: 8137.347: [GC 3471167K-&gt;1829299K(3914560K), 0.0405040 secs] 2013-03-03T20:00:01.829-0300: 8145.426: [GC 3477555K-&gt;1832046K(3914560K), 0.0383070 secs] 2013-03-03T20:00:06.327-0300: 8149.924: [GC 3480302K-&gt;1831567K(3914560K), 0.0450550 secs] 2013-03-03T20:00:11.123-0300: 8154.719: [GC 3479823K-&gt;1843283K(3914560K), 0.0401710 secs] 2013-03-03T20:00:14.360-0300: 8157.957: [GC 3491539K-&gt;1854079K(3914560K), 0.0368560 secs] 2013-03-03T20:00:17.419-0300: 8161.015: [GC 3502335K-&gt;1855130K(3914560K), 0.0375530 secs] 2013-03-03T20:00:20.006-0300: 8163.603: [GC 3503386K-&gt;1861867K(3914560K), 0.0413470 secs] 2013-03-03T20:00:22.726-0300: 8166.323: [GC 3510123K-&gt;1870292K(3914560K), 0.0360600 secs] 2013-03-03T20:00:25.420-0300: 8169.017: [GC 3518548K-&gt;1872701K(3914560K), 0.0326970 secs] 2013-03-03T20:00:27.138-0300: 8170.735: [GC 3520957K-&gt;1873446K(3914560K), 0.0381430 secs] 2013-03-03T20:00:28.748-0300: 8172.345: [GC 3521702K-&gt;1889189K(3914560K), 0.0379160 secs] 2013-03-03T20:00:30.404-0300: 8174.001: [GC 3537445K-&gt;1887193K(3914560K), 0.0407670 secs] 2013-03-03T20:00:32.713-0300: 8176.309: [GC 3535449K-&gt;1892863K(3914560K), 0.0366880 secs] 2013-03-03T20:00:34.791-0300: 8178.388: [GC 3541119K-&gt;1899095K(3914560K), 0.0398270 secs] 2013-03-03T20:00:36.533-0300: 8180.129: [GC 3547351K-&gt;1910071K(3914560K), 0.0373960 secs] 2013-03-03T20:00:39.037-0300: 8182.634: [GC 3558327K-&gt;1904198K(3914560K), 0.0393020 secs] 2013-03-03T20:00:41.548-0300: 8185.144: [GC 3552454K-&gt;1912352K(3914560K), 0.0444060 secs] 2013-03-03T20:00:43.771-0300: 8187.368: [GC 3560608K-&gt;1919304K(3914560K), 0.0427220 secs] 2013-03-03T20:00:47.411-0300: 8191.008: [GC 3566354K-&gt;1918102K(3914560K), 0.0418150 secs] 2013-03-03T20:00:50.925-0300: 8194.522: [GC 3564290K-&gt;1930888K(3914560K), 0.0414700 secs] 2013-03-03T20:00:52.991-0300: 8196.588: [GC 3579144K-&gt;1933251K(3914560K), 0.0349600 secs] 2013-03-03T20:00:53.027-0300: 8196.624: [GC 1939697K(3914560K), 0.0256300 secs] 2013-03-03T20:00:54.208-0300: 8197.804: [GC 2780505K(3914560K), 0.1424860 secs] 2013-03-03T20:00:55.684-0300: 8199.281: [GC 3029503K-&gt;1389766K(3914560K), 0.0370380 secs] 2013-03-03T20:00:58.289-0300: 8201.886: [GC 2213458K-&gt;570843K(3914560K), 0.0413220 secs] 2013-03-03T20:01:00.672-0300: 8204.268: [GC 1962741K-&gt;319619K(3914560K), 0.0410840 secs] 2013-03-03T20:01:02.906-0300: 8206.503: [GC 1966833K-&gt;319605K(3914560K), 0.0453730 secs] 2013-03-03T20:01:06.861-0300: 8210.458: [GC 1967861K-&gt;330864K(3914560K), 0.0425570 secs] 2013-03-03T20:01:10.067-0300: 8213.664: [GC 1979120K-&gt;336541K(3914560K), 0.0479380 secs] 2013-03-03T20:01:12.587-0300: 8216.184: [GC 1984797K-&gt;343203K(3914560K), 0.0376810 secs] </code></pre> <p>Also there are just 2 entries at all (about 1 day) greater than 1 second: grep -oP ", [1-9]..*?secs]$" /shop/logs/live/solr/gc.log , 1.1727270 secs] , 1.0390840 secs]</p> <p>Someone any idea or already had this phenomenon with solr/jvm?</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USGeorg Buske
    1. USGeorg Buske
    plurals
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTLinked
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POSolr (JVM) peak every hour
      1. This table or related slice is empty.
      1. VTApproveEditSuggestion
    2. VO
      singulars
      1. POSolr (JVM) peak every hour
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POSolr (JVM) peak every hour
      1. This table or related slice is empty.
      1. VTUpMod
    1. COtry to disable the last three request handlers in your list, see what happens. Also, how do u trigger the document analysis?
      singulars
      1. POSolr (JVM) peak every hour
      1. USItay Moav -Malimovka
    2. COHave you exclude GC activity? I found you had printed GC activity in `-Xloggc:/shop/logs/live/solr/gc.log`. If you did so, please include it in your question.
      singulars
      1. POSolr (JVM) peak every hour
      1. USericson
    3. COIs it possibly something else that runs every hour on the computer? Or a bot visiting every hour? Or your ISP throttling every hour?
      singulars
      1. POSolr (JVM) peak every hour
      1. USPatashu
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload