Databases
StackOverflow2013
Postsdbo
POGetting an advance warning before full GC
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POGetting an advance warning before full GC
    primarykey
    data
    text
    <p>For the context of a soft real time system that should not pause for more than 200ms, We're looking for a way to have an advance warning before a Full GC is imminent. We realize we might not be able to avoid it, but we'd like to fail over to another node before the system stalls.</p> <p>We've been able to come up with a scheme that will provide us with an advance warning, ahead of imminent full GC that may cause the system to stall for several seconds (which we need to avoid).</p> <p>What we've been able to come up with relies on CMS free list statistics: <code>-XX:PrintFLSStatistics=1</code>. This prints free list statistics into the GC log after every GC cycle, including young GC, so the information is available at short intervals, and will appear even more frequently during intervals of high memory allocation rate. It probably costs a little in terms of performance, but our working assumption is that we can afford it.</p> <p>The output to the log looks like so:</p> <pre><code>Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 382153298 Max Chunk Size: 382064598 Number of Blocks: 28 Av. Block Size: 13648332 Tree Height: 8 </code></pre> <p>In particular, the maximum free chunk size is 382064598 words. With 64-bit words this should amount to just below 2915MB. This number has been decreasing very slowly, at a rate of roughly 1MB per hour.</p> <p>It is our understanding that so long as the maximum free chunk size is larger than the young generation (assuming no humungous object allocation), every object promotion should succeed.</p> <p>Recently, we've run a several-days-long stress tests, and have been seeing that CMS was able to maintain maximum chunk sizes upward of 94% of total old region space. The maximum free chunk size appears to be decreasing at a rate of less than 1MB/hour, which should be fine -- according to this we won't be hitting full GC any time soon, and the servers will likely be down for maintenance more frequently than full GC can occur.</p> <p>In a previous test, at a time when the system was less memory efficient, we've been able to run the system for a good 10 hours. During the first hour, the maximum free chunk size has decreased to 100MB, where it stayed for over 8 hours. During the last 40 minutes of the run, the maximum free chunk size has decreased at a steady rate towards 0, when a full GC occurred -- this was very encouraging, because for that workload we seemed to be able to get a 40 minute advance warning (when the chunk size started a steady decline towards 0).</p> <p><strong>My question to you</strong>: assuming this all reflects a prolonged peak workload (workload at any given point in time in production will only be lower), does this sound like a valid approach? To what degree of reliability do you reckon we should be able to count on the maximum free chunk size statistic from the GC log?</p> <p>We are definitely open for suggestions, but request that they be limited to solutions available on HotSpot (No Azul for us, at least for now). Also, G1 by itself is no solution unless we can come up with a similar metric that will give us advance warning before Full GCs, or any GCs that significantly exceed our SLA (and these can occasionally occur).</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USnadavwr
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POGetting an advance warning before full GC
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POGetting an advance warning before full GC
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POGetting an advance warning before full GC
      1. This table or related slice is empty.
      1. VTUpMod
    1. COIs it feasible for you to test JRockit Deterministic GC? http://docs.oracle.com/cd/E15289_01/doc.40/e15071/intro.htm#i1010645
      singulars
      1. POGetting an advance warning before full GC
      1. USfglez
    2. COWe are aware of it, as well as IBM's and Oracle's other real time offerings. It is important for us to have some weaker guarantees (or even just heuristics) that enable us to deploy on HotSpot as well.
      singulars
      1. POGetting an advance warning before full GC
      1. USnadavwr
    3. COHave you considered forcing periodic full GCs on alternating nodes? This would allow more predictable behaviour. CMS is not predictable in the long term as fragmentation grows.
      singulars
      1. POGetting an advance warning before full GC
      1. USfglez
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload