Databases
StackOverflow2013
Postsdbo
POCan memory pressure on Solaris lead to SIGSEGVs in JVM?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POCan memory pressure on Solaris lead to SIGSEGVs in JVM?
    primarykey
    data
    text
    <p>I’m running several memory hungry Java server processes on Sun JVMs in a Solaris container. Recently I added more processes and may have exhausted the total available memory. Java processes started to crash randomly under load. In every single case it was a SIGSEGV in the JVM in thread GCTaskThread. At one point even <code>top</code> died with a SIGSEGV.</p> <p><strong>Does Solaris kill processes with SIGSEGV when memory gets short?</strong></p> <p>Instead of SIGSEGV I would have expected to get OutOfMemoryErrors at random locations. Also I always thought that segfaults in the JVM indicate bugs (unless JNI is involved). I need to know whether I have hit an additional JVM bug.</p> <p>This is Solaris 10 and 32-bit JRE 1.6u20.</p> <hr> <pre><code># # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0xfeb6297c, pid=26468, tid=6 # # JRE version: 6.0_20-b02 # Java VM: Java HotSpot(TM) Server VM (16.3-b01 mixed mode solaris-sparc ) # Problematic frame: # V [libjvm.so+0x76297c] # # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp # --------------- T H R E A D --------------- Current thread (0x0003bc00): GCTaskThread [stack: 0x8b300000,0x8b380000] [id=6] siginfo:si_signo=SIGSEGV: si_errno=0, si_code=2 (SEGV_ACCERR), si_addr=0x8b828bfb Registers: O0=0xf824d1d0 O1=0xdcdc43e8 O2=0xf824d1d3 O3=0x00000001 O4=0xfee2d800 O5=0x00000003 O6=0x8b37f718 O7=0xfe546d18 G1=0xfe55a854 G2=0x90d7f7b4 G3=0xfffffff4 G4=0x00000000 G5=0xfee2dd5f G6=0x00000000 G7=0xfeed2200 Y=0x00000000 PC=0xfeb6297c nPC=0xfeb62980 Top of Stack: (sp=0x8b37f718) 0x8b37f718: 8bc00000 00033158 fee3f000 fbc00000 0x8b37f728: 70000000 1c000000 8b3a2000 00033200 0x8b37f738: fee3b800 00486bfb 00000011 00000003 0x8b37f748: f824d1d3 dbc00000 8b37f778 fe55a854 0x8b37f758: ed204000 00001000 00000000 00000032 0x8b37f768: 8737f7f0 0003c3a8 8b37f7ec 0191c568 0x8b37f778: 0003c3a8 00000010 00000003 00000003 0x8b37f788: 00000009 8b37f84c 00000004 8b37f84c Instructions: (pc=0xfeb6297c) 0xfeb6296c: ee 06 21 b8 b3 36 60 09 b4 10 20 11 ec 05 e0 4c 0xfeb6297c: f4 2d 80 19 81 c7 e0 08 81 e8 20 00 9d e3 bf a0 Stack: [0x8b300000,0x8b380000], sp=0x8b37f718, free space=1fdfeb6297ck Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0x76297c] V [libjvm.so+0x15a85c] V [libjvm.so+0x486acc] V [libjvm.so+0x169314] V [libjvm.so+0x16bda0] V [libjvm.so+0x24c2a4] V [libjvm.so+0x72a1a0] --------------- P R O C E S S --------------- Java Threads: ( =&gt; current thread ) 0x027bf800 JavaThread "pool-3-thread-31" [_thread_blocked, id=1171, stack(0x90d00000,0x90d80000)] 0x028df000 JavaThread "AWT-EventQueue-0" [_thread_blocked, id=1169, stack(0x91400000,0x91480000)] 0x028dec00 JavaThread "AWT-Shutdown" [_thread_blocked, id=1168, stack(0x91700000,0x91780000)] 0x00dd4800 JavaThread "Product-producer-thread-7" [_thread_blocked, id=1165, stack(0x90b00000,0x90b80000)] 0x03a18400 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" [_thread_in_native, id=580, stack(0x85100000,0x85180000)] 0x00f83c00 JavaThread "BaseComTimerThread-9" daemon [_thread_blocked, id=287, stack(0x88d80000,0x88e00000)] 0x014d5000 JavaThread "PACCommand_7" [_thread_blocked, id=105, stack(0x86500000,0x86580000)] 0x0109b800 JavaThread "PACCommand_6" [_thread_blocked, id=104, stack(0x86600000,0x86680000)] 0x0109b000 JavaThread "PACCommand_5" [_thread_blocked, id=103, stack(0x86700000,0x86780000)] 0x00d75800 JavaThread "PACCommand_4" [_thread_blocked, id=102, stack(0x86800000,0x86880000)] 0x0106ac00 JavaThread "PACCommand_3" [_thread_blocked, id=101, stack(0x86900000,0x86980000)] 0x01529000 JavaThread "PACCommand_2" [_thread_blocked, id=100, stack(0x86a00000,0x86a80000)] 0x0131e800 JavaThread "PACCommand_1" [_thread_blocked, id=99, stack(0x86b00000,0x86b80000)] 0x01163400 JavaThread "BaseComSingleThread-1" daemon [_thread_blocked, id=91, stack(0x86d00000,0x86d80000)] 0x002f6800 JavaThread "FailoverExecutorThread-0" daemon [_thread_blocked, id=90, stack(0x87500000,0x87580000)] 0x00dd6800 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=89, stack(0x86f00000,0x86f80000)] 0x00b8fc00 JavaThread "NonGUI_5" [_thread_blocked, id=87, stack(0x86e00000,0x86e80000)] 0x0091d000 JavaThread "NonGUI_4" [_thread_blocked, id=86, stack(0x87000000,0x87080000)] 0x0191e800 JavaThread "NonGUI_3" [_thread_blocked, id=82, stack(0x87100000,0x87180000)] 0x01fa0400 JavaThread "NonGUI_2" [_thread_blocked, id=81, stack(0x87200000,0x87280000)] 0x018f6800 JavaThread "NonGUI_1" [_thread_blocked, id=80, stack(0x87300000,0x87380000)] 0x00ef0400 JavaThread "Timer legends" [_thread_blocked, id=79, stack(0x87400000,0x87480000)] 0x01179400 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=75, stack(0x87600000,0x87680000)] 0x009bd800 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=73, stack(0x87700000,0x87780000)] 0x00931800 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=71, stack(0x88980000,0x88a00000)] 0x00a6e400 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=69, stack(0x88a80000,0x88b00000)] 0x0113b400 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=67, stack(0x88b80000,0x88c00000)] 0x00c72000 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=65, stack(0x87900000,0x87980000)] 0x001bbc00 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=63, stack(0x87a00000,0x87a80000)] 0x00f45c00 JavaThread "Access Layer MDT Update Thread #0" daemon [_thread_blocked, id=60, stack(0x89900000,0x89980000)] 0x00a70c00 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=59, stack(0x87800000,0x87880000)] 0x00a68800 JavaThread "AWT-XAWT" daemon [_thread_in_native, id=53, stack(0x87b00000,0x87b80000)] 0x008ecc00 JavaThread "ProductTimer" [_thread_blocked, id=52, stack(0x87d00000,0x87d80000)] 0x00030400 JavaThread "DestroyJavaVM" [_thread_blocked, id=2, stack(0xfe300000,0xfe380000)] 0x00e34c00 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" [_thread_in_native, id=48, stack(0x87e00000,0x87e80000)] 0x00e32800 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=47, stack(0x87f00000,0x87f80000)] 0x00f68400 JavaThread "Proxy.cleaner" daemon [_thread_blocked, id=45, stack(0x88000000,0x88080000)] 0x00f67400 JavaThread "Thread-10" [_thread_blocked, id=44, stack(0x88100000,0x88180000)] 0x00f66400 JavaThread "Timer-1" [_thread_blocked, id=43, stack(0x88200000,0x88280000)] 0x00f64c00 JavaThread "CORBA cleaner" daemon [_thread_blocked, id=42, stack(0x88300000,0x88380000)] 0x00f64400 JavaThread "Timer-0" [_thread_blocked, id=41, stack(0x88400000,0x88480000)] 0x0117e800 JavaThread "BaseComExecutorThread-4" [_thread_blocked, id=40, stack(0x88500000,0x88580000)] 0x01fbd400 JavaThread "BaseComExecutorThread-3" [_thread_blocked, id=39, stack(0x88600000,0x88680000)] 0x011ad400 JavaThread "BaseComExecutorThread-2" [_thread_blocked, id=38, stack(0x88700000,0x88780000)] 0x011acc00 JavaThread "Product.scheduler.thread" [_thread_blocked, id=37, stack(0x88800000,0x88880000)] 0x00a9f800 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=35, stack(0x88e80000,0x88f00000)] 0x00e95000 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" [_thread_in_native, id=34, stack(0x89400000,0x89480000)] 0x00c14400 JavaThread "BaseComExecutorThread-1" [_thread_blocked, id=32, stack(0x89500000,0x89580000)] 0x00e92000 JavaThread "ConfigCacheCleaner4" daemon [_thread_blocked, id=31, stack(0x89600000,0x89680000)] 0x00f4e400 JavaThread "ConfigCacheCleaner3" daemon [_thread_blocked, id=30, stack(0x89700000,0x89780000)] 0x00cb6000 JavaThread "Orbacus:GIOPConnectionThreaded:ReceiverThread" daemon [_thread_in_native, id=29, stack(0x89800000,0x89880000)] 0x00ba0800 JavaThread "BaseComTimerThread-1" daemon [_thread_blocked, id=25, stack(0x89a00000,0x89a80000)] 0x00133000 JavaThread "TaskRouterThread-1" daemon [_thread_blocked, id=24, stack(0x89b00000,0x89b80000)] 0x00f3c000 JavaThread "NINJO_CACHE_1333544439197_ScratchConfigCache1.3_2.data" daemon [_thread_blocked, id=23, stack(0x89c00000,0x89c80000)] 0x00c00400 JavaThread "NINJO_CACHE_1333544439197_ScratchConfigCache1.3_1.data" daemon [_thread_blocked, id=22, stack(0x89d00000,0x89d80000)] 0x00ed6c00 JavaThread "NINJO_CACHE_1333544439197_ScratchConfigCache1.3.data" daemon [_thread_blocked, id=21, stack(0x89e00000,0x89e80000)] 0x00ba3c00 JavaThread "com.product.fwk.common.cache.impl.EHCacheManager@b02928" daemon [_thread_blocked, id=20, stack(0x89f00000,0x89f80000)] 0x00aa3400 JavaThread "com.product.fwk.common.cache.impl.EHCacheManager@b02928" daemon [_thread_blocked, id=19, stack(0x8a000000,0x8a080000)] 0x00ab5400 JavaThread "ConfigCacheCleaner2" daemon [_thread_blocked, id=18, stack(0x8a100000,0x8a180000)] 0x00c0d800 JavaThread "ConfigCacheCleaner1" daemon [_thread_blocked, id=17, stack(0x8a200000,0x8a280000)] 0x00e40000 JavaThread "ORB-Thread" [_thread_blocked, id=16, stack(0x8a300000,0x8a380000)] 0x00e3e800 JavaThread "ORBacus:Server:StarterThread" [_thread_in_native, id=15, stack(0x8a400000,0x8a480000)] 0x00127c00 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=13, stack(0x8aa80000,0x8ab00000)] 0x00126000 JavaThread "CompilerThread1" daemon [_thread_blocked, id=12, stack(0x8ab80000,0x8ac00000)] 0x00123800 JavaThread "CompilerThread0" daemon [_thread_blocked, id=11, stack(0x8ac80000,0x8ad00000)] 0x00122000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=10, stack(0x8ad80000,0x8ae00000)] 0x0010d000 JavaThread "Finalizer" daemon [_thread_blocked, id=9, stack(0x8ae80000,0x8af00000)] 0x0010b800 JavaThread "Reference Handler" daemon [_thread_blocked, id=8, stack(0x8af80000,0x8b000000)] Other Threads: 0x00109400 VMThread [stack: 0x8b080000,0x8b100000] [id=7] 0x0012ac00 WatcherThread [stack: 0x8a980000,0x8aa00000] [id=14] =&gt;0x0003bc00 (exited) GCTaskThread [stack: 0x8b300000,0x8b380000] [id=6] VM state:at safepoint (normal execution) VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event]) [0x0002f578] Threads_lock - owner thread: 0x00109400 [0x0002f988] Heap_lock - owner thread: 0x00dd4800 Heap PSYoungGen total 457280K, used 83928K [0xdbc00000, 0xfbc00000, 0xfbc00000) eden space 391808K, 21% used [0xdbc00000,0xe0df6178,0xf3aa0000) from space 65472K, 0% used [0xf3aa0000,0xf3aa0000,0xf7a90000) to space 61632K, 4% used [0xf7fd0000,0xf8254130,0xfbc00000) PSOldGen total 1048576K, used 331538K [0x9bc00000, 0xdbc00000, 0xdbc00000) object space 1048576K, 31% used [0x9bc00000,0xaffc4b68,0xdbc00000) PSPermGen total 49152K, used 45797K [0x8bc00000, 0x8ec00000, 0x9bc00000) object space 49152K, 93% used [0x8bc00000,0x8e8b9690,0x8ec00000) Dynamic libraries: 0x00010000 /opt/product/release_1.6/server/jre/bin/java 0xff3a0000 /lib/libthread.so.1 0xff370000 /opt/product/release_1.6/server/jre/bin/../lib/sparc/jli/libjli.so 0xff350000 /lib/libdl.so.1 0xff200000 /lib/libc.so.1 0xff390000 /platform/SUNW,SPARC-Enterprise/lib/libc_psr.so.1 0xfe400000 /opt/product/release_1.6/server/jre/lib/sparc/server/libjvm.so 0xff1d0000 /lib/libsocket.so.1 0xff1f0000 /usr/lib/libsched.so.1 0xff1b0000 /lib/libm.so.1 0xff180000 /usr/lib/libCrun.so.1 0xff160000 /lib/libdoor.so.1 0xff080000 /lib/libnsl.so.1 0xfef80000 /lib/libm.so.2 0xff050000 /lib/libscf.so.1 0xff140000 /lib/libuutil.so.1 0xff030000 /lib/libgen.so.1 0xfef50000 /lib/libmd.so.1 0xfef30000 /lib/libmp.so.2 0xfeea0000 /opt/product/release_1.6/server/jre/lib/sparc/libverify.so 0xfee60000 /opt/product/release_1.6/server/jre/lib/sparc/libjava.so 0xfe3e0000 /opt/product/release_1.6/server/jre/lib/sparc/native_threads/libhpi.so 0xfe2e0000 /lib/nss_files.so.1 0xfe2b0000 /usr/lib/nss_vas4.so.1 0xfe280000 /opt/quest/lib/libvtsmartcache.so.1 0xfe180000 /opt/quest/lib/libvtutil.so.1 0xfe230000 /opt/quest/lib/libvtcacheipc.so.1 0xfe120000 /lib/libresolv.so.2 0xfe100000 /opt/product/release_1.6/server/jre/lib/sparc/libzip.so 0xfe0b0000 /usr/lib/locale/en_US.ISO8859-1/en_US.ISO8859-1.so.3 0x8ae50000 /opt/product/release_1.6/server/jre/lib/sparc/libnet.so 0x89200000 /opt/product/release_1.6/server/jre/lib/sparc/libawt.so 0x8ad20000 /opt/product/release_1.6/server/jre/lib/sparc/xawt/libmawt.so 0x8ae10000 /usr/openwin//lib/libXext.so.0 0x89080000 /usr/openwin//lib/libX11.so.4 0x8ac50000 /usr/openwin//lib/libXtst.so.1 0x8ac20000 /usr/openwin//lib/libXi.so.5 0x88f80000 /opt/product/release_1.6/server/jre/lib/sparc/libfontmanager.so 0x8b110000 /opt/product/release_1.6/server/jre/lib/sparc/libnio.so 0x8ab60000 /lib/librt.so.1 0x8ab40000 /lib/libaio.so.1 0x8af10000 /usr/lib/libsendfile.so.1 0x887d0000 /opt/product/release_1.6/server/jre/lib/sparc/libmanagement.so 0x887b0000 /usr/openwin/lib/locale/common/xlibi18n.so.2 0x886d0000 /usr/X11/lib/libXcursor.so.1 0x886a0000 /usr/openwin/sfw/lib/libXrender.so.1 0x8a850000 /opt/product/release_1.6/server/jre/lib/sparc/libdcpr.so VM Arguments: jvm_args: -Xmx1536M -Xms1536M -XX:MaxPermSize=256m -XX:-ReduceInitialCardMarks -Djava.ext.dirs=/opt/product/release_1.6/server/lib/ext -Djava.endorsed.dirs=/opt/product/release_1.6/server/lib/boot -Dorg.omg.CORBA.ORBClass=com.ooc.CORBA.ORB -Dorg.omg.CORBA.ORBSingletonClass=com.ooc.CORBA.ORBSingleton -Dcorba.port=23801 -Dserver.name=ProductServer -Dserver.basedir=/data/product -Dproduct.log.path=../../log -Dproduct.i18n.locale.language=en java_command: /opt/product/release_1.6/server/lib/product/ProductMain.jar /opt/product/release_1.6/server/config.properties Launcher Type: SUN_STANDARD Environment Variables: CLASSPATH=/opt/local/opkg/share/oracle/product/10.2.0/jdbc/lib/classes12.zip PATH=/usr/bin:/usr/sbin:/usr/local/bin:/opt/local/opkg/bin:/opt/local/opkg/sbin:/home/ext/user/bin:/opt/local/opkg/share/oracle/product/10.2.0/bin:/usr/bin:/usr/sbin:/usr/local/bin:/opt/local/opkg/bin:/opt/local/opkg/sbin:/home/users/user/bin:/usr/bin:/home/users/user/bin LD_LIBRARY_PATH=/opt/product/release_1.6/server/jre/lib/sparc/server:/opt/product/release_1.6/server/jre/lib/sparc:/opt/product/release_1.6/server/jre/../lib/sparc:/opt/local/opkg/share/oracle/product/10.2.0/lib32:/opt/local/opkg/share/oracle/product/10.2.0/lib:/opt/local/opkg/share/oracle/product/10.2.0/jdbc/lib32 SHELL=/bin/bash DISPLAY=localhost:63.0 Signal Handlers: SIGSEGV: [libjvm.so+0x84dd5c], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c SIGBUS: [libjvm.so+0x84dd5c], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c SIGFPE: [libjvm.so+0x1c0330], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c SIGPIPE: [libjvm.so+0x1c0330], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c SIGXFSZ: [libjvm.so+0x1c0330], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c SIGILL: [libjvm.so+0x1c0330], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000 SIGUSR2: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000 SIGQUIT: [libjvm.so+0x72c934], sa_mask[0]=0xffbffeff, sa_flags=0x00000004 SIGHUP: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000 SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000 SIGTERM: [libjvm.so+0x72c934], sa_mask[0]=0xffbffeff, sa_flags=0x00000004 SIG39: [libjvm.so+0x730360], sa_mask[0]=0x00000000, sa_flags=0x00000008 SIG40: [libjvm.so+0x1c0330], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c --------------- S Y S T E M --------------- OS: Solaris 10 10/09 s10s_u8wos_08a SPARC Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Use is subject to license terms. Assembled 16 September 2009 uname:SunOS 5.10 Generic_142909-17 sun4u (T2 libthread) rlimit: STACK 8192k, CORE infinity, NOFILE 8192, AS infinity load average:7.79 3.58 3.31 CPU:total 4 has_v8, has_v9, popc, has_vis1, has_vis2, is_ultra3 Memory: 8k page, physical 15728640k(1399920k free) vm_info: Java HotSpot(TM) Server VM (16.3-b01) for solaris-sparc JRE (1.6.0_20-b02), built on Apr 12 2010 14:27:21 by "" with Workshop 5.8 time: Wed Apr 4 13:50:18 2012 elapsed time: 2982 seconds </code></pre>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. UShfs
    1. UShfs
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. COSince you wrote even top failed with SIGSEGV , it is unlikely a jvm issue. A possibility is you somehow changed shared library dependency. What is there is in the hs_err_<pid>.log file?
      singulars
      1. POCan memory pressure on Solaris lead to SIGSEGVs in JVM?
      1. USJayan
    2. COIn some testing that I've done before, the JVM actually uses the SIGSEGV signal to determine if there's a NullPointerException. Based on your statement that 'top' is also failing with SIGSEGV, my best guess would be that the programs are simply assuming that malloc(or some variant thereof) is never failing; they get a NULL pointer back, and that causes a SIGSEGV. Possibly Java is doing the same thing, since it only happens in GCTaskThread?
      singulars
      1. POCan memory pressure on Solaris lead to SIGSEGVs in JVM?
      1. USrm5248
    3. CO@rm5248: Yes, many _handled_ SIGSEGVs are used for memory management, see e.g. [this question](http://stackoverflow.com/questions/3731784/a-lot-of-sigsegv-while-straceing-java-process). The canonical error message when memory allocation fails is "java.lang.OutOfMemoryError: requested … bytes for …. Out of swap space?", so I guess allocation is actually checked.
      singulars
      1. POCan memory pressure on Solaris lead to SIGSEGVs in JVM?
      1. UShfs
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload