This section provides a number of examples which demonstrate how the error log can be used to find the cause of the crash, and suggests some tips for troubleshooting the problem depending on the cause.
The error log header indicates the type of error and the problematic frame, while the thread stack indicates the current thread and stack trace. See Header Format.
The following are possible causes for the crash.
If the fatal error log indicates the problematic frame to be a native library, there might be a bug in native code or the Java Native Interface (JNI) library code. The crash could of course be caused by something else, but analysis of the library and any core file or crash dump is a good starting place. Consider the extract in Example 5-1 from the header of a fatal error log.
Example 5-1 Extract from the Header of a Fatal Error Log
# An unexpected error has been detected by HotSpot Virtual Machine: # # SIGSEGV (0xb) at pc=0x417789d7, pid=21139, tid=1024 # # Java VM: Java HotSpot(TM) Server VM (6-beta2-b63 mixed mode) # Problematic frame: # C [libApplication.so+0x9d7]
In this case a SIGSEGV occurred with a thread executing in the library libApplication.so.
In some cases a bug in a native library manifests itself as a crash in Java VM code. Consider the crash in Example 5-2, where a JavaThread fails while in the _thread_in_vm state (meaning that it is executing in Java VM code).
Example 5-2 Example for a Crash
# An unexpected error has been detected by HotSpot Virtual Machine: # # EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x08083d77, pid=3700, tid=2896 # # Java VM: Java HotSpot(TM) Client VM (1.5-internal mixed mode) # Problematic frame: # V [jvm.dll+0x83d77] --------------- T H R E A D --------------- Current thread (0x00036960): JavaThread "main" [_thread_in_vm, id=2896] : Stack: [0x00040000,0x00080000), sp=0x0007f9f8, free space=254k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [jvm.dll+0x83d77] C [App.dll+0x1047] <========= C/native frame j Test.foo()V+0 j Test.main([Ljava/lang/String;)V+0 v ~StubRoutines::call_stub V [jvm.dll+0x80f13] V [jvm.dll+0xd3842] V [jvm.dll+0x80de4] V [jvm.dll+0x87cd2] C [java.exe+0x14c0] C [java.exe+0x64cd] C [kernel32.dll+0x214c7] :
In this case, although the problematic frame is a VM frame, the thread stack shows that a native routine in App.dll has called into the VM (probably with JNI).
The first step to solving a crash in a native library is to investigate the source of the native library where the crash occurred.
If the native library is provided by your application, then investigate the source code of your native library. A significant number of issues with JNI code can be identified by running the application with the -Xcheck:jni option added to the command line. See The -Xcheck:jni Option.
If the native library has been provided by another vendor and is used by your application, then file a bug report against this third-party application and provide the fatal error log information.
If the native library where the crash occurred is part of the Java Runtime Environment (JRE) (for example awt.dll, net.dll, and so forth), then it is possible that you have encountered a library or API bug. If so, gather as much data as possible and submit a bug or report, indicating the library name. You can find JRE libraries in the jre/lib or jre/bin directories of the JRE distribution. See Submit a Bug Report.
You can troubleshoot a crash in a native application library by attaching the native debugger to the core file or crash dump, if it is available. Depending on the OS, the native debugger is dbx, gdb, or windbg. See Native Operating System Tools.
If the fatal error log indicates that the crash occurred in compiled code, then it is possible that you have encountered a compiler bug that has resulted in incorrect code generation. You can recognize a crash in compiled code if the type of the problematic frame is J (meaning a compiled Java frame). Example 5-3 shows such a crash.
Example 5-3 Crash in Compiled Code
# An unexpected error has been detected by HotSpot Virtual Machine: # # SIGSEGV (0xb) at pc=0x0000002a99eb0c10, pid=6106, tid=278546 # # Java VM: Java HotSpot(TM) 64-Bit Server VM (1.6.0-beta-b51 mixed mode) # Problematic frame: # J org.foobar.Scanner.body()V # : Stack: [0x0000002aea560000,0x0000002aea660000), sp=0x0000002aea65ddf0, free space=1015k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) J org.foobar.Scanner.body()V [error occurred during error reporting, step 120, id 0xb]
Note: A complete thread stack is not available. The output line "error occurred during error reporting" means that a problem arose trying to obtain the stack trace (this might indicate stack corruption).
It might be possible to temporarily work around the issue by switching the compiler (for example, by using the HotSpot Client VM instead of the HotSpot Server VM, or visa versa) or by excluding from compilation the method that provoked the crash. In this specific example it might not be possible to switch the compiler as it was taken from a 64-bit Server VM and hence it might not be feasible to switch to a 32-bit Client VM.
For more information on possible workarounds, see Working Around Crashes in the HotSpot Compiler Thread or Compiled Code.
If the fatal error log output shows that the current thread is JavaThread named CompilerThread0, CompilerThread1, or AdapterCompiler, then it is possible that you have encountered a compiler bug. In this case it might be necessary to temporarily work around the issue by switching the compiler (for example, by using the HotSpot Client VM instead of the HotSpot Server VM, or visa versa), or by excluding from compilation the method that provoked the crash.
For more information on possible workarounds, see Working Around Crashes in the HotSpot Compiler Thread or Compiled Code.
If the fatal error log output shows that the current thread is VMThread, then look for the line containing VM_Operation in the THREAD section. VMThread is a special thread in the HotSpot VM. It performs special tasks in the VM such as garbage collection (GC). If the VM_Operation suggests that the operation is a GC, then it is possible that you have encountered an issue such as heap corruption.
Besides a GC issue, it could equally be something else (such as a compiler or runtime bug) that leaves object references in the heap in an inconsistent or incorrect state. In this case, collect as much information as possible about the environment and try possible workarounds. If the issue is related to GC, you might be able to temporarily work around the issue by changing the GC configuration.
For more information on possible workarounds, see Working Around Crashes during Garbage Collection.
A stack overflow in Java language code will normally result in the offending thread throwing the java.lang.StackOverflowError exception. On the other hand, C and C++ write past the end of the stack and provoke a stack overflow. This is a fatal error which causes the process to terminate.
In the HotSpot implementation, Java methods share stack frames with C/C++ native code, namely user native code and the virtual machine itself. Java methods generate code that checks whether stack space is available a fixed distance towards the end of the stack so that the native code can be called without exceeding the stack space. This distance towards the end of the stack is called "Shadow Pages". The size of the shadow pages is between 3 and 20 pages, depending on the platform. This distance is tunable, so that applications with native code needing more than the default distance can increase the shadow page size. The option to increase shadow pages is -XX:StackShadowPages=n, where n is greater than the default stack shadow pages for the platform.
If your application gets a segmentation fault without a core file or fatal error log file see Appendix A, or a STACK_OVERFLOW_ERROR on Windows, or the message "An irrecoverable stack overflow has occurred", this indicates that the value of StackShadowPages was exceeded and more space is needed.
If you increase the value of StackShadowPages, you might also need to increase the default thread stack size using the -Xss parameter. Increasing the default thread stack size might decrease the number of threads that can be created, so be careful in choosing a value for the thread stack size. The thread stack size varies by platform from 256 KB to 1024 KB.
Example 5-4 Stack Overflow Exception
# An unexpected error has been detected by HotSpot Virtual Machine: # # EXCEPTION_STACK_OVERFLOW (0xc00000fd) at pc=0x10001011, pid=296, tid=2940 # # Java VM: Java HotSpot(TM) Client VM (1.6-internal mixed mode, sharing) # Problematic frame: # C [App.dll+0x1011] # --------------- T H R E A D --------------- Current thread (0x000367c0): JavaThread "main" [_thread_in_native, id=2940] : Stack: [0x00040000,0x00080000), sp=0x00041000, free space=4k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) C [App.dll+0x1011] C [App.dll+0x1020] C [App.dll+0x1020] : C [App.dll+0x1020] C [App.dll+0x1020] ...<more frames>... Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) j Test.foo()V+0 j Test.main([Ljava/lang/String;)V+0 v ~StubRoutines::call_stub
You can interpret the following information from Example 5-4.
The exception is EXCEPTION_STACK_OVERFLOW.
The thread state is _thread_in_native, which means that the thread is executing native or JNI code.
In the stack information, the free space is only 4 KB (a single page on a Windows system). In addition, the stack pointer (sp) is at 0x00041000, which is close to the end of the stack at 0x00040000.
The printout of the native frames shows that a recursive native function is the issue in this case. The output notation ...<more frames>... indicates that additional frames exist but were not printed. The output is limited to 100 frames.
