HPROF - Memory leak analysis

Monitoring and troubleshooting tools 

• Quest Foglight (JVM and garbage collection monitoring)
• jmap (hprof / Heap Dump generation tool)
• Memory Analyzer 1.1 via IBM support assistant (hprof Heap Dump analysis)
• Platform type: Middle tier

Step #1 – WLS 9.2 Admin server JVM monitoring and leak confirmation 

The Quest Foglight Java EE monitoring tool was quite useful to identify a Java Heap leak from our Weblogic Admin server. As you can see below, the Java Heap memory is growing over time. 

If you are not using any monitoring tool for your Weblogic environment, my recommendation to you is to at least enable verbose:gc of your HotSpot VM. Please visit my Java 7 verbose:gc tutorial on this subject for more detailed instructions. 

Step #2 – Generate a Heap Dump from your leaking JVM 

Following the discovery of a JVM memory leak, the goal is to generate a Heap Dump file (binary format) by using the Sun JDK jmap utility. 

** please note that jmap Heap Dump generation will cause your JVM to become unresponsive so please ensure that no more traffic is sent to your affected / leaking JVM before running the jmap utility ** 

1	/bin/jmap -heap:format=b

This command will generate a Heap Dump binary file (heap.bin) of your leaking JVM. The size of the file and elapsed time of the generation process will depend of your JVM size and machine specifications / speed. 

For our case study, a binary Heap Dump file of ~ 2 GB was generated in about 1 hour elapsed time. 

Sun HotSpot 1.5/1.6/1.7 Heap Dump file will also be generated automatically as a result of a OutOfMemoryError and by adding -XX:+HeapDumpOnOutOfMemoryError in your JVM start-up arguments. 

Step #3 – Load your Heap Dump file in Memory Analyzer tool 

It is now time to load your Heap Dump file in the Memory Analyzer tool. The loading process will take several minutes depending of the size of your Heap Dump and speed of your machine. 

Step #4 – Analyze your Heap Dump 

The Memory Analyzer provides you with many features, including a Leak Suspect report. For this case study, the Java Heap histogram was used as a starting point to analyze the leaking objects and the source. 

For our case study, java.lang.String and char[] data were found as the leaking Objects. Now question is what is the source of the leak e.g. references of those leaking Objects. Simply right click over your leaking objects and select >> List Objects > with incoming references 

As you can see, javax.management.ObjectName objects were found as the source of the leaking String & char[] data. The Weblogic Admin server is communicating and pulling stats from its managed servers via MBeans / JMX which create javax.management.ObjectName for any MBean object type. Now question is why Weblogic 9.2 is not releasing properly such Objects… 

Root cause: Weblogic javax.management.ObjectName leak! 

Following our Heap Dump analysis, a review of the Weblogic known issues was performed which did reveal the following Weblogic 9.2 bug below: 

• Weblogic Bug ID: CR327368
• Description: Memory leak of javax.management.ObjectName objects on the Administration Server used to cause OutOfMemory error on the Administration Server.
• Affected Weblogic version(s): WLS 9.2
• Fixed in: WLS 10 MP1

http://download.oracle.com/docs/cd/E11035_01/wls100/issues/known_resolved.html
This finding was quite conclusive given the perfect match of our Heap Dump analysis, WLS version and this known problem description. 

Conclusion 

I hope this tutorial along with case study has helped you understand how you can pinpoint the source of a Java Heap leak using jmap and the Memory Analyzer tool. 


Reference
http://www.javacodegeeks.com

JAVA Heap Analysis - GC limit exeeded

Troubleshooting tools

** all these tools can be downloaded for free **

•  Eclipse Indigo Release 
•  Memory Analyzer via IBM Support Assistant 4.1 (HotSpot Heap Dump analysis) 
• Java VM: Windows HotSpot JRE 1.6.0_24 64-bit

Sample Java program

The simple sample Java program below will be used to triggered an OutOfMemoryError; allowing us to analyze the generated HotSpot Heap Dump file. Simply create a new Java class : JVMOutOfMemoryErrorSimulator.java to the Eclipse project of your choice and either rename or keep the current package as is.

This program is basically creating multiple String instances within a Map data structure until the Java Heap depletion.

** please make sure your Eclipse compiler and JRE is 1.6 **

01	package org.ph.javaee.javaheap;

02

03	import java.util.Map;

04	import java.util.HashMap;

05

06

/**

07	* JVMOutOfMemoryErrorSimulator

08

*

09	* @author PH

10

*

11	*/public class JVMOutOfMemoryErrorSimulator {

12

13	private final static int NB_ITERATIONS = 500000;

14

15	// ~1 KB data footprint

16	private final static String LEAKING_DATA_PREFIX = "datadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadat

17	adatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadat

18	adatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadat

19	adatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadat

20	adatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadat

21	adatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadata";

22

23	// Map used to stored our leaking String instances

24	private static Map leakingMap;

25

26

static {

27	leakingMap = new HashMap();

28

}

29

30

/**

31	* @param args

32

*/

33	public static void main(String[] args) {

34

35	System.out.println("JVM OutOfMemoryError Simulator 1.0");

36	System.out.println("Author: Pierre-Hugues Charbonneau");

37	System.out.println(" http ://javaeesupportpatterns.blogspot.com/");

38

39

try {

40

41	for (int i = 0; i < NB_ITERATIONS; i++) {

42

43	String data = LEAKING_DATA_PREFIX + i;

44

45	// Add data to our leaking Map data structure...

46	leakingMap.put(data, data);

47

48

}

49

50	} catch (Throwable any) {

51	if (any instanceof java.lang.OutOfMemoryError){

52	System.out.println("OutOfMemoryError triggered! "

53	+ any.getMessage() + " [" + any + "]");

54

55

} else {

56	System.out.println("Unexpected Exception! " +

57	any.getMessage() + " [" + any + "]");

58

}

59

}

60

61	System.out.println("simulator done!");

62

}

63

64

}

Step #1 - Setup your JVM start-up arguments

First, setup your Eclipse Java runtime arguments as per below. For our example, we used an external JRE 1.6 outside the Eclipse IDE with a Java Heap maximum capacity of 512 MB.

The key JVM argument allowing us to generate a Heap Dump is -XX:+HeapDumpOnOutOfMemoryError which tells the JVM to generate a Heap Dump following an OutOfMemoryError condition.

Step #2 - Run the sample Java program

The next step is to run our Java program. Depending on your computer specs, this program will run between 5-30 seconds before existing with an OutOfMemoryError.

As you can see, the JVM generated a Heap Dump file java_pid3880.hprof . It is now time to fire the Memory Analyzer tool and analyze the JVM Heap Dump.

Step #3 - Load the Heap Dump

Analyzing a Heap Dump is an analysis activity that can be simple or very complex. The goal of this tutorial is to give you the basics of Heap Dump analysis. For more Heap Dump analysis, please refer to the other case studies of this Blog.

Step #4 - Analyze Heap Dump

Below are the snapshots and analysis steps that you can follow to understand the memory leak that we simulated in our sample Java program.

As you can see, the Heap Dump analysis using the Memory Analyzer tool was able to easily identify our primary leaking Java class and data structure.

Conclusion

I hope this simple Java program and Heap Dump analysis tutorial has helped you understand the basic principles of Java Heap analysis using the raw Heap Dump data. This analysis is critical when dealing with OutOfMemoryError: GC overhead problems since those are symptoms of either Java Heap leak of Java Heap footprint / tuning problem.

Reference
http://www.javacodegeeks.com