Sunday, November 24, 2013

Analyzing large Java heap dumps when Eclipse Memory Analyzer (MAT) UI fails

If you find yourself trying to analyze a big heap dump (20-30GB) downloaded from your production server to your staging/test machines.. only to find out that X-over-SSH is too slow then this article is for you.

As of Nov 2013, we have 2 options - Eclipse MAT and a hidden gem called Bheapsampler.

Option 1:
Eclipse Memory Analyzer is obviously the best tool for this job. However, trying to get the UI to run remotely is very painful. Launching Eclipse and updating the UI is an extra load on the JVM that is already busy analyzing a 30G heap dump. Fortunately, there is a script that comes with MAT to parse the the heap dump and generate HTML reports without ever having to launch Eclipse! It's just that the command line option is not well advertised.

Command line heap analysis using Eclipse MAT:

Assuming Eclipse MAT is installed and we are inside the mat/ directory, modify MemoryAnalyzer.ini heap settings to use a large heap to handle large dumps:

    -startup
    plugins/org.eclipse.equinox.
launcher_1.2.0.v20110502.jar
    --launcher.library
    plugins/org.eclipse.equinox.launcher.gtk.linux.x86_64_1.1.
100.v20110505
    -vmargs
    -Xms24g
    -Xmx24g


Run MAT against the heap dump:

    ./ParseHeapDump.sh ../today_heap_dump/jvm.hprof

This takes a while to execute and generates indices and other files to make repeated analysis faster. Then use the indices created in the previous step and run a "Leak suspects" report on the heap dump.

    ./ParseHeapDump.sh ../today_heap_dump/jvm.hprof org.eclipse.mat.api:suspects

The output is a small and easy to download jvm_Leak_Suspects.zip. This has HTML files just like the MAT Eclipse UI. It can be easily SCP'ed/emailed around.

Other report types possible.

    org.eclipse.mat.api:suspects
    org.eclipse.mat.api:overview
    org.eclipse.mat.api:top_
components
   
More details - http://wiki.eclipse.org/index.php/MemoryAnalyzer/FAQ.

Option 2:
http://dr-brenschede.de/bheapsampler is something I chanced upon. It is a sampling heap dump reader and so it works for very large heap dumps where MAT sometimes fails. Being a sampling reader, the output is also a little imprecise but helps a great deal when you have nothing else. The tool seems to be closed source and is very sensitive to heap dump corruptions.

As an aside, here's something that might be useful for the initial heap dump quickly - https://blogs.atlassian.com/2013/03/so-you-want-your-jvms-heap/.

5 comments:

Charles Roth said...

Sounds great... except when I try it (ssh'd through to bash on a linux box), I get "Unable to initialize GTK+". Sounds like it really wants to start up some kind of UI. Any thoughts?

Ashwin Jayaprakash said...

I haven't tried it on Linux in a while. I think your best bet would be to post on the Eclipse MAT mailing list.

Charles Roth said...

The problem seems to be that the (newer?) versions of ParseHeapDump.sh call the MemoryAnalyzer program directly, which in turn expects GTK+ to be there.

I made my own version of ParseHeapDump.sh that does this instead:
/tools/java/java/bin/java -Xmx16g -Xms16g \
-jar plugins/org.eclipse.equinox.launcher_1.3.100.v20150511-1540.jar \
-consoleLog -consolelog -application org.eclipse.mat.api.parse "$@"

And that seems to work fine. (Substitute the location of your own 'java' for the /tools/java... above.)

Ashwin Jayaprakash said...

Thanks for posting the solution!

Unknown said...

Use "Memory Analyzer 1.0.0". It works perfect as described by Ashwin.