.. Or: Friends don't let friends Grok and Drive
I will admit two personal things about JVM tuning: 1) I thought I was pretty hot stuff at JVM tuning 2) I was recently completely humbled and admit I was naive.
On a long road trip recently driving up I95 (23 hours of driving to be exact) I listened to approximately 16+ hours of JVM tuning talks by: Gil Tene, Charlie Hunt, Attila Szegedi, and a few others. These guys are the JVM tuning giants of the industry. I learned a bunch from them in these terse talks - challenged by not getting lost or hitting stuff while driving.
I will lay out here some practical things I learned while listening to them. The talks I listened to ranged in time period from 2009 to present time. What was interesting to "see" was the slow adaptation of theirs talks over time. Mostly the same corpus of info, with slight tweaks as JVM technology has progressed. I will save one point for last. Well the last point is not exactly a suggestion, but really more of a "revelation". It made me somewhat sad initially, but I relate to it in some manner.
Don't bother tuning until you have measured. I would say that attempting to tune a Java application without measurement is like tweaking a race car for performance without ever driving it. Get a baseline of performance and use a profiler to attempt to find issues before you tune, and based on what you find, then start tuning things.
Don't use: finalizers, WeakReferences, and try to size your Collection sizes appropriately. All Garbage Collectors will absolutely find 100% of "dead" Object's without needing a "finalizer" or help from a developer. I think these are pretty common suggestions: I knew most of these before I listened to the talks. Finalizers especially. The story Gil relates is a C++ architect writing his first Java application. The proper paradigm for development in C++ is to make sure Objects are reclaimed with delete. To ensure this happened in his new Java application, he used finalizers in every single class. Because the architect used this paradigm, all of the developers followed suit. What resulted was multi hour STW (stop the world) GC pauses. Not exactly ideal performance.
Be careful using class extension or inner classes. While these may be necessary, a composition model may be better when it's possible. Class extension can cause certain references to be maintained for extra cycles after the containing class has gone out of scope. The same goes for inner classes. This can cause some classes that would have been GC-ed in the Young Gen to be promoted to the Old Gen, and then GC-ed from there. You don't want that to happen.
Be practical, but don't worry about creating many short lived Objects. It takes very few CPU cycles (I think just a couple) to instantiate a new Object. You want as many Objects created to die in the Young Generation due to "increased" GC performance of the Young Gen. Additionally, GC collection of them is free: only Live Objects are promoted, and the "dead" ones nothing need be done to them. A corollary, don't create large Objects. Obviously these are a drag on performance due to the increase in CPU cycles to create these Objects, as well as GC.
A young GC is performant because it is usually a fraction of the size of the Old Gen. This seems practical and I had never thought about it in this way. I had always accepted it as "Gospel" or some kind of magical property that GC for Young Gen was fast. It is simply because the size is usually small. Increase the overall heap and the Young Gen as well, GC times will increase. This is one of the two drivers in keeping overall small heap sizes: to keep Young Gen GC times low, as well as STW GC's.
This one makes sense as well: double the memory and you will "cut in half" the time spent in GC... Well not really. Eventually you are going to have a GC. It's inevitable. When that GC happens, you will double the time spent in GC due to the doubling in size of the Heap.
Your application will pause about 1 second per live Gigabyte of memory during its lifetime, and there is very little you will ever be able to do about this. This was one of the revelations for me, and was a little disconcerting. It made me ask the question: "If true, then why would I try to tune a JVM then?"
Because of JVM enhancements, it is no longer valid that heap min and max should be set to the same size. A corollary to this would be that when / if you use memory timing flags to tune your JVM don't use absolute values like setting the min and max heap size. This defeats the dynamic tuning algorithms of the JVM. Similar to this for setting the Young Gen, use percentages instead of concrete values. You want the JVM to run the performance algorithms to dynamically tune itself while the application runs. The question one of the speakers asked (Gil or Charlie) was: Do you think you are smarter than the JVM?
I will stop at this one: there were many more. This one knocked the keys out of my keyboard: when you tune an application you are really tuning the "current state" of your application. As soon as you make source code changes, you cannot rely on the tuning you performed previously because your application has now changed. Agreed, if the changes are not significant then maybe this doesn't apply. However, how many times during your development cycle do you make changes for a release that were not significant? So I thought: I have to completely retune my application at every release? Yeah, I don't think so...
This last point really took the enthusiasm out of me for JVM tuning. What I thought about was how much fun it used to be building 8088 IBM XT systems with full height MFM or RLL drives, setting IRQ's on network, video and modem cards. Woohoo! After your two dozenth system you just want the damn things to work. You don't want to mess with the innerds anymore. Well, at least me anyway.
I think my takeaway from all of these talks was that there is a lot more going on in the JVM than I really understood. Most of the "tuning" is going to be a bit more development approach and application profiling than JVM switch tweaking. One other takeaway was that there is no magic in any of the JVM switches to make GC pauses go away with the current state publicly available of JVM collectors.