ResourceBundle ClassLoader lock contention under heavy load can soft-lock the system on Java 6

Stash's I18nService uses ResourceBundle.getBundle to load internationalized messages. Internally, ResourceBundle.getBundle uses ClassLoader.loadClass to load locale-specific bundles. The synchronization around ClassLoader.loadClass in Java 6 (and previous) can result in lock contention hotspots which cause the system to starve to a near-halt. The ClassLoader internals were significantly retooled for Java 7, greatly reducing this contention. It is highly recommended to run Stash on a Java 7 JVM.

The tops of affected call stacks look like this:

"http-bio-7990-exec-10236" daemon prio=10 tid=0x56c75400 nid=0x48df waiting for monitor entry [0x3c3e0000]
   java.lang.Thread.State: BLOCKED (on object monitor)
  at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1593)
  - waiting to lock <0x74203f70> (a org.apache.catalina.loader.WebappClassLoader)
  at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
  at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2289)
  at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1364)
  at java.util.ResourceBundle.findBundle(ResourceBundle.java:1328)
  at java.util.ResourceBundle.findBundle(ResourceBundle.java:1282)
  at java.util.ResourceBundle.findBundle(ResourceBundle.java:1282)
  at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1224)
  at java.util.ResourceBundle.getBundle(ResourceBundle.java:952)
  at com.atlassian.stash.internal.i18n.I18nServiceImpl.getBundle(I18nServiceImpl.java:197)
  at com.atlassian.stash.internal.i18n.I18nServiceImpl.getText(I18nServiceImpl.java:102)

The WebappClassLoader is a primary hotspot, since it's the parent ClassLoader for Stash itself. A jstack dump on a server with this issue showed 108 different http-bio threads waiting for the lock on that ClassLoader.

Systems which are under heavy polling load (CI servers checking for changes, FishEye indexing, etc) are particularly susceptible to this, depending on the polling interval. Due to how git fetch/git clone handle authantication handshaking, each one produces a series of 3-4 HTTP requests. Each of those, in turn, result in many ResourceBundle-provided messages being loaded. If the load becomes heavy, the lock contention can push out request times to a point where the next wave of polling requests cascade onto the first, further increasing contention. The system is not deadlocked; requests do still drain, but too slowly. The rising load eventually triggers some further form of failure, like an OutOfMemoryError due to the resources being held open in so many starving threads.