Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-16932

Memory Leak in Sun JVM tripped by Plugins2

    XMLWordPrintable

Details

    Description

      The Plugins2 system in JIRA4 appears to be leaking heap and permgen when restarting it.

      Restarting Plugins2 (in JIRA this happens when doing a full system import) when there are active Plugins2 plugins does not allow all memory previously referenced to be available for GC, most particularly there are a number of ClassLoaders left active with no obvious GC root the classes of whom are themselves roots for a whole graph of circularly referenced garbage. The plugins do not have to have been used for the leak to occur. There is not a one-one correspondence to restarts and leaks however (we generally see about 6 dead plugin systems although 30 or more restarts may have occured).

      In the process of diagnosing this we have tracked down and removed every possible reference to a large section of memory (a plugin framework) that we could find. Starting with all strong references and proceeding to remove soft and weak references - even things like clearing the java.lang.reflect.Proxy cache - and even Finalizer references until YourKit, Eclipse MAT, JProfiler and jhat profilers all report that the memory in question is dead and should be collectable, but inexplicably the JVM still holds on to it. There are no JNI Global references either, yet this memory remains uncollectable!

      This happens on all JVMs we have tried so far, which are the 1.5 and 1.6 JVMs on Linux and Mac, and the IBM 1.6 JDK on Linux.

      In practice this doesn't hurt a standard JIRA customer as a full system import is generally a one-off operation. It seriously affects runs of our functional test suite.

      This is a JVM GC bug (4957990). No known memory analysis tool can find the heap root (i.e. according to "the rules" there is no heap root). Are there any known GC memory leaks caused by ClassLoaders being dropped for instance?

      The application is creating and disposing a lot of ClassLoaders via OSGi (Apache Felix) with Spring OSGi. It creates a lot of java.lang.reflect.Proxy class instances.

      Strictly speaking this is not actually a memory leak as the additional memory use is bounded - memory use does not go up forever.

      Attachments

        1. java_pid10050.hprof.gz
          126.64 MB
          Jed Wesley-Smith
        2. java_pid10050-gc-debug-log.gz
          147 kB
          Jed Wesley-Smith
        3. java_pid2881.hprof.gz
          39.23 MB
          Jed Wesley-Smith
        4. tomcat-sun6-macos-cms.hprof.gz
          45.46 MB
          Jed Wesley-Smith

        Issue Links

          Activity

            People

              Unassigned Unassigned
              jed Jed Wesley-Smith (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: