In the last several weeks we've been seeing a lot of confluence instabilities at wikis.sun.com - all of them were related to running out of heap space. Several iterations of increasing Xmx didn't help (we started at 3GB and now we are at 5GB and 64bit JVM).

      I took several memory dumps during outages and analyzed them with Eclipse Memory Analyzer, which repeatedly found two issues:

      • Something is storing Xerces SaxParser objects as ThreadLocal variables, this results in up to 90MB being retained per thread and I see several instances of this size being held in memory causing total of 800-1200MB of the memory to be retained
      • Hundreds of instances of net.sf.hibernate.impl.SessionImpl retain additional ~780MB of memory - I'll document this as a separate issue

      Just before taking the heap dump, I also took a thread dump. By comparing the two I found that threads that were holding on the the huge thread local variables were currently in the containers thread pool and were not processing any requests - thus should have minimal memory requirements.

      I'm attaching some annotated screenshots from Eclipse Memory Analyzer and a thread dump that proves that the misbehaving threads were idle.

        1. wikis-threaddump-090320_1106.txt
          274 kB
        2. SAXParserInstanceListing.jpg
          SAXParserInstanceListing.jpg
          151 kB
        3. ThreadInstanceListing.jpg
          ThreadInstanceListing.jpg
          166 kB
        4. ThreadSuspectSummary.png
          ThreadSuspectSummary.png
          121 kB
        5. ThreadInstanceDrilldown.png
          ThreadInstanceDrilldown.png
          266 kB
        6. XMLReaderManager.class
          3 kB

            [CONFSERVER-14988] SAXParser memory leaks

            Katherine Yabut made changes -
            Workflow Original: JAC Bug Workflow v3 [ 2893207 ] New: CONFSERVER Bug Workflow v4 [ 2985212 ]
            Owen made changes -
            Workflow Original: JAC Bug Workflow v2 [ 2793167 ] New: JAC Bug Workflow v3 [ 2893207 ]
            Status Original: Resolved [ 5 ] New: Closed [ 6 ]
            Owen made changes -
            Workflow Original: JAC Bug Workflow [ 2727664 ] New: JAC Bug Workflow v2 [ 2793167 ]
            Owen made changes -
            Workflow Original: Confluence Workflow - Public Facing - Restricted v5 - TEMP [ 2392371 ] New: JAC Bug Workflow [ 2727664 ]
            Katherine Yabut made changes -
            Workflow Original: Confluence Workflow - Public Facing - Restricted v5 [ 2273771 ] New: Confluence Workflow - Public Facing - Restricted v5 - TEMP [ 2392371 ]
            Katherine Yabut made changes -
            Workflow Original: Confluence Workflow - Public Facing - Restricted v5.1 - TEMP [ 2228127 ] New: Confluence Workflow - Public Facing - Restricted v5 [ 2273771 ]
            Katherine Yabut made changes -
            Workflow Original: Confluence Workflow - Public Facing - Restricted v5 - TEMP [ 2184196 ] New: Confluence Workflow - Public Facing - Restricted v5.1 - TEMP [ 2228127 ]
            Katherine Yabut made changes -
            Workflow Original: Confluence Workflow - Public Facing - Restricted v5 [ 1952787 ] New: Confluence Workflow - Public Facing - Restricted v5 - TEMP [ 2184196 ]
            Katherine Yabut made changes -
            Workflow Original: Confluence Workflow - Public Facing - Restricted v3 [ 1745839 ] New: Confluence Workflow - Public Facing - Restricted v5 [ 1952787 ]
            Katherine Yabut made changes -
            Workflow Original: CONF Bug Subtask WF (TEMP) [ 1708051 ] New: Confluence Workflow - Public Facing - Restricted v3 [ 1745839 ]

              alynch Andrew Lynch (Inactive)
              15d9a6950818 Igor Minar
              Affected customers:
              0 This affects my team
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: