Uploaded image for project: 'Confluence Data Center'
  1. Confluence Data Center
  2. CONFSERVER-38699

Workbox Plugin loads full HTML of JIRA comment, leads to GC loop of death on large comment

    XMLWordPrintable

Details

    Description

      To reproduce:

      • start Confluence with GC logging enabled (optional, but helps)
      • Link Confluence and JIRA
      • create an issue in JIRA
      • watch it
      • add a large comment to the JIRA issue, e.g. paste a 7.7MB log file between {code} tags
      • open the workbox in Confluence (optional: in network tab of web developer tools)
      • check
        • CPU usage of Confluence: it switches between one http thread using 100% CPU to render the comment and as many threads as the GC uses to collect the garbage that the thread produces
        • GC log of Confluence: you will see frequent full garbage collections, all freeing big parts of the heap (in our case for example: Full GC every 5 seconds, heap 5G -> 1G, taking 2-5 seconds)
        • if Confluence is not dead yet, check the network tab in the browser: /rest/mywork/latest/notification/nested?limit=30 loads a huge file (examples in our case: 21MB, 43MB before gzip), as it loads the whole comment in html
        • take a heap dump of Confluence just before a garbage collection. When analyzing in Eclipse Memory Analyzer, use -keep_unreachable_objects
          • you will find big char arrays/Strings containing the various parts of the comment, embedded in org.apache.xerces.dom.TextImpl, which suggests that the whole comment is parsed as html/xml and kept in memory at once
      • check the table AO_9412A1_AONOTIFICATION in the Confluence database. It contains several entries with a large DESCRIPTION field, even though only one large comment was added. It seems that one comment triggers new entries in this table again and again, which is also reflected in the workbox that says "9 comments on <ISSUEKEY>", although there is only one comment.

      On a production system with a reasonable load, enough request queue up during the GC pauses to render the system unusable - garbage is produced faster than it can be collected.

      Note: this has security implications as well, as any user that can comment on issues can DOS the system easily - they do not even require another user to watch the ticket, as they are notified of their own JIRA actions in Confluence, see CONF-30081.

      Suggestions for mitigation:

      • JIRA should not allow comments that big
      • JIRA should not send the whole comment to Confluence
      • Confluence should stop processing the comment after a few lines, and show a "read more" link instead of the whole comment (like the JIRA activity stream does)

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              3bea9b9e5330 Martin Sander
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: