Page comparison in Confluence has a high heap usage and might contribute to OOM errors

Issue Summary

Comparing two versions of a page consumes a high amount of memory and might contribute to out of memory errors.

This is intensified when the Send batched notifications scheduled job runs and needs to send e-mails about changed content.

How to identify if the bug is affecting your instance during periods of high load:

1. High heap usage.
2. High GC activity, including concurrent and Full GC.
3. If a heap dump is taken, you will notice a significant number of instances (reachable or unreachable) of any of the below classes.
   • org.eclipse.compare.rangedifferencer
   • com.atlassian.confluence.diff
   • org.outerj.daisy.diff

Environment

This was validated on Confluence Server on version 6.13.2.
It can definitely impact 6.14.1 since there were no improvements on that area, but wasn't tested on that version.
This affects Confluence Server or Data Center on any version prior (and including) to the highlighted above.

Steps to Reproduce

1. Fresh install of Confluence.
2. Start your preferred live monitoring tool and start to monitor your Confluence instance.
   • Jconsole is used throughout this test.
3. Create a test space named Test Page Comparison.
4. Create a sample page named Sample Page Comparison and publish the 1st version with no contents.
5. Edit and publish a new version of the page with the contents from sample_page_v1.txt storage format.
   • You need to use the Source Editor to edit the storage format of a page.
6. Edit and publish a new version of the page with the contents from sample_page_v2.txt storage format.
7. Access the Page History of Sample Page Comparison and at this point you should have 3 versions as the image below.
   
   • Keep this page opened in a tab.
8. In a new tab, access the Cache Statistics admin page and flush all caches.
9. Go back to the tab with the Page History of Sample Page Comparison opened.
10. Select v. 2 and CURRENT (v. 3) and click on Compare selected versions.

Expected Results

Confluence performs page comparison using minimum server resources.

Actual Results

Confluence uses a lot of heap to perform page comparison.

Notes

The situation may get worse on large Confluence instances, such as on Enterprise deployments with Confluence Data Centers, on which users perform multiple changes on pages of different complexity.
When the Send batched notifications scheduled job runs it will compare versions of watched pages and send e-mails to users.
This requires more heap and CPU resources as shown in the below graph after 5 pages were edited with the same contents as the example above.

Performance on Scale

There are some situations on which the performance is impacted depending on the scale:

1. Heap usage increases depending on the number of watchers on the page – for cases triggered by the batch notifications job.
2. Heap usage increases depending on the number of pages to be processed – for cases triggered by the batch notifications job.
3. Heap usage increases depending on the complexity of the page and the changes between versions.

Number of watchers of the page

1 watcher on a sample page

1,000 watchers on the same page

Number of pages to process

10 pages with 1 watcher per page

250 pages with 1 watcher per page

Workaround

There's no workaround when the user needs to compare page versions from the web UI.
To minimize the impact that could be caused by the Send batched notifications scheduled job, users must disable the Show changed content option in their Email Notifications option.
There's an enhancement request to globally disable this functionality, as described in CONFSERVER-43671. You may want to check one of the workarounds described in this suggestion report.

As a practical example of the benefits of disabling this functionality on all users, the screenshot below shows heap and CPU usage while running the Send batched notifications on the same scenario of the Notes Section, but all users watching all pages had the Show changed content option disabled.

Users will still receive an e-mail notifying changes on the watched page, but no page comparison is run.

Further Reading

Page History and Page Comparison
Email Notifications
How to globally disable e-mail notifications with page content
CONFSERVER-43671 – Ability to disable "Show Changed Content" site-wide