Reindexing large number of issues with a lot of change history can cause an OOME

Summary

Reindexing an JIRA issue with a lot of change history will load all the changes for the issues. If issue has a lot changes, that resulting document will be quite big (30MB+).
If a lot issues requested to be reindex at the same time, that will put a lot of memory pressure for CHANGE_HISTORY-indexQueue and QueueingIndex and can cause an OOME.
Reindexing large number of issues with a lot of change history can cause an OOME.

Environment

• Large number of issues (100K+)
• Issues with a lot of change history (1k+)

Steps to Reproduce

1. Reindex issue

Expected Results

JIRA performs well

Actual Results

Reindex is very slow and JVM heap usage is high (2GB+). In some cases it can cause OOM.

Notes

• SQL to find large number of changes per issue:

  select concat(p.pkey,'-',ji.issuenum) as issue, count (ji.id) from changeitem ci join changegroup cg on cg.id = ci.groupid join jiraissue ji on cg.issueid = ji.id join project p on p.id = ji.project group by ji.issuenum,p.pkey order by count (ji.id) DESC;
     issue    | count
  ------------+-------
   PRG-5      |    50
   AG-4       |    17
   AG-12      |    14
  

• SQL to find large number of actions (comments ... ) per issue:

  select concat(p.pkey,'-',i.issuenum) as issue, count(i.id) from jiraaction a, jiraissue i, project p where i.project = p.id and i.id = a.issueid group by p.pkey,i.issuenum order by count (i.id) DESC;
  

Dominator tree from heap dump:

Workaround

• This is not a good workaround, but you might consider delete issues with large history

Note on fix

In Jira 9.0.0/8.22.4 we have introduced per-issue limits related to the number of indexed comments, worklogs and history.

Problem & solution description:

• https://community.developer.atlassian.com/t/safeguards-in-jira-dc-index
• https://community.atlassian.com/t5/Data-Center-articles/Introducing-Safeguards-for-Jira-index/ba-p/2037880#M433
• https://confluence.atlassian.com/jirakb/jira-indexing-limits-stats-1125876330.html