Jira Data Center local indexes get inconsistent over time

Summary

Difference in number of issues in index and the database increase over time, until a lock & re-index operation is done to get all indexes in sync again.

Environment

Jira Data Center

Steps to Reproduce

Still unknown

Expected Results

Difference between number of issues in the database and in the nodes local indexes is eventually zero.

Actual Results

Difference in the number of issues in the database and in the nodes local indexes start to drift after a lock & re-index operation, and indexes will not catch up as expected.

The charts below were generated using the results of the REST API endpoint /rest/api/2/index/summary. Note how the delta between issues in the database and issues in the local index starts to drift. Indexes will remain inconsistent in relation to the database until a lock & re-index operation is performed.

Note on fix

• The following contributors to this issue that we've identified:
  • JRASERVER-64922
  • JRASERVER-65564
    • have been fixed and are awaiting release (fix versions: 7.6.11, 7.13.1, 8.0.0).
  • JRASERVER-68642 - pending
• As of 2019-01-02, we were unable to identify any other contributors to this issue.

Workaround

There's no workaround to prevent the issue from happening. It's necessary to perform a lock & re-index operation to make the indexes consistent again.