DC index replication delays are affecting end-users due to single thread processing and re-computation of CF values

Summary

JIRA Datacenter will automatically keep all the copies of the index up to date automatically. This synchronisation is not fully synchronous, but aims for eventual consistency. This means that there is some delay before index changes are seen on other nodes in the cluster.

Each indexing operation writes a row to the database table replicatedindexoperation. All of the nodes then look for entries in this table that were inserted by nodes other than themselves. They then apply the changes to their local Lucene index by recollecting the CF values and recreating Lucene document.

see Lucene Index Synchronization in Jira Data Center for more details.

In some cases, DC index replication takes longer due to the high volume of changes and expensive Custom Field computations. This causes the replication queue to grow significantly and delay changes applied to the node. Those delays are affecting end-users.

Environment

• Jira DC
• Large number of Custom Fields

Steps to Reproduce

1. TBD

Expected Results

Index replication is relatively fast and doesn't cause problem to end-users (eg, within 5min)

Actual Results

Index replication queue grows significantly. Delay is 20+ min.

Notes

• To check the replication queue size, you can run API call /rest/api/2/index/summary and check queueSize value.

Workaround

• Reduce the number of CF
• Reduce complexity of expensive CF.