LexoRank Rebalance can cause index replication delays in JIRA Datacenter

Summary

LexoRank Rebalance can cause index replication delays in JIRA Datacenter

Environment

• JIRA Datacenter
• Large number of issues 1M+
• Large number of custom fields: 1k+

Steps to Reproduce

1. Setup JIRA Datacenter
2. Trigger or wait for LexoRank Rebalance

Expected Results

JIRA Datacenter performance will not experience replication lag

Actual Results

There will be replication lag. That will cause data discrepancy between nodes.
You will have following health-check error:

["Index replication for cluster node 'node3' is behind by 2,991 seconds.","Index replication for cluster node 'node1' is behind by 1,501 seconds.","Index replication for cluster node 'node2_0004' is behind by 2,123 seconds."]

Notes

During LexoRank Rebalance, node that is performing Rebalance can issue extensive number of requests to reindex issues on other nodes in the cluster. In some cases those other nodes might not be able to keep up with amount of work to be performed. This in turn can cause replication lag. That will cause temporary data discrepancy between nodes.

Workarounds

• Leave only one node in LB to prevent serving stale data from other nodes. This negates the high availability value of Data Center so is considered a last resort. This also requires that each node is capable of handling the full concurrent user traffic for your organization, as is the best practice for an HA cluster.

Full details on workarounds and solutions are available at JIRAKB/JIRA Software Data Center Lexorank Indexing Lag.

Note on fix

Since JIRA Server 7.2.8 LexoRank Rebalancing will pause and wait until indexes on all live nodes in cluster are up to date. This should prevent LexoRank Rebalancing from causing index replication lag.