Cluster Cache Replication Health Check can fail with a false positive in JIRA Data Center

Summary

Depending on the order of startup in a clustered JIRA instance, it's possible the cluster cache replication health check may fail initially on startup. However when accessing the health check page it shows as a pass.

Environment

JIRA Data Center (JDC) only.

Steps to Reproduce

This is a race condition so will only occur on certain times that JDC is started. It is exacerbated by Zero Downtime Upgrades as this involves more frequent restarting of JIRA nodes.

Expected Results

The health check does not fail, as the order of heartbeat propagation is correct.

Actual Results

The health check reports failure, however when executing the check again it passes.

Notes

This is due to the timing of startup - the health check implements its own heartbeat that is used to propagate cache data across nodes. This heart beat scheduled is started after the plugin has started and the plugin framework is available. We also have a health check scheduler is scheduled to execute a minute after startup. Due to the timing of the various starts it's possible the health check scheduler is executed before the heartbeat has propagated the cache. Due to this the check will initially fail.