Summary

While DC is put under load project cache is sometimes not replicated on some nodes. It is not even eventually consistent - the project is still not visible long after load is gone.

Environment

This is DC specific only

Steps to Reproduce

1. Generate load on all nodes (constant load over time, causes many hits on project cache e.g. view issue, view boards, view gadgets)
2. Create project on one of the nodes
3. Verify if the project is visible on all other nodes
4. Repeat steps 2 and 3

Actual Results

At some point of time project is not replicated on some node. It will not replicate even after long time when load is gone.

Expected Results

Project is always eventually replicated on all nodes.

Note

• The same behaviour was present since the beginning of Jira DC
• Root cause: There was a race condition when receiving cache replication call on the nodes. If node was busy loading cache value it would ignore incoming cache invalidation calls and a newly loaded value (already stall) would be left in cache indefinitely. That's why it was rarely happening: only during certain load and only on some nodes. Note projects cache is just example here - it could happen to any cache.

Workaround

Create yet one more project on a node that has problem. Then cache gets flushed and all projects are visible again.

Fix

Stop ignoring incoming cache replication request while loading cache