Asynchronous cache replication in Jira Data Center

Problem Definition

Caches in Jira Data Center are synchronous and have no guaranteed delivery. Synchronous replication may cause problems with scalability, as each change in cache needs to be replicated to other nodes. It means each change is blocking other actions for the caller until completed. Second important thing is that replication event is not guaranteed to be delivered. If something goes wrong, there is an exception when replicating - the operation will not be redone and cluster may be inconsistent, until cache will be refreshed. 

Suggested Solution

Caches in Data Center should be replicated asynchronously (async) using persistent layer as a separation of HTTP thread and RMI replication mechanism.

• HTTP thread will be responsible only for persisting replication events to queues and replication itself should happen asynchronously to that thread.
• The order of replication events coming from one HTTP request should be preserved.
• Replication event is removed from queue only when there is confirmation of successful replication to other node.
• If replication can not be done, replication event will remain in queue, waiting for possibility to replicate