Asynchronous cache replication can cause extra overhead in case of large number cache updates and many stale nodes

Summary

Asynchronous cache replication can cause extra overhead in case of large number cache updates and many stale nodes.

Environment

• Jira DC
• A large number of stale nodes (see JRASERVER-42916)
• Plugin (code) generating a large number of cache update events, eg reaching 2000 messages/min.

Steps to Reproduce

1. Open a URL which produces the cache update event while computing the business logic
   • Eg. #* /rest/servicedesk/1/<PRJ>/webfragments/sections/sd-queues-nav,servicedesk.agent.queues,servicedesk.agent.queues.ungrouped
2. Measure response time and number of replication events

Expected Results

Performance doesn't degrade with a number of old nodes.

Actual Results

Performance degrades with a number of old stale nodes.

• While taking thread dumps you can see a lot of threads busy in the following stack:

    java.lang.Thread.State: RUNNABLE
  	at java.io.RandomAccessFile.writeBytes(Native Method)
  	at java.io.RandomAccessFile.write(RandomAccessFile.java:512)
  	at com.squareup.tape.QueueFile.writeHeader(QueueFile.java:184)
  	at com.squareup.tape.QueueFile.add(QueueFile.java:321)
  	- locked <0x00000003ce9b43e0> (a com.squareup.tape.QueueFile)
  	at com.squareup.tape.FileObjectQueue.add(FileObjectQueue.java:46)
  	at com.atlassian.jira.cluster.distribution.localq.tape.TapeLocalQCacheOpQueue.add(TapeLocalQCacheOpQueue.java:151)
  	at com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpQueueWithStats.add(LocalQCacheOpQueueWithStats.java:115)
  	at com.atlassian.jira.cluster.distribution.localq.LocalQCacheManager.addToQueue(LocalQCacheManager.java:370)
  	at com.atlassian.jira.cluster.distribution.localq.LocalQCacheManager.addToAllQueues(LocalQCacheManager.java:354)
  	at com.atlassian.jira.cluster.distribution.localq.LocalQCacheReplicator.replicateToQueue(LocalQCacheReplicator.java:85)
  	at com.atlassian.jira.cluster.distribution.localq.LocalQCacheReplicator.replicatePutNotification(LocalQCacheReplicator.java:65)
  	at com.atlassian.jira.cluster.cache.ehcache.AbstractJiraCacheReplicator.notifyElementUpdated(AbstractJiraCacheReplicator.java:123)
  	at net.sf.ehcache.event.RegisteredEventListeners.internalNotifyElementUpdated(RegisteredEventListeners.java:228)
  	at net.sf.ehcache.event.RegisteredEventListeners.notifyElementUpdated(RegisteredEventListeners.java:206)
  ...
  

• From client's case, we saw 15 - 20% of all threads doing replicateToQueue

Notes

None

Workaround

Clean-up old node data manually, see JRASERVER-42916