Issue Summary

Cluster-wide invalidation loop in JiraDelegatingDirectoryInstanceLoader cache causes high CPU and log volume during production load or rolling restarts.

Background:
The cache com.atlassian.jira.user.directory.loader.JiraDelegatingDirectoryInstanceLoader is initialized without a CacheLoader. This causes atlassian-cache to treat it as a non-self-loading cache, setting replicateUpdates=true.
Because the cache is not self-loading, it lacks the SynchronizedLoadingCacheDecorator. Under production load, concurrent misses on the same key result in multiple threads loading the same entry in parallel. Each subsequent put is classified as an UPDATE and broadcast as a cluster-wide invalidation message to all peer nodes.

This creates a self-sustaining feedback loop:

• Node A misses, repopulates, and broadcasts an invalidation.

• Nodes B through F evict the entry.

• Nodes B through F then miss, repopulate, and broadcast invalidations back to Node A.

• Node A evicts, and the cycle repeats indefinitely.

A node with a cold cache (e.g., during a rolling restart) acts as the strongest invalidation source, often worsening the incident until a full cluster cold restart is performed.

Steps to Reproduce

1. Run a Jira Data Center cluster under high concurrency.
2. Perform a rolling restart of one node and observe the increase in invalidation traffic on peer nodes.
3. Observe the volume of invalidation messages and CPU spikes across nodes.

Note This issue is was consistently reproducible during high concurrency and restart of nodes.

Expected Results

Cache repopulation should be synchronized and should not trigger cluster-wide invalidations for ordinary reads.

Actual Results

Concurrent misses trigger multiple "UPDATE" events, leading to a cluster-wide invalidation storm and high log volume (specifically from TapeLocalQCacheOpQueue).

Stacktrace of the threads stuck on I/O attempting to  write JiraDelegatingDirectoryInstanceLoader cache to the localQ causing high load and incrased I/O activity.

at java.base/java.io.RandomAccessFile.writeBytes0(Native Method)
		at java.base/java.io.RandomAccessFile.writeBytes(RandomAccessFile.java:572)
		at java.base/java.io.RandomAccessFile.write(RandomAccessFile.java:588)
		at com.squareup.tape.QueueFile.writeHeader(QueueFile.java:184)
		at com.squareup.tape.QueueFile.add(QueueFile.java:321)
		at com.squareup.tape.FileObjectQueue.add(FileObjectQueue.java:58)
		at com.atlassian.jira.cluster.distribution.localq.tape.TapeLocalQCacheOpQueue.add(TapeLocalQCacheOpQueue.java:167)
		at com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpQueueWithStats.add(LocalQCacheOpQueueWithStats.java:140)
		at com.atlassian.jira.cluster.distribution.localq.LocalQCacheManager.addToQueue(LocalQCacheManager.java:403)
		at com.atlassian.jira.cluster.distribution.localq.LocalQCacheManager.addToAllQueues(LocalQCacheManager.java:384)
		at com.atlassian.jira.cluster.distribution.localq.LocalQCacheReplicator.replicateToQueue(LocalQCacheReplicator.java:87)
		at com.atlassian.jira.cluster.distribution.localq.LocalQCacheReplicator.replicateRemovalNotification(LocalQCacheReplicator.java:73)
		at com.atlassian.jira.cluster.cache.ehcache.AbstractJiraCacheReplicator.notifyElementUpdated(AbstractJiraCacheReplicator.java:130)
		at net.sf.ehcache.event.RegisteredEventListeners.internalNotifyElementUpdated(RegisteredEventListeners.java:228)
		at net.sf.ehcache.event.RegisteredEventListeners.notifyElementUpdated(RegisteredEventListeners.java:206)
		at net.sf.ehcache.Cache.notifyPutInternalListeners(Cache.java:1646)
		at net.sf.ehcache.Cache.putInternal(Cache.java:1618)
		at net.sf.ehcache.Cache.put(Cache.java:1543)
		at net.sf.ehcache.Cache.getWithLoader(Cache.java:1863)
		at com.atlassian.cache.ehcache.DelegatingCache.get(DelegatingCache.java:108)
		at com.atlassian.cache.impl.metrics.InstrumentedCache.get(InstrumentedCache.java:70)
		at com.atlassian.jira.cache.stats.CacheWithStats.get(CacheWithStats.java:52)
		at com.atlassian.jira.user.directory.loader.JiraDelegatingDirectoryInstanceLoader.getDirectory(JiraDelegatingDirectoryInstanceLoader.java:71)
 

Workaround

• Declare the cache in a custom ehcache.xml under WEB-INF/classes/ehcache.xml with no replicator attached and restart Jira:
   

<cache name="com.atlassian.jira.user.directory.loader.JiraDelegatingDirectoryInstanceLoader"eternal="true" maxEntriesLocalHeap="0"><cacheEventListenerFactory class="com.atlassian.jira.NoopCacheEventListenerFactory"/>

</cache>