Problem Definition

Jira Datacenter uses EHCache as a cache implementation with a local wrapper LoadingCache (com.atlassian.cache.ehcache.LoadingCache), which is configured by default with a low number of stripes (mutexes).

In some cases, we see a very large contention inside the cache due to cross locking of different keys in the same stripe of the cache.

• Due to low number of stripes (64) there is a high chance of collision in the same stripe for large caches.
• End-users observed a performance degradation. Response time spikes are accumulated because of threads are waiting and not being able to utilize the hardware during that time.

Details:

• EhCache protects cache consistency by having a read/write lock for data per stripe (segment). It’s not as granular as by keys
• Reads/Writes to the same stripe will share the same lock, so reads for Key-A might be blocked by writes for Key-B if they are in the same segment.
• In Atlassian cache wrapper we have 64 stripes. The default number of stripes in EhCache is 2048 (DEFAULT_NUMBER_OF_MUTEXES) which will allow higher concurrency and low chance of collision.

Suggested Solution

Increase number of stripes (NUMBER_OF_MUTEXES).

Related code from net.sf.ehcache.concurrent.StripedReadWriteLockSync#getSyncForKey

 /**
     * The default number of locks to use. Must be a power of 2.
     * <p/>
     * The choice of 2048 enables 2048 concurrent operations per cache or cache store, which should be enough for most
     * uses.
     */
    public static final int DEFAULT_NUMBER_OF_MUTEXES = 2048;
    private final ReadWriteLockSync[] mutexes;
...

See related - https://ecosystem.atlassian.net/browse/CACHE-234

Workaround

Manually increase the number of stripes in EHCache. Add to JVM args:

JVM_SUPPORT_RECOMMENDED_ARGS=" $JVM_SUPPORT_RECOMMENDED_ARGS -Dcom.atlassian.cache.ehcache.LoadingCache.DEFAULT_NUMBER_OF_MUTEXES=2048 "