JIRA becomes unresponsive because of growing recency queue in frequently used CacheReference caches from atlassian-cache

Summary

CachedReference in atlassian-cache has a max size set to 1. This means that internal guava cache implementation will maintain recency queue with elements for each read of this cache. This leads to performance problem at large instances as frequently accessed caches might generate more entries to recency queue that can be processed by the system.

Environment

• Large JIRA instance: 1M issues, 20k users, 1k CF

Steps to Reproduce

We don't have specific steps at the moment. Suggested steps

1. Generate high load and let system run for couple of hours
2. Remove load
3. Generate high load again at specific time, eg: 09:00 and then 10:00

Expected Results

JIRA performs well and incoming threads doesn't stall

Actual Results

JIRA very slow and doesn't responde. High numner of concurent runnign threads, incoming http threads busy processing recency queue.

grep -c expireEntries threadDump-20161019-1*

threadDump-20161019-1450-1.txt:21
threadDump-20161019-1450-2.txt:13
threadDump-20161019-1451-2.txt:8
threadDump-20161019-1451-3.txt:20
threadDump-20161019-1455-1.txt:12
threadDump-20161019-1455-2.txt:9
threadDump-20161019-1456-2.txt:5
threadDump-20161019-1456-3.txt:10
threadDump-20161019-1500-1.txt:1
threadDump-20161019-1500-2.txt:1
threadDump-20161019-1500-3.txt:1

Those threads will have stack traces that contain following or similar lines:

at com.google.common.cache.LocalCache$Segment.drainRecencyQueue()V(LocalCache.java:2625)
at com.google.common.cache.LocalCache$Segment.expireEntries(J)V(LocalCache.java:2648)

Notes

None

Workaround

None