projectAndUserToSecurityLevelCache can cause OOM due to large number of elements

Issue Summary

The projectAndUserToSecurityLevelCache cache (com.atlassian.jira.issue.security.IssueSecurityLevelManagerImpl.projectAndUserToSecurityLevelCache) can grow quite large, consuming most of JVM Heap and causing frequent Full GC, potentially leading to OOM and causing Jira to be unavailable.

The size of the cache depends on the number of elements (keys) and the size of each key.

• "Number of elements" is a product of # projects x # user, eg, for 100k users and 4k projects, will be 400M elements
• "Size" is a product of # Security Levels

Environment

1. Have 100K+ users
2. Have 70+ Security Levels
3. Have 4K+ Projects

Expected Results

Cache allocates a reasonable amount of heap, eg. up to 1Gb

Actual Results

• Cache is unbounded
• We observed cache grows up to 13GB.

Note on fix

Cache size is limited to 300k elements, this can be overwritten by jira.project.and.user.to.security.level.cache.max.size application property (in jira-config.properties file, see also jpm.xml).

Workaround

Currently there is no known workaround for this behavior on Server instances as the cache settings are immutable in the runtime.

On DC the size of cache can be constrained in the runtime by executing:

ComponentAccessor.getComponent(CacheManager.class)
.getManagedCache("com.atlassian.jira.issue.security.IssueSecurityLevelManagerImpl.projectAndUserToSecurityLevelCache")
.updateMaxEntries(300000)

We've calculated that the limit of 300000 entries in the cache will constrain its size to about ~4GB max and it still shouldn't create performance issues.