Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-71820

projectAndUserToSecurityLevelCache can cause OOM due to large number of elements

XMLWordPrintable

      Issue Summary

      The projectAndUserToSecurityLevelCache cache (com.atlassian.jira.issue.security.IssueSecurityLevelManagerImpl.projectAndUserToSecurityLevelCache) can grow quite large, consuming most of JVM Heap and causing frequent Full GC, potentially leading to OOM and causing Jira to be unavailable.

      The size of the cache depends on the number of elements (keys) and the size of each key.

      • "Number of elements" is a product of # projects x # user, eg, for 100k users and 4k projects, will be 400M elements
      • "Size" is a product of # Security Levels

      Environment

      1. Have 100K+ users
      2. Have 70+ Security Levels
      3. Have 4K+ Projects

      Expected Results

      Cache allocates a reasonable amount of heap, eg. up to 1Gb

      Actual Results

      • Cache is unbounded
      • We observed cache grows up to 13GB.

      Note on fix

      Cache size is limited to 300k elements, this can be overwritten by jira.project.and.user.to.security.level.cache.max.size application property (in jira-config.properties file, see also jpm.xml).

      Workaround

      Currently there is no known workaround for this behavior on Server instances as the cache settings are immutable in the runtime.

      On DC the size of cache can be constrained in the runtime by executing:

      ComponentAccessor.getComponent(CacheManager.class)
      .getManagedCache("com.atlassian.jira.issue.security.IssueSecurityLevelManagerImpl.projectAndUserToSecurityLevelCache")
      .updateMaxEntries(300000)
      

      We've calculated that the limit of 300000 entries in the cache will constrain its size to about ~4GB max and it still shouldn't create performance issues.

              mciesielski2@atlassian.com Michal Ciesielski
              bsilva@atlassian.com Bruno
              Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

                Created:
                Updated:
                Resolved: