Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-72470

Jira stalls due to contention on CachedEncryptor.decrypt

    XMLWordPrintable

Details

    Description

      Issue Summary

      In Jira 8.14 we added a security feature that made encrypted passwords more secure in how they are stored and handled, see more details Password encryption.

      After enabling this feature, Jira may stall during very heavy application usage due to contention on the LocalCache by the CachedEncryptor.decrypt method.

      Currently, CachedEncryptor.decrypt has a limit of 4 concurrent requests which may cause the bottleneck. Additionally, stored passwords are invalidated after 10 minutes which leads to an increased number of requests.

      The stacktrace looks as follows:

      sun.misc.Unsafe.park(Native Method)
       java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
       java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
       java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
       java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
       java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
       java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
       com.google.common.cache.LocalCache$Segment.compute(LocalCache.java:2194)
       com.google.common.cache.LocalCache.compute(LocalCache.java:4197)
       com.google.common.cache.LocalCache.computeIfAbsent(LocalCache.java:4204)
       com.atlassian.crowd.crypto.CachedEncryptor.decrypt(CachedEncryptor.java:63)
       com.atlassian.crowd.crypto.SaltingEncryptor.decrypt(SaltingEncryptor.java:33)
       com.atlassian.crowd.crypto.MissingKeyHandlingEncryptor.decrypt(MissingKeyHandlingEncryptor.java:31)
       com.atlassian.crowd.crypto.PrefixBasedSwitchableEncryptor.decrypt(PrefixBasedSwitchableEncryptor.java:60)
       com.atlassian.crowd.crypto.ClusterLockingEncryptor.decrypt(ClusterLockingEncryptor.java:32)
       com.atlassian.jira.crowd.embedded.encryptors.JiraEncryptor.decrypt(JiraEncryptor.java:40)
       com.atlassian.crowd.crypto.DirectoryPasswordsEncryptor$$Lambda$1376/1299619370.apply(Unknown Source)
       com.atlassian.crowd.crypto.DirectoryPasswordsEncryptor.lambda$transformPasswordAttributes$0(DirectoryPasswordsEncryptor.java:32)
       com.atlassian.crowd.crypto.DirectoryPasswordsEncryptor$$Lambda$1377/330450352.apply(Unknown Source)
       java.util.HashMap.replaceAll(HashMap.java:1305)
       com.atlassian.crowd.crypto.DirectoryPasswordsEncryptor.transformPasswordAttributes(DirectoryPasswordsEncryptor.java:31)
       com.atlassian.crowd.crypto.DirectoryPasswordsEncryptor.decryptPasswords(DirectoryPasswordsEncryptor.java:20)
       com.atlassian.crowd.crypto.EncryptingDirectoryDAO$$Lambda$1375/1648181244.apply(Unknown Source)
      

      Workaround

      There are 2 workarounds available.

      Workaround 1

      Increase the retention period for the caches to ~30 minutes.

      1. Stop Jira.
      2. Add the following system property: -Dcrowd.encryption.cache.expire.minutes=30
      3. Start Jira

      Workaround 2

      If workaround 1 is not sufficient enough to handle the CachedEncryptor.decrypt requests, a sysadmin may disable the feature.

      1. Navigate to SystemGeneral configurationAdvanced Settings in the admin panel.
      2. Find option crowd.encryption.encryptor.default and set it’s value to DISABLED.

      Attachments

        Issue Links

          Activity

            People

              e7591db3c7db Antoni Kowalski
              sdegroot@atlassian.com Steven de Groot
              Votes:
              3 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: