Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-72909

Universal Plugin Manager flushing jira.properties cache causing contention and performance problem

XMLWordPrintable

    • 8.05
    • 43
    • Severity 2 - Major
    • 26
    • Hide
      Atlassian Update – 23 June 2023

      Thank you for reporting this issue. In the last weeks we have been working hard on fixing it.

      Summary of the problem:

      Currently Jira manages all the jira.properties cache under single cache key, and because of this every change to any number of properties even if it's just a single property change/removal/addition invalidates whole jira.properties cache.

      New behaviour after the change:

      The jira.properties cache is now managing its data under separate cache keys and do not treat this data as single entry so now any change to a number of properties will only invalidate the cache for these properties and not whole jira.properties.

      Status of the fix and Fix Version:

      We cannot commit to any Fix version yet. We will update the ticket once its ready for release.

      The status of backporting this fix to previous versions has not yet been determined. Please monitor this ticket for further updates.

      Show
      Atlassian Update – 23 June 2023 Thank you for reporting this issue. In the last weeks we have been working hard on fixing it. Summary of the problem: Currently Jira manages all the jira.properties cache under single cache key, and because of this every change to any number of properties even if it's just a single property change/removal/addition invalidates whole jira.properties cache. New behaviour after the change: The jira.properties cache is now managing its data under separate cache keys and do not treat this data as single entry so now any change to a number of properties will only invalidate the cache for these properties and not whole jira.properties. Status of the fix and Fix Version: We cannot commit to any Fix version yet. We will update the ticket once its ready for release. The status of backporting this fix to previous versions has not yet been determined. Please monitor this ticket for further updates.

      Issue Summary

      When cached data is updated by UPM, it updates the database and invalidates the CachingOfBizPropertyEntryStore cache. This invalidated all entries with ‘entityName =jira.properties, entityId = 1’ keys, all the requests trying to access any data from jira.properties, tries to load the cache or get blocked by the others.

      Steps to Reproduce

      1. Generate high load
      2. Navigate to Administration > Manage apps > Find new apps to trigger a request to /rest/plugins/1.0/notifications to invalidate the cache or install/uninstall/update an app

      Expected Results

      Jira performing without any performance issue

      Actual Results

      Threads are blocked by the threads running to load the cache.

      Additional logs enabled via ByteCat shows the notification requests from UPM causing cache invalidation on entityName = jira.properties, entityId = 1

      2021-10-07 04:56:25,484+0200 https-jsse-nio-8443-exec-1 url: /rest/plugins/1.0/notifications; user: username WARN username 123x45678x9 abcdef 192.168.1.1 /rest/plugins/1.0/notifications [Logging] CachingOfBizPropertyEntryStore.invalidateCacheEntry entityName = jira.properties, entityId = 1
2021-10-07 04:56:25,521+0200 https-jsse-nio-8443-exec-1 url: /rest/plugins/1.0/notifications; user: username WARN username 123x45678x9 abcdef 192.168.1.1 /rest/plugins/1.0/notifications [Logging] CachingOfBizPropertyEntryStore.loadPropertySetData onCacheMiss entityName = jira.properties, entityId = 1
2021-10-07 04:56:26,727+0200 https-jsse-nio-8443-exec-1 url: /rest/plugins/1.0/notifications; user: username WARN username 123x45678x9 abcdef 192.168.1.1 /rest/plugins/1.0/notifications [Logging] CachingOfBizPropertyEntryStore.invalidateCacheEntry entityName = jira.properties, entityId = 1
2021-10-07 04:56:26,753+0200 https-jsse-nio-8443-exec-1 url: /rest/plugins/1.0/notifications; user: username WARN username 123x45678x9 abcdef 192.168.1.1 /rest/plugins/1.0/notifications [Logging] CachingOfBizPropertyEntryStore.loadPropertySetData onCacheMiss entityName = jira.properties, entityId = 1
2021-10-07 04:56:27,840+0200 https-jsse-nio-8443-exec-1 url: /rest/plugins/1.0/notifications; user: username WARN username 123x45678x9 abcdef 192.168.1.1 /rest/plugins/1.0/notifications [Logging] CachingOfBizPropertyEntryStore.invalidateCacheEntry entityName = jira.properties, entityId = 1
2021-10-07 04:56:27,867+0200 https-jsse-nio-8443-exec-1 url: /rest/plugins/1.0/notifications; user: username WARN username 123x45678x9 abcdef 192.168.1.1 /rest/plugins/1.0/notifications [Logging] CachingOfBizPropertyEntryStore.loadPropertySetData onCacheMiss entityName = jira.properties, entityId = 1
2021-10-07 04:56:28,980+0200 https-jsse-nio-8443-exec-1 url: /rest/plugins/1.0/notifications; user: username WARN username 123x45678x9 abcdef 192.168.1.1 /rest/plugins/1.0/notifications [Logging] CachingOfBizPropertyEntryStore.invalidateCacheEntry entityName = jira.properties, entityId = 1

      Workaround

      UPM may clear the sensitive jira.properties cache when triggering some actions related to apps management. It is recommended to not perform any apps management during business hours.

      Add-on changes should be performed during maintenance windows to minimize the impact on users.

      You may want to block the GET requests to /rest/plugins/1.0/notifications endpoint on your load balancer or proxy. This will cause fewer cache flushes. This will prevent admin messages generated by UPM (Universal Plugin Manager component) regarding Apps on your system while in effect.

        1. cache_2.png
          cache_2.png
          351 kB
        2. IBM_Thread_and_Monitor_Dump_Analyzer_for_Java-7.png
          IBM_Thread_and_Monitor_Dump_Analyzer_for_Java-7.png
          314 kB
        3. tdump-2.png
          tdump-2.png
          338 kB

              f5e0e0ab36de Kamil Bar
              b39100fe004b Kutay Peksen
              Votes:
              29 Vote for this issue
              Watchers:
              57 Start watching this issue

                Created:
                Updated:
                Resolved: