Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-64908

UPM actions may flush internal caches, leading to performance problems

XMLWordPrintable

    • 6.04
    • 142
    • Severity 1 - Critical
    • 2,510
    • Hide
      Atlassian Update – 26 April 2019

      Dear Jira users,

      we’re glad to announce that this issue will be addressed in our upcoming 8.2.0 release!

      We’ve fixed several issues that caused performance degradation in large Jira instances when installing or updating apps. If you’ve encountered this problem before, it’s solved, so feel free to update your apps anytime. For technical details please go this comment.

       

      Looking forward for your comments.

      Thank you,
      Grazyna Kaszkur

      Product Manager,
      Jira Server and Data Center

      Show
      Atlassian Update – 26 April 2019 Dear Jira users, we’re glad to announce that this issue will be addressed in our upcoming 8.2.0 release! We’ve fixed several issues that caused performance degradation in large Jira instances when installing or updating apps. If you’ve encountered this problem before, it’s solved, so feel free to update your apps anytime. For technical details please go this comment.   Looking forward for your comments. Thank you, Grazyna Kaszkur
 Product Manager, Jira Server and Data Center

      Summary

      During plugin (add-on) installs/updates (including license updates) /delete/disable action UPM will flush the large number of caches. Under high load that will cause significant JIRA performance degradation and might cause JIRA to freeze/stall for a long time.

      Steps to Reproduce

      1. Apply high load to JIRA
      2. Upgrade any plugin.

      Expected Results

      • JIRA performs fast, could be slight performance degradation.
      • JIRA admin is notified about the impact

      Actual Results

      • JIRA admin is not notified about the impact
      • Thread dump might show the following thread UpmAsynchronousTaskManager:thread 
        "UpmAsynchronousTaskManager:thread-1" #38891 prio=5 os_prio=0 tid=0x00007fbda8047800 nid=0x6712 waiting on condition [0x00007fbcf9cd5000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000056c734418> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
...
	at net.sf.ehcache.concurrent.ReadWriteLockSync.lock(ReadWriteLockSync.java:50)
	at com.atlassian.cache.ehcache.LoadingCache.remove(LoadingCache.java:190)
	at com.atlassian.cache.ehcache.DelegatingCachedReference.reset(DelegatingCachedReference.java:79)
	at com.atlassian.jira.issue.search.managers.DefaultSearchHandlerManager.refresh(DefaultSearchHandlerManager.java:290)
	at com.atlassian.jira.issue.search.managers.DefaultIssueSearcherManager.refresh(DefaultIssueSearcherManager.java:46)
	at com.atlassian.jira.issue.fields.DefaultFieldManager.refreshSearchersAndIndexers(DefaultFieldManager.java:804)
	at com.atlassian.jira.issue.fields.DefaultFieldManager.refresh(DefaultFieldManager.java:706)
	at com.atlassian.jira.plugin.JiraCacheResetter$Delegate.resetCaches(JiraCacheResetter.java:83)
	at com.atlassian.jira.plugin.JiraCacheResetter$Delegate.onPluginModuleDisabled(JiraCacheResetter.java:53)
...
	at com.atlassian.plugin.event.impl.DefaultPluginEventManager.broadcast(DefaultPluginEventManager.java:73)
	at com.atlassian.plugin.manager.DefaultPluginManager.broadcastIgnoreError(DefaultPluginManager.java:2081)
	at com.atlassian.plugin.manager.DefaultPluginManager.publishModuleDisabledEvents(DefaultPluginManager.java:1955)
	at com.atlassian.plugin.manager.DefaultPluginManager.disablePluginModuleNoPersist(DefaultPluginManager.java:1910)
	at com.atlassian.plugin.manager.DefaultPluginManager.disablePluginModules(DefaultPluginManager.java:1904)
	at com.atlassian.plugin.manager.DefaultPluginManager.disablePluginWithModuleEvents(DefaultPluginManager.java:1860)
	at com.atlassian.plugin.manager.DefaultPluginManager.notifyPluginDisabled(DefaultPluginManager.java:1888)
	at com.atlassian.plugin.manager.DefaultPluginManager.unloadPlugin(DefaultPluginManager.java:1022)
	at com.atlassian.plugin.manager.DefaultPluginManager.uninstallNoEvent(DefaultPluginManager.java:981)
	at com.atlassian.plugin.manager.DefaultPluginManager.updatePlugin(DefaultPluginManager.java:1343)
	at com.atlassian.plugin.manager.DefaultPluginManager.addPlugins(DefaultPluginManager.java:1133)
	at com.atlassian.jira.plugin.JiraPluginManager.addPlugins(JiraPluginManager.java:150)
	at com.atlassian.plugin.manager.DefaultPluginManager.scanForNewPlugins(DefaultPluginManager.java:903)
	at com.atlassian.plugin.manager.DefaultPluginManager.installPlugins(DefaultPluginManager.java:821)
	at com.atlassian.jira.plugin.JiraPluginManager.installPlugins(JiraPluginManager.java:168)
...

      Notes

      Problem is amplified at JIRA Datacenter. Cache flush is triggered during node startup as UPM needs to register and load modules from plugins. See JRASERVER-66839

      Workaround

      Please plan any plugin actions updates/delete/disable (UPM actions) during low peak hours or maintenance windows. See Best Practices for Managing JIRA Application Add-ons for recommendations.
       

        1. plugin_disable_80.png
          plugin_disable_80.png
          341 kB
        2. plugin_disable_82.png
          plugin_disable_82.png
          133 kB

              mswinarski Maciej Swinarski (Inactive)
              ayakovlev@atlassian.com Andriy Yakovlev [Atlassian]
              Votes:
              82 Vote for this issue
              Watchers:
              132 Start watching this issue

                Created:
                Updated:
                Resolved: