Uploaded image for project: 'Jira Service Management Data Center'
  1. Jira Service Management Data Center
  2. JSDSERVER-12653

JSM Chaperone plugin is flushing jira.properties cache causing contention and performance problem

XMLWordPrintable

      Issue Summary

      The JSM chaperone uses the endpoint /rest/chaperone/1/chaperone/<feature.key>, e.g. jsm.rebrand.notification, uses a property that causes a cache flush during the property set. The cacheflush event is quite expensive in large environments and can cause contention.

      Steps to Reproduce

      1. Generate high load
      2. Log in with a user after going through JSM 4.14 upgrade, and browse to a JSM project;
      3. The notification banner will appear in the top right, clicking 'Got it' will set the property:

      Actual Results

      Threads can become blocked by the threads running to load the cache. This can cause performance issues or outages in large instances that have been just upgraded and have gone through the rebranding as multiple users may be setting the property at the same time triggering the invalidation.

      Log evidence:

      2023-04-20 08:27:16,232-0700 http-nio-8081-exec-16 url: /rest/chaperone/1/chaperone/jsm.rebrand.notification; user: xxxxx WARN xxxxxxx 507x6384014x37 v7fp0 xxxxxx /rest/chaperone/1/chaperone/jsm.rebrand.notification [PS-xxxxxx] Stack trace for thread http-nio-8081-exec-16 url: /rest/chaperone/1/chaperone/jsm.rebrand.notification; user: xxxxxxx
          com.atlassian.jira.propertyset.CachingOfBizPropertyEntryStore.invalidateCacheEntry(CachingOfBizPropertyEntryStore.java:-1)
          com.atlassian.jira.propertyset.CachingOfBizPropertyEntryStore.setEntry(CachingOfBizPropertyEntryStore.java:175)
          com.atlassian.jira.propertyset.CachingOfBizPropertySet.setImpl(CachingOfBizPropertySet.java:183)
          com.opensymphony.module.propertyset.AbstractPropertySet.set(AbstractPropertySet.java:502)
          com.opensymphony.module.propertyset.AbstractPropertySet.setString(AbstractPropertySet.java:300)
          com.atlassian.sal.jira.pluginsettings.JiraPluginSettings.putActual(JiraPluginSettings.java:26)
          com.atlassian.sal.core.pluginsettings.AbstractStringPluginSettings.put(AbstractStringPluginSettings.java:71)
          com.atlassian.sal.jira.pluginsettings.ClusterSafePluginSettings.put(ClusterSafePluginSettings.java:50)
          com.atlassian.chaperone.ChaperonePinManager.setValue(ChaperonePinManager.java:36)
          com.atlassian.chaperone.rest.ChaperoneResource.putPins(ChaperoneResource.java:36)
          sun.reflect.GeneratedMethodAccessor5022.invoke (Unknown Source)
          sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          java.lang.reflect.Method.invoke(Method.java:498)
          com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
          com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
          com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
          com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
          com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
          com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
          com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
          com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
          com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
          com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
          com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
          com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
          com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
          com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
          com.atlassian.plugins.rest.module.RestDelegatingServletFilter$JerseyOsgiServletContainer.doFilter(RestDelegatingServletFilter.java:160)
          com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829)
          com.atlassian.plugins.rest.module.RestDelegatingServletFilter.doFilter(RestDelegatingServletFilter.java:70)
          com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
          com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
          com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:56)
          com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:44)
          com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
          com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
          com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
          com.atlassian.plugins.rest.module.servlet.RestServletUtilsUpdaterFilter.doFilterInternal(RestServletUtilsUpdaterFilter.java:23)
          com.atlassian.plugins.rest.module.servlet.RestServletUtilsUpdaterFilter.doFilter(RestServletUtilsUpdaterFilter.java:35)
          com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
          com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
          com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
          com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
          com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
          com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
          com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
          com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
          com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
          com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
          com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
          com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
          com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
          com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
          com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
          com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
          com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
          com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
          com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
          com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
          com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
          com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
          com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
          com.atlassian.servicedesk.internal.web.ExternalCustomerLockoutFilter.doFilter(ExternalCustomerLockoutFilter.java:55)
          com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
          com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
          com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
          com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
          com.atlassian.jira.plugin.mobile.web.filter.MobileAppRequestFilter.doFilter(MobileAppRequestFilter.java:59)
          com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
          com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
          com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
          com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
          com.atlassian.jira.plugin.mobile.login.MobileLoginSuccessFilter.doFilter(MobileLoginSuccessFilter.java:54)
          com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
          com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
          com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
          com.atlassian.diagnostics.internal.platform.monitor.http.HttpRequestMonitoringFilter.doFilter(HttpRequestMonitoringFilter.java:55)
          com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
          com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
          com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
            . . .
      

      Workaround:

      It is possible to disable the pop-up message that adds the property:

      1. Go to Administrator > Manage Apps > Manage Apps
      2. Select "All Apps" on the drop-down
      3. Locate Jira Service Management Frontend Plugin and expand the module
      4. Locate and disable entrypoint-jsm.rebrand.notification

        1. screenshot-1.png
          25 kB
          Pedro Souza

              aba1389a0947 Josh Cameron
              psouza Pedro Souza
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: