Loading...

Details

Type: Bug
Resolution: Fixed
Priority: High
Fix Version/s: 4.21.0
Affects Version/s: 4.14.0, 4.20.12, 4.20.14
Component/s: Installation, Upgrade, Startup, Shutdown
Labels:
None

Support reference count:
2
Symptom Severity:
Severity 2 - Major
UIS:
6
Bug Fix Policy:
View Atlassian Server bug fix policy

Description

Issue Summary

The JSM chaperone uses the endpoint /rest/chaperone/1/chaperone/<feature.key>, e.g. jsm.rebrand.notification, uses a property that causes a cache flush during the property set. The cacheflush event is quite expensive in large environments and can cause contention.

Steps to Reproduce

Generate high load
Log in with a user after going through JSM 4.14 upgrade, and browse to a JSM project;
The notification banner will appear in the top right, clicking 'Got it' will set the property:

Actual Results

Threads can become blocked by the threads running to load the cache. This can cause performance issues or outages in large instances that have been just upgraded and have gone through the rebranding as multiple users may be setting the property at the same time triggering the invalidation.

Log evidence:

2023-04-20 08:27:16,232-0700 http-nio-8081-exec-16 url: /rest/chaperone/1/chaperone/jsm.rebrand.notification; user: xxxxx WARN xxxxxxx 507x6384014x37 v7fp0 xxxxxx /rest/chaperone/1/chaperone/jsm.rebrand.notification [PS-xxxxxx] Stack trace for thread http-nio-8081-exec-16 url: /rest/chaperone/1/chaperone/jsm.rebrand.notification; user: xxxxxxx
    com.atlassian.jira.propertyset.CachingOfBizPropertyEntryStore.invalidateCacheEntry(CachingOfBizPropertyEntryStore.java:-1)
    com.atlassian.jira.propertyset.CachingOfBizPropertyEntryStore.setEntry(CachingOfBizPropertyEntryStore.java:175)
    com.atlassian.jira.propertyset.CachingOfBizPropertySet.setImpl(CachingOfBizPropertySet.java:183)
    com.opensymphony.module.propertyset.AbstractPropertySet.set(AbstractPropertySet.java:502)
    com.opensymphony.module.propertyset.AbstractPropertySet.setString(AbstractPropertySet.java:300)
    com.atlassian.sal.jira.pluginsettings.JiraPluginSettings.putActual(JiraPluginSettings.java:26)
    com.atlassian.sal.core.pluginsettings.AbstractStringPluginSettings.put(AbstractStringPluginSettings.java:71)
    com.atlassian.sal.jira.pluginsettings.ClusterSafePluginSettings.put(ClusterSafePluginSettings.java:50)
    com.atlassian.chaperone.ChaperonePinManager.setValue(ChaperonePinManager.java:36)
    com.atlassian.chaperone.rest.ChaperoneResource.putPins(ChaperoneResource.java:36)
    sun.reflect.GeneratedMethodAccessor5022.invoke (Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:498)
    com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
    com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
    com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
    com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
    com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
    com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
    com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
    com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
    com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
    com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
    com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
    com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
    com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
    com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
    com.atlassian.plugins.rest.module.RestDelegatingServletFilter$JerseyOsgiServletContainer.doFilter(RestDelegatingServletFilter.java:160)
    com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829)
    com.atlassian.plugins.rest.module.RestDelegatingServletFilter.doFilter(RestDelegatingServletFilter.java:70)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:56)
    com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:44)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.plugins.rest.module.servlet.RestServletUtilsUpdaterFilter.doFilterInternal(RestServletUtilsUpdaterFilter.java:23)
    com.atlassian.plugins.rest.module.servlet.RestServletUtilsUpdaterFilter.doFilter(RestServletUtilsUpdaterFilter.java:35)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.servicedesk.internal.web.ExternalCustomerLockoutFilter.doFilter(ExternalCustomerLockoutFilter.java:55)
    com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.jira.plugin.mobile.web.filter.MobileAppRequestFilter.doFilter(MobileAppRequestFilter.java:59)
    com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.jira.plugin.mobile.login.MobileLoginSuccessFilter.doFilter(MobileLoginSuccessFilter.java:54)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.diagnostics.internal.platform.monitor.http.HttpRequestMonitoringFilter.doFilter(HttpRequestMonitoringFilter.java:55)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
      . . .

Workaround:

It is possible to disable the pop-up message that adds the property:

Go to Administrator > Manage Apps > Manage Apps
Select "All Apps" on the drop-down
Locate Jira Service Management Frontend Plugin and expand the module
Locate and disable entrypoint-jsm.rebrand.notification

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List

screenshot-1.png
25 kB
20/Apr/2023 9:19 PM

Issue Links

relates to

JRASERVER-72909 Universal Plugin Manager flushing jira.properties cache causing contention and performance problem

Closed

ACE-3299 Loading...

mentioned in: Page Loading...; Page Loading...

JSM Chaperone plugin is flushing jira.properties cache causing contention and performance problem

Details

Description

Issue Summary

Steps to Reproduce

Actual Results

Log evidence:

Workaround:

Attachments

Attachments

Issue Links

Activity

People

Dates

Backbone Issue Sync