[JSDSERVER-12653] JSM Chaperone plugin is flushing jira.properties cache causing contention and performance problem

Type: Bug
Resolution: Fixed
Priority: High
Fix Version/s: 4.21.0
Affects Version/s: 4.14.0, 4.20.12, 4.20.14
Component/s: Installation, Upgrade, Startup, Shutdown
Labels:
None

Support reference count:
2
Symptom Severity:
Severity 2 - Major
UIS:
6
Bug Fix Policy:
View Atlassian Server bug fix policy

Issue Summary

The JSM chaperone uses the endpoint /rest/chaperone/1/chaperone/<feature.key>, e.g. jsm.rebrand.notification, uses a property that causes a cache flush during the property set. The cacheflush event is quite expensive in large environments and can cause contention.

Steps to Reproduce

Generate high load
Log in with a user after going through JSM 4.14 upgrade, and browse to a JSM project;
The notification banner will appear in the top right, clicking 'Got it' will set the property:

Actual Results

Threads can become blocked by the threads running to load the cache. This can cause performance issues or outages in large instances that have been just upgraded and have gone through the rebranding as multiple users may be setting the property at the same time triggering the invalidation.

Log evidence:

2023-04-20 08:27:16,232-0700 http-nio-8081-exec-16 url: /rest/chaperone/1/chaperone/jsm.rebrand.notification; user: xxxxx WARN xxxxxxx 507x6384014x37 v7fp0 xxxxxx /rest/chaperone/1/chaperone/jsm.rebrand.notification [PS-xxxxxx] Stack trace for thread http-nio-8081-exec-16 url: /rest/chaperone/1/chaperone/jsm.rebrand.notification; user: xxxxxxx
    com.atlassian.jira.propertyset.CachingOfBizPropertyEntryStore.invalidateCacheEntry(CachingOfBizPropertyEntryStore.java:-1)
    com.atlassian.jira.propertyset.CachingOfBizPropertyEntryStore.setEntry(CachingOfBizPropertyEntryStore.java:175)
    com.atlassian.jira.propertyset.CachingOfBizPropertySet.setImpl(CachingOfBizPropertySet.java:183)
    com.opensymphony.module.propertyset.AbstractPropertySet.set(AbstractPropertySet.java:502)
    com.opensymphony.module.propertyset.AbstractPropertySet.setString(AbstractPropertySet.java:300)
    com.atlassian.sal.jira.pluginsettings.JiraPluginSettings.putActual(JiraPluginSettings.java:26)
    com.atlassian.sal.core.pluginsettings.AbstractStringPluginSettings.put(AbstractStringPluginSettings.java:71)
    com.atlassian.sal.jira.pluginsettings.ClusterSafePluginSettings.put(ClusterSafePluginSettings.java:50)
    com.atlassian.chaperone.ChaperonePinManager.setValue(ChaperonePinManager.java:36)
    com.atlassian.chaperone.rest.ChaperoneResource.putPins(ChaperoneResource.java:36)
    sun.reflect.GeneratedMethodAccessor5022.invoke (Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:498)
    com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
    com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
    com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
    com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
    com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
    com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
    com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
    com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
    com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
    com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
    com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
    com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
    com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
    com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
    com.atlassian.plugins.rest.module.RestDelegatingServletFilter$JerseyOsgiServletContainer.doFilter(RestDelegatingServletFilter.java:160)
    com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829)
    com.atlassian.plugins.rest.module.RestDelegatingServletFilter.doFilter(RestDelegatingServletFilter.java:70)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:56)
    com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:44)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.plugins.rest.module.servlet.RestServletUtilsUpdaterFilter.doFilterInternal(RestServletUtilsUpdaterFilter.java:23)
    com.atlassian.plugins.rest.module.servlet.RestServletUtilsUpdaterFilter.doFilter(RestServletUtilsUpdaterFilter.java:35)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.servicedesk.internal.web.ExternalCustomerLockoutFilter.doFilter(ExternalCustomerLockoutFilter.java:55)
    com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.jira.plugin.mobile.web.filter.MobileAppRequestFilter.doFilter(MobileAppRequestFilter.java:59)
    com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.jira.plugin.mobile.login.MobileLoginSuccessFilter.doFilter(MobileLoginSuccessFilter.java:54)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.diagnostics.internal.platform.monitor.http.HttpRequestMonitoringFilter.doFilter(HttpRequestMonitoringFilter.java:55)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
      . . .

Workaround:

It is possible to disable the pop-up message that adds the property:

Go to Administrator > Manage Apps > Manage Apps
Select "All Apps" on the drop-down
Locate Jira Service Management Frontend Plugin and expand the module
Locate and disable entrypoint-jsm.rebrand.notification

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List

screenshot-1.png
25 kB
20/Apr/2023 9:19 PM

relates to

JRASERVER-72909 Universal Plugin Manager flushing jira.properties cache causing contention and performance problem

Closed

ACE-3299 You do not have permission to view this issue

mentioned in: Page Failed to load; Page Failed to load

Form Name

Josh Cameron made changes - 04/Jun/2023 11:30 PM

Resolution		New: Fixed [ 1 ]
Status	Original: In Progress [ 3 ]	New: Closed [ 6 ]

Josh Cameron added a comment - 04/Jun/2023 11:30 PM

Released as part of JSM 5.9.0

Josh Cameron added a comment - 04/Jun/2023 11:30 PM Released as part of JSM 5.9.0

Craig Shannon made changes - 26/May/2023 6:22 AM

Remote Link

New: This issue links to "Page (Confluence)" [ 768325 ]

Josh Cameron made changes - 11/May/2023 2:31 AM

Status

Original: Short Term Backlog [ 12074 ]

New: In Progress [ 3 ]

Josh Cameron made changes - 11/May/2023 2:31 AM

Assignee

New: Josh Cameron [ aba1389a0947 ]

Josh Cameron made changes - 11/May/2023 2:31 AM

Status

Original: Gathering Impact [ 12072 ]

New: Short Term Backlog [ 12074 ]

Maciej Swinarski (Inactive) made changes - 08/May/2023 11:43 AM

Remote Link

New: This issue links to "Page (Confluence)" [ 759404 ]

Craig Shannon made changes - 24/Apr/2023 2:03 AM

Status

Original: Needs Triage [ 10030 ]

New: Gathering Impact [ 12072 ]

Craig Shannon made changes - 24/Apr/2023 2:03 AM

Description

Original: h3. Issue Summary

The endpoint /rest/chaperone/1/chaperone/jsm.rebrand.notification uses a property that causes a cache flush during the property set. The cacheflush event is quite expensive in large environments and can cause contention.
h3. Steps to Reproduce
# Generate high load
# Log in with a user after going through JSM 4.14 upgrade, and browse to a JSM project;
# The notification banner will appear in the top right, clicking 'Got it' will set the property:
!screenshot-1.png|thumbnail!

h3. Actual Results

Threads can become blocked by the threads running to load the cache. This can cause performance issues or outages in large instances that have been just upgraded and have gone through the rebranding as multiple users may be setting the property at the same time triggering the invalidation.

h3. Log evidence:
{noformat}
2023-04-20 08:27:16,232-0700 http-nio-8081-exec-16 url: /rest/chaperone/1/chaperone/jsm.rebrand.notification; user: xxxxx WARN xxxxxxx 507x6384014x37 v7fp0 xxxxxx /rest/chaperone/1/chaperone/jsm.rebrand.notification [PS-xxxxxx] Stack trace for thread http-nio-8081-exec-16 url: /rest/chaperone/1/chaperone/jsm.rebrand.notification; user: xxxxxxx
    com.atlassian.jira.propertyset.CachingOfBizPropertyEntryStore.invalidateCacheEntry(CachingOfBizPropertyEntryStore.java:-1)
    com.atlassian.jira.propertyset.CachingOfBizPropertyEntryStore.setEntry(CachingOfBizPropertyEntryStore.java:175)
    com.atlassian.jira.propertyset.CachingOfBizPropertySet.setImpl(CachingOfBizPropertySet.java:183)
    com.opensymphony.module.propertyset.AbstractPropertySet.set(AbstractPropertySet.java:502)
    com.opensymphony.module.propertyset.AbstractPropertySet.setString(AbstractPropertySet.java:300)
    com.atlassian.sal.jira.pluginsettings.JiraPluginSettings.putActual(JiraPluginSettings.java:26)
    com.atlassian.sal.core.pluginsettings.AbstractStringPluginSettings.put(AbstractStringPluginSettings.java:71)
    com.atlassian.sal.jira.pluginsettings.ClusterSafePluginSettings.put(ClusterSafePluginSettings.java:50)
    com.atlassian.chaperone.ChaperonePinManager.setValue(ChaperonePinManager.java:36)
    com.atlassian.chaperone.rest.ChaperoneResource.putPins(ChaperoneResource.java:36)
    sun.reflect.GeneratedMethodAccessor5022.invoke (Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:498)
    com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
    com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
    com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
    com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
    com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
    com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
    com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
    com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
    com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
    com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
    com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
    com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
    com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
    com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
    com.atlassian.plugins.rest.module.RestDelegatingServletFilter$JerseyOsgiServletContainer.doFilter(RestDelegatingServletFilter.java:160)
    com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829)
    com.atlassian.plugins.rest.module.RestDelegatingServletFilter.doFilter(RestDelegatingServletFilter.java:70)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:56)
    com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:44)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.plugins.rest.module.servlet.RestServletUtilsUpdaterFilter.doFilterInternal(RestServletUtilsUpdaterFilter.java:23)
    com.atlassian.plugins.rest.module.servlet.RestServletUtilsUpdaterFilter.doFilter(RestServletUtilsUpdaterFilter.java:35)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.servicedesk.internal.web.ExternalCustomerLockoutFilter.doFilter(ExternalCustomerLockoutFilter.java:55)
    com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.jira.plugin.mobile.web.filter.MobileAppRequestFilter.doFilter(MobileAppRequestFilter.java:59)
    com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.jira.plugin.mobile.login.MobileLoginSuccessFilter.doFilter(MobileLoginSuccessFilter.java:54)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.diagnostics.internal.platform.monitor.http.HttpRequestMonitoringFilter.doFilter(HttpRequestMonitoringFilter.java:55)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
      . . .
{noformat}
h3. Workaround:

It is possible to disable the pop-up message that adds the property:
# Go to Administrator > Manage Apps > Manage Apps
# Select "All Apps" on the drop-down
# Locate *Jira Service Management Frontend Plugin* and expand the module
# Locate and disable {{entrypoint-jsm.rebrand.notification}}

New: h3. Issue Summary

The JSM chaperone uses the endpoint /rest/chaperone/1/chaperone/<feature.key>, e.g. jsm.rebrand.notification, uses a property that causes a cache flush during the property set. The cacheflush event is quite expensive in large environments and can cause contention.

h3. Steps to Reproduce
# Generate high load
# Log in with a user after going through JSM 4.14 upgrade, and browse to a JSM project;
# The notification banner will appear in the top right, clicking 'Got it' will set the property:
!screenshot-1.png|thumbnail!

h3. Actual Results

Threads can become blocked by the threads running to load the cache. This can cause performance issues or outages in large instances that have been just upgraded and have gone through the rebranding as multiple users may be setting the property at the same time triggering the invalidation.

h3. Log evidence:
{noformat}
2023-04-20 08:27:16,232-0700 http-nio-8081-exec-16 url: /rest/chaperone/1/chaperone/jsm.rebrand.notification; user: xxxxx WARN xxxxxxx 507x6384014x37 v7fp0 xxxxxx /rest/chaperone/1/chaperone/jsm.rebrand.notification [PS-xxxxxx] Stack trace for thread http-nio-8081-exec-16 url: /rest/chaperone/1/chaperone/jsm.rebrand.notification; user: xxxxxxx
    com.atlassian.jira.propertyset.CachingOfBizPropertyEntryStore.invalidateCacheEntry(CachingOfBizPropertyEntryStore.java:-1)
    com.atlassian.jira.propertyset.CachingOfBizPropertyEntryStore.setEntry(CachingOfBizPropertyEntryStore.java:175)
    com.atlassian.jira.propertyset.CachingOfBizPropertySet.setImpl(CachingOfBizPropertySet.java:183)
    com.opensymphony.module.propertyset.AbstractPropertySet.set(AbstractPropertySet.java:502)
    com.opensymphony.module.propertyset.AbstractPropertySet.setString(AbstractPropertySet.java:300)
    com.atlassian.sal.jira.pluginsettings.JiraPluginSettings.putActual(JiraPluginSettings.java:26)
    com.atlassian.sal.core.pluginsettings.AbstractStringPluginSettings.put(AbstractStringPluginSettings.java:71)
    com.atlassian.sal.jira.pluginsettings.ClusterSafePluginSettings.put(ClusterSafePluginSettings.java:50)
    com.atlassian.chaperone.ChaperonePinManager.setValue(ChaperonePinManager.java:36)
    com.atlassian.chaperone.rest.ChaperoneResource.putPins(ChaperoneResource.java:36)
    sun.reflect.GeneratedMethodAccessor5022.invoke (Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:498)
    com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
    com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
    com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
    com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
    com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
    com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
    com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
    com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
    com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
    com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
    com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
    com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
    com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
    com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
    com.atlassian.plugins.rest.module.RestDelegatingServletFilter$JerseyOsgiServletContainer.doFilter(RestDelegatingServletFilter.java:160)
    com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829)
    com.atlassian.plugins.rest.module.RestDelegatingServletFilter.doFilter(RestDelegatingServletFilter.java:70)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:56)
    com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:44)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.plugins.rest.module.servlet.RestServletUtilsUpdaterFilter.doFilterInternal(RestServletUtilsUpdaterFilter.java:23)
    com.atlassian.plugins.rest.module.servlet.RestServletUtilsUpdaterFilter.doFilter(RestServletUtilsUpdaterFilter.java:35)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.servicedesk.internal.web.ExternalCustomerLockoutFilter.doFilter(ExternalCustomerLockoutFilter.java:55)
    com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.jira.plugin.mobile.web.filter.MobileAppRequestFilter.doFilter(MobileAppRequestFilter.java:59)
    com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.jira.plugin.mobile.login.MobileLoginSuccessFilter.doFilter(MobileLoginSuccessFilter.java:54)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
    com.atlassian.diagnostics.internal.platform.monitor.http.HttpRequestMonitoringFilter.doFilter(HttpRequestMonitoringFilter.java:55)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
    com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
    com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.lambda$doFilter$0(DelegatingPluginFilter.java:57)
      . . .
{noformat}
h3. Workaround:

It is possible to disable the pop-up message that adds the property:
# Go to Administrator > Manage Apps > Manage Apps
# Select "All Apps" on the drop-down
# Locate *Jira Service Management Frontend Plugin* and expand the module
# Locate and disable {{entrypoint-jsm.rebrand.notification}}

Craig Shannon made changes - 24/Apr/2023 2:02 AM

Summary

Original: jsm.rebrand.notification flushing jira.properties cache causing contention and performance problem

New: JSM Chaperone plugin is flushing jira.properties cache causing contention and performance problem

Assignee:: Josh Cameron

Reporter:: Pedro Souza

Affected customers:: 1 This affects my team

Watchers:: 6 Start watching this issue

Created:: 20/Apr/2023 7:50 PM

Updated:: 04/Jun/2023 11:30 PM

Resolved:: 04/Jun/2023 11:30 PM

Details

Description

Issue Summary

Steps to Reproduce

Actual Results

Log evidence:

Workaround:

Attachments

Attachments

Issue Links

Forms

Activity

Collapse comment: Josh Cameron added a comment - 04/Jun/2023 11:30 PM

Expand comment: Josh Cameron added a comment - 04/Jun/2023 11:30 PM

People

Dates

Backbone Issue Sync