[JRASERVER-44236] The Favourite Filters REST Resource retrieves all users eagerly causing JIRA to perform poorly under concurrent load

Type: Bug
Resolution: Fixed
Priority: Medium (View bug fix roadmap)
Fix Version/s: 6.4.10, 7.0.0-OD-02
Affects Version/s: 6.3.11, 6.4.7
Component/s: Navigation - Filters
Labels:

Introduced in Version:
6.03
Bug Fix Policy:
View Atlassian Server bug fix policy

Summary

When the Issue Navigator is loaded, it makes a call to the favourite filter resource, e.g.: https://jira.atlassian.com/rest/api/2/filter/favourite. FilterResource.getFavouriteFilters is currently inefficient such that it performs a lookup to retrieve every single user which is unnecessary and causes performance problems.

Environment

Java 8

Database stats:

  <database-statistics>
    <Attachments>291235</Attachments>
    <Comments>1535357</Comments>
    <Components>7276</Components>
    <Custom-Fields>576</Custom-Fields>
    <Groups>17478</Groups>
    <Issue-Security-Levels>1</Issue-Security-Levels>
    <Issue-Types>77</Issue-Types>
    <Issues>765676</Issues>
    <Priorities>5</Priorities>
    <Projects>1392</Projects>
    <Resolutions>24</Resolutions>
    <Screen-Schemes>64</Screen-Schemes>
    <Screens>207</Screens>
    <Statuses>296</Statuses>
    <Users>80433</Users>
    <Versions>7505</Versions>
    <Workflows>60</Workflows>
  </database-statistics>

4x Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz (64 threads)

Steps to Reproduce

Load up a huge instance with a large number of users, such as above.
Perform a large number of current accesses to the Issue Navigator.

Expected Results

The instance performs smoothly.

Actual Results

The CPU spikes and the instance performs poorly. Analysis of thread dumps shows a number of long running threads with the below stack trace:

"http-bio-8080-exec-1249" #43426 daemon prio=5 tid=0x00007f95e9eee800 nid=0x3917 runnable [0x00007f9518456000]
   java.lang.Thread.State: RUNNABLE
	at com.google.common.cache.CacheBuilder$3.read(CacheBuilder.java:197)
	at com.google.common.cache.LocalCache$Segment.cleanUp(LocalCache.java:3475)
	at com.google.common.cache.LocalCache$Segment.postReadCleanup(LocalCache.java:3452)
	at com.google.common.cache.LocalCache$HashIterator.advanceTo(LocalCache.java:4369)
	at com.google.common.cache.LocalCache$HashIterator.nextInTable(LocalCache.java:4344)
	at com.google.common.cache.LocalCache$HashIterator.advance(LocalCache.java:4308)
	at com.google.common.cache.LocalCache$HashIterator.nextEntry(LocalCache.java:4382)
	at com.google.common.cache.LocalCache$KeyIterator.next(LocalCache.java:4397)
	at com.atlassian.jira.crowd.embedded.ofbiz.UserOrGroupCache.visitAllInDirectory(UserOrGroupCache.java:215)
	at com.atlassian.jira.crowd.embedded.ofbiz.UserOrGroupCache.getAllInDirectory(UserOrGroupCache.java:232)
	at com.atlassian.jira.crowd.embedded.ofbiz.UserOrGroupCache.getAllInDirectory(UserOrGroupCache.java:204)
	at com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao.getAllUsersFromCache(OfBizUserDao.java:854)
	at com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao.search(OfBizUserDao.java:827)
	at com.atlassian.crowd.directory.AbstractInternalDirectory.searchUsers(AbstractInternalDirectory.java:780)
	at com.atlassian.crowd.directory.DbCachingRemoteDirectory.searchUsers(DbCachingRemoteDirectory.java:602)
	at com.atlassian.crowd.manager.directory.DirectoryManagerGeneric.searchUsers(DirectoryManagerGeneric.java:301)
	at com.atlassian.jira.user.util.DefaultUserManager.getAllUsersFromDirectory(DefaultUserManager.java:116)
	at com.atlassian.jira.user.util.DefaultUserManager.getAllUsersFromCrowd(DefaultUserManager.java:155)
	at com.atlassian.jira.user.util.DefaultUserManager.getUsers(DefaultUserManager.java:94)
	at sun.reflect.GeneratedMethodAccessor2062.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.atlassian.plugin.osgi.hostcomponents.impl.DefaultComponentRegistrar$ContextClassLoaderSettingInvocationHandler.invoke(DefaultComponentRegistrar.java:134)
	at com.sun.proxy.$Proxy76.getUsers(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor2062.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
	at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:58)
	at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:62)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:56)
	at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:39)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.osgi.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:59)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
	at com.sun.proxy.$Proxy2808.getUsers(Unknown Source)
	at com.atlassian.jira.rest.v2.search.UserListResolver.getAllActiveUsers(UserListResolver.java:106)
	at com.atlassian.jira.rest.v2.search.UserListResolver.getShareUsers(UserListResolver.java:69)
	at com.atlassian.jira.rest.v2.search.FilterBeanBuilder.build(FilterBeanBuilder.java:169)
	at com.atlassian.jira.rest.v2.search.FilterResource$SearchRequestToFilterBean.apply(FilterResource.java:650)
	at com.atlassian.jira.rest.v2.search.FilterResource$SearchRequestToFilterBean.apply(FilterResource.java:622)
	at com.google.common.collect.Iterators$8.next(Iterators.java:812)
	at com.google.common.collect.Lists.newArrayList(Lists.java:139)
	at com.google.common.collect.Lists.newArrayList(Lists.java:119)
	at com.atlassian.jira.rest.v2.search.FilterResource.getFavouriteFilters(FilterResource.java:422)
	at sun.reflect.GeneratedMethodAccessor2548.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper$TypeOutInvoker$1.invoke(DispatchProviderHelper.java:205)
	at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper$1.intercept(DispatchProviderHelper.java:100)
	at com.atlassian.plugins.rest.common.interceptor.impl.DefaultMethodInvocation.invoke(DefaultMethodInvocation.java:61)
	at com.atlassian.plugins.rest.common.expand.interceptor.ExpandInterceptor.intercept(ExpandInterceptor.java:38)
	at com.atlassian.plugins.rest.common.interceptor.impl.DefaultMethodInvocation.invoke(DefaultMethodInvocation.java:61)
	at com.atlassian.jira.rest.exception.ExceptionInterceptor.intercept(ExceptionInterceptor.java:59)
	at com.atlassian.plugins.rest.common.interceptor.impl.DefaultMethodInvocation.invoke(DefaultMethodInvocation.java:61)
	at com.atlassian.jira.rest.v2.issue.scope.RequestScopeInterceptor.intercept(RequestScopeInterceptor.java:43)
	at com.atlassian.plugins.rest.common.interceptor.impl.DefaultMethodInvocation.invoke(DefaultMethodInvocation.java:61)
	at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper.invokeMethodWithInterceptors(DispatchProviderHelper.java:132)
	at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper$TypeOutInvoker._dispatch(DispatchProviderHelper.java:201)
	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
	at com.atlassian.plugins.rest.module.RestDelegatingServletFilter$JerseyOsgiServletContainer.doFilter(RestDelegatingServletFilter.java:178)
	at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
	at com.atlassian.plugins.rest.module.RestDelegatingServletFilter.doFilter(RestDelegatingServletFilter.java:73)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)
	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.plugins.rest.module.servlet.RestServletUtilsUpdaterFilter.doFilterInternal(RestServletUtilsUpdaterFilter.java:26)
	at com.atlassian.plugins.rest.module.servlet.RestServletUtilsUpdaterFilter.doFilter(RestServletUtilsUpdaterFilter.java:40)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:25)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.servicedesk.internal.web.RedirectExternalCustomerToPortalFilter.doFilter(RedirectExternalCustomerToPortalFilter.scala:18)
	at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.mywork.client.filter.ServingRequestsFilter.doFilter(ServingRequestsFilter.java:37)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.plugins.cors.CorsFilter.doFilter(CorsFilter.java:65)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFixupFilter.doFilter(PrettyUrlsSiteMeshFixupFilter.java:36)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.prettyurls.filter.PrettyUrlsDispatcherFilter.doFilter(PrettyUrlsDispatcherFilter.java:60)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFilter.doFilter(PrettyUrlsSiteMeshFilter.java:92)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.prettyurls.filter.PrettyUrlsMatcherFilter.doFilter(PrettyUrlsMatcherFilter.java:56)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.labs.botkiller.BotKillerFilter.doFilter(BotKillerFilter.java:36)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)
	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.jira.web.filters.accesslog.AccessLogFilter.executeRequest(AccessLogFilter.java:107)
	at com.atlassian.jira.web.filters.accesslog.AccessLogFilter.doFilter(AccessLogFilter.java:91)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.jira.security.xsrf.XsrfTokenAdditionRequestFilter.doFilter(XsrfTokenAdditionRequestFilter.java:54)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.jira.web.filters.pagebuilder.PageBuilderFilter.doFilter(PageBuilderFilter.java:90)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.jira.web.filters.CommittedResponseHtmlErrorRecoveryFilter.doFilter(CommittedResponseHtmlErrorRecoveryFilter.java:63)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)
	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.seraph.filter.SecurityFilter.doFilter(SecurityFilter.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:100)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.seraph.filter.BaseLoginFilter.doFilter(BaseLoginFilter.java:172)
	at com.atlassian.jira.web.filters.JiraLoginFilter.doFilter(JiraLoginFilter.java:70)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:79)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.plugins.rest.module.servlet.RestSeraphFilter.doFilter(RestSeraphFilter.java:40)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)
	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.johnson.filters.AbstractJohnsonFilter.doFilter(AbstractJohnsonFilter.java:71)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176)
	at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)
	at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)
	at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:394)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:78)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)
	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.jira.web.filters.steps.ChainedFilterStepRunner.doFilter(ChainedFilterStepRunner.java:87)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.core.filters.cache.AbstractCachingFilter.doFilter(AbstractCachingFilter.java:33)
	at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.core.filters.encoding.AbstractEncodingFilter.doFilter(AbstractEncodingFilter.java:41)
	at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	at com.atlassian.jira.web.filters.PathMatchingEncodingFilter.doFilter(PathMatchingEncodingFilter.java:45)
	at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.jira.startup.JiraStartupChecklistFilter.doFilter(JiraStartupChecklistFilter.java:79)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.jira.web.filters.MultipartBoundaryCheckFilter.doFilter(MultipartBoundaryCheckFilter.java:41)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.jira.web.filters.steps.ChainedFilterStepRunner.doFilter(ChainedFilterStepRunner.java:87)
	at com.atlassian.jira.web.filters.JiraFirstFilter.doFilter(JiraFirstFilter.java:61)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.gzipfilter.GzipFilter.doFilterInternal(GzipFilter.java:115)
	at com.atlassian.gzipfilter.GzipFilter.doFilter(GzipFilter.java:92)
	at com.atlassian.jira.web.filters.gzip.JiraGzipFilter.doFilter(JiraGzipFilter.java:56)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
	- locked <0x000000021c622bc8> (a org.apache.tomcat.util.net.SocketWrapper)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
	- <0x0000000113316680> (a java.util.concurrent.ThreadPoolExecutor$Worker)

Verification

Analysis of the thread dumps can indicate this is a problem, for example searching for the amount of threads in that filter method, specifically:

grep FilterResource.getFavouriteFilters *threads*  |awk '{print $1}' |sort |uniq -c
  29 jira_threads.1436473916.txt:
  31 jira_threads.1436473932.txt:
  28 jira_threads.1436473946.txt:
  29 jira_threads.1436473960.txt:
  29 jira_threads.1436473974.txt:
  31 jira_threads.1436473988.txt:

This shows us there's roughly 30 threads in each of those thread dumps that are calling this method, and this is likely a problem. Further investigation into the dumps and matching them with the stack trace above will verify it as a cause.

is caused by

JRASERVER-33220 /filter&expand=sharedUsers don't report size properly

Closed

is related to

JRASERVER-65892 Favourite filters REST Resource is needlessly retrieving associated share users

Closed

caused: PIR-206 You do not have permission to view this issue

mentioned in: Page Failed to load; Page Failed to load; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...

relates to: HOT-36420 Loading...

(5 mentioned in, 1 relates to)

scmenthusiast added a comment - 03/Aug/2015 8:22 AM - edited

Thank-you for update, Oswaldo.
I will raise a support request for further questions.

scmenthusiast added a comment - 03/Aug/2015 8:22 AM - edited Thank-you for update, Oswaldo. I will raise a support request for further questions.

Oswaldo Hernandez (Inactive) added a comment - 28/Jul/2015 3:37 AM

Hi vprasad79,

We are planning on delivering this fix in 6.4.10, large instances with high concurrent load are the ones affected by this hotspot.

We suggest doing your own performance / load testing on a test environment before proceeding to do and upgrade to 6.4.8 (or any version of JIRA).

Should you have any further questions on your upgraded or need further assistance, could you please raise a support request at https://support.atlassian.com? That way we can provide you with support specific to your circumstances and with higher security. The credentials should be the same as for this site (https://jira.atlassian.com).

Regards,

Oswaldo Hernández.
JIRA Bugmaster.
[Atlassian].

Oswaldo Hernandez (Inactive) added a comment - 28/Jul/2015 3:37 AM Hi vprasad79 , We are planning on delivering this fix in 6.4.10, large instances with high concurrent load are the ones affected by this hotspot. We suggest doing your own performance / load testing on a test environment before proceeding to do and upgrade to 6.4.8 (or any version of JIRA). Should you have any further questions on your upgraded or need further assistance, could you please raise a support request at https://support.atlassian.com? That way we can provide you with support specific to your circumstances and with higher security. The credentials should be the same as for this site ( https://jira.atlassian.com ). Regards, Oswaldo Hernández. JIRA Bugmaster. [Atlassian] .

scmenthusiast added a comment - 24/Jul/2015 8:47 AM

We are planning to upgrade our production jira to 6.4.8 but this issue sounds scary.
Can you please let us know the timeline to fix this issue?

scmenthusiast added a comment - 24/Jul/2015 8:47 AM We are planning to upgrade our production jira to 6.4.8 but this issue sounds scary. Can you please let us know the timeline to fix this issue?

Emiliano (Inactive) added a comment - 20/Jul/2015 4:46 AM

Fix includes getting the count from the current Jira API instead of calculating this each time. Since value is already calculated, it's just a matter of retrieving it.

Emiliano (Inactive) added a comment - 20/Jul/2015 4:46 AM Fix includes getting the count from the current Jira API instead of calculating this each time. Since value is already calculated, it's just a matter of retrieving it.

Jira Data Center

Details

Description

Summary

Environment

Steps to Reproduce

Expected Results

Actual Results

Verification

Attachments

Issue Links

Forms

Activity

Collapse comment: scmenthusiast added a comment - 03/Aug/2015 8:22 AM, Edited by scmenthusiast - 03/Aug/2015 8:23 AM

Expand comment: scmenthusiast added a comment - 03/Aug/2015 8:22 AM, Edited by scmenthusiast - 03/Aug/2015 8:23 AM

Collapse comment: Oswaldo Hernandez (Inactive) added a comment - 28/Jul/2015 3:37 AM

Expand comment: Oswaldo Hernandez (Inactive) added a comment - 28/Jul/2015 3:37 AM

Collapse comment: scmenthusiast added a comment - 24/Jul/2015 8:47 AM

Expand comment: scmenthusiast added a comment - 24/Jul/2015 8:47 AM

Collapse comment: Emiliano (Inactive) added a comment - 20/Jul/2015 4:46 AM

Expand comment: Emiliano (Inactive) added a comment - 20/Jul/2015 4:46 AM

People

Dates