Uploaded image for project: 'Jira Software Data Center'
  1. Jira Software Data Center
  2. JSWSERVER-9611

performance very poor for kanban boards when instance has large amount of projects/issues

    • 6.02
    • Hide
      Atlassian Status as of Monday November 3
      • Changes have been made in JIRA Agile to improve the performance of calculations of status changes – this issue (GHS-9611) tracks the progress on this. This will improve performance on loading Kanban Boards (when Days In Column are shown) as well as charts such as Burndown, Control Chart, CFD.
      • A separate task for JIRA Agile to temporarily disable the "days in column" functionality (which will provide benefits to all supported JIRA versions) is being tracked in GHS-10520. This can be used to reduce the performance impact of this problem.
      • There is also a known issue with the unreleasedVersions() JQL, which is the default work sub-filter, that is fixed in JIRA 6.3.1 as tracked in JRA-38883.

      The fix for this bug has now been shipped in JIRA Agile 6.6.11.

      Show
      Atlassian Status as of Monday November 3 Changes have been made in JIRA Agile to improve the performance of calculations of status changes – this issue ( GHS-9611 ) tracks the progress on this. This will improve performance on loading Kanban Boards (when Days In Column are shown) as well as charts such as Burndown, Control Chart, CFD. A separate task for JIRA Agile to temporarily disable the "days in column" functionality (which will provide benefits to all supported JIRA versions) is being tracked in GHS-10520 . This can be used to reduce the performance impact of this problem. There is also a known issue with the unreleasedVersions() JQL, which is the default work sub-filter, that is fixed in JIRA 6.3.1 as tracked in JRA-38883 . The fix for this bug has now been shipped in JIRA Agile 6.6.11.

      After moving an issue on the board the board is refreshed by calling xboard/work/allData.json. On our production instance this takes 6 seconds. Updating the issue is relatively quick, refreshing the board data is not.

      This is down to com.atlassian.greenhopper.service.lucene.ChangeHistorySearchHelper#findChangeLogFilter. Specifically, computing the cross-references from two queries, one of which may contain up to half a million issues, is very slow. I'm sure there is a quicker way, even if it's just checking the permission for each issue retrieved from the change history index.

      As a workaround I am just disabling loading the days in column data in com.atlassian.greenhopper.web.rapid.list.RapidIssueEntryQueryServiceImpl#collectWorkModeIssues, which makes things passably fast.

      Areas of JIRA Agile Affected

      • Generating JIRA Agile charts - notably Burndown, Cumulative Flow Diagram & Control Chart.
      • Loading days in column for a list of issues (workaround to disable this in GHS-10520).

      Workaround for Work Sub-Filter

      This is fixed in JIRA 6.3.1 as tracked in JRA-38883, so this workaround will only apply for previous versions of JIRA.

      It's possible that on larger instances the default work sub-filter may be causing performance issues of its own. This is not related to com.atlassian.greenhopper.service.lucene.ChangeHistorySearchHelper however does present with similar symptoms. Specifically, the work sub-filter default is:

      fixVersion in unreleasedVersions() OR fixVersion is EMPTY
      

      This unreleasedVersions() JQL may be slow for larger instances that process a large number of projects, as it goes through all of them to find out which versions are relevant. This can be sped up by specifying a list of projects that are only relevant to the JIRA Agile board being used. For example if we wanted to only return issues for projects GHS and JRA (because they are in the board filter), we would use the following:

      fixVersion in unreleasedVersions(GHS,JRA) OR fixVersion is EMPTY
      

      Please make the above changes and verify if the boards are faster to render or not. If they are faster, it implies the unreleasedVersions() was slow to process due to the large number of projects. If it does not it would indicate that the com.atlassian.greenhopper.service.lucene.ChangeHistorySearchHelper is the cause of slowness.

      It's possible that both unreleasedVersions() and com.atlassian.greenhopper.service.lucene.ChangeHistorySearchHelper are slow to process and this workaround only addresses one part of the problem.

      Sample Thread Dump

      "http-bio-8080-exec-29" daemon prio=10 tid=0x00007f6bc0034000 nid=0x6705 runnable [0x00007f6bb4ba0000]
         java.lang.Thread.State: RUNNABLE
      	at java.util.BitSet.nextSetBit(Unknown Source)
      	at com.atlassian.greenhopper.service.lucene.BitSetRightJoinEnum.hasNext(BitSetRightJoinEnum.java:65)
      	at com.atlassian.greenhopper.service.lucene.LeftJoin.computeResult(LeftJoin.java:52)
      	at com.atlassian.greenhopper.service.lucene.ChangeHistorySearchHelper.findChangeLogFilter(ChangeHistorySearchHelper.java:81)
      	at com.atlassian.greenhopper.service.charts.IssueStatusHistoryServiceImpl.collectStatusHistory(IssueStatusHistoryServiceImpl.java:79)
      	at com.atlassian.greenhopper.web.rapid.list.RapidIssueEntryQueryServiceImpl.loadDaysInColumn(RapidIssueEntryQueryServiceImpl.java:820)
      	at com.atlassian.greenhopper.web.rapid.list.RapidIssueEntryQueryServiceImpl.collectIssues(RapidIssueEntryQueryServiceImpl.java:716)
      	at com.atlassian.greenhopper.web.rapid.list.RapidIssueEntryQueryServiceImpl.collectIssues(RapidIssueEntryQueryServiceImpl.java:627)
      	at com.atlassian.greenhopper.web.rapid.list.RapidIssueEntryQueryServiceImpl.collectWorkModeIssues(RapidIssueEntryQueryServiceImpl.java:178)
      	at com.atlassian.greenhopper.web.rapid.work.WorkDataFactory.getOrderedIssuesData(WorkDataFactory.java:511)
      	at com.atlassian.greenhopper.web.rapid.work.WorkDataFactory.getAllData(WorkDataFactory.java:177)
      	at com.atlassian.greenhopper.web.rapid.work.WorkResource$1.call(WorkResource.java:70)
      	at com.atlassian.greenhopper.web.rapid.work.WorkResource$1.call(WorkResource.java:59)
      	at com.atlassian.greenhopper.web.util.RestCall.response(RestCall.java:48)
      	at com.atlassian.greenhopper.web.AbstractResource.response(AbstractResource.java:60)
      	at com.atlassian.greenhopper.web.rapid.work.WorkResource.getPoolData(WorkResource.java:58)
      	at sun.reflect.GeneratedMethodAccessor1993.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      	at java.lang.reflect.Method.invoke(Unknown Source)
      	at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper$ResponseOutInvoker$1.invoke(DispatchProviderHelper.java:234)
      	at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper$1.intercept(DispatchProviderHelper.java:100)
      	at com.atlassian.plugins.rest.common.interceptor.impl.DefaultMethodInvocation.invoke(DefaultMethodInvocation.java:61)
      	at com.atlassian.plugins.rest.common.expand.interceptor.ExpandInterceptor.intercept(ExpandInterceptor.java:38)
      	at com.atlassian.plugins.rest.common.interceptor.impl.DefaultMethodInvocation.invoke(DefaultMethodInvocation.java:61)
      	at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper.invokeMethodWithInterceptors(DispatchProviderHelper.java:132)
      	at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper$ResponseOutInvoker._dispatch(DispatchProviderHelper.java:230)
      	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
      	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
      	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
      	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
      	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
      	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
      	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
      	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
      	at com.atlassian.plugins.rest.module.RestDelegatingServletFilter$JerseyOsgiServletContainer.doFilter(RestDelegatingServletFilter.java:178)
      	at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
      	at com.atlassian.plugins.rest.module.RestDelegatingServletFilter.doFilter(RestDelegatingServletFilter.java:73)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)
      	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
      	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)
      	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)
      	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)
      	at com.atlassian.plugins.rest.module.servlet.RestServletUtilsUpdaterFilter.doFilterInternal(RestServletUtilsUpdaterFilter.java:26)
      	at com.atlassian.plugins.rest.module.servlet.RestServletUtilsUpdaterFilter.doFilter(RestServletUtilsUpdaterFilter.java:40)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)
      	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)
      	at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:25)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)
      	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)
      	at com.atlassian.mywork.client.filter.ServingRequestsFilter.doFilter(ServingRequestsFilter.java:37)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)
      	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)
      	at com.atlassian.plugins.cors.CorsFilter.doFilter(CorsFilter.java:65)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)
      	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)
      	at com.atlassian.prettyurls.filter.PrettyUrlsDispatcherFilter.doFilter(PrettyUrlsDispatcherFilter.java:60)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)
      	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)
      	at com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFilter.doFilter(PrettyUrlsSiteMeshFilter.java:92)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)
      	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)
      	at com.atlassian.prettyurls.filter.PrettyUrlsMatcherFilter.doFilter(PrettyUrlsMatcherFilter.java:56)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)
      	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)
      	at com.atlassian.labs.botkiller.BotKillerFilter.doFilter(BotKillerFilter.java:36)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)
      	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
      	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)
      	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
      	at com.atlassian.jira.web.filters.accesslog.AccessLogFilter.executeRequest(AccessLogFilter.java:103)
      	at com.atlassian.jira.web.filters.accesslog.AccessLogFilter.doFilter(AccessLogFilter.java:87)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
      	at com.atlassian.jira.security.xsrf.XsrfTokenAdditionRequestFilter.doFilter(XsrfTokenAdditionRequestFilter.java:54)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
      	at com.atlassian.jira.web.filters.steps.ChainedFilterStepRunner.doFilter(ChainedFilterStepRunner.java:87)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
      	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)
      	at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)
      	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
      	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)
      	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
      	at com.atlassian.seraph.filter.SecurityFilter.doFilter(SecurityFilter.java:234)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
      	at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:100)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
      	at com.atlassian.seraph.filter.BaseLoginFilter.doFilter(BaseLoginFilter.java:169)
      	at com.atlassian.jira.web.filters.JiraLoginFilter.doFilter(JiraLoginFilter.java:70)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
      	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)
      	at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:55)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)
      	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)
      	at com.atlassian.plugins.rest.module.servlet.RestSeraphFilter.doFilter(RestSeraphFilter.java:40)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)
      	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)
      	at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)
      	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
      	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)
      	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
      	at com.atlassian.johnson.filters.AbstractJohnsonFilter.doFilter(AbstractJohnsonFilter.java:71)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
      	at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176)
      	at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)
      	at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)
      	at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:394)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
      	at com.atlassian.gzipfilter.GzipFilter.doFilterInternal(GzipFilter.java:80)
      	at com.atlassian.gzipfilter.GzipFilter.doFilter(GzipFilter.java:51)
      	at com.atlassian.jira.web.filters.gzip.JiraGzipFilter.doFilter(JiraGzipFilter.java:55)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
      	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)
      	at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)
      	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)
      	at com.sysbliss.jira.plugins.workflow.servlet.JWDSendRedirectFilter.doFilter(JWDSendRedirectFilter.java:25)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)
      	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:66)
      	at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:202)
      	at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:180)
      	at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:85)
      	at net.bull.javamelody.JiraMonitoringFilter.doFilter(JiraMonitoringFilter.java:104)
      	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:74)
      	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:42)
      	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:77)
      	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:63)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
      	at com.atlassian.jira.web.filters.steps.ChainedFilterStepRunner.doFilter(ChainedFilterStepRunner.java:87)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
      	at com.atlassian.core.filters.cache.AbstractCachingFilter.doFilter(AbstractCachingFilter.java:33)
      	at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
      	at com.atlassian.core.filters.encoding.AbstractEncodingFilter.doFilter(AbstractEncodingFilter.java:41)
      	at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
      	at com.atlassian.jira.web.filters.PathMatchingEncodingFilter.doFilter(PathMatchingEncodingFilter.java:45)
      	at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
      	at com.atlassian.jira.startup.JiraStartupChecklistFilter.doFilter(JiraStartupChecklistFilter.java:78)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
      	at com.atlassian.jira.web.filters.steps.ChainedFilterStepRunner.doFilter(ChainedFilterStepRunner.java:87)
      	at com.atlassian.jira.web.filters.JiraFirstFilter.doFilter(JiraFirstFilter.java:57)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
      	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:225)
      	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
      	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
      	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
      	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
      	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
      	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
      	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
      	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1001)
      	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:585)
      	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
      	- locked <0x00007f70d75e5e70> (a org.apache.tomcat.util.net.SocketWrapper)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      	at java.lang.Thread.run(Unknown Source)
      

            [JSWSERVER-9611] performance very poor for kanban boards when instance has large amount of projects/issues

            @Jamie and @Qualcomm support,

            Also could you please tell me which tool you have used to do Invocation trace as per attached screenshots?
            Our jira is configured on linux.

            Thanks and regards,

            Pankaj

            Pankaj Pimple added a comment - @Jamie and @Qualcomm support, Also could you please tell me which tool you have used to do Invocation trace as per attached screenshots? Our jira is configured on linux. Thanks and regards, Pankaj

            Hi All,

            We are using Agile 6.7.11 with JIRA 6.4.10 and could see same performance issue. While loading dashboards between 9-10 LDN time when all users are connecting and may be starting their standups, timeouts out errors are getting.
            We checked our Hardware and as per admins the hardware is sufficient and we need to look at the application side.
            i am aware that we have many issues on boards and many teams accessing board at the same time causing this issue.

            Hi jamie, Can I try your patch and if yes could you please tell how to deploy it?

            Thanks and regards,
            pankaj

            Pankaj Pimple added a comment - Hi All, We are using Agile 6.7.11 with JIRA 6.4.10 and could see same performance issue. While loading dashboards between 9-10 LDN time when all users are connecting and may be starting their standups, timeouts out errors are getting. We checked our Hardware and as per admins the hardware is sufficient and we need to look at the application side. i am aware that we have many issues on boards and many teams accessing board at the same time causing this issue. Hi jamie, Can I try your patch and if yes could you please tell how to deploy it? Thanks and regards, pankaj

            Thank you everyone for your patience. We have been investigating this issue and we believe we have a fix prepared. Please watch this issue for further updates on when the fix will be released. We expect this to be some time soon, within the next few weeks.

            Regards,
            JIRA Agile Team

            Michael Tokar added a comment - Thank you everyone for your patience. We have been investigating this issue and we believe we have a fix prepared. Please watch this issue for further updates on when the fix will be released. We expect this to be some time soon, within the next few weeks. Regards, JIRA Agile Team

            intersol_OLD that caching setting is only applied to the Classic Boards.

            Michael Tokar added a comment - intersol_OLD that caching setting is only applied to the Classic Boards.

            intersol_old added a comment -

            Does this has anything to do with the issue caching in Agile plugin, now is configured to 50.000 but we do have ~300.000 issues on our JIRA instance. The documentation doesn't say much about it https://confluence.atlassian.com/display/AGILE/Specifying+your+JIRA+Agile+General+Configuration?focusedCommentId=664995483&#comment-664995483

            intersol_old added a comment - Does this has anything to do with the issue caching in Agile plugin, now is configured to 50.000 but we do have ~300.000 issues on our JIRA instance. The documentation doesn't say much about it https://confluence.atlassian.com/display/AGILE/Specifying+your+JIRA+Agile+General+Configuration?focusedCommentId=664995483&#comment-664995483

            intersol_old added a comment -

            Please take a look at https://answers.atlassian.com/questions/8620924/how-to-make-agile-board-planning-loading-to-a-decent-performance-level as it presents the performance problem as being more like a design issue.

            intersol_old added a comment - Please take a look at https://answers.atlassian.com/questions/8620924/how-to-make-agile-board-planning-loading-to-a-decent-performance-level as it presents the performance problem as being more like a design issue.

            Dieter : Same thing here: After a fresh start, Jira take about 2s to load our big board (not as big as your, but big too !). Just wait some days and you will see your 2s goes up to 4s and 6s and... after ~ 2 month our take 16s to load ! Just restart Jira and Bang : back to 2s.

            Actually on Jira 6.2.3. Planning to passe at 6.3.4 today.

            Yannick Bilodeau added a comment - Dieter : Same thing here: After a fresh start, Jira take about 2s to load our big board (not as big as your, but big too !). Just wait some days and you will see your 2s goes up to 4s and 6s and... after ~ 2 month our take 16s to load ! Just restart Jira and Bang : back to 2s. Actually on Jira 6.2.3. Planning to passe at 6.3.4 today.

            Dieter Greiner added a comment - - edited

            It's not clear to me why the same board on JIRA 6.3.4 / Agile 6.5.0 takes so much longer than on JIRA 5.2.11/Agile 6.3.6.1. It's 12s on JIRA 6 and 2s on JIRA5. Restricting the work sub filter to a project didn't help.

            The JIRA 6.3.4 is an exact copy of the JIRA 5.2.11 system (1020 projects, 2000 fields, 600000 issues). Agile 6.3.6.1 should be affected by the bug as well but it doesn't seem to.

            What could be the reason for JIRA 5.2.11 being so much faster? I can confirm that i see the mentioned stack trace while the board is being loaded.

            Update
            After i have restarted the JIRA 6 system performance is the same as in JIRA 5. Strange ...

            Dieter Greiner added a comment - - edited It's not clear to me why the same board on JIRA 6.3.4 / Agile 6.5.0 takes so much longer than on JIRA 5.2.11/Agile 6.3.6.1. It's 12s on JIRA 6 and 2s on JIRA5. Restricting the work sub filter to a project didn't help. The JIRA 6.3.4 is an exact copy of the JIRA 5.2.11 system (1020 projects, 2000 fields, 600000 issues). Agile 6.3.6.1 should be affected by the bug as well but it doesn't seem to. What could be the reason for JIRA 5.2.11 being so much faster? I can confirm that i see the mentioned stack trace while the board is being loaded. Update After i have restarted the JIRA 6 system performance is the same as in JIRA 5. Strange ...

            Can we running Jira 6.3.1 and hitting severe performance issues because of this, can we please get a backport to this issue asap.
            -am

            Ashok Manthina added a comment - Can we running Jira 6.3.1 and hitting severe performance issues because of this, can we please get a backport to this issue asap. -am

            Cool. Perhaps "Manage Boards" and "Go To Board" issues will be fixed in this unknown release as well.

            Also, please consider making board refresh frequency configurable and/or make it stop refreshing after a while, like Structure plugin does – these polls take away valuable resources from the app.

            Sergey Svishchev added a comment - Cool. Perhaps "Manage Boards" and "Go To Board" issues will be fixed in this unknown release as well. Also, please consider making board refresh frequency configurable and/or make it stop refreshing after a while, like Structure plugin does – these polls take away valuable resources from the app.

            Dave C added a comment -

            svs the work sub-filter problem was fixed in JIRA 6.3.1 as per JRA-38883.

            I can't provide any information on when this would be fixed.

            Dave C added a comment - svs the work sub-filter problem was fixed in JIRA 6.3.1 as per JRA-38883 . I can't provide any information on when this would be fixed.

            Will "Work Sub-Filter" still be a problem after new change history index is used by Agile?

            Is it safe to say that next major Agile release (6.5) will have the fix for this issue and GHS-10520?

            Sergey Svishchev added a comment - Will "Work Sub-Filter" still be a problem after new change history index is used by Agile? Is it safe to say that next major Agile release (6.5) will have the fix for this issue and GHS-10520 ?

            Dave C added a comment -

            An update on this:

            • A change in JIRA's change history index to store additional information was included in JIRA 6.3.1. This change by itself does not have any performance gains, however when JIRA Agile is updated to use that additional information there will be performance gains.
            • JIRA Agile has not yet been updated to consume the new information in the index – this issue tracks the progress on this.
            • A separate task for JIRA Agile to temporarily disable the "days in column" functionality (which will provide benefits to all supported JIRA versions) is being tracked in GHS-10520. This can be used to reduce the performance impact of this problem.

            Dave C added a comment - An update on this: A change in JIRA's change history index to store additional information was included in JIRA 6.3.1. This change by itself does not have any performance gains, however when JIRA Agile is updated to use that additional information there will be performance gains. JIRA Agile has not yet been updated to consume the new information in the index – this issue tracks the progress on this. A separate task for JIRA Agile to temporarily disable the "days in column" functionality (which will provide benefits to all supported JIRA versions) is being tracked in GHS-10520 . This can be used to reduce the performance impact of this problem.

            The core driver of this issue based on our research is poor performance of the Days in Column display in Kanban when used over a large number of issues. The short-term plan is to allow administrators to disable that feature as a work-around for this issue, and that work is tracked in GHS-10520. The eventual resolution will be to improve the performance of this tabulation function, which continues to be tracked here.

            Please continue to watch this issue for updates.

            John Garcia (Inactive) added a comment - The core driver of this issue based on our research is poor performance of the Days in Column display in Kanban when used over a large number of issues. The short-term plan is to allow administrators to disable that feature as a work-around for this issue, and that work is tracked in GHS-10520 . The eventual resolution will be to improve the performance of this tabulation function, which continues to be tracked here. Please continue to watch this issue for updates.

            I'm sorry, but what's the fix? Upgrading?

            IT Operations added a comment - I'm sorry, but what's the fix? Upgrading?

            JamieA added a comment -

            I can give you our production database, it's jira 6.2. Will check the JAG version when I'm in the office tomorrow, think it might be 6.2.5.1. If that's not too old let me know. I should be able to upload via your write-only ftp.

            JamieA added a comment - I can give you our production database, it's jira 6.2. Will check the JAG version when I'm in the office tomorrow, think it might be 6.2.5.1. If that's not too old let me know. I should be able to upload via your write-only ftp.

            jamieechlin, do you have a dataset you could share with us which reproduces the issue with a modern version of JIRA and JAG? I would be happy to do the work to reproduce it on our end.

            Boris Berenberg (Inactive) added a comment - jamieechlin , do you have a dataset you could share with us which reproduces the issue with a modern version of JIRA and JAG? I would be happy to do the work to reproduce it on our end.

            JamieA added a comment -

            I do use it, surprise surprise, and have had no problems, although I haven't ported it to very recent versions. The thing is, this issue does not depend on server load or number of connections etc, so you should be able to reproduce fairly easily that it a) returns the same data and b) is much faster, by copying your current production data to a UAT system.

            JamieA added a comment - I do use it, surprise surprise, and have had no problems, although I haven't ported it to very recent versions. The thing is, this issue does not depend on server load or number of connections etc, so you should be able to reproduce fairly easily that it a) returns the same data and b) is much faster, by copying your current production data to a UAT system.

            Does anyone use Jamie's patch in production? Since Atlassian is taking their sweet time fixing this...

            Sergey Svishchev added a comment - Does anyone use Jamie's patch in production? Since Atlassian is taking their sweet time fixing this...

            Same issue on Agile Boards by the way!

            Yann Grenier added a comment - Same issue on Agile Boards by the way!

            Does that mean it will be fixed soon?

            Sergey Svishchev added a comment - Does that mean it will be fixed soon?

            Found a similar problem with a JIRA DC Pioneer customer

            MichaelA (Inactive) added a comment - Found a similar problem with a JIRA DC Pioneer customer

            Devu Heda added a comment -

            This is a major issue at out end as well .. we are not able to load the Kanban board at all .... it is literally unusable
            Atlassian folks this is the ID JSP-190471 if anyone is interested ...

            we need a faster resolution on this

            Devu Heda added a comment - This is a major issue at out end as well .. we are not able to load the Kanban board at all .... it is literally unusable Atlassian folks this is the ID JSP-190471 if anyone is interested ... we need a faster resolution on this

            I found that the release 6.3.12 is significantly faster than the original 6.3.9.2 (vs the patched 6.3.9.2).
            On our test system with Jira 6.1.7 (around 200 000 issues, 1000 projects) the loading time is around the same as the patched 6.3.9.2.
            Of course there is not much user activity so I don't know how will behave with significant user activity.

            Balint Zsolt added a comment - I found that the release 6.3.12 is significantly faster than the original 6.3.9.2 (vs the patched 6.3.9.2). On our test system with Jira 6.1.7 (around 200 000 issues, 1000 projects) the loading time is around the same as the patched 6.3.9.2. Of course there is not much user activity so I don't know how will behave with significant user activity.

            We are affected by this as well on an instance with ~ 1000 projects / 500000 issues and board with 35 issues. We have measured this with identical data:

            • JIRA 5.2.11 / JIRA Agile 6.3.6.1 => Loading time: 2s
            • JIRA 6.2.2 ( JIRA Agile 6.3.12 => Loading time: 7s

            Strange enough that sometimes the board only reloads after several attempts with F5.

            This is a blocker for rollout of JIRA 6.2.2 in conjunction with this and that. Like Jamie we have made fixes on our own for part of these issues but this is not the way we can continue.

            Dieter Greiner added a comment - We are affected by this as well on an instance with ~ 1000 projects / 500000 issues and board with 35 issues. We have measured this with identical data: JIRA 5.2.11 / JIRA Agile 6.3.6.1 => Loading time: 2s JIRA 6.2.2 ( JIRA Agile 6.3.12 => Loading time: 7s Strange enough that sometimes the board only reloads after several attempts with F5. This is a blocker for rollout of JIRA 6.2.2 in conjunction with this and that . Like Jamie we have made fixes on our own for part of these issues but this is not the way we can continue.

            Many thanks for reporting this issue, and for submitting a patch too. We plan to look at this and address it in the near future.

            Apparently they couldn't load their Kanban board with this issue ...

            Sergey Svishchev added a comment - Many thanks for reporting this issue, and for submitting a patch too. We plan to look at this and address it in the near future. Apparently they couldn't load their Kanban board with this issue ...

            We're seeing the same issue as @jamieechlin on one of our instances. Here is the snippet of the invocation trace I captured from one of our production systems:

            ...
            RapidIssueEntryQueryServiceImpl.loadDaysInColumn:823 (0ms self time, 38050 ms total time)
                IssueStatusHistoryServiceImpl.collectStatusHistory:79 (0ms self time, 38050 ms total time)
                    ChangeHistorySearchHelper.findChangeLogFilter:81 (0ms self time, 38050 ms total time)
                        LeftJoin.computeResult:52 (0ms self time, 38050 ms total time)
                            BitSetRightJoinEnum.hasNext:65 (38050ms self time, 38050 ms total time)
            ....
            

            There is a whopping 38s just in this one calculation – and this with a decently powered box with indexes stored on fusion i/o cards. Given that someone has already provided a patch, it would be nice to see some movement on this issue – in our case this is flat out unusable.

            Qualcomm Support added a comment - We're seeing the same issue as @jamieechlin on one of our instances. Here is the snippet of the invocation trace I captured from one of our production systems: ... RapidIssueEntryQueryServiceImpl.loadDaysInColumn:823 (0ms self time, 38050 ms total time) IssueStatusHistoryServiceImpl.collectStatusHistory:79 (0ms self time, 38050 ms total time) ChangeHistorySearchHelper.findChangeLogFilter:81 (0ms self time, 38050 ms total time) LeftJoin.computeResult:52 (0ms self time, 38050 ms total time) BitSetRightJoinEnum.hasNext:65 (38050ms self time, 38050 ms total time) .... There is a whopping 38s just in this one calculation – and this with a decently powered box with indexes stored on fusion i/o cards. Given that someone has already provided a patch, it would be nice to see some movement on this issue – in our case this is flat out unusable.

            David Yu added a comment -

            We also see more board requests taking 7 ~ 10 seconds to complete. Approaching 500K issues.

            David Yu added a comment - We also see more board requests taking 7 ~ 10 seconds to complete. Approaching 500K issues.

            Konstantin Shtennikov added a comment - - edited

            Have problem too
            But do not know it's the same problem or not

            Problem begin after change JIRA to 6.1 from 5.2, and Agile 6.3.5 (from 6.3.1)
            Before it all working fast

            Now, ~250 000 issue, board with 1 issue opened 3-10 sec
            Is that request: http://jira/rest/greenhopper/1.0/xboard/work/allData.json?rapidViewId=107&_=1387373084863

            Konstantin Shtennikov added a comment - - edited Have problem too But do not know it's the same problem or not Problem begin after change JIRA to 6.1 from 5.2, and Agile 6.3.5 (from 6.3.1) Before it all working fast Now, ~250 000 issue, board with 1 issue opened 3-10 sec Is that request: http://jira/rest/greenhopper/1.0/xboard/work/allData.json?rapidViewId=107&_=1387373084863

            Many thanks for reporting this issue, and for submitting a patch too. We plan to look at this and address it in the near future.

            Kind regards

            Tom Kotecki
            Product Manager, GreenHopper

            Tom Kotecki (Inactive) added a comment - Many thanks for reporting this issue, and for submitting a patch too. We plan to look at this and address it in the near future. Kind regards Tom Kotecki Product Manager, GreenHopper

            JamieA added a comment -

            I may have got the diagnosis a bit wrong, although the symptoms are correct. I thought it was down to the number of docs in the issue index and the "permissions query" returning too much. Actually I think it's because the changehistory index has 2.5MM docs, so even if you reduce the matches from the issue index it's still necessary to join on the 2.5MM set.

            I have an alternative implementation which doesn't change the functionality. Patch attached. It runs a query on the changehistory index rather than seeking through it looking for matching issue IDs.

            As always, I'm attaching a patch not to invite a code critique or to say you should implement it the same way, just to illustrate an alternative approach which appears to work fine. This reduces the allData.json call on kanban boards from 3.5s to under 0.5s.

            JamieA added a comment - I may have got the diagnosis a bit wrong, although the symptoms are correct. I thought it was down to the number of docs in the issue index and the "permissions query" returning too much. Actually I think it's because the changehistory index has 2.5MM docs, so even if you reduce the matches from the issue index it's still necessary to join on the 2.5MM set. I have an alternative implementation which doesn't change the functionality. Patch attached. It runs a query on the changehistory index rather than seeking through it looking for matching issue IDs. As always, I'm attaching a patch not to invite a code critique or to say you should implement it the same way, just to illustrate an alternative approach which appears to work fine. This reduces the allData.json call on kanban boards from 3.5s to under 0.5s.

            JamieA added a comment -

            That's probably relevant, but I think that's a different performance issue than I have.

            In my case there are only a handful of issues on the board, but the work is in computeResult() called by findChangeLogFilter().

            This is taking a BitSet containing approx half a million issue IDs and "inner joining" it on the board filter issues, to check which issues the user has permission to see. As the javadoc notes, the performance is O(l+r), where l and r are the number of bits in each set. Therefore, if you operate an instance where users by default have permissions to see almost everything this is always going to be slow. (In contrast, there are good reasons to use permission schemes which contain the Anyone pseudo-group).

            I don't really quite see why this needs to be done... if the issues aren't on the board the user is not going to see them.

            Also 25s, depending on what instrumentation you are using, may not be significant. What's the speed when the profiler is not recording?

            JamieA added a comment - That's probably relevant, but I think that's a different performance issue than I have. In my case there are only a handful of issues on the board, but the work is in computeResult() called by findChangeLogFilter(). This is taking a BitSet containing approx half a million issue IDs and "inner joining" it on the board filter issues, to check which issues the user has permission to see. As the javadoc notes, the performance is O(l+r), where l and r are the number of bits in each set. Therefore, if you operate an instance where users by default have permissions to see almost everything this is always going to be slow. (In contrast, there are good reasons to use permission schemes which contain the Anyone pseudo-group). I don't really quite see why this needs to be done... if the issues aren't on the board the user is not going to see them. Also 25s, depending on what instrumentation you are using, may not be significant. What's the speed when the profiler is not recording?

            Yilin (Inactive) added a comment - - edited

            Able to reproduce the issue with 269 issue in one KanBan board, which took 25 seconds to load the board after moving one issue.

            It seems that there are two very consuming class (please check the jprofiler output in the attachment):

            1. ChangeHistoryCollector
            2. findChangeLogFilter

            Yilin (Inactive) added a comment - - edited Able to reproduce the issue with 269 issue in one KanBan board, which took 25 seconds to load the board after moving one issue. It seems that there are two very consuming class (please check the jprofiler output in the attachment): ChangeHistoryCollector findChangeLogFilter

              Unassigned Unassigned
              714061b26763 JamieA
              Affected customers:
              73 This affects my team
              Watchers:
              99 Start watching this issue

                Created:
                Updated:
                Resolved: