Uploaded image for project: 'Jira Software Data Center'
  1. Jira Software Data Center
  2. JSWSERVER-20445

Sprint picker field causing too much CPU

XMLWordPrintable

      Issue Summary

      The Sprint field drawdown suggestion can be CPU hungry and take a long time to calculate due to the following hotspot
      https://stash.atlassian.com/projects/jswserver/repos/jira-agile/browse/greenhopper/src/main/java/com/atlassian/greenhopper/web/rapid/sprint/SprintResource.java?at=230fa87c55fdb18d803279df30b333fa58086f4f#99

        Set<Sprint> allMatches = check(sprintHelper.findSprintPickerAllMatches(user, suggestions, searchQuery, maxResults, excludeCompleted));

      The Sprint field suggestion is calculated based on users' last 5 visited Sprint. If a user has not to visit a minimum of 5 Sprint, the suggestion will be calculated by retrieving  ALL SPRINT available to the user and perform some calculation to return number issues associated with each Sprint.

      In large instances, with 3M issues and 52K Sprints, this operation alone can take 2 minutes to complete and consume significant CPU. 

       

      In the thread dump, you will see the following thread exist in multiple threads (ie. longer than 60-180s)

      url:/rest/greenhopper/1.0/sprint/picker?query username:vkharisma
java.util.TreeMap.getEntry(TreeMap.java:359)
java.util.TreeMap.get(TreeMap.java:278)
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.terms(PerFieldPostingsFormat.java:315)
org.apache.lucene.index.CodecReader.terms(CodecReader.java:106)
org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:122)
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:90)
org.apache.lucene.search.Weight.scorerSupplier(Weight.java:113)
org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:720)
org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:329)
org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:295)
org.apache.lucene.search.Weight.bulkScorer(Weight.java:147)
org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:289)
org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:795)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:657)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)
com.atlassian.jira.index.DelegateSearcher.search(DelegateSearcher.java:169)
com.atlassian.jira.index.DelegateSearcher.search(DelegateSearcher.java:169)
com.atlassian.jira.index.UnmanagedIndexSearcher.search(UnmanagedIndexSearcher.java:9)
com.atlassian.jira.index.DelegateSearcher.search(DelegateSearcher.java:169)
com.atlassian.jira.index.ManagedIndexSearcher.search(ManagedIndexSearcher.java:15)
com.atlassian.jira.issue.search.providers.LuceneSearchProvider.getHitCount(LuceneSearchProvider.java:178)
com.atlassian.jira.issue.search.providers.LuceneSearchProvider.getHitCount(LuceneSearchProvider.java:145)
com.atlassian.jira.bc.issue.search.DefaultSearchService.searchCount(DefaultSearchService.java:130)
......
com.atlassian.greenhopper.service.sprint.SprintPermissionServiceImpl.getIssueCountForSprint(SprintPermissionServiceImpl.java:250)
com.atlassian.greenhopper.service.sprint.SprintPermissionServiceImpl.canViewSprint(SprintPermissionServiceImpl.java:90)
com.atlassian.greenhopper.customfield.sprint.SprintResolver.checkSprintAccessible(SprintResolver.java:31)
com.atlassian.greenhopper.customfield.sprint.SprintResolver$SprintAccessible.apply(SprintResolver.java:139)
com.atlassian.greenhopper.customfield.sprint.SprintResolver$SprintAccessible.apply(SprintResolver.java:125)
com.google.common.base.Predicates$AndPredicate.apply(Predicates.java:353)

      The bottleneck appears to be in the following code which will fetch all Sprints available to the user

        Set<Sprint> allMatches = check(sprintHelper.findSprintPickerAllMatches(user, suggestions, searchQuery, maxResults, excludeCompleted));

      Steps to Reproduce

      1. Create 500 Sprint with 50,000 issues each (to exaggerate the problem)
      2. Go to Issue Navigator > Click on Sprint dropdown
      3. Notice the dropdown appear slowly 

      Expected Results

      A dropdown appears reasonably instantly. 

      Actual Results

      In the extreme, where Jira instance has 3.5M issues and a lot of Sprint, this sprint picker field can take up to 2 minutes to return:

      xxx.xxx.xxx [14/Feb/2020:08:02:47 +0000] "GET /jira/rest/greenhopper/1.0/sprint/picker?query=&_=1581667257699 HTTP/1.1" 200 1233 93952 "https://localhost/jira/browse/FBVRDS-1449" 
xxx.xxx.xxx [14/Feb/2020:08:02:12 +0000] "GET /jira/rest/greenhopper/1.0/sprint/picker?query=&_=1581667222695 HTTP/1.1" 200 1233 95313 "https://localhost/jira/browse/FBVRDS-1448" 
xxx.xxx.xxx [14/Feb/2020:08:01:04 +0000] "GET /jira/rest/greenhopper/1.0/sprint/picker?query=&_=1581667061798 HTTP/1.1" 200 1213 126270 "https://localhost/jira/secure/RapidBoard.jspa?rapidView=88532&selectedIssue=CMBGSAI-10743&quickFilter=129584&quickFilter=142240" 
xxx.xxx.xxx [14/Feb/2020:08:01:03 +0000] "GET /jira/rest/greenhopper/1.0/sprint/picker?query=&_=1581592861732 HTTP/1.1" 200 1262 130847 "https://localhost/jira/secure/RapidBoard.jspa?rapidView=43317&projectKey=HMS&view=planning.nodetail&selectedIssue=HMS-7980&issueLimit=100" 
xxx.xxx.xxx [14/Feb/2020:08:00:57 +0000] "GET /jira/rest/greenhopper/1.0/sprint/picker?query=s&_=1581667061799 HTTP/1.1" 200 1185 118021 "https://localhost/jira/secure/RapidBoard.jspa?rapidView=88532&selectedIssue=CMBGSAI-10743&quickFilter=129584&quickFilter=142240" 
xxx.xxx.xxx [14/Feb/2020:07:56:12 +0000] "GET /jira/rest/greenhopper/1.0/sprint/picker?query=&_=1581666825794 HTTP/1.1" 200 1255 94268 "https://localhost/jira/browse/GCM-8057" 
xxx.xxx.xxx [14/Feb/2020:07:23:57 +0000] "GET /jira/rest/greenhopper/1.0/sprint/picker?query=&_=1581664229894 HTTP/1.1" 200 1329 101854 "https://localhost/jira/secure/CreateIssue.jspa"

      Workaround

      • Instead of using the basic JQL search function that allows for the dropdown, switch to the "Advanced" JQL view. When typing out the sprint name, Jira will provide a relevant list of suggestions.

        1. a.png
          a.png
          245 kB
        2. Screen Shot 2020-02-19 at 4.11.39 pm.png
          Screen Shot 2020-02-19 at 4.11.39 pm.png
          671 kB

            [JSWSERVER-20445] Sprint picker field causing too much CPU

            Dibyandu Roy added a comment -

            @szarazinski would you please confirm if there any bug fix happened for this issue resolution or shall we consider provided workaround as the fix?

            Dibyandu Roy added a comment - @szarazinski would you please confirm if there any bug fix happened for this issue resolution or shall we consider provided workaround as the fix?

            Michael Tse added a comment -

            Will the fix be ported to LTS 8.13.x?

            Michael Tse added a comment - Will the fix be ported to LTS 8.13.x?

            shrivatsaa (Inactive) added a comment -

            https://getsupport.atlassian.com/browse/GHS-206868
            https://getsupport.atlassian.com/browse/PS-66466

            This can also cause timeouts and make the basic search unusable.

            shrivatsaa (Inactive) added a comment - https://getsupport.atlassian.com/browse/GHS-206868 https://getsupport.atlassian.com/browse/PS-66466 This can also cause timeouts and make the basic search unusable.

            Utkarsh Agarwal added a comment -

            The new workaround:

            • Instead of using the basic JQL search function that allows for the dropdown, switch to the "Advanced" JQL view. When typing out the sprint name, Jira will provide a relevant list of suggestions.

            Is not at all practical. This bug is specifically affecting new user accounts, we have a user base of more than 33k users and asking every new user to use advance jql functionality is not a practical solution. Moreover, not all users have the skills to work on the advance jql and prefer to use basic jql. 

            Utkarsh Agarwal added a comment - The new workaround: Instead of using the basic JQL search function that allows for the dropdown, switch to the "Advanced" JQL view. When typing out the sprint name, Jira will provide a relevant list of suggestions. Is not at all practical. This bug is specifically affecting new user accounts, we have a user base of more than 33k users and asking every new user to use advance jql functionality is not a practical solution. Moreover, not all users have the skills to work on the advance jql and prefer to use basic jql. 

            David Yu added a comment -

            As others have mentioned, I tested the feature flag for two days and received enough complaints to disable the flag. Hoping for a better fix. Thanks.

            David Yu added a comment - As others have mentioned, I tested the feature flag for two days and received enough complaints to disable the flag. Hoping for a better fix. Thanks.

            David Yu added a comment - - edited

            I noticed today that a user trigger a few heavy CPU cpikes when all they did was do a JQL search that utilized a sprint in the search. They didn't even have to use the Sprint Picker at all...just performing the search did it. I was able to reproduce...here's the stacktrace in case it's helpful:

                    at com.atlassian.greenhopper.service.sprint.SprintPermissionServiceImpl.getIssueCountForSprint(SprintPermissionServiceImpl.java:250)
        at com.atlassian.greenhopper.service.sprint.SprintPermissionServiceImpl.canViewSprint(SprintPermissionServiceImpl.java:90)
        at com.atlassian.greenhopper.customfield.sprint.SprintResolver.checkSprintAccessible(SprintResolver.java:31)
        at com.atlassian.greenhopper.customfield.sprint.SprintResolver$SprintAccessible.apply(SprintResolver.java:139)
        at com.atlassian.greenhopper.customfield.sprint.SprintResolver$SprintAccessible.apply(SprintResolver.java:125)
        at com.google.common.base.Predicates$AndPredicate.apply(Predicates.java:353)
        at com.google.common.collect.Iterators$5.computeNext(Iterators.java:639)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
        at com.google.common.collect.Iterators.addAll(Iterators.java:357)
        at com.google.common.collect.Lists.newArrayList(Lists.java:147)
        at com.google.common.collect.Lists.newArrayList(Lists.java:133)
        at com.atlassian.greenhopper.customfield.sprint.SprintResolver.findSprintByPredicate(SprintResolver.java:78)
        at com.atlassian.greenhopper.customfield.sprint.SprintResolver.findSprintByPredicate(SprintResolver.java:97)
        at com.atlassian.greenhopper.service.sprint.SprintServiceImpl.findSprintsByName(SprintServiceImpl.java:159)
        at com.atlassian.greenhopper.web.rapid.sprint.SprintHelper.findSprintPickerAllMatches(SprintHelper.java:291)
        at com.atlassian.greenhopper.customfield.sprint.SprintSearchRenderer.getEditHtml(SprintSearchRenderer.java:59)
        at com.atlassian.jira.components.query.DefaultSearcherService.getEditHtml(DefaultSearcherService.java:422)
        at com.atlassian.jira.components.query.DefaultSearcherService.getValueResults(DefaultSearcherService.java:237)
        at com.atlassian.jira.components.query.DefaultSearcherService.getSearchResults(DefaultSearcherService.java:169)
        at com.atlassian.jira.components.query.DefaultSearcherService.searchWithJql(DefaultSearcherService.java:164)

            On Jira 8.5.8 without using the Flag workaround.

            David Yu added a comment - - edited I noticed today that a user trigger a few heavy CPU cpikes when all they did was do a JQL search that utilized a sprint in the search. They didn't even have to use the Sprint Picker at all...just performing the search did it. I was able to reproduce...here's the stacktrace in case it's helpful: at com.atlassian.greenhopper.service.sprint.SprintPermissionServiceImpl.getIssueCountForSprint(SprintPermissionServiceImpl.java:250) at com.atlassian.greenhopper.service.sprint.SprintPermissionServiceImpl.canViewSprint(SprintPermissionServiceImpl.java:90) at com.atlassian.greenhopper.customfield.sprint.SprintResolver.checkSprintAccessible(SprintResolver.java:31) at com.atlassian.greenhopper.customfield.sprint.SprintResolver$SprintAccessible.apply(SprintResolver.java:139) at com.atlassian.greenhopper.customfield.sprint.SprintResolver$SprintAccessible.apply(SprintResolver.java:125) at com.google.common.base.Predicates$AndPredicate.apply(Predicates.java:353) at com.google.common.collect.Iterators$5.computeNext(Iterators.java:639) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) at com.google.common.collect.Iterators.addAll(Iterators.java:357) at com.google.common.collect.Lists.newArrayList(Lists.java:147) at com.google.common.collect.Lists.newArrayList(Lists.java:133) at com.atlassian.greenhopper.customfield.sprint.SprintResolver.findSprintByPredicate(SprintResolver.java:78) at com.atlassian.greenhopper.customfield.sprint.SprintResolver.findSprintByPredicate(SprintResolver.java:97) at com.atlassian.greenhopper.service.sprint.SprintServiceImpl.findSprintsByName(SprintServiceImpl.java:159) at com.atlassian.greenhopper.web.rapid.sprint.SprintHelper.findSprintPickerAllMatches(SprintHelper.java:291) at com.atlassian.greenhopper.customfield.sprint.SprintSearchRenderer.getEditHtml(SprintSearchRenderer.java:59) at com.atlassian.jira.components.query.DefaultSearcherService.getEditHtml(DefaultSearcherService.java:422) at com.atlassian.jira.components.query.DefaultSearcherService.getValueResults(DefaultSearcherService.java:237) at com.atlassian.jira.components.query.DefaultSearcherService.getSearchResults(DefaultSearcherService.java:169) at com.atlassian.jira.components.query.DefaultSearcherService.searchWithJql(DefaultSearcherService.java:164) On Jira 8.5.8 without using the Flag workaround.

            Utkarsh Agarwal added a comment - - edited

            The workaround seems to create another issue:

            Starting from version 8.5.5, 8.8.1, there is a new workaround available (see JSWSERVER-20494).
            It is activated by using this feature flag:
            com.atlassian.jira.agile.darkfeature.sprint.picker.allsprints.suggestion.disabled
            As soon as this flag is added, "All sprints" suggestion will not be calculated/shown.

            The Sprint picker field does not seems to populate Sprints at all<even if the Sprint name are searched with exact name & case>, users are unable to assign issues to the Sprint(on the create/edit issue screen) if the dark feature is enabled. 

            We specifically updated Jira to version 8.5.5 to avoid the long running threads caused due to this bug, however this has made Jira not usable.

            Utkarsh Agarwal added a comment - - edited The workaround seems to create another issue: Starting from version 8.5.5, 8.8.1, there is a new workaround available (see  JSWSERVER-20494 ). It is activated by using this feature flag: com.atlassian.jira.agile.darkfeature.sprint.picker.allsprints.suggestion.disabled As soon as this flag is added, "All sprints" suggestion will not be calculated/shown. The Sprint picker field does not seems to populate Sprints at all<even if the Sprint name are searched with exact name & case>, users are unable to assign issues to the Sprint(on the create/edit issue screen) if the dark feature is enabled.  We specifically updated Jira to version 8.5.5 to avoid the long running threads caused due to this bug, however this has made Jira not usable.

            Matt Doar added a comment - - edited

            Doesn't see to affect us using Basic or Advanced JQL with a user who has not logged in before. 5.5M issues, 40K sprints, Jira 8.5.4

            Matt Doar added a comment - - edited Doesn't see to affect us using Basic or Advanced JQL with a user who has not logged in before. 5.5M issues, 40K sprints, Jira 8.5.4

            Stanislav Natkovskyi (Inactive) added a comment -

            gonchik, I'm not sure if making this workaround an easily accessible option is a good idea, as I'm afraid this may lower the motivation to resolve the root cause of this issue

            Stanislav Natkovskyi (Inactive) added a comment - gonchik , I'm not sure if making this workaround an easily accessible option is a good idea, as I'm afraid this may lower the motivation to resolve the root cause of this issue

            Gonchik Tsymzhitov added a comment -

            snatkovskyi a big thank you for the workaround. Any plans to make as option ?

            Gonchik Tsymzhitov added a comment - snatkovskyi a big thank you for the workaround. Any plans to make as option ?

              szarazinski Sławomir Zaraziński
              vkharisma vkharisma
              Affected customers:
              38 This affects my team
              Watchers:
              54 Start watching this issue

                Created:
                Updated:
                Resolved: