Uploaded image for project: 'Jira Software Data Center'
  1. Jira Software Data Center
  2. JSWSERVER-20452

Epic Link query causes timeouts

XMLWordPrintable

      Problem Definition

      While Jira 8 has made great strides in performance, there remain a few operations that are regularly 5+ seconds. I'm singling out Epic Link query performance because I'm noticing this is one of our top-ranked common operations. This can cause delays and block stories from being created when the Epic Link is a required field on the create screen.

      Steps to reproduce

      You can experience this when you:

      1. Edit an Issue
      2. fill out the `E_pic Link_` value
        • Jira will generate the XHR request -/rest/greenhopper/1.0/epics

      Suggested Solution

      Epic data loads within 3 seconds.

      Notes

      • Data from the access logs: 
        5.719 /rest/greenhopper/1.0/epics?searchQuery=&projectKey=XXX&maxResults=100&hideDone=true&_=1582316504312
5.830 /rest/greenhopper/1.0/epics?searchQuery=&projectKey=XXX&maxResults=100&hideDone=true&_=1582317208392
5.832 /rest/greenhopper/1.0/epics?searchQuery=&projectKey=XXX&maxResults=100&hideDone=false&_=1582305640419
5.861 /rest/greenhopper/1.0/epics?searchQuery=&projectKey=XXX&maxResults=100&hideDone=true&_=1582318961711
5.916 /rest/greenhopper/1.0/epics?searchQuery=&projectKey=XXX&maxResults=100&hideDone=true&_=1582305972667
5.948 /rest/greenhopper/1.0/epics?searchQuery=&projectKey=XXX&maxResults=100&hideDone=true&_=1582297025081
5.963 /rest/greenhopper/1.0/epics?searchQuery=&projectKey=XXX&maxResults=100&hideDone=false&_=1582311915016
6.012 /rest/greenhopper/1.0/epics?searchQuery=&projectKey=XXX&maxResults=100&hideDone=true&_=1582314026469
6.161 /rest/greenhopper/1.0/epics?searchQuery=&projectKey=XXX&maxResults=100&hideDone=true&_=1582314877488
6.313 /rest/greenhopper/1.0/epics?searchQuery=&projectKey=XXX&maxResults=100&hideDone=true?searchQuery=&projectKey=XXX&maxResults=10&hideDone=true
6.440 /rest/greenhopper/1.0/epics?searchQuery=&projectKey=XXX&maxResults=100&hideDone=true?searchQuery=&projectKey=XXX&maxResults=100&hideDone=true
      • Here are our Epic Statistics. Total Epics: 28K
        • issuetype=Epic and statusCategory = Done : count: 14k
        • issuetype=Epic and statusCategory != Done : count: 14k

      Some sample thread dumps in case they are helpful

      • Snippet1 
        "http-nio-8082-exec-107" #2634 daemon prio=5 os_prio=0 tid=0x00007f5dbca36800 nid=0x54fe runnable [0x00007f5d5cc0b000]
   java.lang.Thread.State: RUNNABLE
	at java.util.ArrayList.indexOf(ArrayList.java:321)
	at java.util.ArrayList.contains(ArrayList.java:304)
	at com.atlassian.greenhopper.service.issue.callback.EpicLabelAndKeyMatchingCallback.lambda$orderEpicByName$5(EpicLabelAndKeyMatchingCallback.java:252)
	at com.atlassian.greenhopper.service.issue.callback.EpicLabelAndKeyMatchingCallback$$Lambda$8850/1016207412.apply(Unknown Source)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
	at com.atlassian.greenhopper.service.issue.callback.EpicLabelAndKeyMatchingCallback.orderEpicByName(EpicLabelAndKeyMatchingCallback.java:253)
	at com.atlassian.greenhopper.service.issue.callback.EpicLabelAndKeyMatchingCallback.getIssues(EpicLabelAndKeyMatchingCallback.java:233)
	at com.atlassian.greenhopper.service.issuelink.EpicPickerServiceImpl.listEpicNames(EpicPickerServiceImpl.java:69)
	at com.atlassian.greenhopper.web.rapid.issue.issuelink.EpicResource$4.call(EpicResource.java:225)
	at com.atlassian.greenhopper.web.rapid.issue.issuelink.EpicResource$4.call(EpicResource.java:218)
	at com.atlassian.greenhopper.web.util.RestCall.response(RestCall.java:42)
	at com.atlassian.greenhopper.web.AbstractResource.createResponse(AbstractResource.java:111)
	at com.atlassian.greenhopper.web.AbstractResource.responseWithoutAccessCheck(AbstractResource.java:105)
..
      • Snippet2 
        "http-nio-8082-exec-107" #2634 daemon prio=5 os_prio=0 tid=0x00007f5dbca36800 nid=0x54fe runnable [0x00007f5d5cc0b000]
   java.lang.Thread.State: RUNNABLE
	at java.util.ArrayList.indexOf(ArrayList.java:321)
	at java.util.ArrayList.contains(ArrayList.java:304)
	at com.atlassian.greenhopper.service.issue.callback.EpicLabelAndKeyMatchingCallback.lambda$orderEpicByName$5(EpicLabelAndKeyMatchingCallback.java:252)
	at com.atlassian.greenhopper.service.issue.callback.EpicLabelAndKeyMatchingCallback$$Lambda$8850/1016207412.apply(Unknown Source)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
	at com.atlassian.greenhopper.service.issue.callback.EpicLabelAndKeyMatchingCallback.orderEpicByName(EpicLabelAndKeyMatchingCallback.java:253)
	at com.atlassian.greenhopper.service.issue.callback.EpicLabelAndKeyMatchingCallback.getIssues(EpicLabelAndKeyMatchingCallback.java:233)
	at com.atlassian.greenhopper.service.issuelink.EpicPickerServiceImpl.listEpicNames(EpicPickerServiceImpl.java:69)
	at com.atlassian.greenhopper.web.rapid.issue.issuelink.EpicResource$4.call(EpicResource.java:225)
	at com.atlassian.greenhopper.web.rapid.issue.issuelink.EpicResource$4.call(EpicResource.java:218)
	at com.atlassian.greenhopper.web.util.RestCall.response(RestCall.java:42)
	at com.atlassian.greenhopper.web.AbstractResource.createResponse(AbstractResource.java:111)
	at com.atlassian.greenhopper.web.AbstractResource.responseWithoutAccessCheck(AbstractResource.java:105)
...

      Code samples

      • com.atlassian.greenhopper.web.rapid.issue.issuelink.EpicResource#listEpics 
        /**
 * Resource for changing epic-issue links.
 */
@Path("epics")
@AnonymousAllowed
@Consumes({MediaType.APPLICATION_JSON})
@Produces({MediaType.APPLICATION_JSON})

...
public Response listEpics(@QueryParam("searchQuery") final String searchQuery, @QueryParam("maxResults") final int maxResults, @QueryParam("projectKey") final String projectKeys, @QueryParam("hideDone") final boolean hideDone, @QueryParam("query") final String query, @QueryParam("filterEpicsByGivenProjects") final boolean filterEpicByGivenProjects)
 ...

                List<EpicNamesResult> epicNamesResults = check(epicPickerService.listEpicNames(user, searchTerm, maxResults, projectKeys, hideDone, filterEpicByGivenProjects));

                FindEpicNamesResponse response = new FindEpicNamesResponse();
                for (EpicNamesResult epicResult : epicNamesResults)
                {
                    EpicNameListModel list = new EpicNameListModel(epicResult.getListDescriptor());
                    for (EpicModel epicName : epicResult.getNames())
                    {
                        String key = epicName.getEpicKey();
                        String label = epicName.getEpicName();
                        label = epicLabelProvider.getEpicLabel(key, label);
                        list.epicNames.add(new EpicNameModel(key, label, epicName.isDone()));
                    }
                    response.epicLists.add(list);
                    response.total = epicResult.getTotal();
                }

                return createOkResponse(response);
            }
        });
    }

        Workaround:

        There is no workaround currently for this issue

          Form Name

            [JSWSERVER-20452] Epic Link query causes timeouts

            Andriy Yakovlev [Atlassian] added a comment -

            606a5dfdb6fa
            Thanks for reporting this, yes the problem is fixed in all versions post 8.5.7, so 8.5.9 should be ok also.

            Regarding your case, there might be an additional problem(s) related to the Epic Link slow loading, can you please open a support case with us, so we can look into this?
            Thanks.

            Andriy Yakovlev [Atlassian] added a comment - 606a5dfdb6fa Thanks for reporting this, yes the problem is fixed in all versions post 8.5.7, so 8.5.9 should be ok also. Regarding your case, there might be an additional problem(s) related to the Epic Link slow loading, can you please open a support case with us, so we can look into this? Thanks.

            Nitin Chhabra added a comment -

            Hello,

            We just upgraded to 8.5.9 and issue seems to be still there.

             

            Should we upgrade to 8.5.7, we thought it should be fixed in all versions post 8.5.7.

             

            Thanks

            Nitin Chhabra

            Nitin Chhabra added a comment - Hello, We just upgraded to 8.5.9 and issue seems to be still there.   Should we upgrade to 8.5.7, we thought it should be fixed in all versions post 8.5.7.   Thanks Nitin Chhabra

            Stanislav Natkovskyi (Inactive) added a comment -

            Hello wtuscano,

            Thanks for contacting Atlassian.
            The fix was applied to 8.5.7, but not 8.5.6.
            I understand that you might wonder why the fix didn't get into the 8.5.6 even though it was released after the fix was applied.
            The answer to this lies in the way the release process is constructed - there is some grace period after the code freeze when the version is not yet released, but the changes are no longer possible.
            So please expect the fix in 8.5.7.
            Hope this answers your question.

            Cheers,
            Stanislav Natkovskyi
            Jira Server Developer

            Stanislav Natkovskyi (Inactive) added a comment - Hello wtuscano , Thanks for contacting Atlassian. The fix was applied to 8.5.7, but not 8.5.6. I understand that you might wonder why the fix didn't get into the 8.5.6 even though it was released after the fix was applied. The answer to this lies in the way the release process is constructed - there is some grace period after the code freeze when the version is not yet released, but the changes are no longer possible. So please expect the fix in 8.5.7. Hope this answers your question. Cheers, Stanislav Natkovskyi Jira Server Developer

            Wesley Tuscano added a comment -

            Hello Atlassian,

            I noticed that this issue is closed but our fix version is 8.5.7 which is not released yet.  Was this implemented as part of 8.5.6 which was released on July 19, 2020?

            Wesley Tuscano added a comment - Hello Atlassian, I noticed that this issue is closed but our fix version is 8.5.7 which is not released yet.  Was this implemented as part of 8.5.6 which was released on July 19, 2020?

            David Yu added a comment -

            🥳 thanks for the backport to Enterprise release! This will be a real productivity gain for users.

            David Yu added a comment - 🥳 thanks for the backport to Enterprise release! This will be a real productivity gain for users.

            Stanislav Natkovskyi (Inactive) added a comment - - edited

            I have investigated this in terms of PSR-432.
            Both JSWSERVER-20452 and JSWSERVER-20272 have the same root cause - wrong selection of data structure that is used to store Epics with status = Done.
            The data structure currently used has O(n ) search complexity. To fix, we need to switch it to the one that has O(1) complexity.
            I will post the fix ASAP.

            Stanislav Natkovskyi (Inactive) added a comment - - edited I have investigated this in terms of PSR-432. Both JSWSERVER-20452 and JSWSERVER-20272 have the same root cause - wrong selection of data structure that is used to store Epics with status = Done. The data structure currently used has O(n ) search complexity. To fix, we need to switch it to the one that has O(1) complexity. I will post the fix ASAP.

            Wesley Tuscano added a comment -

            Hello Atlassian,

            This issue is severely impacting our user base.  We have received complaints of people not being able to create a story since Epic Link is a required field in our instance.  The epic link dropdown never finishes and times out.  We cannot increase the time out value since it would cause other impacts to the application such as allowing users to make longer calls which may cause issues.  It seems when a project has closer to 1 epic, the call is the slowest and may never finish in some cases.

             

            Since the call never finishes, the users try opening additional tabs to create a story which compounds the issue. 

             

            Is there a way to optimize the current query or create a feature (dark feature?) to turn off suggestions only for epic link?  It would be best if this could be back ported to an Enterprise Release which will be easier for us to deploy rather than a major release. 

            Wesley Tuscano added a comment - Hello Atlassian, This issue is severely impacting our user base.  We have received complaints of people not being able to create a story since Epic Link is a required field in our instance.  The epic link dropdown never finishes and times out.  We cannot increase the time out value since it would cause other impacts to the application such as allowing users to make longer calls which may cause issues.  It seems when a project has closer to 1 epic, the call is the slowest and may never finish in some cases.   Since the call never finishes, the users try opening additional tabs to create a story which compounds the issue.    Is there a way to optimize the current query or create a feature (dark feature?) to turn off suggestions only for epic link?  It would be best if this could be back ported to an Enterprise Release which will be easier for us to deploy rather than a major release. 

              snatkovskyi Stanislav Natkovskyi (Inactive)
              a38518e05741 David Yu
              Affected customers:
              25 This affects my team
              Watchers:
              37 Start watching this issue

                Created:
                Updated:
                Resolved: