Uploaded image for project: 'Jira Software Data Center'
  1. Jira Software Data Center
  2. JSWSERVER-20336

Searching and Mentioning users may cause performance issues and high CPU load

      Issue Summary

      Certain functionalities in Jira, such as trying to mention users or otherwise typing usernames into user fields, allow Jira to return user suggestions as usernames are being typed out.

      As the searching user types, Jira will use a thread to perform a search against what they've typed and return possible results back to the user. In instances with many users, directories, or nested groups, these searches may be computationally expensive and may run for several seconds in order to return results.

      It is possible for the searching user to continue typing and end up making multiple user search requests before the previous requests ever complete and return. Jira will not cancel or otherwise terminate previously ongoing search threads in this case.

      The instance may experience high CPU load and thread utilization if too many of these threads are working on these searches concurrently.

      Steps to Reproduce

      1. Have many users, groups/nested groups, and directories.
      2. Attempt to search for a user (i.e. by mentioning or perhaps trying to assign an issue), by typing a character out at a pace of 2-3 characters/second.

      Expected Results

      Suggested users will be returned, and no other behaviors (high CPU/thread usage) are observed.

      Actual Results

      Suggested users may be returned, but CPU and thread usage run high. Excessive usage may cause the system to become unresponsive.

      Thread Dumps show many threads like the following, which showcase a user mention being searched against nested groups in this particular example:

      "http-nio-8080-exec-20 url:/rest/internal/2/user/mention/search username:" #49 daemon prio=5 os_prio=0 tid=0x00007f8210b43000 nid=0x5d85 runnable [0x00007f810e1e1000]
         java.lang.Thread.State: RUNNABLE
      	...
      	at com.atlassian.crowd.search.EntityDescriptor.toString(EntityDescriptor.java:103)
      	at java.lang.String.valueOf(String.java:2994)
      	at java.lang.StringBuilder.append(StringBuilder.java:131)
      	at com.atlassian.crowd.search.query.membership.MembershipQuery.(MembershipQuery.java:60)
      	at com.atlassian.crowd.search.query.membership.GroupMembershipQuery.(GroupMembershipQuery.java:19)
      	at com.atlassian.crowd.search.builder.QueryBuilder.createMembershipQuery(QueryBuilder.java:177)
      	at com.atlassian.crowd.search.builder.QueryBuilder$PartialMembershipQueryWithNameToMatch.returningAtMost(QueryBuilder.java:285)
      	at com.atlassian.crowd.manager.directory.RemoteDirectorySearcher.findNestedGroupMembershipsIncludingGroups(RemoteDirectorySearcher.java:278)
      	at com.atlassian.crowd.manager.directory.RemoteDirectorySearcher.findNestedGroupMembershipsOfUser(RemoteDirectorySearcher.java:300)
      	at com.atlassian.crowd.manager.directory.RemoteDirectorySearcher.getParentDirectoryEntities(RemoteDirectorySearcher.java:203)
      	at com.atlassian.crowd.manager.directory.RemoteDirectorySearcher.getRelatedDirectoryEntities(RemoteDirectorySearcher.java:191)
      	at com.atlassian.crowd.manager.directory.RemoteDirectorySearcher.searchNestedGroupRelationships(RemoteDirectorySearcher.java:170)
      	at com.atlassian.crowd.manager.directory.DirectoryManagerGeneric.searchNestedGroupRelationships(DirectoryManagerGeneric.java:750)
      	at com.atlassian.crowd.manager.application.SingleDirectorySearchStrategy.searchNestedGroupRelationships(SingleDirectorySearchStrategy.java:71)

      Access Logs show individual HTTP requests being made for each character typed:

      /rest/api/latest/user/assignable/search?username=j
      /rest/api/latest/user/assignable/search?username=ji
      /rest/api/latest/user/assignable/search?username=jir
      /rest/api/latest/user/assignable/search?username=jira
      /rest/api/latest/user/assignable/search?username=jiraa

      Workaround

      Reducing the complexity and dataset of users, groups, nested groups, and directories may help reduce the calculation cost and time of these searches.

      Suggested Solution

      Allow administrators to configure the frequency at which the browser may submit search requests.

          Form Name

            [JSWSERVER-20336] Searching and Mentioning users may cause performance issues and high CPU load

            Is this fix included in 8.20.9 or the latest LTS version

            Noni Khutane added a comment - Is this fix included in 8.20.9 or the latest LTS version

            Hi!  ayakovlev@atlassian.com

            Thanks to upgraded to the latest 8.20.5, starting to better

            Gonchik Tsymzhitov added a comment - Hi!  ayakovlev@atlassian.com ,  Thanks to upgraded to the latest 8.20.5, starting to better

            Hey folks 04cd9c419c73, 4555db5a5f8f
            Unfortunately, both 8.13.x/8.14.x are affected.

            Andriy Yakovlev [Atlassian] added a comment - Hey folks 04cd9c419c73 , 4555db5a5f8f Unfortunately, both 8.13.x/8.14.x are affected.

            Gonchik Tsymzhitov added a comment - - edited

            Does is it related to the 8.14.1 release?
            we met with problems for 99к users for the mention rest request

            Gonchik Tsymzhitov added a comment - - edited Does is it related to the 8.14.1 release? we met with problems for 99к users for the mention rest request

            This is present on 8.13.13

            Felix Martineau added a comment - This is present on 8.13.13

            Hey All,

            We are happy to announce that the bug is fixed and it will be available in the next Jira LTS 8.20.0 version.
            Due to the complexity of the fix, it will be too risky to backport it to 8.13.x line, so we encourage our clients to upgrade to the upcoming LTS version.

            Cheers.

            Best regards,
            Andriy | SET

            Andriy Yakovlev [Atlassian] added a comment - Hey All, We are happy to announce that the bug is fixed and it will be available in the next Jira LTS 8.20.0 version. Due to the complexity of the fix, it will be too risky to backport it to 8.13.x line, so we encourage our clients to upgrade to the upcoming LTS version. Cheers. Best regards, Andriy | SET

            How many memberships (rows in cwd_membership table) did the impacted customer have?

            Boris Berenberg - Atlas Authority added a comment - How many memberships (rows in cwd_membership table) did the impacted customer have?

              Unassigned Unassigned
              kliou Kevin Liou
              Affected customers:
              30 This affects my team
              Watchers:
              52 Start watching this issue

                Created:
                Updated:
                Resolved: