Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-69446

Removing actor from project role can make Jira unresponsive

XMLWordPrintable

      Summary

      In specific situations with few roles defined in the instance and large amount of users assigned to them, removing actor from a specific role can take several seconds causing all incoming requests that require permission validation to hang.

      Steps to reproduce

      1. Set up a vanilla Jira instance
      2. Populate projectroleactor table to get ~100k results for the following query 
        SELECT ID, PID, PROJECTROLEID, ROLETYPE, ROLETYPEPARAMETER FROM projectroleactor WHERE PROJECTROLEID='10100'

        e.g. by:

        1. Generating ~100k users and assigning them to a single role (e.g. "Developers") in a single project, or
        2. Generating any amount of users, assigned to a single role in multiple projects so that users*projects=~100k
      2. Remove a user from role

      Expected result

      Action completes successfully and lasts up to 100ms

      Actual result

      The action is executed for several seconds which in total includes:

      1. Query to remove user from projectroleactor table: 
        DELETE FROM projectroleactor WHERE ID IN ('?')

        executed for several seconds.

      1. Removing entries for specified role from projectRoleActorCache
      2. Loading all users for specified role into cache by executing:
      SELECT ID, PID, PROJECTROLEID, ROLETYPE, ROLETYPEPARAMETER FROM projectroleactor WHERE PROJECTROLEID='?

      which lasts for additional several seconds.

      If the request is triggered several times, this can lead to minutes of system unresponsiveness as described in: PS-40349

      Workaround

      1. Add index on PROJECTROLEID column in projectroleactor table
      2. Limit number of actors assigned to a given project role by:
        1. Archiving unused projects
        2. Splitting roles - e.g distributing users from large "Developers" roles as in example above into "Developers", "Testers", "IT Ops" etc.
      3. Threshold 
        DELETE /jira02/rest/api/2/project/?/role/

        requests or temporarily block them

      4. Do not trigger removing role actors in peak hours and do not trigger removing same actor

          Form Name

            [JRASERVER-69446] Removing actor from project role can make Jira unresponsive

            Kamil Kolonko added a comment -

            mdoar2, the example listed in the issue description is for technical purposes of reproducing the problem. Please mind that managing 100,000 users with a single project role is not the only possible option to be affected by the problem. This is clarified in the issue description.

            Now, to answer your question precisely, I can't see a valid real-life use case of adding 100k users to a single project role.

            Regards
            Kamil

            Kamil Kolonko added a comment - mdoar2 , the example listed in the issue description is for technical purposes of reproducing the problem. Please mind that managing 100,000 users with a single project role is not the only possible option to be affected by the problem. This is clarified in the issue description. Now, to answer your question precisely, I can't see a valid real-life use case of adding 100k users to a single project role. Regards Kamil

            Matt Doar added a comment -

            Thank you for the increased detail, but please read this: why would a customer want to add 100,000 users to a project role?
            If you have no real-life use case then this is interesting to know but much less likely to ever occur.

            Matt Doar added a comment - Thank you for the increased detail, but please read this: why would a customer want to add 100,000 users to a project role? If you have no real-life use case then this is interesting to know but much less likely to ever occur.

            Kamil Kolonko added a comment -

            Hi mdoar2,

            I've added more detail to the "Steps to reproduce" section to better visualize the problem. The important number here is the result size of following query:

            SELECT ID, PID, PROJECTROLEID, ROLETYPE, ROLETYPEPARAMETER FROM projectroleactor WHERE PROJECTROLEID='?

            This grows with the number of projects and users. In case of synthetic environment where the problem was reproduced 100k generated users were assigned to a single project role to a single project which resulted in 100k query results. In order to get the same result size for the distribution could be 10k users assigned to a single role in 10 projects.

            Regards
            Kamil

            Kamil Kolonko added a comment - Hi mdoar2 , I've added more detail to the "Steps to reproduce" section to better visualize the problem. The important number here is the result size of following query: SELECT ID, PID, PROJECTROLEID, ROLETYPE, ROLETYPEPARAMETER FROM projectroleactor WHERE PROJECTROLEID='? This grows with the number of projects and users. In case of synthetic environment where the problem was reproduced 100k generated users were assigned to a single project role to a single project which resulted in 100k query results. In order to get the same result size for the distribution could be 10k users assigned to a single role in 10 projects. Regards Kamil

            Matt Doar added a comment -

            Sure, all enterprise software should have limits set for all actions. So the vendor can test to those limits and support them.

            But again, why would a customer add 100,000 users to a project role?

            Matt Doar added a comment - Sure, all enterprise software should have limits set for all actions. So the vendor can test to those limits and support them. But again, why would a customer add 100,000 users to a project role?

            Kamil Kolonko added a comment -

            Hi mdoar2,

            as much as we recommend distributing users across roles to best represent organizational structure and easily manage permissions it is possible in Jira to assign unlimited number of users to a single role. We've noticed such configurations at some customer instances which then results in the symptoms described in the issue.

            This bug is to report potential problems that may be caused by that and possible workarounds for the future.

            Regards
            Kamil

            Kamil Kolonko added a comment - Hi mdoar2 , as much as we recommend distributing users across roles to best represent organizational structure and easily manage permissions it is possible in Jira to assign unlimited number of users to a single role. We've noticed such configurations at some customer instances which then results in the symptoms described in the issue. This bug is to report potential problems that may be caused by that and possible workarounds for the future. Regards Kamil

            Matt Doar added a comment -

            Why would I want to add 100,000 users to a project role in the first place?

            Matt Doar added a comment - Why would I want to add 100,000 users to a project role in the first place?

              twieckowski Tomasz Wieckowski (Inactive)
              kkolonko Kamil Kolonko
              Affected customers:
              2 This affects my team
              Watchers:
              19 Start watching this issue

                Created:
                Updated:
                Resolved: