Uploaded image for project: 'Jira Service Management Data Center'
  1. Jira Service Management Data Center
  2. JSDSERVER-6889

Typing query into JSD "Alert user" automation "THEN" action causes constant GC and service instability

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Low
    • None
    • 4.5.4, 3.16.4
    • Automation
    • None

    Description

      Issue Summary

      With a large number of users (in our case, north of 1 million), typing to search for users in the "Alert user" field of a "THEN" automation workflow action causes a spike in CPU utilization, followed by constant garbage collection at a level that renders that node in the cluster unable to respond to any other requests until the service is manually restarted.

      Steps to Reproduce

      1. Navigate to the "Project settings" area of a Service Desk project
      2. Click "Automation" in the sidebar
      3. Either edit an existing rule or create a new one
      4. If necessary (if creating a new rule), add an applicable "WHEN" action (such that adding a "THEN" action is possible)
      5. Add a "THEN" action; select the "Alert user" action
      6. In the resulting text field, type some characters to select a user
      7. Wait for list of users to return; see that the response takes a long time (30+ sec) and may not ever return (receiving error message: "The call to the Jira server did not complete within the timeout period. We are unsure of the result of this operation.")

      Expected Results

      Searching for a user returns results in a reasonable amount of time for the specificity of the query and the service should remain available.

      Actual Results

      1. Each character typed into the field results in a request like the following: /rest/servicedesk/automation/servicedeskmodules/1/then/alertuser/search?usernameOrEmail=b&projectKey=ABC&_=1591399088734
      2. Every time the text field gains/loses focus, the same request is sent again
      3. If the user interprets the long response time as an indication that the user they're searching for doesn't exist, they may choose to clear the field with their backspace key; each press to erase a character also sends a request
      4. While a search for one or two characters, or even a copy/pasted username (a single request) may return successfully (though after a long delay; 30+ sec), a user who types a long username or email address, waits, sees no response, then backspaces their way back through the query can easily generate 20-30 requests in just a few seconds; this activity is enough to cause:
        • High CPU
        • High JVM memory usage
        • Continuous garbage collection
        • Service instability and potential unavailability (if the user attempting these searches bounces to other nodes in the cluster (as a result of nodes becoming unhealthy) and attempts their search again)

      Workaround

      At the load balancer or other proxy in front of Jira, block requests on the path /rest/servicedesk/automation/servicedeskmodules/1/then/alertuser/search to prevent these search requests from reaching Jira. Obviously this has the side-effect of disabling the functionality entirely, but it does prevent users from triggering this issue.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              lgoolsbee Lee Goolsbee
              Votes:
              2 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:

                Backbone Issue Sync