Uploaded image for project: 'Jira Server and Data Center'
  1. Jira Server and Data Center
  2. JRASERVER-61937

Atlassian HTTP client might stop working at high load

    XMLWordPrintable

    Details

      Description

      NOTE: This bug report is for JIRA Server. Using JIRA Cloud? See the corresponding bug report.

      Summary

      Atlassian HTTP Client's callbacks' ThreadPoolExecutor doesn't queue callbacks, so it might reject tasks submitted to it during high load (when no threads available right at the moment of submitting a task).
      In such cases it throws java.util.concurrent.RejectedExecutionException from the execute method in a submitting thread.
      Apache HTTP Client doesn't expect that, so it throws org.apache.http.nio.reactor.IOReactorException when catching RejectedExecutionException and changes IOReactor status to STOPPED therefore preventing further requests being sent by this client bean.
      One of the consequences is that WebHooks stop working after it happens.

      Environment

      Any JIRA up to 7.2.0

      Steps to Reproduce

      1. Enable DEBUG logging for com.atlassian.httpclient
      2. Set up an HTTP endpoint (preferably having a timeout of 30+ sec before responding)
      3. Configure a WebHook against it
      4. Make a bulk edit of 1000 issues
      5. Cross the fingers
      6. Repeat bulk edits until you see java.util.concurrent.RejectedExecutionException in the logs.

      Expected Results

      • No java.util.concurrent.RejectedExecutionException thrown
      • No org.apache.http.nio.reactor.IOReactorException thrown
      • All JIRA's outgoing HTTP requests (including webhooks) continue working

      Actual Results

      • Apache HTTP client's reactor status is STOPPED
      • Webhooks (as well as some other functionality) do not work

      Workaround

      That might only decrease the probability of this issue, but not fix it completely:

      • Decrease the number of webhooks
      • Decrease the number of errors when calling webhooks (e.g. get rid of webhooks often throwing TimeoutExceptions, etc.)
      • Better hardware might also help slightly

      Actions that help if the issue actually happened:

      • Disabling/enabling the Atlassian HTTP Client plugin
        OR
      • Restarting JIRA.

      Note:

      • Disabling/enabling the Atlassian HTTP Client Plugin will trigger other plugins to be restarted. Schedule downtime to make sure it is not affecting users that using JIRA.
      • Disabling/enabling the Atlassian HTTP Client Plugin will disable other Jira apps (Jira Software, Jira Service Desk). You need to reenable them manually from Administrations > Add-ons > Manage add-ons after that.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              apawelczyk Artur Pawelczyk
              Reporter:
              ibruzgin Ivan Bruzgin (Inactive)
              Votes:
              11 Vote for this issue
              Watchers:
              29 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: