Uploaded image for project: 'Jira Cloud'
  1. Jira Cloud
  2. JRACLOUD-61937

Atlassian HTTP client might stop working at high load

    XMLWordPrintable

Details

    Description

      NOTE: This bug report is for JIRA Cloud. Using JIRA Server? See the corresponding bug report.

      Summary

      Atlassian HTTP Client's callbacks' ThreadPoolExecutor doesn't queue callbacks, so it might reject tasks submitted to it during high load (when no threads available right at the moment of submitting a task).
      In such cases it throws java.util.concurrent.RejectedExecutionException from the execute method in a submitting thread.
      Apache HTTP Client doesn't expect that, so it throws org.apache.http.nio.reactor.IOReactorException when catching RejectedExecutionException and changes IOReactor status to STOPPED therefore preventing further requests being sent by this client bean.
      One of the consequences is that WebHooks stop working after it happens.

      Environment

      Any JIRA up to 7.2.0

      Steps to Reproduce

      1. Enable DEBUG logging for com.atlassian.httpclient
      2. Set up an HTTP endpoint (preferably having a timeout of 30+ sec before responding)
      3. Configure a WebHook against it
      4. Make a bulk edit of 1000 issues
      5. Cross the fingers
      6. Repeat bulk edits until you see java.util.concurrent.RejectedExecutionException in the logs.

      Expected Results

      • No java.util.concurrent.RejectedExecutionException thrown
      • No org.apache.http.nio.reactor.IOReactorException thrown
      • All JIRA's outgoing HTTP requests (including webhooks) continue working

      Actual Results

      • Apache HTTP client's reactor status is STOPPED
      • Webhooks (as well as some other functionality) do not work

      Workaround

      That might only decrease the probability of this issue, but not fix it completely:

      • Decrease the number of webhooks
      • Decrease the number of errors when calling webhooks (e.g. get rid of webhooks often throwing TimeoutExceptions, etc.)
      • Better hardware might also help slightly

      Actions that help if the issue actually happened:

      • Disabling/enabling the Atlassian HTTP Client plugin
        OR
      • Restarting JIRA.

      Note:

      • Disabling/enabling the Atlassian HTTP Client Plugin will trigger other plugins to be restarted. Scheduled a downtime to make sure it is not affecting users that using JIRA.

      Attachments

        Issue Links

          Activity

            People

              apawelczyk Artur Pawelczyk (Inactive)
              ibruzgin Ivan Bruzgin (Inactive)
              Votes:
              11 Vote for this issue
              Watchers:
              27 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: