Summary

Atlassian HTTP Client's callbacks' ThreadPoolExecutor doesn't queue callbacks, so it might reject tasks submitted to it during high load (when no threads available right at the moment of submitting a task).
In such cases it throws java.util.concurrent.RejectedExecutionException from the execute method in a submitting thread.
Apache HTTP Client doesn't expect that, so it throws org.apache.http.nio.reactor.IOReactorException when catching RejectedExecutionException and changes IOReactor status to STOPPED therefore preventing further requests being sent by this client bean.
One of the consequences is that WebHooks stop working after it happens.

Environment

Any JIRA up to 7.2.0

Steps to Reproduce

1. Enable DEBUG logging for com.atlassian.httpclient
2. Set up an HTTP endpoint (preferably having a timeout of 30+ sec before responding)
3. Configure a WebHook against it
4. Make a bulk edit of 1000 issues
5. Cross the fingers
6. Repeat bulk edits until you see java.util.concurrent.RejectedExecutionException in the logs.

Expected Results

• No java.util.concurrent.RejectedExecutionException thrown
• No org.apache.http.nio.reactor.IOReactorException thrown
• All JIRA's outgoing HTTP requests (including webhooks) continue working

Actual Results

• Apache HTTP client's reactor status is STOPPED
• Webhooks (as well as some other functionality) do not work

Workaround

That might only decrease the probability of this issue, but not fix it completely:

• Decrease the number of webhooks
• Decrease the number of errors when calling webhooks (e.g. get rid of webhooks often throwing TimeoutExceptions, etc.)
• Better hardware might also help slightly

Actions that help if the issue actually happened:

• Disabling/enabling the Atlassian HTTP Client plugin
  OR
• Restarting JIRA.

Note:

• Disabling/enabling the Atlassian HTTP Client Plugin will trigger other plugins to be restarted. Schedule downtime to make sure it is not affecting users that using JIRA.
• Disabling/enabling the Atlassian HTTP Client Plugin will disable other Jira apps (Jira Software, Jira Service Desk). You need to reenable them manually from Administrations > Add-ons > Manage add-ons after that.