Details
High Description
Issue Summary
Jira webhooks suddenly stop working, after an error happens in the httpcore component in the org.apache.httpcomponents library.
The error is documented in the bug https://issues.apache.org/jira/browse/HTTPCORE-695:
2021-10-12 00:22:19,326-0400 httpclient-io:thread-1 ERROR anonymous [o.a.h.i.nio.client.InternalHttpAsyncClient] I/O reactor terminated abnormally org.apache.http.nio.reactor.IOReactorException: I/O dispatch worker terminated abnormally at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:356) at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221) at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.nio.channels.CancelledKeyException at java.base/sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:71) at java.base/sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:96) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processPendingInterestOps(AbstractIOReactor.java:459) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:298) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
When it happens, the I/O dispatcher # threads with # from 1-10 are ended, and as a consequence the webhooks stop working.
This broken state can be verified by inspecting thread dumps or running the command below:
JIRA_PID=`ps aux | grep -i jira | grep -i [j]ava | awk -F '[ ]*' '{print $2}'`; jstack -l $JIRA_PID |grep "I/O dispatcher"
There are no errors in the logs, but we simply can't find the I/O dispatcher thread logs as shown here with the appropriate debug settings.
Steps to Reproduce
Can't reproduce.
Expected Results
Webhooks work normally.
Actual Results
Apparently at random, the webhooks stop working.
Workaround
Restart the affected nodes.