Uploaded image for project: 'Jira Server and Data Center'
  1. Jira Server and Data Center
  2. JRASERVER-70780

Webhooks intermittently stop working with Java 11

    XMLWordPrintable

    Details

      Description

      Solution

      This problem is caused by a bug in the Java runtime: JDK-8214418 HttpClient falls in running with 100% cpu usage after an error signalled on channel (Backported as JDK-8241054)

      The fix has been verified to be available in AdoptOpenJDK 11.0.8 - https://github.com/AdoptOpenJDK/openjdk-jdk11u/commit/8d1b63a4db2c6348a97b3cf45bd4d2caa7cad6b5

      Issue Summary

      Similar to BSERV-12131, the HTTP Client will stop working on occasion whenever using Java 11.

      Steps to Reproduce

      1. Configure Jira to run on Java 11
      2. Register and configure a webhook to the application.
      3. Execute the webhook.

      Expected Results

      The webhook will trigger successfully.

      Actual Results

      When using Java 11, webhooks will work briefly and eventually fail. Thread dumps will show these webhook publishers in a BLOCKED or WAITING state:

      "Web-Hook-Publisher-0" daemon prio=5 tid=0x0000000000000127 nid=0 waiting on condition 
         java.lang.Thread.State: WAITING (parking)
        at java.base/jdk.internal.misc.Unsafe.park(Native Method)
        - parking to wait for <0x00000000a4586577> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
         owned by Web-Hook-Publisher-1 id=0x000000000000012f
        at java.base/java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source)
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Unknown Source)
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source)
        at java.base/java.util.concurrent.locks.ReentrantLock.lock(Unknown Source)
        at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:278)
        at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.java:295)
        at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377)
        at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129)
        at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141)
        at com.atlassian.httpclient.apache.httpcomponents.BoundedHttpAsyncClient.execute(BoundedHttpAsyncClient.java:49)
        at org.apache.http.impl.client.cache.CachingHttpAsyncClient.callBackend(CachingHttpAsyncClient.java:691)
        at org.apache.http.impl.client.cache.CachingHttpAsyncClient.execute(CachingHttpAsyncClient.java:323)
        at org.apache.http.impl.client.cache.CachingHttpAsyncClient.execute(CachingHttpAsyncClient.java:281)
        at com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient.execute(SettableFuturePromiseHttpPromiseAsyncClient.java:34)
        at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.doExecute(ApacheAsyncHttpClient.java:339)
        at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.execute(ApacheAsyncHttpClient.java:292)
        at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.execute(DefaultRequest.java:258)
        at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.post(DefaultRequest.java:226)
        at com.atlassian.webhooks.plugin.PublishTaskFactoryImpl$PublishTaskImpl.run(PublishTaskFactoryImpl.java:119)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
      

      This issue is suspected to be caused by the following JDK bug:

      Solution

      Upgrade AdoptOpenJDK to version 11.0.8 which has this issue solved.

      Workaround

      1. Run Jira Server using Java 8 or;
      2. The issue is observed with Java 11 that uses TLS v1.3. TLS v1.2 is unaffected by this. It's possible to force Java 11 to use TLS v1.2 by adding the following JVM parameter: -Dhttps.protocols=TLSv1.2 to your startup properties.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              jsanchez2@atlassian.com Jeremy S
              Votes:
              8 Vote for this issue
              Watchers:
              22 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: