High CPU usage from IO Dispatcher threads when using Java 11

XMLWordPrintable

    • 9
    • Severity 3 - Minor
    • 1

      Issue Summary

      • High CPU usage is noticed on I/O Dispatcher threads
      • Confluence is using Java 11

      Steps to Reproduce

      No particular steps to reproduce, issue happens randomly even when Confluence is not under load.

      Expected Results

      I/O Dispatcher threads shouldn't be using lots of CPU.

      Actual Results

      At least one I/O Dispatcher thread is using up 99% CPU.
      You can see from the below Thread dump snapshot that the I/O Dispatcher is stuck in runnable and is using a lot of CPU time:

      "I/O dispatcher 12" #94 daemon prio=5 os_prio=0 cpu=3439222.56ms elapsed=5220.64s tid=0x0000ff0e98033000 nid=0x3cf5 runnable  [0x0000ff0e98033000]
   java.lang.Thread.State: RUNNABLE
        at sun.security.ssl.SSLEngineImpl.wrap(java.base@11.0.4/SSLEngineImpl.java:136)
        - locked <0x00000006dfc4cb18> (a sun.security.ssl.SSLEngineImpl)
        at sun.security.ssl.SSLEngineImpl.wrap(java.base@11.0.4/SSLEngineImpl.java:116)
        - locked <0x00000006dfc4cb18> (a sun.security.ssl.SSLEngineImpl)
        at javax.net.ssl.SSLEngine.wrap(java.base@11.0.4/SSLEngine.java:479)
        at org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(SSLIOSession.java:263)
        at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:301)
        at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:503)
        - locked <0x00000006dfc4c5a8> (a org.apache.http.nio.reactor.ssl.SSLIOSession)
        at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
        at java.lang.Thread.run(java.base@11.0.4/Thread.java:834)

   Locked ownable synchronizers:
        - None

      The CPU data also shows the same thread is using more than 90% of CPU:

        PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                     
15605 tcjira    20   0 11.8g 5.2g  68m R 99.2 44.5  58:02.42 I/O dispatcher                                              
15321 tcjira    20   0 11.8g 5.2g  68m S  1.9 44.5   0:06.70 VM Periodic Tas

      Solution

      Upgrade AdoptOpenJDK to version 11.0.8 which has this issue solved.

      Notes

      Further investigation leads to the conclusion that there is a problem with OpenJDK's implementation of TLS 1.3:

      Workaround

      • Downgrade your JDK version from jdk 11 to jdk 1.8, Java 1.8 is not known to cause this problem.
      • The issue is observed with Java 11 that uses TLS v1.3. TLS v1.2 is unaffected by this. By default, Confluence 7.1.0+ ships with this as default JVM flags: 
        -Djdk.tls.server.protocols=TLSv1.1,TLSv1.2 -Djdk.tls.client.protocols=TLSv1.1,TLSv1.2
        • Add these JVM flags back into Confluence if they are missing

            Assignee:
            Unassigned
            Reporter:
            Eric L
            Votes:
            2 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: