[JRASERVER-70169] High CPU usage from IO Dispatcher threads when using Java 11 - Create and track feature requests for Atlassian products.

Type: Bug
Resolution: Answered
Priority: High (View bug fix roadmap)
Fix Version/s: None
Affects Version/s: 8.4.2
Component/s: Environment - Java
Labels:

Introduced in Version:
8.04
Support reference count:
13
Symptom Severity:
Severity 3 - Minor
UIS:
1
Bug Fix Policy:
View Atlassian Server bug fix policy

Solution

This problem is caused by a bug in the Java runtime: JDK-8214418 HttpClient falls in running with 100% cpu usage after an error signalled on channel (Backported as JDK-8241054)

The fix has been verified to be available in AdoptOpenJDK 11.0.8 - https://github.com/AdoptOpenJDK/openjdk-jdk11u/commit/8d1b63a4db2c6348a97b3cf45bd4d2caa7cad6b5

Issue Summary

High CPU usage is noticed on I/O Dispatcher threads.
Jira is up on Java 11.

Steps to Reproduce

No particular steps to reproduce, issue happens randomly even when Jira is not under load.

Expected Results

I/O Dispatcher threads shouldn't be using lots of CPU.

Actual Results

At least One I/O Dispatcher thread is using up 99% CPU.
You can see from the below Thread dump snapshot that the I/O Dispatcher is stuck in runnable and is using a lot of CPU time:

"I/O dispatcher 12" #94 daemon prio=5 os_prio=0 cpu=3439222.56ms elapsed=5220.64s tid=0x0000ff0e98033000 nid=0x3cf5 runnable  [0x0000ff0e98033000]
   java.lang.Thread.State: RUNNABLE
        at sun.security.ssl.SSLEngineImpl.wrap(java.base@11.0.4/SSLEngineImpl.java:136)
        - locked <0x00000006dfc4cb18> (a sun.security.ssl.SSLEngineImpl)
        at sun.security.ssl.SSLEngineImpl.wrap(java.base@11.0.4/SSLEngineImpl.java:116)
        - locked <0x00000006dfc4cb18> (a sun.security.ssl.SSLEngineImpl)
        at javax.net.ssl.SSLEngine.wrap(java.base@11.0.4/SSLEngine.java:479)
        at org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(SSLIOSession.java:263)
        at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:301)
        at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:503)
        - locked <0x00000006dfc4c5a8> (a org.apache.http.nio.reactor.ssl.SSLIOSession)
        at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
        at java.lang.Thread.run(java.base@11.0.4/Thread.java:834)

   Locked ownable synchronizers:
        - None

The CPU data also shows the same thread is using more than 90% of CPU:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                     
15605 tcjira    20   0 11.8g 5.2g  68m R 99.2 44.5  58:02.42 I/O dispatcher                                              
15321 tcjira    20   0 11.8g 5.2g  68m S  1.9 44.5   0:06.70 VM Periodic Tas

Note:

The problem is related to a bug in the httpcore library that ships with Apache Httpclient used by Jira to communicate with remote services: https://issues.apache.org/jira/browse/HTTPCORE-583.
The bug in httpcore was fixed in 4.4.12.
The bug is triggered when Java 11 is used in combination with a version of httpcore that has the problem.
Jira ships with multiple versions of the httpcore library, main Jira 8.4.2 uses httpcore-4.4.9.jar, while some other bundled plugins use other versions:
- jira-workflow-sharing-plugin-2.2.3.jar-embedded uses httpcore-4.4.1.jar
- atlassian-remote-event-common-plugin-6.1.0.jar-embedded uses httpcore-4.4.10.jar

Solution

Upgrade AdoptOpenJDK to version 11.0.8 which has this issue solved.

Workaround

Downgrade your JDK version from jdk 11 to jdk 1.8, Java 1.8 is not known to cause this problem.
The issue is observed with Java 11 that uses TLS v1.3. TLS v1.2 is unaffected by this. It's possible to force Java 11 to use TLS v1.2 by adding the following JVM parameter: -Dhttps.protocols=TLSv1.2 to your startup properties.

is duplicated by

JRASERVER-70189 When Jira is running against Java 11, TLS 1.3 requests can cause httpclient threads to become blocked

Closed

mentioned in: Page Failed to load; Page Failed to load

relates to: HL-1277 You do not have permission to view this issue; HL-1410 You do not have permission to view this issue

Form Name

Loading...

Type: Bug
Resolution: Answered
Priority: High
Fix Version/s: None
Affects Version/s: 8.4.2
Component/s: Environment - Java
Labels:

Introduced in Version:
8.04
Support reference count:
13
Symptom Severity:
Severity 3 - Minor
UIS:
1
Bug Fix Policy:
View Atlassian Server bug fix policy

Solution

This problem is caused by a bug in the Java runtime: JDK-8214418 HttpClient falls in running with 100% cpu usage after an error signalled on channel (Backported as JDK-8241054)

The fix has been verified to be available in AdoptOpenJDK 11.0.8 - https://github.com/AdoptOpenJDK/openjdk-jdk11u/commit/8d1b63a4db2c6348a97b3cf45bd4d2caa7cad6b5

Issue Summary

High CPU usage is noticed on I/O Dispatcher threads.
Jira is up on Java 11.

Steps to Reproduce

No particular steps to reproduce, issue happens randomly even when Jira is not under load.

Expected Results

I/O Dispatcher threads shouldn't be using lots of CPU.

Actual Results

At least One I/O Dispatcher thread is using up 99% CPU.
You can see from the below Thread dump snapshot that the I/O Dispatcher is stuck in runnable and is using a lot of CPU time:

"I/O dispatcher 12" #94 daemon prio=5 os_prio=0 cpu=3439222.56ms elapsed=5220.64s tid=0x0000ff0e98033000 nid=0x3cf5 runnable  [0x0000ff0e98033000]
   java.lang.Thread.State: RUNNABLE
        at sun.security.ssl.SSLEngineImpl.wrap(java.base@11.0.4/SSLEngineImpl.java:136)
        - locked <0x00000006dfc4cb18> (a sun.security.ssl.SSLEngineImpl)
        at sun.security.ssl.SSLEngineImpl.wrap(java.base@11.0.4/SSLEngineImpl.java:116)
        - locked <0x00000006dfc4cb18> (a sun.security.ssl.SSLEngineImpl)
        at javax.net.ssl.SSLEngine.wrap(java.base@11.0.4/SSLEngine.java:479)
        at org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(SSLIOSession.java:263)
        at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:301)
        at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:503)
        - locked <0x00000006dfc4c5a8> (a org.apache.http.nio.reactor.ssl.SSLIOSession)
        at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
        at java.lang.Thread.run(java.base@11.0.4/Thread.java:834)

   Locked ownable synchronizers:
        - None

The CPU data also shows the same thread is using more than 90% of CPU:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                     
15605 tcjira    20   0 11.8g 5.2g  68m R 99.2 44.5  58:02.42 I/O dispatcher                                              
15321 tcjira    20   0 11.8g 5.2g  68m S  1.9 44.5   0:06.70 VM Periodic Tas

Note:

The problem is related to a bug in the httpcore library that ships with Apache Httpclient used by Jira to communicate with remote services: https://issues.apache.org/jira/browse/HTTPCORE-583.
The bug in httpcore was fixed in 4.4.12.
The bug is triggered when Java 11 is used in combination with a version of httpcore that has the problem.
Jira ships with multiple versions of the httpcore library, main Jira 8.4.2 uses httpcore-4.4.9.jar, while some other bundled plugins use other versions:
- jira-workflow-sharing-plugin-2.2.3.jar-embedded uses httpcore-4.4.1.jar
- atlassian-remote-event-common-plugin-6.1.0.jar-embedded uses httpcore-4.4.10.jar

Solution

Upgrade AdoptOpenJDK to version 11.0.8 which has this issue solved.

Workaround

Downgrade your JDK version from jdk 11 to jdk 1.8, Java 1.8 is not known to cause this problem.
The issue is observed with Java 11 that uses TLS v1.3. TLS v1.2 is unaffected by this. It's possible to force Java 11 to use TLS v1.2 by adding the following JVM parameter: -Dhttps.protocols=TLSv1.2 to your startup properties.

is duplicated by

JRASERVER-70189 When Jira is running against Java 11, TLS 1.3 requests can cause httpclient threads to become blocked

Closed

mentioned in: Page Loading...; Page Loading...

relates to: HL-1277 Loading...; HL-1410 Loading...

Assignee:: Artur Gniadzik

Reporter:: Sherif Abdelfattah (Inactive)

Votes:: 3 Vote for this issue

Watchers:: 12 Start watching this issue

Created:: 25/Oct/2019 12:27 PM

Updated:: 29/May/2024 8:26 PM

Resolved:: 04/Sep/2020 2:39 PM

Details

Description

Issue Summary

Steps to Reproduce

Expected Results

Actual Results

Note:

Solution

Workaround

Attachments

Issue Links

Forms

Activity

Details

Description

Issue Summary

Steps to Reproduce

Expected Results

Actual Results

Note:

Solution

Workaround

Attachments

Issue Links

Forms

Activity

People

Dates

People

Dates