Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-12131

Webhooks intermittently stop working with Java 11 JRE

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Answered
    • Icon: Highest Highest
    • None
    • 6.0.0
    • Webhooks

      Solution

      This problem is caused by a bug in the Java runtime: JDK-8214418 HttpClient falls in running with 100% cpu usage after an error signalled on channel (Backported as JDK-8241054)

      The fix has been verified to be available in AdoptOpenJDK 11.0.8 - https://github.com/AdoptOpenJDK/openjdk-jdk11u/commit/8d1b63a4db2c6348a97b3cf45bd4d2caa7cad6b5

      Webhooks are likely impacted by the same problem as is described here: https://jira.atlassian.com/browse/BSERV-11796
      In fixing BSERV-11796, all that was done was to disable TLSv1.3 in the HTTP Client, however that client instance is specific to the repository importer.

      The above is suspected to relate to the following JDK bug:

      • JDK-8214418 HttpClient falls in running with 100% cpu usage after an error signalled on channel

      Workaround

      1) Run Bitbucket Server with Java 8; or
      2) (Not yet verified) The issue is observed with Java 11 that uses TLS v1.3. It is likely that TLS v1.2 works fine and will not cause this issue. This can be done by setting the following JVM parameter: -Dhttps.protocols=TLSv1.2 while starting bitbucket server up.

          Form Name

            [BSERV-12131] Webhooks intermittently stop working with Java 11 JRE

            Ben Humphreys added a comment - - edited

            I'm happy to be able to report that the Java bug that has been causing this problem has been backported from OpenJDK Java 12 to OpenJDK Java 11. It looks like it will be released in AdoptOpenJDK 11.0.8 which is likely to be released July 2020

            https://github.com/AdoptOpenJDK/openjdk-jdk11u/commit/8d1b63a4db2c6348a97b3cf45bd4d2caa7cad6b5

            $ git tag --contains 8d1b63a4db2c6348a97b3cf45bd4d2caa7cad6b5
jdk-11.0.8+1
jdk-11.0.8+1_adopt
jdk-11.0.8+2
jdk-11.0.8+2_adopt
jdk-11.0.8+3
jdk-11.0.8+3_adopt
jdk-11.0.8+4
jdk-11.0.8+4_adopt

            Once AdoptOpenJDK 11.0.8 ships we will perform some testing and update this bug report. I'm fairly confident this will fix the problem, as some time ago I backported the change from Java 12 and tested. I was unable to reproduce the problem after the fix was applied.

            Ben Humphreys added a comment - - edited I'm happy to be able to report that the Java bug that has been causing this problem has been backported from OpenJDK Java 12 to OpenJDK Java 11. It looks like it will be released in AdoptOpenJDK 11.0.8 which is likely to be released July 2020 https://github.com/AdoptOpenJDK/openjdk-jdk11u/commit/8d1b63a4db2c6348a97b3cf45bd4d2caa7cad6b5 $ git tag --contains 8d1b63a4db2c6348a97b3cf45bd4d2caa7cad6b5 jdk-11.0.8+1 jdk-11.0.8+1_adopt jdk-11.0.8+2 jdk-11.0.8+2_adopt jdk-11.0.8+3 jdk-11.0.8+3_adopt jdk-11.0.8+4 jdk-11.0.8+4_adopt Once AdoptOpenJDK 11.0.8 ships we will perform some testing and update this bug report. I'm fairly confident this will fix the problem, as some time ago I backported the change from Java 12 and tested. I was unable to reproduce the problem after the fix was applied.

              behumphreys Ben Humphreys
              behumphreys Ben Humphreys
              Affected customers:
              6 This affects my team
              Watchers:
              25 Start watching this issue

                Created:
                Updated:
                Resolved: