Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-11796

Failure while importing repositories when running on Java 11

      Issue Summary

      Importing repositories from Bitbucket Cloud / Github fails on Bitbucket Server running on Java 11. Additionally, it causes abnormal CPU utilization on the server and queues up additional import requests.

      Environment

      Java 11 (Even the latest version, `11.0.3`)

      $java -version
      openjdk version "11.0.3" 2019-04-16
      OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.3+7)
      OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.3+7, mixed mode)
      

      Steps to Reproduce

      1. Use Java 11 JRE to run bitbucket server (any version >= 6.0)
      2. Try to import a repository from github/bitbucket cloud

      Expected Results

      No errors are found in the server logs and the UI

      Actual Results

      In the server logs,

      2019-06-06 14:38:51,984 WARN  [http-nio-7990-exec-2] admin @1BEE1W4x878x6x0 ovgau 0:0:0:0:0:0:0:1 "POST /rest/importer/latest/external-source/owners HTTP/1.1" c.a.b.i.i.r.ExternalRepositoryResource Error making external request: Request to the external repository source failed with error: 'peer not authenticated'
      

      Attaching the screenshot of what happens on the UI.

      Additionally, subsequent import requests start getting queued on the server and they get timed out on the UI (504- Gateway error). Also, on the server side, cpu utilization goes to crazy high levels and never comes down till the app is restarted.

      Workaround

      1) Run Bitbucket Server with Java 8; or
      2) Run Bitbucket Server with Java 11.0.8 or later; or
      2) The issue is observed with Java 11 that uses TLS v3. Using TLS v2 works fine and doesn't cause this issue. This can be done by setting the following JVM parameter: -Dhttps.protocols=TLSv1.2 while starting bitbucket server up.

      Note

      This might also effect other functionalities of the app that use the HttpClient , most notably webhooks and some third party apps.

          Form Name

            [BSERV-11796] Failure while importing repositories when running on Java 11

            Ben Humphreys added a comment - - edited

            I'm happy to be able to report that the Java bug that has been causing this problem has been backported from OpenJDK Java 12 to OpenJDK Java 11. It looks like it will be released in AdoptOpenJDK 11.0.8 which is likely to be released July 2020

            https://github.com/AdoptOpenJDK/openjdk-jdk11u/commit/8d1b63a4db2c6348a97b3cf45bd4d2caa7cad6b5

            $ git tag --contains 8d1b63a4db2c6348a97b3cf45bd4d2caa7cad6b5
            jdk-11.0.8+1
            jdk-11.0.8+1_adopt
            jdk-11.0.8+2
            jdk-11.0.8+2_adopt
            jdk-11.0.8+3
            jdk-11.0.8+3_adopt
            jdk-11.0.8+4
            jdk-11.0.8+4_adopt
            

            Once that fix is available, upgrading to 11.0.8 is another solution for customers unable to upgrade to a version of Bitbucket Server that has the workaround applied. It also means we will be able to consider reverting the in-product workaround shipped here; which was obviously to disable TLSv1.3 for the repository importer.

            Ben Humphreys added a comment - - edited I'm happy to be able to report that the Java bug that has been causing this problem has been backported from OpenJDK Java 12 to OpenJDK Java 11. It looks like it will be released in AdoptOpenJDK 11.0.8 which is likely to be released July 2020 https://github.com/AdoptOpenJDK/openjdk-jdk11u/commit/8d1b63a4db2c6348a97b3cf45bd4d2caa7cad6b5 $ git tag --contains 8d1b63a4db2c6348a97b3cf45bd4d2caa7cad6b5 jdk-11.0.8+1 jdk-11.0.8+1_adopt jdk-11.0.8+2 jdk-11.0.8+2_adopt jdk-11.0.8+3 jdk-11.0.8+3_adopt jdk-11.0.8+4 jdk-11.0.8+4_adopt Once that fix is available, upgrading to 11.0.8 is another solution for customers unable to upgrade to a version of Bitbucket Server that has the workaround applied. It also means we will be able to consider reverting the in-product workaround shipped here; which was obviously to disable TLSv1.3 for the repository importer.

            What was the fix and does this same fix apply to BSERV-12180 and BSERV-12131?

            Jonathan Hult added a comment - What was the fix and does this same fix apply to  BSERV-12180  and  BSERV-12131 ?

              khughes@atlassian.com Kristy
              csubraveti Chandravadan
              Affected customers:
              0 This affects my team
              Watchers:
              8 Start watching this issue

                Created:
                Updated:
                Resolved: