Uploaded image for project: 'Bitbucket Server'
  1. Bitbucket Server
  2. BSERV-12285

Fetching and pushing via HTTP(S) can hang

    XMLWordPrintable

Details

    Description

      Issue Summary

      When fetching (not cloning; fetches only, where the local repository already has some refs) or pushing via HTTP(S), the git http-backend process on the server can exhibit odd behaviors. Some observed behaviors include:

      • Hanging at high CPU usage
      • Unexpected termination

      This only affects instances which are using an SSL connector in Tomcat. Additionally, it only impacts Bitbucket Server 7.1.0, specifically. Bitbucket Server 7.0.x releases, and Bitbucket Server 7.1.1 and newer will not be affected.

      Steps to Reproduce

      There are no specific steps to reliably reproduce this behavior, at least so far. Whether or not it happens depends on several environmental factors. The triggers are either git fetch or git push using an HTTP(S) remote, but simply doing those is not guaranteed to cause an issue consistently.

      Expected Results

      Fetching and pushing via HTTP(S) should work stably.

      Actual Results

      Sporadic failures occur, with a range of symptoms, including:

      • "Hung" git http-backend processes, which sometimes spin at high CPU usage
      • Exceptions in the log file (see below)
      • Running out of "scm-hosting" tickets

      For some environments the failures appear to be more consistent than for others, and many environments don't appear to have any issues at all.

      Exceptions observed include the following:

      2020-04-02 10:54:13,784 WARN  [https-jsse-nio-8443-exec-3] jdoe @117G4D4x585x253x0 1.1.1.1 "POST /scm/KEY/slug.git/git-receive-pack HTTP/1.1" c.a.s.i.s.g.p.http.CgiStdoutHandler [C:\Program Files\Git\cmd\git.exe http-backend] has completed (2147483647), but stdout was never closed
      
      2020-03-30 20:10:46,072 ERROR [https-jsse-nio-7990-exec-3] jdoe @1C78VOBx1206x3x0 1.1.1.1 "POST /scm/KEY/slug.git/git-receive-pack HTTP/1.1" c.a.s.i.s.g.p.h.GitSmartExitHandler KEY/slug[1]: Write request from 2.2.2.2 failed: com.atlassian.utils.process.ProcessException: Non-zero exit code: 1
      

      However, this failure can happen with no exceptions on the server at all.

      Clients may observe errors like the following:

      $ git push https://bitbucket.example.com:8443/scm/KEY/slug.git
      Enumerating objects: 1894301, done.
      Counting objects: 100% (1894301/1894301), done.
      Delta compression using up to 16 threads
      Compressing objects: 100% (698045/698045), done.
      error: RPC failed; curl 55 SSL_write() returned SYSCALL, errno = 32
      fatal: the remote end hung up unexpectedly
      Writing objects: 100% (1894301/1894301), 287.72 MiB | 65.37 MiB/s, done.
      Total 1894301 (delta 878225), reused 1820448 (delta 810526)
      fatal: the remote end hung up unexpectedly
      Everything up-to-date
      
      $ git push https://bitbucket.example.com:8443/scm/KEY/slug.git
      Enumerating objects: 935, done.
      Counting objects: 100% (935/935), done.
      Delta compression using up to 4 threads
      Compressing objects: 100% (374/374), done.
      fatal: the remote end hung up unexpectedlyB | 34.08 MiB/s
      Writing objects: 100% (929/929), 672.39 MiB | 30.69 MiB/s, done.
      Total 929 (delta 613), reused 858 (delta 555)
      fatal: the remote end hung up unexpectedly
      Everything up-to-date
      

      The error can vary from these; these are just some examples of the failures that have been observed on the client side.

      Workaround

      For organizations that require SSL on Tomcat, and require developers/CI/etc. to use HTTPS for Git hosting, there are no workarounds for this issue. The issue can be avoided by:

      • Switching to SSH, or
      • Introducing a load balancer or reverse proxy in front of Bitbucket Server to terminate SSL

      (Those workarounds may not be readily possible for all organizations.)

      Warning

      Do not attempt to downgrade from Bitbucket Server 7.1.0 to 7.0.x without restoring from a backup. There are database schema changes in 7.1.0 that will break 7.0.x.

      Attachments

        Issue Links

          Activity

            People

              bturner Bryan Turner
              bturner Bryan Turner
              Votes:
              1 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: