Fetching and pushing via HTTP(S) can hang

Issue Summary

When fetching (not cloning; fetches only, where the local repository already has some refs) or pushing via HTTP(S), the git http-backend process on the server can exhibit odd behaviors. Some observed behaviors include:

• Hanging at high CPU usage
• Unexpected termination

This only affects instances which are using an SSL connector in Tomcat. Additionally, it only impacts Bitbucket Server 7.1.0, specifically. Bitbucket Server 7.0.x releases, and Bitbucket Server 7.1.1 and newer will not be affected.

Steps to Reproduce

There are no specific steps to reliably reproduce this behavior, at least so far. Whether or not it happens depends on several environmental factors. The triggers are either git fetch or git push using an HTTP(S) remote, but simply doing those is not guaranteed to cause an issue consistently.

Expected Results

Fetching and pushing via HTTP(S) should work stably.

Actual Results

Sporadic failures occur, with a range of symptoms, including:

• "Hung" git http-backend processes, which sometimes spin at high CPU usage
• Exceptions in the log file (see below)
• Running out of "scm-hosting" tickets

For some environments the failures appear to be more consistent than for others, and many environments don't appear to have any issues at all.

Exceptions observed include the following:

2020-04-02 10:54:13,784 WARN  [https-jsse-nio-8443-exec-3] jdoe @117G4D4x585x253x0 1.1.1.1 "POST /scm/KEY/slug.git/git-receive-pack HTTP/1.1" c.a.s.i.s.g.p.http.CgiStdoutHandler [C:\Program Files\Git\cmd\git.exe http-backend] has completed (2147483647), but stdout was never closed

2020-03-30 20:10:46,072 ERROR [https-jsse-nio-7990-exec-3] jdoe @1C78VOBx1206x3x0 1.1.1.1 "POST /scm/KEY/slug.git/git-receive-pack HTTP/1.1" c.a.s.i.s.g.p.h.GitSmartExitHandler KEY/slug[1]: Write request from 2.2.2.2 failed: com.atlassian.utils.process.ProcessException: Non-zero exit code: 1

However, this failure can happen with no exceptions on the server at all.

Clients may observe errors like the following:

$ git push https://bitbucket.example.com:8443/scm/KEY/slug.git
Enumerating objects: 1894301, done.
Counting objects: 100% (1894301/1894301), done.
Delta compression using up to 16 threads
Compressing objects: 100% (698045/698045), done.
error: RPC failed; curl 55 SSL_write() returned SYSCALL, errno = 32
fatal: the remote end hung up unexpectedly
Writing objects: 100% (1894301/1894301), 287.72 MiB | 65.37 MiB/s, done.
Total 1894301 (delta 878225), reused 1820448 (delta 810526)
fatal: the remote end hung up unexpectedly
Everything up-to-date

$ git push https://bitbucket.example.com:8443/scm/KEY/slug.git
Enumerating objects: 935, done.
Counting objects: 100% (935/935), done.
Delta compression using up to 4 threads
Compressing objects: 100% (374/374), done.
fatal: the remote end hung up unexpectedlyB | 34.08 MiB/s
Writing objects: 100% (929/929), 672.39 MiB | 30.69 MiB/s, done.
Total 929 (delta 613), reused 858 (delta 555)
fatal: the remote end hung up unexpectedly
Everything up-to-date

The error can vary from these; these are just some examples of the failures that have been observed on the client side.

Workaround

For organizations that require SSL on Tomcat, and require developers/CI/etc. to use HTTPS for Git hosting, there are no workarounds for this issue. The issue can be avoided by:

• Switching to SSH, or
• Introducing a load balancer or reverse proxy in front of Bitbucket Server to terminate SSL

(Those workarounds may not be readily possible for all organizations.)

Warning

Do not attempt to downgrade from Bitbucket Server 7.1.0 to 7.0.x without restoring from a backup. There are database schema changes in 7.1.0 that will break 7.0.x.