Issue Summary

When using an HTTP Access Token with git on repositories that have LFS enabled, the LFS download fails if there is a reverse proxy in front of Bitbucket Server.
This only happens if the -c http.extraHeader command line option is used in git to specify the HTTP Access Token for authentication via a Bearer token.

If Apache HTTP Server is used it fails with a JwtSignatureMismatchException, and the reason is that the client (git) will send two Authorization headers (one with the JWT for LFS download and one with the Bearer HTTP Access Token), and Apache will merge any incoming request headers that appear more than once (separating them by a comma) before sending the request to the backend server.

The Authorization header value received by Bitbucket will look something like this:

JWT eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIyIiwicmVwbyI6InByb2pcL3JlcG8iLCJxc2giOiJlMGJiYTg4YTQxMmUyZDU5N2Q1NGY5YmRlYmVkMmJkNTY3YTg1MGQ4MTk5ZTcyMzJjYzY2NWU2M2JkMGY3OWU1IiwiaXNzIjoiY29tLmF0bGFzc2lhbi5iaXRidWNrZXQuc2VydmVyLmJpdGJ1Y2tldC1naXQtbGZzLXN0b3JhZ2UiLCJjb250ZXh0Ijp7InVzZXIiOnsiaGlnaGVzdFBlcm1pc3Npb24iOiJTWVNfQURNSU4iLCJkaXNwbGF5TmFtZSI6IkNHIiwic2x1ZyI6ImNnIiwidXNlcktleSI6IjIiLCJ1c2VybmFtZSI6ImNnIn19LCJleHAiOjE2NzIzODkxNjMsImlhdCI6MTY3MjMwMjc2M30.Mgd0PDWsxGSfoMgPnQ3L9AqsZ652sWEKs5LXGunsLaQ, Bearer BBDC-NDM5MzAxNTc4MzA1OqEEPqHU4oTScTVkDs/JUzB7ptIf

Bitbucket Server does not correctly handle the concatenated values and interprets the entire header value as the JWT, which is incorrect.

If nginx is used it fails with a similar "Smudge error" because unlike Apache nginx does not merge the headers but discards one of the two headers and only forwards the HTTP Access token instead of the the JWT to Bitbucket Server, therefore authentication fails for the LFS operation.

This is reproducible on Data Center: yes

Steps to Reproduce

1. Use a Bitbucket Server installation behind a reverse proxy server
2. Create an HTTP Access Token
3. Try cloning a repository with LFS data using the command

   git -c http.extraHeader='Authorization: Bearer <token>' clone <repoUrl>
   

   where <token> is the HTTP Access Token generated in step 2 and <repoUrl> is the HTTP (i.e. non-SSH) URL of the repository

Expected Results

The repository is cloned without error

Actual Results

The clone fails with the error(s) described above.

The below exception may be thrown in the atlassian-bitbucket.log file (if Apache HTTP Server is used):

2022-12-30 10:22:35,913 WARN  [http-nio-7990-exec-7] @K84A0Dx622x477x0 192.168.1.254,127.0.0.1 "GET /rest/git-lfs/storage/PROJ/repo/57cc928f8f924606af3674d00a64e6dd9a2fa152fc7f5bd0062632c87261439b HTTP/1.1" c.a.j.i.s.DefaultAuthenticationResultHandler Signature mismatch during JWT authentication, issuer: com.atlassian.bitbucket.server.bitbucket-git-lfs-storage
com.atlassian.jwt.exception.JwtSignatureMismatchException: eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIyIiwicmVwbyI6InByb2pcL3JlcG8iLCJxc2giOiJlMGJiYTw4YTQxMmUyZDU5N2Q1NGY5YmRlYmVkMmJkNTY3YTg1MGQ4MTk5ZTcyMzJjYzY2NWU2M2JkMGY3OWU1IiwiaXNzIjoiY29tLmF0bGFzc2lhbi5iaXRidWNrZXQuc2VydmVyLmJpdGJ1Y2tldC1naXQtbGZzLXN0b3JhZ2UiLCJjb250ZXh0Ijp7InVzZXIiOnsiaGlnaGVzdFBlcm1pc3Npb24iOiJTWVNfQURNSU4iLCJkaXNwbGF5TmFtZSI6IkNHIiwic2x1ZyI6ImNnIiwidXNlcktleSI6IjIiLCJ1c2VybmFtZSI6ImNnIn19LCJleHAiOjE2NzI0ODIxNTUsImlhdCI6MTY3MjM5NTc1NX0.xPVOoEKHDJlL_EwEz9JsmYoS0bDQMLNwtVNr7HhMwKs, Bearer BBDC-NDM5MzAxNTc4MzA1OqEEPqHU4oTScTVkDs/JUzB7ptIf

Workaround

Instead of providing the HTTP Access Token as a Bearer token in the Authorization header provide the token as the password of the user when using git.

If you're unable to specify the token as the password of the user:

• When using Apache HTTP Server you can change the configuration such that the Bearer HTTP Access Token is removed from the headers sent to Bitbucket Server. See "Smudge error" occurs when performing git operations when using token authentication with Bitbucket Server or Data Center for details on how to make these changes.
• For nginx, we have a workaround as specified on the KB article - https://support.atlassian.com/bitbucket-data-center/kb/smudge-error-in-git-lfs-with-bearer-token-on-bitbucket-data-center/
• For other reverse proxies no other workaround is currently available.