Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-9388

Pre-receive hooks and branch permissions reject valid pushes on git 2.11

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: High High
    • 4.13.0
    • 4.11.2, 4.12.0
    • None

      Note: This bug is fixed in Bitbucket Server 4.13 but third-party plugins (particularly plugins that check new commits as they are pushed) may also be impacted by the changes in git 2.11.

      Summary

      When the "Reject Force Push" hook is enabled and the Git version is 2.11.0, all pushes to the repository fail even when the --force option is not used.

      Steps to reproduce

      1. Spin up Bitbucket Server using git v2.11.0
      2. Create a new repository and push an initial commit to it
      3. Turn the "Reject Force Push" pre-receive hook on OR setup branch permissions to prevent writing history on your branch
      4. Create a new commit and attempt to push it to Bitbucket Server

      Expected Results

      Push is successful.

      Actual Results

      Push is blocked by a pre-receive hook (or branch permissions).

      Cause

      From the git 2.11 release notes:

      In order for the receiving end of "git push" to inspect the
      received history and decide to reject the push, the objects sent
      from the sending end need to be made available to the hook and
      the mechanism for the connectivity check, and this was done
      traditionally by storing the objects in the receiving repository
      and letting "git gc" expire them. Instead, store the newly
      received objects in a temporary area, and make them available by
      reusing the alternate object store mechanism to them only while we
      decide if we accept the check, and once we decide, either migrate
      them to the repository or purge them immediately.

      Workaround

      Downgrade to Git 2.10.x or older. Any version after Git 2.11.0 (including future 2.12+ releases) cannot be used with Bitbucket Server.

      Bitbucket Server 4.12 will fail on startup if Git 2.11+ is detected. However, older versions of Bitbucket Server (and Stash) are not aware of the 2.11 issues (since it was released after they were), but will still be broken by them. Administrators are strongly encouraged to only upgrade to Git versions which are explicitly marked supported on our Supported Platforms for the version of the product they're running.

            [BSERV-9388] Pre-receive hooks and branch permissions reject valid pushes on git 2.11

            Bryan Turner (Inactive) added a comment - - edited

            Mark,

            Unfortunately, from looking at its source code, YACC uses JGit to retrieve commits. As I mentioned when I closed this issue, hooks which use JGit instead of Bitbucket Server's APIs do not work with Git 2.11, and cannot be made to work without switching away from JGit. I've raised an issue on the plugin's repository to see if I can work with the add-on developer to get it fixed.

            Until the hook can be updated according, YACC will only work with Git 2.10 or earlier, regardless of the Bitbucket Server version it's deployed into.

            Best regards,
            Bryan Turner
            Atlassian Bitbucket

            Bryan Turner (Inactive) added a comment - - edited Mark, Unfortunately, from looking at its source code , YACC uses JGit to retrieve commits. As I mentioned when I closed this issue, hooks which use JGit instead of Bitbucket Server's APIs do not work with Git 2.11, and cannot be made to work without switching away from JGit. I've raised an issue on the plugin's repository to see if I can work with the add-on developer to get it fixed. Until the hook can be updated according, YACC will only work with Git 2.10 or earlier, regardless of the Bitbucket Server version it's deployed into. Best regards, Bryan Turner Atlassian Bitbucket

            markgillespien added a comment - - edited

            It would be really nice if Bitbucket actually had it's own JIRA enforcement plugin, so we don't need to go third party shopping..  Seems like a really common usage scenario that all commits must have a valid JIRA ticket.

            markgillespien added a comment - - edited It would be really nice if Bitbucket actually had it's own JIRA enforcement plugin, so we don't need to go third party shopping..  Seems like a really common usage scenario that all commits must have a valid JIRA ticket.

            There was more fallout from this for us.  When we want back to Git 1.9 (which was all that was on offer on Ubuntu Trusty), this caused other Bitbucket Server issues

            All out Pull Requests showed.  Bitbucket Server could not create the merge diff for this pull request

             

            We had to emergency compile GIT 2.10 from source to get up and running again.  less than ideal.   My experiences are:

             

            • Don't upgrade to GIT 2.11 if you run plugins with hooks
            • Don't downgrade from GIT 2.x to GIT1.x

             

            As we now have a frozen GIT 2.10, we won't be updating it, until there is some confidence this is fully resolved.

            markgillespien added a comment - There was more fallout from this for us.  When we want back to Git 1.9 (which was all that was on offer on Ubuntu Trusty), this caused other Bitbucket Server issues All out Pull Requests showed.   Bitbucket Server could not create the merge diff for this pull request   We had to emergency compile GIT 2.10 from source to get up and running again.  less than ideal.   My experiences are:   Don't upgrade to GIT 2.11 if you run plugins with hooks Don't downgrade from GIT 2.x to GIT1.x   As we now have a frozen GIT 2.10, we won't be updating it, until there is some confidence this is fully resolved.

            Brent P added a comment -

            Thanks for the note mark.gillespie1 - I'll add a note to the top of the issue that plugins may also be affected by this problem.

            Brent P added a comment - Thanks for the note mark.gillespie1 - I'll add a note to the top of the issue that plugins may also be affected by this problem.

            A word of warning on this.  We got caught out by this.  GIT was upgraded on Ubuntu to 2.11 and suddenly nobody could push   Turns out it was Yet Another Commit Checker   Plugin that was falling over.  Had to emergency downgrade GIT to 1.9x (as that was all that Ubuntu was finding as a downgrade path).

             

            2017-01-24 15:29:56,388 WARN  [threadpool:thread-4] user123@1QRQRBLx929x487433x1 172.20.9.82 "POST /scm/fut/projectname.git/git-receive-pack HTTP/1.1" c.a.s.i.h.r.PreReceiveRepositoryHookAdapter Receive Hook com.isroot.stash.plugin.YaccHook failed.

            markgillespien added a comment - A word of warning on this.  We got caught out by this.  GIT was upgraded on Ubuntu to 2.11 and suddenly nobody could push   Turns out it was  Yet Another Commit Checker    Plugin that was falling over.  Had to emergency downgrade GIT to 1.9x (as that was all that Ubuntu was finding as a downgrade path).   2017-01-24 15:29:56,388 WARN   [threadpool:thread-4] user123@1QRQRBLx929x487433x1 172.20.9.82 "POST /scm/fut/projectname.git/git-receive-pack HTTP/1.1" c.a.s.i.h.r.PreReceiveRepositoryHookAdapter Receive Hook com.isroot.stash.plugin.YaccHook failed.

            Bryan Turner (Inactive) added a comment - - edited

            Git 2.11's quarantining functionality for pre-receive hooks will be supported by the upcoming 4.13 release. We've also extended our internal testing matrix for Git functionality to ensure we pick up on other changes to hook behavior much earlier, should something similar happen again.

            Quarantined pushes mean Git now sets GIT_ALTERNATE_OBJECT_DIRECTORIES and GIT_OBJECT_DIRECTORY while pre-receive hooks are running. Bitbucket Server automatically mirrors this behavior for any commands which are created using GitScmCommandBuilder--which includes all of the Command instances created by all of our *CommandFactory interfaces, which are in turn used by all of the services we export for plugins. This means any PreReceiveHook or PreReceiveRepositoryHook which uses any of the APIs Bitbucket Server exports for plugin developers should work on Git 2.11 without changes.

            Any pre-receive hook which executes git processes without using our GitScmCommandBuilder, or that exports the GIT_OBJECT_DIRECTORY environment variable, will still not work on Git 2.11. Any pre-receive hook which uses JGit to interact with repositories will also not work (Bitbucket Server does not use JGit, or provide it for plugin developers to use, so any plugin using it must be bundling it for itself). Unfortunately, it's not possible for us to fix such hooks on our end–the best solution is for their developers to update them to use GitScmCommandBuilder, or one of our other services.

            Best regards,
            Bryan Turner
            Atlassian Bitbucket

            Bryan Turner (Inactive) added a comment - - edited Git 2.11's quarantining functionality for pre-receive hooks will be supported by the upcoming 4.13 release. We've also extended our internal testing matrix for Git functionality to ensure we pick up on other changes to hook behavior much earlier, should something similar happen again. Quarantined pushes mean Git now sets GIT_ALTERNATE_OBJECT_DIRECTORIES and GIT_OBJECT_DIRECTORY while pre-receive hooks are running. Bitbucket Server automatically mirrors this behavior for any commands which are created using GitScmCommandBuilder --which includes all of the Command instances created by all of our *CommandFactory interfaces, which are in turn used by all of the services we export for plugins. This means any PreReceiveHook or PreReceiveRepositoryHook which uses any of the APIs Bitbucket Server exports for plugin developers should work on Git 2.11 without changes . Any pre-receive hook which executes  git processes without using our  GitScmCommandBuilder , or that exports the GIT_OBJECT_DIRECTORY environment variable, will still not work on Git 2.11 . Any  pre-receive  hook which uses JGit to interact with repositories will also not work (Bitbucket Server does not use JGit, or provide it for plugin developers to use, so any plugin using it must be bundling it for itself). Unfortunately, it's not possible for us to fix such hooks on our end–the best solution is for their developers to update them to use GitScmCommandBuilder , or one of our other services. Best regards, Bryan Turner Atlassian Bitbucket

            Antonio Anzivino added a comment - - edited

            I don't use the BitBucket's default "Reject force push" hook. However, I do use Branch permissions to reject force pushes on certain branches (with the "disallow rewriting history except for...").

            Is this compatible with Git 2.11? Can I live with it until Atlassian releases an update?

            I am currently in the middle of a disaster recovery and cannot install git 2.10 on SLES 11 SP4 at the moment, so I can't bring BB server back to life until I either get git 2.10 or confirmation that BB server can operate with 2.11 and workarounds (if any).

             -- 

            Any ideas?

             

            Self deleted. I didn't read that the bug is activated also by using branch permissions

            Antonio Anzivino added a comment - - edited I don't use the BitBucket's default "Reject force push" hook. However, I do use Branch permissions to reject force pushes on certain branches (with the "disallow rewriting history except for..."). Is this compatible with Git 2.11? Can I live with it until Atlassian releases an update? I am currently in the middle of a disaster recovery and cannot install git 2.10 on SLES 11 SP4 at the moment, so I can't bring BB server back to life until I either get git 2.10 or confirmation that BB server can operate with 2.11 and workarounds (if any).  --  Any ideas?   Self deleted. I didn't read that the bug is activated also by using branch permissions

            The affected server runs Ubuntu 12.04 LTS (it's scheduled to be replaced during the next few months), so the stock git version is a bit dated.

            Pushing works fine though with 4.11. I can live with disabling my pre-receive hooks until this is resolved, but 4.12 doesn't even let me start up with the pre-receive hooks disabled, or start up to a UI that lets me disable them. Executing a total denial of service to myself even for read access is not a good workaround.

            Maik Musall added a comment - The affected server runs Ubuntu 12.04 LTS (it's scheduled to be replaced during the next few months), so the stock git version is a bit dated. Pushing works fine though with 4.11. I can live with disabling my pre-receive hooks until this is resolved, but 4.12 doesn't even let me start up with the pre-receive hooks disabled, or start up to a UI that lets me disable them. Executing a total denial of service to myself even for read access is not a good workaround.

            maik4,

            Apologies for that. I've updated the "Known Issues" (on the Upgrade Notes) for 4.12 to include this bug. We'll get the release notes updated.

            I just want to be clear that downgrading to Bitbucket Server 4.11 will allow the system to start, but your pre-receive hooks are still broken for Git 2.11.0. 4.12 doesn't break the hooks; it's just blocking the Git versions that do. If you're using Ubuntu 14.04 LTS or newer you might consider simply removing the Git maintainers' PPA (for now) and running the version that comes with Ubuntu. 1.9.1 is far from new, but it's new enough to be supported (and 2.7.4 for Ubuntu 16.04 LTS is, of course, even better).

            Again, my apologies for the hassle.

            Best regards,
            Bryan Turner
            Atlassian Bitbucket

            Bryan Turner (Inactive) added a comment - maik4 , Apologies for that. I've updated the "Known Issues" (on the Upgrade Notes) for 4.12 to include this bug. We'll get the release notes updated. I just want to be clear that downgrading to Bitbucket Server 4.11 will allow the system to start, but your pre-receive hooks are still broken for Git 2.11.0. 4.12 doesn't break the hooks; it's just blocking the Git versions that do. If you're using Ubuntu 14.04 LTS or newer you might consider simply removing the Git maintainers' PPA (for now) and running the version that comes with Ubuntu. 1.9.1 is far from new, but it's new enough to be supported (and 2.7.4 for Ubuntu 16.04 LTS is, of course, even better). Again, my apologies for the hassle. Best regards, Bryan Turner Atlassian Bitbucket

            Maik Musall added a comment - - edited

            This should be prominently mentioned in the 4.12 Release Notes. This ticket is not even listed in the change log. However, if you use the launchnet ppa feed to keep git updated on Ubuntu, there is no easy way to downgrade git to 2.10 because they don't keep older versions around. Since I can't extend the downtime right now to deal with this, my only resort is to go back to Bitbucket 4.11 until I can resolve that.

            Please, you need to warn users about this in the Release Notes.

            Maik Musall added a comment - - edited This should be prominently mentioned in the 4.12 Release Notes. This ticket is not even listed in the change log. However, if you use the launchnet ppa feed to keep git updated on Ubuntu, there is no easy way to downgrade git to 2.10 because they don't keep older versions around. Since I can't extend the downtime right now to deal with this, my only resort is to go back to Bitbucket 4.11 until I can resolve that. Please, you need to warn users about this in the Release Notes.

              bturner Bryan Turner (Inactive)
              fhaehnel Felix (Inactive)
              Affected customers:
              20 This affects my team
              Watchers:
              42 Start watching this issue

                Created:
                Updated:
                Resolved: