• Icon: Suggestion Suggestion
    • Resolution: Done
    • 5.5.0
    • Git Hosting
    • None
    • We collect Bitbucket feedback from various sources, and we evaluate what we've collected when planning our product roadmap. To understand how this piece of feedback will be reviewed, see our Implementation of New Features Policy.

      Git v2.5.0 introduced uploadpack.allowReachableSHA1InWant, which allows fetching any commit from a remote repository as long as that commit is reachable from at least one ref. This can be useful for grabbing smaller subsets of large repositories at specific commits.

      Since allowReachableSHA1InWant only allows access to commits reachable from at least one ref, it's not exposing any commits to which a remote client would not already have access; it just means they can fetch such commits directly, rather than having to fetch a ref (which may include a significant amount of unnecessary data in the resulting pack) and then checkout the desired commit.

            [BSERV-8268] Enable uploadpack.allowReachableSHA1InWant

            Bryan Turner (Inactive) added a comment - - edited

            Bitbucket Server 5.5 enables uploadpack.allowReachableSHA1InWant for all repositories.

            Enabling uploadarchive.allowUnreachable is a riskier, and therefore more contentious, change. Unlike uploadpack.allowReachableSHA1InWant, which only allows access to commits that are reachable from at least one ref (and can therefore be fetched, indirectly, anyway), uploadarchive.allowUnreachable allows access to any commit, including those which are not reachable. For some organizations, that may pose an unacceptable security risk. For example, consider the case where a developer pushes a commit which adds a clear-text password, or other piece of sensitive information, to a repository, and creates a pull request. One of their reviewers notes the issue, and so the ref and, for added safety, the entire pull request are deleted. However, until the offending commit is pruned from the repository, it would still be accessible to an archive request.

            Rather than hold up the less controversial uploadpack.allowReachableSHA1InWant change, I have split out BSERV-10179 for the uploadarchive.allowUnreachable setting. Those who have use cases for that setting should comment on the new issue.

            Best regards,
            Bryan Turner
            Atlassian Bitbucket

            Bryan Turner (Inactive) added a comment - - edited Bitbucket Server 5.5 enables uploadpack.allowReachableSHA1InWant for all repositories . Enabling uploadarchive.allowUnreachable is a riskier, and therefore more contentious, change. Unlike uploadpack.allowReachableSHA1InWant , which only allows access to commits that are reachable from at least one ref (and can therefore be fetched, indirectly, anyway), uploadarchive.allowUnreachable allows access to any commit, including those which are not reachable. For some organizations, that may pose an unacceptable security risk. For example, consider the case where a developer pushes a commit which adds a clear-text password, or other piece of sensitive information, to a repository, and creates a pull request. One of their reviewers notes the issue, and so the ref and, for added safety, the entire pull request are deleted. However, until the offending commit is pruned from the repository, it would still be accessible to an archive request. Rather than hold up the less controversial uploadpack.allowReachableSHA1InWant change, I have split out BSERV-10179 for the uploadarchive.allowUnreachable setting. Those who have use cases for that setting should comment on the new issue. Best regards, Bryan Turner Atlassian Bitbucket

            allowReachableSHA1InWant is fine for me.  Thanks to the engineer looking into this — it'll save more time for me than any improvement in Bitbucket I can imagine.

            Jonah Petri added a comment - allowReachableSHA1InWant is fine for me.  Thanks to the engineer looking into this — it'll save more time for me than any improvement in Bitbucket I can imagine.

            Hi all,

            An engineer on the team is taking a look at this as part of 20% time and it looks like we can ship something to cover most use cases.

            Specifically we think we can enable uploadpack.allowReachableSHA1InWant for everyone, allowing fetches for reachable SHAs.

            There are subtle security/privacy implications for providing other options, including uploadarchive.allowUnreachable.

            This leads me to a question for you, dear watchers: do any of you have a recurring use case where uploadpack.allowReachableSHA1InWant absolutely wouldn't be useable and you'd need uploadarchive.allowUnreachable as well? If so, could you elaborate as to the context and why?

            Thanks!

            Roger Barnes (Inactive) added a comment - Hi all, An engineer on the team is taking a look at this as part of 20% time and it looks like we can ship something to cover most use cases. Specifically we think we can enable uploadpack.allowReachableSHA1InWant for everyone, allowing fetches for reachable SHAs. There are subtle security/privacy implications for providing other options, including uploadarchive.allowUnreachable. This leads me to a question for you, dear watchers: do any of you have a recurring use case where uploadpack.allowReachableSHA1InWant absolutely wouldn't be useable and you'd need uploadarchive.allowUnreachable as well? If so, could you elaborate as to the context and why? Thanks!

            Bryan Turner (Inactive) added a comment - - edited

            pawel.defee,

            We use bitbucket.com in corporate environment...

            This would actually benefit Atlassian as well as traffic to Bitbucket.com would be reduced (by a great amount in our case).

            I just want to clarify that this issue tracker, and by extension this issue, is for Bitbucket Server, not Bitbucket Cloud (bitbucket.org). Any change we make in response to this request will make zero difference on Bitbucket Cloud unless they choose to make a similar change; the two products are 100% separate and distinct and share no common code. You'd need to request your change in their issue tracker, rather than voting for this issue, so that the change will be made where it helps you.

            Best regards,
            Bryan Turner
            Atlassian Bitbucket

            Bryan Turner (Inactive) added a comment - - edited pawel.defee , We use bitbucket.com in corporate environment... This would actually benefit Atlassian as well as traffic to Bitbucket.com would be reduced (by a great amount in our case). I just want to clarify that this issue tracker, and by extension this issue, is for Bitbucket Server , not Bitbucket Cloud (bitbucket.org). Any change we make in response to this request will make zero difference on Bitbucket Cloud unless they choose to make a similar change; the two products are 100% separate and distinct and share no common code. You'd need to request your change in their issue tracker , rather than voting for this issue, so that the change will be made where it helps you. Best regards, Bryan Turner Atlassian Bitbucket

            Yeah, it saves us 2 hours per clean build if this option is enabled.  Plus about 2 GB of Bitbucket traffic per build.  You guys could save so much of my time if you simply enabled this.

            Jonah Petri added a comment - Yeah, it saves us 2 hours per clean build if this option is enabled.  Plus about 2 GB of Bitbucket traffic per build.  You guys could save so much of my time if you simply enabled this.

            PawelDefee added a comment -

            We use bitbucket.com in corporate environment and uploadpack.allowReachableSHA1InWant is very much needed for the following reason:

            Shallow clones for repositories that have submodules require this option (i.e. running git clone --shallow-submodules).

            Why is it important and beneficial to use shallow clones? Well:

            1) When working on a repository, I am rarely interested in full history of the submodules. Having the possibility to use shallow cloning would greatly speed up the clone operation.

            2) CI agents similarly need no history for builds - right now we must retrieve repos with full history as well as submodules with full history as well. Enabling the option would mean we could fetch a single commit of the repo as well as submodules.

            This would actually benefit Atlassian as well as traffic to Bitbucket.com would be reduced (by a great amount in our case).

             

            PawelDefee added a comment - We use bitbucket.com in corporate environment and uploadpack.allowReachableSHA1InWant is very much needed for the following reason: Shallow clones for repositories that have submodules require this option (i.e. running git clone --shallow-submodules). Why is it important and beneficial to use shallow clones? Well: 1) When working on a repository, I am rarely interested in full history of the submodules. Having the possibility to use shallow cloning would greatly speed up the clone operation. 2) CI agents similarly need no history for builds - right now we must retrieve repos with full history as well as submodules with full history as well. Enabling the option would mean we could fetch a single commit of the repo as well as submodules. This would actually benefit Atlassian as well as traffic to Bitbucket.com would be reduced (by a great amount in our case).  

            kaliprasad.ruddarraju1568970106 added a comment -

            @Joshua Burt - do you have any suggestion or workaround in the case no dedicated git server configured?

            How can we set this setting at a repository level or entire projects on the server level?

            Thanks in advance for any suggestions.

            kaliprasad.ruddarraju1568970106 added a comment - @Joshua Burt - do you have any suggestion or workaround in the case no dedicated git server configured? How can we set this setting at a repository level or entire projects on the server level? Thanks in advance for any suggestions.

            kaliprasad.ruddarraju1568970106 added a comment -

            Here is our use case:

            We have a requirement in our team where we need to retrieve a specific commit from a remote Git repository.

            Use Case description:

            User submits commit SHA-1 hash from front-end and our API should retrieve only that particular commit .

            Generally, we need to clone the entire repo first and then do a checkout:

            git clone ssh://git@xxxxxxxx:7999/abc/pqr.git

            git checkout 7695a5cc0f730e060f2e5e15b81a5f21dccc9659

            But as commits increases and history increases, git repo size will also increase which will lead to increased bandwidth usage as well as disk space usage.

            I went through this forum: http://stackoverflow.com/questions/14872486/retrieve-specific-commit-from-a-remote-git-repository/30701724#30701724

            I found out that if uploadpack.allowReachableSHA1InWant is set to true on server side then we can perform a fetch by giving a SHA-1 hash.

            Finally, we can do something like this:

            git –depth=1 clone ssh://git@xxxxxxxx:7999/abc/pqr.git --no-single-branch

            git fetch origin 7695a5cc0f730e060f2e5e15b81a5f21dccc9659

            git checkout FETCH_HEAD

            Let me know if you have any questions.

            Thanks,
            Kali

            kaliprasad.ruddarraju1568970106 added a comment - Here is our use case: We have a requirement in our team where we need to retrieve a specific commit from a remote Git repository. Use Case description: User submits commit SHA-1 hash from front-end and our API should retrieve only that particular commit . Generally, we need to clone the entire repo first and then do a checkout: git clone ssh://git@xxxxxxxx:7999/abc/pqr.git git checkout 7695a5cc0f730e060f2e5e15b81a5f21dccc9659 But as commits increases and history increases, git repo size will also increase which will lead to increased bandwidth usage as well as disk space usage. I went through this forum: http://stackoverflow.com/questions/14872486/retrieve-specific-commit-from-a-remote-git-repository/30701724#30701724 I found out that if uploadpack.allowReachableSHA1InWant is set to true on server side then we can perform a fetch by giving a SHA-1 hash. Finally, we can do something like this: git –depth=1 clone ssh://git@xxxxxxxx:7999/abc/pqr.git --no-single-branch git fetch origin 7695a5cc0f730e060f2e5e15b81a5f21dccc9659 git checkout FETCH_HEAD Let me know if you have any questions. Thanks, Kali

            I concur. It's very useful to be able to pull from a bitbucket hosted repo by a specific commit ID, just like one would do with a Subversion export:

            svn export -r123 https://repo/project/trunk/subdir@123 project
            

            Somewhat equivalent git archive call:

            git archive --format=tar --remote ssh://git@somebitbucket/repo.git 83ed0638fb698133ae137d900546b6a49426e278 subdir | tar -xf - -C project
            

            Deleted Account (Inactive) added a comment - - edited I concur. It's very useful to be able to pull from a bitbucket hosted repo by a specific commit ID, just like one would do with a Subversion export: svn export -r123 https: //repo/project/trunk/subdir@123 project Somewhat equivalent git archive call: git archive --format=tar --remote ssh: //git@somebitbucket/repo.git 83ed0638fb698133ae137d900546b6a49426e278 subdir | tar -xf - -C project

            Josh Burt added a comment - - edited
            I know this is closed but wanted to discuss (I'm late to the party).

            git commit that added the functionality
            https://github.com/git/git/commit/68ee628932c2196742b77d2961c5e16360734a62

            Thoughtworks (GO) is also discussing this
            https://github.com/gocd/gocd/issues/313#issuecomment-174103478

            I'm currently preparing to provide a patch to Opscode/Chef (https://www.chef.io/) to allow support of this within their product as well.

            We will be using this feature but are going to have to maintain a dedicated git server for the repositories that need this enabled since Bitbucket does not currently support it.

            What is required

            • the server-side MUST have git 2.5 or greater
            • This must be run server-side on the repostiory:
                git config --local uploadpack.allowReachableSHA1InWant true
              

            Use Case
            We have a number of instances were we manage web assets (think javascript, etc). We use git to handle the checkout of the reposiory. Up until now we have been requiring either branches or tags to be targeted for checkout. With the addition of this feature we will now be able to fetch/checkout unadvertised refs on our remote servers.

            In conjunction with prune and depth options it allows us to keep the git footprint small on the front-end servers. To accomplish this another route will require a full clone before checkout of a specified unadvertised ref.

            Josh Burt added a comment - - edited I know this is closed but wanted to discuss (I'm late to the party). git commit that added the functionality https://github.com/git/git/commit/68ee628932c2196742b77d2961c5e16360734a62 Thoughtworks (GO) is also discussing this https://github.com/gocd/gocd/issues/313#issuecomment-174103478 I'm currently preparing to provide a patch to Opscode/Chef ( https://www.chef.io/ ) to allow support of this within their product as well. We will be using this feature but are going to have to maintain a dedicated git server for the repositories that need this enabled since Bitbucket does not currently support it. What is required the server-side MUST have git 2.5 or greater This must be run server-side on the repostiory: git config --local uploadpack.allowReachableSHA1InWant true Use Case We have a number of instances were we manage web assets (think javascript, etc). We use git to handle the checkout of the reposiory. Up until now we have been requiring either branches or tags to be targeted for checkout. With the addition of this feature we will now be able to fetch/checkout unadvertised refs on our remote servers. In conjunction with prune and depth options it allows us to keep the git footprint small on the front-end servers. To accomplish this another route will require a full clone before checkout of a specified unadvertised ref.

              bturner Bryan Turner (Inactive)
              xguan Xiao Di Guan (Inactive)
              Votes:
              12 Vote for this issue
              Watchers:
              16 Start watching this issue

                Created:
                Updated:
                Resolved: