• 12
    • Our product teams collect and evaluate feedback from a number of different sources. To learn more about how we use customer feedback in the planning process, check out our new feature policy.

      If using a private image for Pipes was available with the same syntax used for using private step images, this would be super great for wrapping up company specific logic shared across our pipelines.

            [BCLOUD-18270] Use private Docker images as Pipes

            It's really weird to me that this is not supported. Granted, it seems like that's just Bitbucket in general. I've never had to roll so much custom tooling to support a CI solution before, it's nuts.

            Anyways, for any unfortunate soul that stumbles upon this, just wanted to confirm the 'workaround' still works.

            I've provided an example below for how I did this, for context we have self-hosted Bitbucket pipeline runners on swarm and our registry uses https. This approach should work for ECR as well.

            pipelines:
              default:
                - step:
                    name: Test Custom Pipe
                    runs-on:
                      - 'self.hosted'
                      - 'linux'
                    script:
                      - echo "${DOCKER_PASSWORD}" | docker login --username ${DOCKER_USERNAME} --password-stdin <registry URL>
                      - pipe: docker://<registry URL>/<custom pipe image>:<tag>
                        variables:
                          ENVIORMENT_VAR1: test
                          ENVIORMENT_VAR2: test2 

             

            Drew Penrod added a comment - It's really weird to me that this is not supported. Granted, it seems like that's just Bitbucket in general. I've never had to roll so much custom tooling to support a CI solution before, it's nuts. Anyways, for any unfortunate soul that stumbles upon this, just wanted to confirm the 'workaround' still works. I've provided an example below for how I did this, for context we have self-hosted Bitbucket pipeline runners on swarm and our registry uses https. This approach should work for ECR as well. pipelines:   default :     - step:         name: Test Custom Pipe         runs-on:           - 'self.hosted'           - 'linux'         script:           - echo "${DOCKER_PASSWORD}" | docker login --username ${DOCKER_USERNAME} --password-stdin <registry URL>           - pipe: docker: //<registry URL>/<custom pipe image>:<tag>             variables:               ENVIORMENT_VAR1: test               ENVIORMENT_VAR2: test2  

            Can we also add caching for private images at the same time (or pipe images in general) if not already covered? 

            john.lister added a comment - Can we also add caching for private images at the same time (or pipe images in general) if not already covered? 

            Misha Znak added a comment -

            +1 feature are required

            Misha Znak added a comment - +1 feature are required

            yoanis added a comment -

            Pipes that can use private ECR images would be extremely helpful.

            yoanis added a comment - Pipes that can use private ECR images would be extremely helpful.

            any update?

            Carlos Clavijo added a comment - any update?

            Jiwon Jeon added a comment -

            In order to use Bitbucket Pipeline more flexibly, support for the feature is urgently needed. I hope you guys will look forward to this support.

            Jiwon Jeon added a comment - In order to use Bitbucket Pipeline more flexibly, support for the feature is urgently needed. I hope you guys will look forward to this support.

            희석 안 added a comment -

            any update? My team also uses ECR as a private registry and wants to use a custom pipe, but this doesn't seem to work... this is strange

            희석 안 added a comment - any update? My team also uses ECR as a private registry and wants to use a custom pipe, but this doesn't seem to work... this is strange

            This issue has gathered enough interest to be moved automatically to Reviewing status, where it will be reviewed to someone in the relevant product development team and moved on to the appropriate status.

            Mike Howells added a comment - This issue has gathered enough interest to be moved automatically to Reviewing status, where it will be reviewed to someone in the relevant product development team and moved on to the appropriate status.

            Elias Balasis added a comment - - edited

            I have done this in the past following a recipe that involves the following:

            1.
            The docker://<image coordinates> syntax for the "pipe" call
            see https://support.atlassian.com/bitbucket-cloud/docs/configure-bitbucket-pipelinesyml/

            2.
            A private "insecure" Docker registry

            3.
            A custom pipeline Docker image derived from the Atlassian pipeline image and deployed to the "insecure" Docker registry

            4.
            A custom docker-in-docker image derived from "docker:dind"
            with --insecure-registry pointing to the "insecure" Docker registry

            FROM docker:dind
            
            ENV CUSTOM_DOCKER_OPTS="--insecure-registry=myinsecureregistryhost:myinsecureregistryport $DOCKER_OPTS"
            
            ENTRYPOINT [ "sh", "-c", "dockerd-entrypoint.sh $CUSTOM_DOCKER_OPTS" ]
            

            5.
            The following in the bitbucket-pipelines.yml

            definitions:
              services:
                docker:
                  image: myinsecureregistryhost:myinsecureregistryport/mydockerindockerimage
            
                        - pipe: docker://myinsecureregistryhost:myinsecureregistryport/mypipelineimage
                          variables:
                            BITBUCKET_USERNAME: ...
                            BITBUCKET_APP_PASSWORD: ...
                            REPOSITORY: ...
                            REF_TYPE: ...
                            REF_NAME: ...
            

            The complete recipe includes:
            runners on private Docker servers
            using custom images for "pause", "auth_proxy" and "clone" operations
            all derived from the originals and deployed to the "insecure" registry mentioned above.
            see https://support.atlassian.com/bitbucket-cloud/docs/use-your-docker-images-in-self-hosted-runners/
            This way all the executable parts of the build are privately hosted and secure.

            I hope this helps everyone.

            However, I have to agree, even though I have been using my recipe successfully for over a year now, this is indeed a strange limitation and rather unfriendly or complicated to setup for company environments which are expected to have less permissive policies.

            Elias Balasis added a comment - - edited I have done this in the past following a recipe that involves the following: 1. The docker://<image coordinates>  syntax for the "pipe" call see  https://support.atlassian.com/bitbucket-cloud/docs/configure-bitbucket-pipelinesyml/ 2. A private "insecure" Docker registry 3. A custom pipeline Docker image derived from the Atlassian pipeline image and deployed to the "insecure" Docker registry 4. A custom docker-in-docker image derived from "docker:dind" with --insecure-registry pointing to the "insecure" Docker registry FROM docker:dind ENV CUSTOM_DOCKER_OPTS= "--insecure-registry=myinsecureregistryhost:myinsecureregistryport $DOCKER_OPTS" ENTRYPOINT [ "sh" , "-c" , "dockerd-entrypoint.sh $CUSTOM_DOCKER_OPTS" ] 5. The following in the bitbucket-pipelines.yml definitions: services: docker: image: myinsecureregistryhost:myinsecureregistryport/mydockerindockerimage - pipe: docker: //myinsecureregistryhost:myinsecureregistryport/mypipelineimage variables: BITBUCKET_USERNAME: ... BITBUCKET_APP_PASSWORD: ... REPOSITORY: ... REF_TYPE: ... REF_NAME: ... The complete recipe includes: runners on private Docker servers using custom images for "pause", "auth_proxy" and "clone" operations all derived from the originals and deployed to the "insecure" registry mentioned above. see https://support.atlassian.com/bitbucket-cloud/docs/use-your-docker-images-in-self-hosted-runners/ This way all the executable parts of the build are privately hosted and secure. I hope this helps everyone. However, I have to agree, even though I have been using my recipe successfully for over a year now, this is indeed a strange limitation and rather unfriendly or complicated to setup for company environments which are expected to have less permissive policies.

            This is such a strange limitiation, to put it lightly. Why is this not in progress? As some already mentioned this is blocking people from using shared libraries for pipelines, whose company policy is less permissive.

            Mark Hegedus added a comment - This is such a strange limitiation, to put it lightly. Why is this not in progress? As some already mentioned this is blocking people from using shared libraries for pipelines, whose company policy is less permissive.

              57465700c4e1 Edmund Munday
              2171c8f727a3 Satvik Sharma
              Votes:
              276 Vote for this issue
              Watchers:
              142 Start watching this issue

                Created:
                Updated: