Hi 257cb8fcc776 ,

As discussed in the comment, we have released a variable BITBUCKET_SSH_KEY_FILE to the provide the path to the ssh private key file.

Please refer our documentation for more details.

Regards,
Jayant

I am closing this ticket since BuildKit is available now for everyone to use.
Thank you everyone for the support.

Hi Egor Yurtaev.

Persisting mounted build caches across diferent CI agents is very interesting but is currently an open issue for docker buildkit: https://github.com/moby/buildkit/issues/1512

I tried to outsmart the pipelines and setup the buildx CLI plugin like

pipelines:
  default:
    - step:
        services: [docker]
        script:
          - wget --no-verbose https://github.com/docker/buildx/releases/latest/download/buildx-v0.8.2.linux-amd64 --output-document ~/.docker/cli-plugins/docker-buildx
          - chmod a+x ~/.docker/cli-plugins/docker-buildx
          - docker buildx create --use
          - docker buildx build . --pull --cache-from registry.my.org/foo/bar --build-arg BUILDKIT_INLINE_CACHE=1 --progress=plain 

but this fails with

WARNING: No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
#1 [internal] booting buildkit
#1 pulling image moby/buildkit:buildx-stable-1
#1 pulling image moby/buildkit:buildx-stable-1 2.3s done
#1 creating container buildx_buildkit_busy_haibt0 done
#1 ERROR: Error response from daemon: authorization denied by plugin pipelines: --privileged=true is not allowed
------
 > [internal] booting buildkit:
------
error: Error response from daemon: authorization denied by plugin pipelines: --privileged=true is not allowed 

Fat chance.

I never requested a privileged container, though. Am I doing something wrong?

Hi 257cb8fcc776 ,

Thanks for your feedback. I agree with your point, makes total sense. We still need to think about it properly, but it'd probably make sense to provide an environment variable with the path to the configured SSH key, so that also the configuration experience between Cloud and self-hosted is similar. I'll create a ticket in our backlog to look into that.

Regards,

Raul

Hi Raul,

I have managed to successfully use BuildKit but I have found to my dismay that on Runners (at least the Linux Runner) the SSH key is not present in /opt/atlassian/pipelines/agent/ssh/id_rsa but instead it's in the following directory:

/tmp/{runner-uuid}/ssh/id_rsa

This is problematic as the pipeline could run on any runner so UUID will change, and in addition the runner UUID is not exposed as an environment variable during the build process. As highlighted by Patrick Mead in an earlier comment, the only way currently to discover where the key is is to run something like this:

$(head -1 ~/.ssh/config | cut -f2 -d " ") # This isn't ideal as it depends heavily on how bitbucket sets the ssh config, if it changes builds will stop working 

Which is a fairly brittle method.

Would it be possible to do either of the following:

• symlink /tmp/{runner-uuid}/ssh/ directory during the build setup to /opt/atlassian/pipelines/agent/ssh/ if it detects it's on a runner?

• provide SSH directory as an environment variable during the build?
• provide runner UUID, as an environment variable during the build?

Hi 126a025855f3 , 

Thanks for your feedback. Unfortunately, since we provide a fresh environment for every build, that particular cache isn't persisted between steps, and could also be subject to GC clean-up. In order to get the best experience speeding up your builds and caching Docker layers (with or without Buildkit) I'd recommend using --cache-from, which allows to specify one or more tagged images as a cache source, using your own registry as the cache source between steps. This might also help to improve the performance of your build and save build minutes, without having the limitation of 1GB that our own implementation of docker caching currently has.

For example:

docker build \
--cache-from $IMAGE:latest \
--tag $IMAGE:$BITBUCKET_BUILD_NUMBER \
--tag $IMAGE:latest \
--file ./Dockerfile \
--build-arg BUILDKIT_INLINE_CACHE=1 \
"."

You can read the official docker documentation for more details. We'll clarify this to our public docs so that other users could benefit.

Thanks again for your feedback,  

Raul

Hi Raul, this is good news. But cache doesn't work between pipeline jobs if I use something like this:

RUN --mount=type=cache,id=yarn,target=/app/root/.yarn/cache yarn install --immutable

 
 Pipelines only cached docker layouts without buildkit cache

Hi everyone, 

Good news! we have rolled out an early access version of BuildKit support. We'd love you to try and share your feedback.

To use Buildkit, set the DOCKER_BUILDKIT=1 environment variable in the pipeline configuration (bitbucket-pipelines.yml) or as a workspace / repository variable. For example:

DOCKER_BUILDKIT=1 docker image build -t my/image:tag . 

There are some considerations and limitations when using Buildkit in Bitbucket Pipelines:

• Due to security reasons, we don't currently support -network=host or -security=insecure. 
• In order to use Buildkit SSH forwarding in Bitbucket Pipelines, we recommend using the path to the SSH key available in the build filesystem. For Bitbucket Pipelines SSH configured keys, use the path /opt/atlassian/pipelines/agent/ssh/id_rsa as shown in the following example:

DOCKER_BUILDKIT=1 docker image build -t my/image:tag --ssh default=/opt/atlassian/pipelines/agent/ssh/id_rsa --no-cache . 

For more information on Docker BuildKit, visit: Docker Docs — Build images with BuildKit.

Thanks for all the interest and feedback on this issue so far.

Regards,

Raul

Hi, moby team fixed their issue, are there any timelines when it will be implemented? 

Can anyone confirm that the bug (https://github.com/moby/buildkit/issues/2087#issuecomment-1121740254) has already been fixed?

Has anyone faced this error?

—

Docker Compose version v2.4.1

Docker version 19.03.15, build 99e3ed8

docker-compose

  --env-file ./public.env

  -f docker-compose.yml

  up -d

postgres Pulling

df9b9388f04a Pulling fs layer

7902437d3a12 Pulling fs layer

...

01a8f18ebd09 Extracting [==================================================>] 4.701kB/4.701kB

01a8f18ebd09 Pull complete 

postgres Pulled

listing workers for Build: failed to list workers: Unavailable: connection error: desc = "transport: Error while dialing unable to upgrade to h2c, received 403"

—

Thanks! can you suggest the best work around for those of us trying to use buildkit, specifically RUN --mount ssh, we are mostly concerned about security. 

a9346e13a9d6 we are waiting on a bugfix from the moby team (https://github.com/moby/buildkit/pull/2723) which is currently preventing us from enabling BuildKit, as it is currently not compatible with userns-remapping, a security feature which pipelines uses. 

Once this change has been merged, then we can get a better idea of timelines. We've got BuildKit tentatively scheduled in the next quarter or so.

@Graham Gatus Just want to make sure I am clear, buildkit is still not supported by Bitbucket pipelines, but  builds that are attempting to use buildkit will now fail instead of it being ignored?

is there any timeline for adding buildkit to Bitbucket pipelines?

42faa9d40d58 we are in the process of rolling out an upgrade to the Docker in Docker service, upgrading from 20.10.5 to 20.10.6 (see https://support.atlassian.com/bitbucket-cloud/docs/infrastructure-changes-in-bitbucket-pipelines/ for details).

Docker will now enforce the (incorrect) use of Buildkit related directives inside of Dockerfiles, where Buildkit is not enabled. Previously, docker build would incorrectly skip over Buildkit directives, however it now fails.

You can fix the problem by removing any Buildkit directives from your Dockerfile.

We can temporarily exclude your workspace from the upgrade if you would like time to fix things, please raise a support case via https://support.atlassian.com/contact/#/ and we can arrange this.

My builds also started failing today. I am seeing the "--mount option" error you referenced. Explicitly setting DOCKER_BUILDKIT=0 doesn't fix it. Can you let us know what exactly changed so we can take appropriate steps?

Correction: Adding `DOCKER_BUILDKIT=0` won't help to fix the error, the only way to resolve this error is to remove buildkit option from the command.

Hey d6dd686662ff ,

Sorry for the inconvenience it cause,  that change was push to public user by mistake, and we reverted the change the day after your comments.

Having said that, we are planning to roll out same change again(Upgrade docker dind to 20.10.6) these two weeks, and we would like to understand what error did you get on 27/Jan, just to make sure it doesn't break your builds when we rollout the changes.

Just to confirm, did you see the following error?

"the --mount option BuildKit. Refer to https://docs.docker.com/go/buildkit/ to learn how to build images with BuildKit enabled."

If so, I believed you can remove the unsupported buildkit option and add `DOCKER_BUILDKIT=0` in your command fix the breaking changes.

If this is not what you saw, do you mind to create a support ticket in here: https://support.atlassian.com/contact/#/, so we can identify the issue and make sure the upgrade of dind won't cause failed builds for you again.

Cheers

Leo Liang

Looks like today an update was pushed and the BuildKit syntax is no longer ignored, breaking all our builds. Would be good to have an opt-in/opt-out flag or something that we can temporarily use to control when this sort of changes happen in the pipeline.

New link is https://github.com/moby/buildkit/issues/2087 as other issue was closed (mark as duplicate)

@josh, not closed, just a bad hyperlink... https://github.com/moby/buildkit/issues/2200

This does need to happen sooner than later. Seems that the issue above has been closed now. 

Hi everyone,

As rgomis  mentioned in his previous comment, we found that Buildkit is impacted by bug caused by a security patch in Docker v20.10.3, which prevents it from using secret and ssh mount with userns-remap (https://github.com/moby/buildkit/issues/2200) , and that is the major blocker for us to rollout Buildkit support for everyone.

And just like patrick.mead point out in his comment, you can use Buildkit along with self-hosted runner as temporary workaround. We also provide a blog post to describe how to do it:
https://community.atlassian.com/t5/Bitbucket-Pipelines-articles/Use-buildkit-with-Bitbucket-Pipelines-runners/ba-p/1842426

In the meantime, we will actively check the issue we raised to docker and see what we can do to get it fix.

Cheers

Leo Liang

So long Bitbucket. I'm off to Github with the magnificent Actions.

"Why so slow"? Because Bitbucket team did some architectural decision back in the days, while designing pipelines. And now, as a consequence of that, buildkit is not available. Ball is bounced to docker team, so expect a few more years for this feature to come. It does not seem like Bitbucket team has any serious priority on making this work. Solution is easy, all other major CI providers support buildkit since years, so why stick to "ancient" pipelines and Atlassian's pace of development?

Any good news about this support?

Docker Buildkit is not something recent - Docker 18.04 was released in April 2018!

There are some very good features that should be available.

Is there an ETA?

docker compose v2.0.0 was released yesterday and switched to BuildKit by default, so it's unusable for pipelines. Any hopes of Bitbucket supporting this in the near future?

FYI Atlassian: we've migrated all of our critical repos to GitHub and are moving the last few ones to consolidate there.  This was a solid contributing factor.  

Three years down the road, but still no buildkit... Why so slow?

After a bit of further experimenting, I have managed to successfully use buildkit and its ssh mount features to pull a private repo.

Custom runners give us the ability to control the docker daemon version and docker in docker image used for building, swapping out Bitbucket's image for docker:dind meant I could use some of the buildkit features such as mounting ssh without the error This gets rid of "no active session (...) failed to dial gRPC: unable to upgrade to h2c, received 403"

The next hurdle is making an SSH auth socket with the correct ssh-key mounted available to the buildkit command, Bitbucket pipelines doesn't start these by default and instead places the identity in a generated temp folder and adds the path to config. This is fine for most processes like git and ssh running inside of the runner but not for buildkit (and subsequently git and ssh running inside of our docker build).

Here's some steps to help others, YMMV.

Step 1 - Setup a custom runner 

Step 2 - Configure your pipeline to use a custom docker in docker image, add this to the top of your pipelines.yml

definitions:
  services:
    docker:
      image: docker:dind  # Note this can ONLY be used with a self-hosted runner

pipelines:
...

Step 3 - Add pipeline steps to Start an SSH Agent with the correct identity added so that buildkit can use it

...
pipelines:
  default:
    - step:
        name: build
        runs-on: 
          - 'self.hosted'
        script:
          - eval `ssh-agent`
          - ssh-add $(head -1 ~/.ssh/config | cut -f2 -d " ") # This isn't ideal as it depends heavily on how bitbucket sets the ssh config, if it changes builds will stop working
...

Step 4 - Add destination (e.g. bitbucket.org) to .ssh/known_hosts in the Dockerfile (setting it on the runner doesn't help, but it could be mounted from there instead).

Example **Dockerfile

# syntax=docker/dockerfile:1.2
FROM alpine


# Install ssh client and git (your image may come with it already installed)
RUN apk add --no-cache openssh-client git# Add bitbucket servers (or wherever hosts your repo) to the known hosts


# If there are a lot of hosts an alternative would be mounting the known hosts from the runner
RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan bitbucket.org >> ~/.ssh/known_hosts


# Run any commands that will use SSH Keys e.g. bundle install, git clone, npm install, etc
RUN --mount=type=ssh git clone git@bitbucket.org:dubit/some_project.git

This is stopping us from moving from our current provider to using Bitbucket pipelines.

We use dind to build images from a dockerfile and those builds require private repos from within bitbucket.

@Raul Given that custom pipeline runners are now available and you can specify a different dind image,  shouldn't it be possible to do this with custom runners?

I've experimented and it seems I can get past the original buildkit errors but the ssh just fails as if buildkit isn't getting passed the correct AUTH SOCK.

You can find all Atlassian Cloud public IP ranges at https://ip-ranges.atlassian.com

More about Bitbucket specifics at https://support.atlassian.com/bitbucket-cloud/docs/what-are-the-bitbucket-cloud-ip-addresses-i-should-use-to-configure-my-corporate-firewall/

@Chris Burke, thanks for the DOCKER_HOST suggestion! Do you happen to know which IPs/ports to whitelist (ie Bitbucket's)?

We currently host our own server with Docker installed and set the following environment variable:

DOCKER_HOST=ssh://user@docker.example.com

This allows us to use BuildKit today and works quite well - maybe others will find this useful until official support is added.

Raul, could you please explain what is the current status? I noticed that we already got docker 20 on our accounts, but are still getting "no active session (...) failed to dial gRPC: unable to upgrade to h2c, received 403" message, like build kit would not be supported at all.
We are not affected by the bug above, because we don't use secret/ssh mounts. We, however, completely depend on buildkit and can't build images of our vendor's software without this. Is there a chance for us to use Buildkit features already now, before the moby/buildkit issue 2200 gets resolved?

Hi everyone,

I'd like to share an update on this feature request:

We have successfully rolled out Docker v20 support to all users and we spent some time in the last weeks to conduct an initial security assessment with positive results.

Unfortunately, during the testing phase, we found that Buildkit is impacted by bug caused by a security patch in Docker v20.10.3, which prevents it from using secret and ssh mount with userns-remap. We have opened a bug with Buildkit / Docker maintainers  (https://github.com/moby/buildkit/issues/2200) and let them know about it.

We'll keep checking for status updates and let you know as soon as we have an ETA for the fix.

Thanks for your patience.

Regards,

Raul 

Yes, please make Bitbucket Cloud support BuildKit! It would help a lot to be able to build ARMv7 images. 

+1

Hi cosgrove.tyler.98,

We will ship first Docker v20 and roll it out progressively to all our customers in June as described here. After that, we will look into shipping BuildKit support, as we need to perform a security assessment to identify potential security risks with Docker v20 before shipping it.

We understand how important BuildKit is for you, so I'll share an update as soon as I have more news about rollout, next steps, etc.

Regards,

Raul

Justin, for clarification can you tell us whether

A. BuildKit will be supported when support for Docker v20 is released

B. Work on BuildKit will be started after Docker v20 is released

17ec64851f7f We will be rolling out Docker v20 in June, further details can be found here

@Justin Thomas - is there any update regarding Docker v20 support from Bitbucket Pipelines pov?

0fce5f4b4b28 Yes, that makes sense. I have updated the status of the current issue to Under Consideration and created a new issue for Docker v20 support.

50b64633b7c1 We had a blocker with Docker v19 not supporting secure registries, but I guess we could have done better at communication by providing better visibility into what's happening behind the scenes. Thanks for the feedback.

Thanks for update, Justing. It's great to hear. Sadly, we've already abandoned Bitbucket as there was no feedback for long time about this issue, so for continuity of our development we stopped using pipelines. It's great that you enhance your feature set by adding features like windows runners etc., that will definitetly help you build new user base. But don't forget about your existing users - for technologies you're offering, we need continuous support and upgrades. And communication of what you're doing behind the curtain - which is highly appreciated

Thanks for the update Justin. Should the status of this issue be updated to reflect the current progress? Or is there another issue tracking the Docker v20 rollout?

Hi everyone,

This is Justin from the Bitbucket team. Thank you for your feedback on BuildKit support; I apologize for the lack of update on this issue.

We understand how important BuildKit support is for you. We have been testing BuildKit internally with Docker v20, and we hit a few blockers. They have been resolved now, and we are planning to roll out Docker v20 next month. Once Docker v20 is rolled out, we will be enabling support for BuildKit.

Sorry for the delay, the team has been busy building Bitbucket Pipelines Runners, and then we will be looking at adding Windows support to runners; the public roadmap can be found here.

Thank you again for your feedback.

Best,
Justin Thomas

I ended up digging into this - this article explains a lot about why this is hard for Atlassian to support:

• https://blog.jessfraz.com/post/building-container-images-securely-on-kubernetes/

• and this Atlassian pres about the underlying infrastructure: https://www.slideshare.net/NathanBurrell1/bitbucket-pipelines-powered-by-kubernetes is interesting in terms of understanding the underlying setup and how it can handle potentially dangerous containers

I looked at non-privileged build tools, but they (buildah/podman, buildkit itself, and kaniko) all want to run in privileged containers. Looked at rootless options, like https://github.com/moby/buildkit/blob/master/docs/rootless.md but they need specific security settings on the docker run command, eg. ` --security-opt seccomp=unconfined --security-opt apparmor=unconfined ` which we don't have permission to use in pipelines.

I ended up trying this pipeline based on https://github.com/genuinetools/img referred to in the top article:

options:
  docker: true
pipelines:
  custom:
    manual-build:
      - step:
          name: Build image using buildkit via img
          script:
            - TAG=`echo $BITBUCKET_REPO_SLUG | sed "s@/@-@"`
            - docker login -u $DOCKER_HUB_USER -p $DOCKER_HUB_PASSWORD
            - cp -Rp $HOME/.docker .
            - docker run -t --volume $BITBUCKET_CLONE_DIR:/home/user/src:ro --workdir /home/user/src --volume $BITBUCKET_CLONE_DIR/.docker:/root/.docker:ro r.j3ss.co/img build -t getjohn/$BITBUCKET_REPO_SLUG:$TAG .
 

Which gets as far as starting the build, but since we're not allowed to set the `--security-opt` flags needed, it can't complete the build.

Our next workaround is to use pipelines for the scripting but connect to a remote private docker daemon, and run the build on that.

(edit: fix formatting) 

Definitely did not expect this when going all in with Bitbucket pipelines.

Totally agree with prior comment, how much interest does this need to gather before Atlassian will even bother to comment on it, let alone plan to do the work?  

Sigh, +1 on another Atlassian issue that's years old with no response. I'll be sure to mention this on the yearly 'hey we really care about your feedback' email no one reads obviously. Off to rework my whole build pipeline again because of this...

+1

+1

Finished a pretty complex multistage docker build, just to find out that Bitbucket doesn't support BuildKit. I need to put in a significant effort to rethink the pipeline, use artifacts, split the Dockerfiles... Not ideal at all.

+1