With all due respect, self hosted runners are not the solution here. We're not paying to run this on a second machine. I get the issue with running it as privileged, but by saying run it on your own machine, you're really saying the pipelines is basically useless when it comes to docker.

I'm also trying to get this to work and am having problems. I have configured a self hosted runner and am trying to get this pipeline to work:

definitions:
 services:
   docker: # can only be used with a self-hosted runner
     image: docker:dind 

pipelines: 
  default: 
    - step: 
      name: Compile 
      runs-on: 
        - 'self.hosted'
      clone: skip-ssl-verify: true 
      size: 2x 
      services:
        - docker 
      script: 
        - docker version 
        - docker info 
        - docker run --rm --privileged multiarch/qemu-user-static --reset -p yes; docker buildx create --use  

The pipeline fails at the last step due to `–privileged`.
 
Here is the log:

Runner matching labels:
    - linux
    - self.hosted
Runner name: dontthinkpad
Runner labels: self.hosted, linux
Runner version:
    current: 1.252
    latest: 1.252
+ umask 000

...

Images used:
    build: atlassian/default-image@sha256:3a09dfec7e36fe99e3910714c5646be6302ccbca204d38539a07f0c2cb5902d4
    docker: docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-docker-daemon@sha256:5f95befdbd73f8a85ec3b7fb5a88d52a651979aff97b1355efc18df8a9811aef

+ docker version
Client: Docker Engine - Community
 Version:           19.03.15
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        99e3ed8
 Built:             Sat Jan 30 03:11:43 2021
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          20.10.5
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       363e9a8
  Built:            Tue Mar  2 20:18:31 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

+ docker info
Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 20.10.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Authorization: pipelines
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
  userns
 Kernel Version: 5.8.0-59-generic
 Operating System: Alpine Linux v3.13 (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.37GiB
 Name: fe0fc7a1fae5
 ID: YE3P:ABKU:L4T5:FFN3:YH6R:74DT:Q7EG:NEZV:SJV5:E7WM:76PB:YD2K
 Docker Root Dir: /var/lib/docker/165536.165536
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  http://localhost:5000/
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface

+ docker run --rm --privileged multiarch/qemu-user-static --reset -p yes; docker buildx create --use
docker: Error response from daemon: authorization denied by plugin pipelines: --privileged=true is not allowed.
See 'docker run --help'.
Searching for test report files in directories named [test-reports, test-results, surefire-reports, failsafe-reports] down to a depth of 4
Finished scanning for test reports. Found 0 test report files.
Merged test suites, total number tests is 0, with 0 failures and 0 errors.

I already tried running the docker container that I'm running on the my machine with `--privileged`, but that didn't help:

docker container run -it --rm --privileged ... docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner:1

@Justin Thomas, how do I allow running `docker run --privileged` inside the runner pipeline?

bc42126cf290 Can you please get in touch with Atlassian Support, they should be able to help you with the error.

Hi Justin Thomas,

Sorry for the late message, but I have a need to run just one step as self-hosted for arm architecture build.

The other steps would normally run in bitbucket using your structure.

However, when I put in "definitions" the image docker:dind in the docker service it gives an error when compiling the other steps, as this option only works in self-hosted.

I even tried to create a "docker-custom" and include the image only in it, but I gave a memory error when running, not getting the definition of 3072.

I can't isolate this setting only for the "master" branch, as the homologation machine I have is ARM64 (which is not supported by Bitbucket for compilation), but the development machine is a normal AMD64.

The only way I thought was to have two bitbucket-pipelines.yml for each branch, however when merging it overwrites the content even though I include it in .gitignore and setting the merge method in .gitaatributes. Everything is ignored.

How to solve this problem?

I'm doing automated deploy with Rancher and Continuous Delivery (Fleet integrated), which reads a specific branch that I change at the end of the build in the pipeline.

Below my bitbucket-pipelines.yml

options:

  docker: true
 

pipelines:

  branches:

    develop:

      - step:

          name: Build

          image:

            name: golang:stretch

            username: $DOCKER_HUB_USERNAME

            password: $DOCKER_HUB_PASSWORD

            email: $DOCKER_HUB_EMAIL

          services:

            - docker

          condition:

            changesets:

              includePaths:

              - "app/**"

          script:

          - export APP_NAME=`echo ${BITBUCKET_REPO_SLUG} | sed 's/_/-/ig'`

          - echo $APP_NAME

          - export DEPLOY_BRANCH="deploy-dev"

          - export DEPLOY_TYPE="DEV"

          - export DEPLOY_VERSION="$BITBUCKET_BUILD_NUMBER"

          - export DEPLOY_TAG="$DEPLOY_TYPE-$DEPLOY_VERSION"

          - export IMAGE_NAME=$DOCKER_HUB_USERNAME/$APP_NAME:$DEPLOY_TAG

          - echo "Deploying to environment"

          - cd app/

          - ls -l

          - docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD

          - docker build -t $IMAGE_NAME .

          - docker push $IMAGE_NAME

          - git config --global user.email email@dominio

          - git config --global user.name "Desenvolvimento"

          - echo ''$BITBUCKET_GIT_SSH_ORIGIN''

          - git remote set-url origin ${BITBUCKET_GIT_SSH_ORIGIN}

          - cd /opt/atlassian/pipelines/agent/build

          - git clone --branch="$DEPLOY_BRANCH" --depth 5 ${BITBUCKET_GIT_SSH_ORIGIN} /$DEPLOY_BRANCH

          - cd /$DEPLOY_BRANCH

          - sed -i 's/image:\ dockerhubuser.*$/image:\ dockerhubuser\/'$APP_NAME':'$DEPLOY_TAG'/' $APP_NAME-deployment.yaml

          - git add --all

          - git commit -m 'Deploy '$APP_NAME' '$DEPLOY_TAG''

          - git push --set-upstream origin $DEPLOY_BRANCH

    master:

      - step:

          name: Build

          image:

            name: guglio/dind-buildx:latest

            username: $DOCKER_HUB_USERNAME

            password: $DOCKER_HUB_PASSWORD

            email: $DOCKER_HUB_EMAIL

          runs-on: self.hosted

          services:

            - docker

          condition:

            changesets:

              includePaths:

              - "app/**"

          script:

          - export APP_NAME=`echo ${BITBUCKET_REPO_SLUG} | sed 's/_/-/ig'`

          - echo $APP_NAME

          - export DEPLOY_BRANCH="deploy-dev"

          - export DEPLOY_TYPE="DEV"

          - export DEPLOY_VERSION="$BITBUCKET_BUILD_NUMBER"

          - export DEPLOY_TAG="$DEPLOY_TYPE-$DEPLOY_VERSION"

          - export IMAGE_NAME=$DOCKER_HUB_USERNAME/$APP_NAME:$DEPLOY_TAG

          - echo "Deploying to environment"

          - cd app/

          - ls -l

          - docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD

          - docker run --rm --privileged multiarch/qemu-user-static --reset -p yes; docker buildx create --use --name $APP_NAME

          - docker buildx build -t "$IMAGE_NAME" --platform linux/amd64,linux/arm64 --push .

          - docker buildx imagetools inspect "$IMAGE_NAME"

          - git config --global user.email email@dominio

          - git config --global user.name "Desenvolvimento"

          - echo ''$BITBUCKET_GIT_SSH_ORIGIN''

          - git remote set-url origin ${BITBUCKET_GIT_SSH_ORIGIN}

          - cd /opt/atlassian/pipelines/agent/build

          - git clone --branch="$DEPLOY_BRANCH" --depth 5 ${BITBUCKET_GIT_SSH_ORIGIN} /$DEPLOY_BRANCH

          - cd /$DEPLOY_BRANCH

          - sed -i 's/image:\ dockerhubuser.*$/image:\ dockerhubuser\/'$APP_NAME':'$DEPLOY_TAG'/' $APP_NAME-deployment.yaml

          - git add --all

          - git commit -m 'Deploy '$APP_NAME' '$DEPLOY_TAG''

          - git push --set-upstream origin $DEPLOY_BRANCH

definitions:

  services:

    docker:

      image: docker:dind

      memory: 3072

I appreciate any help.

Best regards,

Carlos

29434169bd5d The image on the step is used to start a container for executing your scripts, while the service docker container is used to start the docker daemon against which the docker commands are executed. Hope this helps.

e698327a1ed7 Unfortunately, we currently only support building multi-arch images using self-hosted runners.

Hi Justin,

i got the build process running. Thanks for the advice.
What i am not understanding is: there are two definitions of image. One in the step and one in the service definition. The one in service is not used as far as i tested it. It only worked for me when i added the image definition within the step.

...
 - step:
 runs-on: self.hosted
 image: docker:dind
 name: create Docker Image
 script:
...

definitions: 
 services: 
 docker: # default docker service
 memory: 512
 docker-custom:
 type: docker 
 image: docker:dind

@Monica, from my understanding you need the „runs-on:“ definition point to one of your runners within a step to use a custom docker service. And then additionally you need the definition of the custom docker service within the services.

hmmm... I'm getting this:

The 'services' section in your bitbucket-pipelines.yml file contains a custom docker service. Remove 'image', 'variables' and 'environment' from the docker service definition or use a self-hosted runner for the step.

Is there any way around this without the runner?

Hi 29434169bd5d,

An example bitbucket-pipelines.yml

pipelines:
  default:
    - step:
        name: Macbook Runner 
        image: guglio/dind-buildx:latest
        runs-on:
          - macbook
          - self.hosted
        services:
          - docker
        script:
          - echo "Executing on self-hosted runner" 
          - docker run --rm --privileged multiarch/qemu-user-static --reset -p yes; docker buildx create --use
          - docker buildx build --platform linux/amd64,linux/arm64,linux/arm/v7 .
definitions:
  services:
    docker: # can only be used with a self-hosted runner
      image: docker:dind

Hope this helps, please let me know if it doesn't work.

Cheers,

Justin Thomas 

@Justin, i tried to use a self.hosted runner to get a buildx to work (my plan was to use the docker:dind image and get the latest buildx bin installed) but it always fails due to an authorization restriction. The plugin that denies the call to docker with the privileged flag set, is ‚pipeline‘ and enabled within the runner container. Can you please show me how to use a self.hosted runner to use buildx?
I always get this error `docker: Error response from daemon: authorization denied by plugin pipelines: --privileged=true is not allowed.`

thanks

Hi everyone,

This is Justin from the Bitbucket team. Thank you for your feedback on building multi-architecture images.

We do understand the importance of this feature to you, but due to the security implications of allowing privileged docker command, we won't be adding support for building multi-architecture images in the short term.

The good news is that we just released self-hosted runners, which allows you to run privileged commands. You can use self-hosted runners to build multi-architecture images.

Thank you again for your feedback.

Best,
Justin Thomas

+1

+1

+1

switch to aws codebuild, it has way better support for this kind of thing

+1

+1

+1

+1

+1

+1

+1

This would be incredibly useful, as is we have no option other than to find another provider for our pipeline.

RasperiPi?

Might not be a solution for everyone but we now switched to using https://github.com/GoogleContainerTools/jib to build our cross-platform Docker images. This way you avoid dependency on the actual Docker daemon to build your images. ps: you do still need to the Docker service in your pipeline step.

Still Atlassian should just fix this issue I think.

Moved to hosted Jenkins.

This issue made us moving to Gitlab (hey, it's free to use own runners there).

Sorry @ Atlassian for advertise a competing product here - but you enforced my to login in order to stop watching this issue (no direct unsubscribe button in your notification mails 😬)

I think before Atlassian could enable buildx (which is what we need) they will need to upgrade their Bitbucket Cloud Pipelines Docker runtime. I can't believe they are still on version 19..

Server Version: 19.03.15 

+1 we really need this feature because without it we cannot use Bitbucket Cloud to build our Docker images to run a Raspberry Pi. I.e. we cannot use Bitbucket Cloud in our CI/CD pipeline..

+1

+1

+1

Bumped, please enable buildx

I will add my voice to @Alexander_Sadovskyi ↑  please enable buildx

Can you just enable buildx? That will resolve all problems...

A simple fix for a major problem.

This could be resolved by having the bifmt configuration for qemu-arm-static registered in the host system. 

The qemu-arm-static can be supplied by image itself then. 

The problem with the register scripts mentioned above (mount failed) is due to the fact that the script tries to mount something in /proc to then register the binfmt settings.
Essentially that steps tells the kernel what interpreter to use when it encounters an ELF binary for ARM.
https://ownyourbits.com/2018/06/13/transparently-running-binaries-from-any-architecture-in-linux-with-qemu-and-binfmt_misc/

+1

+1

+1

+1

+1  Follow too

We're looking for this also !

any updates on this???

as a commercial user, I'm planning move to Github.

A real shame this isn't supported... We may end having to move all our pipelines away for this reason too, due to migration to ARM based stacks.

Solved this by moving build pipelines to AWS CodeBuild. Very flexible! At first glance I think we will move all pipelines there. For now I see no reason not to.

We have started using GitHub Actions as they have ARM support. It's a bit of a pain as we have been pushing our relevant repositories to two remotes. We will likely move all of our repositories over there in the long run so everything is in one place.

I can't believe this is still in status "gathering interest". Enough people develop for arm and the architecture is clearly the future! Even Microsoft has ported win10 to arm and apple will completely switch to it. So how is arm support still not important to Atlassian ???

Cannot use Bitbucket pipelines as is for building docker containers for ARM. Tried
export DOCKER_CLI_EXPERIMENTAL=enabled
but it does not work...

Can someone give a date for when this feature will be available. ARM is basically king for embedded systems, I can't believe there is no support for it...

Isn't it possible to just make the multi-arch build in another docker container and use this new docker container in the pipeline?

Hi Bitbucket team,

we can not move all our embedded code to Bitbucket yet because this is a missing feature in the pipelines. Is there any update when this will be possible? It would be great to have the code all in one place, which is not possible at the moment with this feature missing at Bitbucket. Does anyone have a workaround working?

Thanks,

Florian

Just a heads up, cross compilation with docker got a lot simples with `docker buildx`

- export DOCKER_CLI_EXPERIMENTAL=enabled 
- docker buildx build --platform linux/arm/v7 -t my-image:my-tag --push .

Unfortunately, docker experimental feature cannot be enabled in Bitbucket and the command above it doesn't work.

Please, consider enabling experimental features in Bitbucket's Docker engine.

Thanks,
Franklin

Same here, we need to build for Raspberry Pi. Would be nice, if we could build ARM via pipeline. The native way using an ARM Hosted Instance on EC2 seems preferable...

Hello,

We have the same situation, currently we are doing our builds in ci but we will prefer to use Bitbucket pipelines. 

Thanks

Hello Bitbucket support team,

This is very obvious need for our organization.  There must have a solution. 

We need this too to deploy software to RaspberryPis, why has this thread fallen silent?

Any update on this? I believe this is also relevant for multi-architecture builds in pipelines, which is not currently possible.

99b7cb543ca8 said:

Bumping this thread as ARM64 is a present need on Bitbucket pipelines. Please give an update on availability of QEMU or native ARM64 instances.

Original comment missed during migration to jira.atlassian.com due to timing of export.

Any updates on multi-architecture support?

@mryall_atlassian Amazon EC2 now officially supports ARM64 hosted instances.

@reijosirila I will give this a try.

Our engineer where able to build ARM images using Modified version of qemu:
https://github.com/balena-io/qemu

There you have a additional QEMU_EXECVE flag , so with that you can use in pipeline:

docker build -t abc -f Dockerfile.qemu .

and Dockerfile.qemu includes something like....

FROM arm32v7 ... 
...
COPY qemu-arm-static /usr/bin/
SHELL ["/usr/bin/qemu-arm-static", "-execve", "/bin/sh",  "-c"]
RUN build-for-arm-script

@mryall_atlassian I completely understand. Scaleway is another Cloud Compute Platform which provides ARM based hosts (the only one AFAIK).

@rramchandar - good question. We currently run on Amazon EC2, which only runs Intel hosts. A switch to another hosting provider is doable, but is not something we're willing to consider right now.

So QEMU or another similar emulation tool seems like the best path forward for building or running these images on Pipelines in the near future.

Just an idea: @mryall_atlassian could the Kubernetes cluster expand to arm64v8 (and even other) architectures? This way we can add a step to our pipelines YAML to specify the underlying bare-metal architecture.

This way we wouldn't need to use QEMU to build our arm64 images, since it's already an arm64 environment.

Thanks for the suggestion, I've renamed this to be a bit broader.

To move forward with this, we need a proof of concept for how this can work without using privileged Docker commands, which we unfortunately can't support on the shared Kubernetes cluster inside Pipelines.

If someone can get a build working similar to @ramchandar's example above (I'm unsure what it is trying to mount - maybe this can be switched off?), we can look at how we can improve support for this directly in Pipelines.

Note that the Linux capabilities available to containers in Pipelines are those in the default set used by Docker, as referenced in their security documentation. (There used to be a full list there, but it now links to their source code for the canonical list.) So getting this working locally with Docker with the default capability set (and no privileged commands) would also be something we could work from.

I would suggest renaming the title of this issue to "Allow building multi-architecture docker images" or similar.

This is a highly desired feature for us. We can avoid the --privileged command by running the QEMU files directly.

I've attempted to do that below but am given an error when one of the scripts calls mount:

bitbucket-pipelines.yml:

#!yaml
# enable Docker for all steps
options:
  docker: true

pipelines:
  custom: # Pipelines that are triggered manually
    deploy:
      - step:
          script:
            - docker version
            # QEMU setup (for cross platform compilation)
            # unsupported --> docker run --rm --privileged multiarch/qemu-user-static:register
            - wget https://raw.githubusercontent.com/multiarch/qemu-user-static/master/register/register.sh
            - wget https://raw.githubusercontent.com/multiarch/qemu-user-static/master/register/qemu-binfmt-conf.sh
            - chmod +x register.sh qemu-binfmt-conf.sh
            - ./register.sh
            # Build for each architecture
            - docker build arm64/ -t image-arm64
            - docker build amd64/ -t image-amd64
            # Push to registry
            - docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
            - docker push image-arm64
            - docker push image-amd64

Build fails with:

+ ./register.sh
mount: permission denied
./register.sh: 31: exec: /qemu-binfmt-conf.sh: not found

Thanks for raising this.

The privileged flag means that Docker will allow access to all other builds on the machine. For security reasons, we currently don't support this. We will need to do additional investigation to determine whether this is something that Pipelines will support in future. However, the team are currently working on other higher priority features, so this isn't something that we'll be working on anytime soon.

In the meantime, I'll open this issue to gauge the interest of other users on this functionality.

Thanks,
Aneita

++