Is there any way around this without the runner?

Hi 29434169bd5d,

An example bitbucket-pipelines.yml

pipelines:
  default:
    - step:
        name: Macbook Runner 
        image: guglio/dind-buildx:latest
        runs-on:
          - macbook
          - self.hosted
        services:
          - docker
        script:
          - echo "Executing on self-hosted runner" 
          - docker run --rm --privileged multiarch/qemu-user-static --reset -p yes; docker buildx create --use
          - docker buildx build --platform linux/amd64,linux/arm64,linux/arm/v7 .
definitions:
  services:
    docker: # can only be used with a self-hosted runner
      image: docker:dind

Hope this helps, please let me know if it doesn't work.

Cheers,

Justin Thomas 

@Justin, i tried to use a self.hosted runner to get a buildx to work (my plan was to use the docker:dind image and get the latest buildx bin installed) but it always fails due to an authorization restriction. The plugin that denies the call to docker with the privileged flag set, is ‚pipeline‘ and enabled within the runner container. Can you please show me how to use a self.hosted runner to use buildx?
I always get this error `docker: Error response from daemon: authorization denied by plugin pipelines: --privileged=true is not allowed.`

thanks

Hi everyone,

This is Justin from the Bitbucket team. Thank you for your feedback on building multi-architecture images.

We do understand the importance of this feature to you, but due to the security implications of allowing privileged docker command, we won't be adding support for building multi-architecture images in the short term.

The good news is that we just released self-hosted runners, which allows you to run privileged commands. You can use self-hosted runners to build multi-architecture images.

Thank you again for your feedback.

Best,
Justin Thomas

+1

+1

+1

switch to aws codebuild, it has way better support for this kind of thing

+1

+1

+1

+1

+1

+1

+1

This would be incredibly useful, as is we have no option other than to find another provider for our pipeline.

RasperiPi?

Might not be a solution for everyone but we now switched to using https://github.com/GoogleContainerTools/jib to build our cross-platform Docker images. This way you avoid dependency on the actual Docker daemon to build your images. ps: you do still need to the Docker service in your pipeline step.

Still Atlassian should just fix this issue I think.

Moved to hosted Jenkins.

This issue made us moving to Gitlab (hey, it's free to use own runners there).

Sorry @ Atlassian for advertise a competing product here - but you enforced my to login in order to stop watching this issue (no direct unsubscribe button in your notification mails 😬)

I think before Atlassian could enable buildx (which is what we need) they will need to upgrade their Bitbucket Cloud Pipelines Docker runtime. I can't believe they are still on version 19..

Server Version: 19.03.15 

+1 we really need this feature because without it we cannot use Bitbucket Cloud to build our Docker images to run a Raspberry Pi. I.e. we cannot use Bitbucket Cloud in our CI/CD pipeline..

+1

+1

+1

Bumped, please enable buildx

I will add my voice to @Alexander_Sadovskyi ↑  please enable buildx

Can you just enable buildx? That will resolve all problems...

A simple fix for a major problem.

This could be resolved by having the bifmt configuration for qemu-arm-static registered in the host system. 

The qemu-arm-static can be supplied by image itself then. 

The problem with the register scripts mentioned above (mount failed) is due to the fact that the script tries to mount something in /proc to then register the binfmt settings.
Essentially that steps tells the kernel what interpreter to use when it encounters an ELF binary for ARM.
https://ownyourbits.com/2018/06/13/transparently-running-binaries-from-any-architecture-in-linux-with-qemu-and-binfmt_misc/

+1

+1

+1

+1

+1  Follow too

We're looking for this also !

any updates on this???

as a commercial user, I'm planning move to Github.

A real shame this isn't supported... We may end having to move all our pipelines away for this reason too, due to migration to ARM based stacks.

Solved this by moving build pipelines to AWS CodeBuild. Very flexible! At first glance I think we will move all pipelines there. For now I see no reason not to.

We have started using GitHub Actions as they have ARM support. It's a bit of a pain as we have been pushing our relevant repositories to two remotes. We will likely move all of our repositories over there in the long run so everything is in one place.

I can't believe this is still in status "gathering interest". Enough people develop for arm and the architecture is clearly the future! Even Microsoft has ported win10 to arm and apple will completely switch to it. So how is arm support still not important to Atlassian ???

Cannot use Bitbucket pipelines as is for building docker containers for ARM. Tried
export DOCKER_CLI_EXPERIMENTAL=enabled
but it does not work...

Can someone give a date for when this feature will be available. ARM is basically king for embedded systems, I can't believe there is no support for it...

Isn't it possible to just make the multi-arch build in another docker container and use this new docker container in the pipeline?

We as well have migrated to another cloud build system because Bitbucket doesn't support this. Please allow enabling experimental features in order to support these architectures.

Hi Bitbucket team,

we can not move all our embedded code to Bitbucket yet because this is a missing feature in the pipelines. Is there any update when this will be possible? It would be great to have the code all in one place, which is not possible at the moment with this feature missing at Bitbucket. Does anyone have a workaround working?

Thanks,

Florian

Just a heads up, cross compilation with docker got a lot simples with `docker buildx`

- export DOCKER_CLI_EXPERIMENTAL=enabled 
- docker buildx build --platform linux/arm/v7 -t my-image:my-tag --push .

Unfortunately, docker experimental feature cannot be enabled in Bitbucket and the command above it doesn't work.

Please, consider enabling experimental features in Bitbucket's Docker engine.

Thanks,
Franklin

Same here, we need to build for Raspberry Pi. Would be nice, if we could build ARM via pipeline. The native way using an ARM Hosted Instance on EC2 seems preferable...

Hello,

We have the same situation, currently we are doing our builds in ci but we will prefer to use Bitbucket pipelines. 

Thanks

Hello Bitbucket support team,

This is very obvious need for our organization.  There must have a solution. 

We need this too to deploy software to RaspberryPis, why has this thread fallen silent?

99b7cb543ca8 said:

Bumping this thread as ARM64 is a present need on Bitbucket pipelines. Please give an update on availability of QEMU or native ARM64 instances.

Original comment missed during migration to jira.atlassian.com due to timing of export.

Any updates on multi-architecture support?

@mryall_atlassian Amazon EC2 now officially supports ARM64 hosted instances.

@reijosirila I will give this a try.

Our engineer where able to build ARM images using Modified version of qemu:
https://github.com/balena-io/qemu

There you have a additional QEMU_EXECVE flag , so with that you can use in pipeline:

docker build -t abc -f Dockerfile.qemu .

and Dockerfile.qemu includes something like....

FROM arm32v7 ... 
...
COPY qemu-arm-static /usr/bin/
SHELL ["/usr/bin/qemu-arm-static", "-execve", "/bin/sh",  "-c"]
RUN build-for-arm-script

@mryall_atlassian I completely understand. Scaleway is another Cloud Compute Platform which provides ARM based hosts (the only one AFAIK).

@rramchandar - good question. We currently run on Amazon EC2, which only runs Intel hosts. A switch to another hosting provider is doable, but is not something we're willing to consider right now.

So QEMU or another similar emulation tool seems like the best path forward for building or running these images on Pipelines in the near future.

Just an idea: @mryall_atlassian could the Kubernetes cluster expand to arm64v8 (and even other) architectures? This way we can add a step to our pipelines YAML to specify the underlying bare-metal architecture.

This way we wouldn't need to use QEMU to build our arm64 images, since it's already an arm64 environment.

Thanks for the suggestion, I've renamed this to be a bit broader.

To move forward with this, we need a proof of concept for how this can work without using privileged Docker commands, which we unfortunately can't support on the shared Kubernetes cluster inside Pipelines.

If someone can get a build working similar to @ramchandar's example above (I'm unsure what it is trying to mount - maybe this can be switched off?), we can look at how we can improve support for this directly in Pipelines.

Note that the Linux capabilities available to containers in Pipelines are those in the default set used by Docker, as referenced in their security documentation. (There used to be a full list there, but it now links to their source code for the canonical list.) So getting this working locally with Docker with the default capability set (and no privileged commands) would also be something we could work from.

I would suggest renaming the title of this issue to "Allow building multi-architecture docker images" or similar.

This is a highly desired feature for us. We can avoid the --privileged command by running the QEMU files directly.

I've attempted to do that below but am given an error when one of the scripts calls mount:

bitbucket-pipelines.yml:

#!yaml
# enable Docker for all steps
options:
  docker: true

pipelines:
  custom: # Pipelines that are triggered manually
    deploy:
      - step:
          script:
            - docker version
            # QEMU setup (for cross platform compilation)
            # unsupported --> docker run --rm --privileged multiarch/qemu-user-static:register
            - wget https://raw.githubusercontent.com/multiarch/qemu-user-static/master/register/register.sh
            - wget https://raw.githubusercontent.com/multiarch/qemu-user-static/master/register/qemu-binfmt-conf.sh
            - chmod +x register.sh qemu-binfmt-conf.sh
            - ./register.sh
            # Build for each architecture
            - docker build arm64/ -t image-arm64
            - docker build amd64/ -t image-amd64
            # Push to registry
            - docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
            - docker push image-arm64
            - docker push image-amd64

Build fails with:

+ ./register.sh
mount: permission denied
./register.sh: 31: exec: /qemu-binfmt-conf.sh: not found

Thanks for raising this.

The privileged flag means that Docker will allow access to all other builds on the machine. For security reasons, we currently don't support this. We will need to do additional investigation to determine whether this is something that Pipelines will support in future. However, the team are currently working on other higher priority features, so this isn't something that we'll be working on anytime soon.

In the meantime, I'll open this issue to gauge the interest of other users on this functionality.

Thanks,
Aneita

++