Docker Multi-Platform Build With Buildx remote builder node

devops

You want to make a multi platform docker image but the emulation on your Apple M1 is not bulletproof. How to get around this issue is explained in this blog post…

Problem

I have the new Apple M1 MacBook fancy smancy laptop workhorse. The M1 (aarch64) processor is very fast and very friendly on the battery, but there are also some - not so small - issues associated with working on a new chip architecture.

I’ve tried to create a few multi-platform docker images with buildx and some are just not correctly emulated on the M1.
The parallelism of buildx was also an issue but the emulation gave me the most trouble.

Solution

Use buildx (remote) nodes to build on native architecture.

What do you need

  • More than one docker enabled device in your network like:
    • Nas
    • RaspberryPi
    • Other laptop / PC
  • ssh access to these devices

In my case I have a Synology NAS that is amd64/x86_64 based and my own M1 that is arm64/aarch64 based.

How

Step 1: Installation

Buildx (buildkit) is since the newer versions of Docker a buildin function. You can check this with the following command.

1
2
$ docker buildx version
github.com/docker/buildx v0.8.1 5fac64c2c49dae1320f2b51f1a899ca451935554

On my nas docker version 20.10.3 is running but no buildx is available. This is not an issue as my nas will be a worker node.
We will revisit that in a later step.

Step 2: Setup ssh for your docker worker node

The remote host where we also want to run Docker needs to be configured with a password-less ssh connections. This blog will not explain how to do that as there are many blogs explaining how to set up a public/private key access to a device.

if you already have the keys you can copy them to your target device like so:

1
ssh-copy-id username@target_host

In order to run a command with the user environment available you need to make sure your sshd service allows it.
When running a command through ssh you will have a limited environment and that needs to be adjusted.

In order to have a correct environment where docker is known you need to set the #PermitUserEnvironment no property to PermitUserEnvironment yes in the /etc/ssh/sshd_conf file on your NAS. When this is adjusted you need to restart the sshd service

on my Synology nas I used this command. It might be different for your device:

1
sudo systemctl restart sshd.service

Now you have to set the environment for ssh:

  • login to your nas through the terminal (ssh USER_HERE@NAS_HOST_HERE)
  • cd ~/.ssh
  • create a file called environment with the following value in it
1
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin

the last /usr/local/bin is where the docker command lives. So now it will be available.

Check if you can run docker with the command below. It should work:

1
docker -H ssh://USERNAME_HERE@NAS_HOST_HERE info

Do not forget to change the values in the command to appropriate values for your situation.

Step 3: Create a buildx local node

Let’s first create the local platform. In my case that is the Apple M1 node. It should build all the arm64/aarch64 targets.

1
2
3
4
5
6
docker buildx create \
  --name local_remote_builder \
  --node local_remote_builder \
  --platform linux/arm64,linux/riscv64,linux/ppc64le,linux/s390x,linux/mips64le,linux/mips64,linux/arm/v7,linux/arm/v6 \
  --driver-opt env.BUILDKIT_STEP_LOG_MAX_SIZE=10000000 \
  --driver-opt env.BUILDKIT_STEP_LOG_MAX_SPEED=10000000

I’ve played a bit with the settings but this seems to work best for me. I will update as needed or if I find settings that work even better.

Now we have a local config for a builder called local_remote_builder .

You can check if you have it by running docker buildx ls:

1
2
3
NAME/NODE             DRIVER/ENDPOINT             STATUS  PLATFORMS
local_remote_builder * docker-container
  local_remote_builder unix:///var/run/docker.sock running linux/arm64*, linux/riscv64*, linux/ppc64le*, linux/s390x*, linux/mips64le*, linux/mips64*, linux/arm/v7*, linux/arm/v6*, linux/amd64, linux/amd64/v2, linux/386

Step 4: Create a target buildx remote node

In my case this will be the amd64/x86_64 builder node. The platforms corresponding to that architecture should be send there.
We have already configured the ssh access and the environment and tested that docker can access it.

Now we need to add it to the buildx target

1
2
3
4
5
6
7
8
docker buildx create \
  --name local_remote_builder \
  --append \
  --node intelarch \
  --platform linux/amd64,linux/386 \
  ssh://USERNAME_HERE@NAS_HOST_HERE \
  --driver-opt env.BUILDKIT_STEP_LOG_MAX_SIZE=10000000 \
  --driver-opt env.BUILDKIT_STEP_LOG_MAX_SPEED=10000000

Do not forget to change the values in the command to appropriate values for your situation.

This command will append the remote node to the local_remote_builder and call the node intelarch

Step 5: buildx use and bootstrap

In order to start using this builder setup we need to tell buildx to start using it and we need to bootstrap it.

1
2
docker buildx use local_remote_builder
docker buildx inspect --bootstrap

output

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
#1 [intelarch internal] booting buildkit
#1 starting container buildx_buildkit_intelarch
#1 ...

#2 [localremote_builder internal] booting buildkit
#2 pulling image moby/buildkit:buildx-stable-1 1.6s done
#2 creating container buildx_buildkit_local_remote_builder 0.6s done
#2 DONE 2.3s

#1 [intelarch internal] booting buildkit
#1 starting container buildx_buildkit_intelarch 3.6s done
#1 DONE 3.6s
Name:   localremote_builder
Driver: docker-container

Nodes:
Name:      local_remote_builder
Endpoint:  unix:///var/run/docker.sock
Status:    running
Platforms: linux/arm64*, linux/riscv64*, linux/ppc64le*, linux/s390x*, linux/mips64le*, linux/mips64*, linux/arm/v7*, linux/arm/v6*, linux/amd64, linux/amd64/v2, linux/386

Name:      intelarch
Endpoint:  ssh://USER@192.168.X.X
Status:    running
Platforms: linux/amd64*, linux/386*, linux/amd64/v2

Step 6: build a multi-platform image

The command below will build for amd64 and arm64 but will direct the amd64 build to the remote node and the arm64 build will be done on the local machine.

1
docker buildx build --platform=linux/amd64,linux/arm64/v8 --push -t name/target:tag .

This will result in something like this on the docker hub:

Conclusion

The most obvious issue in this setup is that I can only build for multiple platforms when within my own network.
That is not necessarily true if you used a remote ssh accessible host or IP. I did not do that and that limits me to these builds when I have my remote node available in the network. For now that is not a real issue for me.

My NAS is quite a bit slower that my Apple M1 so the builds are slower.

It solved my concurrency problem. I had one build where I had to start the server, in order to configure it, during the docker build. Buildx does this in parallel and that gave me a port conflict as one was a bit faster but not ready when the other also tried to start. That was a problem when I tried to build all on only my local node. When I added the remote node this problem went away as the port was used on a completely different machine.

All in all I am very happy with this solution for now.