How to Build Multi-Arch Docker Image on NixOS

(Updated: )
, , , ,

With the release of Apple silicon, the adoption of ARM-based CPUs is surging. Many developers are facing a new challenge: compiling their software for ARM-based CPUs. And it’s not just about the binary – Docker images are also getting a multi-architecture makeover.

If you search build multi arch docker image on Google, most of the articles will reference using buildx. While this tool makes building multi-architecture images much easier, I’m always curious to explore alternative solutions, especially when I was told it is the ONLY solution. Since I’m running NixOS, I decided to dive deeper and learn what I can do with it.However, like many things in nix, while it is very powerful, it’s often difficult to find good examples that can be used straight away. Even when tools or solutions are available, there is often little documentation explaining HOW to use them.

In today’s post, I will show you how to use a single flake to build both x86_64 and arm64 images.

Here are some resources that I encourage you to check out to understand why we use Nix to build docker images and what is available.

Let’s get started.

IMPORTANT: dockerTools only works on NixOS. [source]

Cross compilation

If you are running NixOS on a x86_64-linux and want to build amd64 images, we need to add below config to the nixos configuration.nix.

This option will also be useful if you use colmena to build and deploy derivations to the hosts running different architectures.

1
boot.binfmt.emulatedSystems = [ "aarch64-linux" ];

Get template

I’ve created a template to help you get started. You can generate this flake.nix with the following command:

1
nix flake init -t github:liyangau/flake-templates#docker

The main part of this flake is shown below, taken from the examples.

1
2
3
4
5
6
7
8
9
pkgs.dockerTools.buildImage {
  name = imageName;
  tag = "${imageTag}-${archSuffix}";
  copyToRoot = pkgs.buildEnv {
    name = "image-root";
    paths = [ pkgs.bashInteractive ];
    pathsToLink = [ "/bin" ];
  };
};

We will modify the flake to build different images in the next few sections.

Images

Interactive container

buildImage

If you are fine with the default image name and tag, you can directly use the flake template to build an image. This template creates an image with pkgs.bashInteractive installed. Simply run nix build .#amd64 or nix build .#arm64 (if your CPU is ARM), and you should get a result in the same folder.

To check your current system, run nix run github:nix-systems/current-system.

Load the result into Docker with docker load < ./result and then run the image with docker run --rm -it example:latest-<arm64 or amd64> bash.

1
2
[fomm@nixos:~]$ docker run --rm -it example:latest-arm64 bash
bash-5.2#

If you want bash to run automatically when container starts, simply add the following configuration under the copyToRoot section. config is used to specify the configuration of the containers that will be started off the built image in Docker. The available options are listed in the Docker Image Specification v1.2.0.

1
2
3
config = {
  Entrypoint = ["/bin/bash"];
};

buildNixShellImage

As we can see the buildImage function is similar to using a Dockerfile, where you need to specify what packages to be installed and the entrypoint. If you already use Nix to set up your dev environmet, you are likely to have a flakes for your dev enviromnent. To package that environment in docker image, you can reuse your flake and build the image with buildNixShellImage.

For more information on using Nix to set up dev environment, please check out my blog post.

Let me use the example from my flake shell template.

1
2
3
4
5
default = pkgs.mkShellNoCC {
  packages = with pkgs; [
    hello
  ];
};

We can build our new image as shown below.

1
2
3
4
5
6
7
8
9
pkgs.dockerTools.buildNixShellImage {
  name = imageName;
  tag = "${imageTag}-${archSuffix}";
  drv = pkgs.mkShellNoCC {
    packages = with pkgs; [
      hello
    ];
  };
};

After loading the result and running it, we should get a nix-shell with hello installed.

1
2
3
4
[fomm@nixos:~]$ docker run --rm -it example:latest-arm64

[nix-shell:~]$ hello
Hello, world!

CLI container

If your application is a CLI tool, you can build a docker image for the tool without an OS or interative shell.

Package available on Nix

Let’s start with something simple - a CLI tool that is already available on Nix. Here we use the neo-cowsay package.

1
2
3
4
5
6
7
8
9
10
11
12
pkgs.dockerTools.buildImage {
  name = imageName;
  tag = "${imageTag}-${archSuffix}";
  copyToRoot = pkgs.buildEnv {
    name = "image-root";
    paths = [ pkgs.neo-cowsay ];
    pathsToLink = [ "/bin" ];
  };
  config = {
    Entrypoint = [ "cowsay" ];
  };
}

After loading the result and running it, we can use the cowsay command with the new image.

1
2
3
4
5
6
7
8
9
10
11
[fomm@nixos:~]$ docker run --rm -it example:latest-arm64 hello world --random
 _____________
< hello world >
 -------------
     \
      \
          oO)-.                       .-(Oo
         /__  _\                     /_  __\
         \  \(  |     ()~()         |  )/  /
          \__|\ |    (-___-)        | /|__/
          '  '--'    ==`-'==        '--'  '

Package not available on Nix

Next, let’s try to create a package that is not available on Nix. We’ll use the case-cli as an example.

Firstly, download the default.nix from here.

Next, use the pkgs.callPackage to bring this package into our flake.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
let
  archSuffix = if targetSystem == "x86_64-linux" then "amd64" else "arm64";
  case-cli = pkgs.callPackage ./default.nix { };
in
pkgs.dockerTools.buildImage {
  name = imageName;
  tag = "${imageTag}-${archSuffix}";
  copyToRoot = pkgs.buildEnv {
    name = "image-root";
    paths = [ case-cli ];
    pathsToLink = [ "/bin" ];
  };
  config = {
    EntryPoint = [ "case-cli" ];
  };
};

The build should complete successfully, but we’ll get some errors. To resolve these issues, add pkgs.genconf and pkgs.bash in copyToRoot.paths. After making these changes, the container should be able to run the case-cli tool successfully. As you may have noticed, case-cli is a Node.js application that requires Node.js to be present in the environment at runtime. With Nix, we don’t need to worry about setting up the runtime environment because Nix takes care of it.

Layer image

buildLayeredImage

All the examples we’ve covered so far use buildImage, which package everything into a single layer. You can also use buildLayeredImage to build your docker images which separate layers for each nix store path.

Let’s reuse the previous example and use buildLayeredImage instead.

1
2
3
4
5
6
7
8
9
10
11
12
let
  archSuffix = if targetSystem == "x86_64-linux" then "amd64" else "arm64";
  case-cli = pkgs.callPackage ./default.nix { };
in
pkgs.dockerTools.buildLayeredImage {
  name = imageName;
  tag = "${imageTag}-${archSuffix}";
  contents = [ pkgs.bash pkgs.getconf case-cli ];
  config = {
    EntryPoint = [ "case-cli" ];
  };
};

When we load this new image to docker, we can see multiple layers.

Multiple buildImage

We can also use create layers with buildImage. Below is an example where I define a base image layer bash and build another layer on top of it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
let
  archSuffix = if targetSystem == "x86_64-linux" then "amd64" else "arm64";
  bash = pkgs.dockerTools.buildImage {
    name = "bash";
    tag = "latest";
    copyToRoot = pkgs.buildEnv {
      name = "image-root";
      paths = [ pkgs.bashInteractive ];
      pathsToLink = [ "/bin" ];
    };
  };
in
pkgs.dockerTools.buildImage {
  name = imageName;
  tag = "${imageTag}-${archSuffix}";
  fromImage = bash;
  copyToRoot = pkgs.buildEnv {
    name = "image-root";
    paths = [
      pkgs.busybox
      pkgs.vim
      pkgs.micro
      pkgs.helix
    ];
    pathsToLink = [ "/bin" ];
  };
  config = {
    EntryPoint = [ "/bin/bash" ];
  };
};

When we import this image, we can see there are two layers.

Use image from Dockerhub

Now that we know we can build image based on another image, is there a way to build ontop of an image on DockerHub? The answer is yes. Let’s say I want to use alpine:3.20.2 as my based, I can use a tool called nix-prefetch-docker to prepare the the image data for us.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[fomm@nixos:~]$ nix run nixpkgs#nix-prefetch-docker -- alpine --image-tag 3.20.2
Getting image source signatures
Copying blob c6a83fedfae6 done   |
Copying config 324bc02ae1 done   |
Writing manifest to image destination
-> ImageName: alpine
-> ImageDigest: sha256:0a4eaa0eecf5f8c050e5bba433f58c052be7587ee8af3e8b3910ef9ab5fbe9f5
-> FinalImageName: alpine
-> FinalImageTag: 3.20.2
-> ImagePath: /nix/store/nigmwc81p2x09fybr8a6g5r7zwmyiafy-docker-image-alpine-3.20.2.tar
-> ImageHash: 1cbdj05qsf6417cq2xxdjr1frca76jdnjx6yn8ziambwags6hfd3
{
  imageName = "alpine";
  imageDigest = "sha256:0a4eaa0eecf5f8c050e5bba433f58c052be7587ee8af3e8b3910ef9ab5fbe9f5";
  sha256 = "1cbdj05qsf6417cq2xxdjr1frca76jdnjx6yn8ziambwags6hfd3";
  finalImageName = "alpine";
  finalImageTag = "3.20.2";
}

Then we can use the data as below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
let
  archSuffix = if targetSystem == "x86_64-linux" then "amd64" else "arm64";
  alpine = pkgs.dockerTools.pullImage {
    imageName = "alpine";
    imageDigest = "sha256:0a4eaa0eecf5f8c050e5bba433f58c052be7587ee8af3e8b3910ef9ab5fbe9f5";
    sha256 = "1cbdj05qsf6417cq2xxdjr1frca76jdnjx6yn8ziambwags6hfd3";
    finalImageName = "alpine";
    finalImageTag = "3.20.2";
  };
in
pkgs.dockerTools.buildImage {
  name = imageName;
  tag = "${imageTag}-${archSuffix}";
  fromImage = alpine;
  copyToRoot = pkgs.buildEnv {
    name = "image-root";
    paths = [
      pkgs.busybox
      pkgs.bashInteractive
    ];
    pathsToLink = [ "/bin" ];
  };
  config = {
    EntryPoint = [ "/bin/bash" ];
  };
};

Once we load and run this image, we can see the OS is alpine.

1
2
3
4
5
6
7
8
9
bash-5.2# cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.20.2
PRETTY_NAME="Alpine Linux v3.20"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"
bash-5.2# exit
exit

Create multi arch image

Now we understand how to build docker images, let’s explore how to create multi arch image. To simply the process, you can simply type nix run in the same folder as your flake.nix, which will build two results for you. This is handled by below section of the flake.nix.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
apps = forEachSupportedSystem (system: {
  default = {
    type = "app";
    program = toString (
      nixpkgs.legacyPackages.${system}.writeScript "build-multi-arch" ''
        #!${nixpkgs.legacyPackages.${system}.bash}/bin/bash
        set -e
        echo "Building x86_64-linux image..."
        nix build .#amd64 --out-link result-${system}-amd64
        echo "Building aarch64-linux image..."
        nix build .#arm64 --out-link result-${system}-arm64
      ''
    );
  };
});

Push image

${system} is the result of nix run github:nix-systems/current-system
${imageName} and ${imageTag} are defined on the flake.nix.
Please subsitute these variables accordingly or put these commands on the flake under apps section above.

After the images are build and fully tested, time to push the images to a image registry. For Dockerhub, you can simply load the image to docker and push them to DockerHub.

1
2
3
4
docker load < result-${system}-amd64
docker load < result-${system}-arm64
docker push ${imageName}:${imageTag}-amd64
docker push ${imageName}:${imageTag}-arm64

Now, these two images should be available on DockerHub separately.

Tag multi arch

To create 1 tag to servces both type of images, we need to use docker manifest to create a new manifest and associate these two images with the manifest.

1
2
3
docker manifest create ${imageName}:${imageTag} ${imageName}:${imageTag}-amd64 ${imageName}:${imageTag}-arm64
docker manifest annotate ${imageName}:${imageTag} ${imageName}:${imageTag}-amd64 --arch amd64
docker manifest annotate ${imageName}:${imageTag} ${imageName}:${imageTag}-arm64 --arch arm64

If you want to use your latest tag as well, just create a manifest for it.

1
2
3
docker manifest create ${imageName}:latest ${imageName}:${imageTag}-amd64 ${imageName}:${imageTag}-arm64
docker manifest annotate ${imageName}:latest ${imageName}:${imageTag}-amd64 --arch amd64
docker manifest annotate ${imageName}:latest ${imageName}:${imageTag}-arm64 --arch arm64

The last step is to simply push the manifest.

1
2
docker manifest push ${imageName}:${imageTag}
docker manifest push ${imageName}:latest

Summary

Nix is clearly a great tool. One of its streight is flexibility, but unfortunately, complexity often comes with it. One of the most challenging aspects of using Nix is that it is not opinionated—there are many ways to achieve the same outcome. As a result, when searching for answers, we often encounter different solutions, making it difficult to determine which one to use. My template might be considered incorrect by advanced users, but it serves as a starting point for new users like myself to begin using Nix to solve real-world problems. I really hope to see more advanced users create examples that new users can apply immediately. It’s okay to be opinionated when we want to encourage more users to try Nix. Nix can only thrive when new users experience more success than frustration. Its growth also depends on the community sharing their use cases and ensuring there are resources for others to follow.

That’s all I wanted to share with you today, see you in the next one!