Building multi-arch images with nerdctl

25 2 月, 2024 1752点热度 0人点赞 0条评论

背景：

上一次写多架构镜像的文章还是2020年（见这篇），马上快 4年了，当时还是用 docker 命令，22年开始，k8s集群底层运行时变成 containerd 了，那么命令也开始用 nerdctl ，算是docker 命令的平替吧，貌似有些方面还是不如docker，这是题外话，今天要说的是 nerdctl 命令如何构建多架构镜像。

官方介绍见这篇 docs/multi-platform.md

前置条件：

主角 nerdctl &gt;= 0.13 本机 nerdctl version 1.7.1
containerd 环境已就绪 本机环境 v1.7.11
隐藏条件依然是 系统内核大于 4.8，我这个环境是 5.4
uname -msr
Linux 5.4.268-1.el7.elrepo.x86_64 x86_64d

主角 nerdctl >= 0.13 本机 nerdctl version 1.7.1

containerd 环境已就绪本机环境 v1.7.11

隐藏条件依然是系统内核大于 4.8，我这个环境是 5.4

uname -msr

Linux 5.4.268-1.el7.elrepo.x86_64 x86_64d

步骤：

部署跨平台模拟器 `binfmt`

nerdctl run --privileged --rm tonistiigi/binfmt --install all

installing: mips64 OK
installing: arm OK
installing: s390x OK
installing: ppc64le OK
installing: riscv64 OK
installing: arm64 OK
installing: mips64le OK
{
"supported": [
"linux/amd64",
"linux/arm64",
"linux/riscv64",
"linux/ppc64le",
"linux/s390x",
"linux/386",
"linux/mips64le",
"linux/mips64",
"linux/arm/v7",
"linux/arm/v6"
],
"emulators": [
"qemu-aarch64",
"qemu-arm",
"qemu-mips64",
"qemu-mips64el",
"qemu-ppc64le",
"qemu-riscv64",
"qemu-s390x"
]
}

nerdctl run --privileged --rm tonistiigi/binfmt --install all

installing: mips64 OK

installing: arm OK

installing: s390x OK

installing: ppc64le OK

installing: riscv64 OK

installing: arm64 OK

installing: mips64le OK

{

"supported": [

"linux/amd64",

"linux/arm64",

"linux/riscv64",

"linux/ppc64le",

"linux/s390x",

"linux/386",

"linux/mips64le",

"linux/mips64",

"linux/arm/v7",

"linux/arm/v6"

"emulators": [

"qemu-aarch64",

"qemu-arm",

"qemu-mips64",

"qemu-mips64el",

"qemu-ppc64le",

"qemu-riscv64",

"qemu-s390x"

]

}

没错，还是docker 时代的 binfmt 跨平台模拟器镜像

前面安装成功后，可以再检查一下是否加载正常了

ls -1 /proc/sys/fs/binfmt_misc/qemu*
/proc/sys/fs/binfmt_misc/qemu-aarch64
/proc/sys/fs/binfmt_misc/qemu-arm
/proc/sys/fs/binfmt_misc/qemu-mips64
/proc/sys/fs/binfmt_misc/qemu-mips64el
/proc/sys/fs/binfmt_misc/qemu-ppc64le
/proc/sys/fs/binfmt_misc/qemu-riscv64
/proc/sys/fs/binfmt_misc/qemu-s390x

ls -1 /proc/sys/fs/binfmt_misc/qemu*

/proc/sys/fs/binfmt_misc/qemu-aarch64

/proc/sys/fs/binfmt_misc/qemu-arm

/proc/sys/fs/binfmt_misc/qemu-mips64

/proc/sys/fs/binfmt_misc/qemu-mips64el

/proc/sys/fs/binfmt_misc/qemu-ppc64le

/proc/sys/fs/binfmt_misc/qemu-riscv64

/proc/sys/fs/binfmt_misc/qemu-s390x

部署buildkitd

访问 https://github.com/moby/buildkit/releases/

找到一个最新 2024-02-16 发布的 buildkit 版本 0.13.rc1

wget -c https://github.com/moby/buildkit/releases/download/v0.13.0-rc1/buildkit-v0.13.0-rc1.linux-amd64.tar.gz
mkdir /usr/local/containerd/
tar zxvf buildkit-v0.13.0-rc1.linux-amd64.tar.gz -C /usr/local/containerd/
ln -s /usr/local/containerd/bin/buildkitd /usr/local/bin/buildkitd
ln -s /usr/local/containerd/bin/buildctl /usr/local/bin/buildctl

wget -c https://github.com/moby/buildkit/releases/download/v0.13.0-rc1/buildkit-v0.13.0-rc1.linux-amd64.tar.gz

mkdir /usr/local/containerd/

tar zxvf buildkit-v0.13.0-rc1.linux-amd64.tar.gz -C /usr/local/containerd/

ln -s /usr/local/containerd/bin/buildkitd /usr/local/bin/buildkitd

ln -s /usr/local/containerd/bin/buildctl /usr/local/bin/buildctl

增加 buildkit 并启动服务

vi /etc/systemd/system/buildkit.socket
[Unit]
Description=BuildKit
Documentation=https://github.com/moby/buildkit

[Socket]
ListenStream=%t/buildkit/buildkitd.sock

[Install]
WantedBy=sockets.target


vi /etc/systemd/system/buildkit.service
[Unit]
Description=BuildKit
Requires=buildkit.socket
After=buildkit.socketDocumentation=https://github.com/moby/buildkit

[Service]
ExecStart=/usr/local/bin/buildkitd --oci-worker=false --containerd-worker=true

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl enable buildkit
systemctl start buildkitk

vi /etc/systemd/system/buildkit.socket

[Unit]

Description=BuildKit

Documentation=https://github.com/moby/buildkit

[Socket]

ListenStream=%t/buildkit/buildkitd.sock

[Install]

WantedBy=sockets.target

vi /etc/systemd/system/buildkit.service

[Unit]

Description=BuildKit

Requires=buildkit.socket

After=buildkit.socketDocumentation=https://github.com/moby/buildkit

[Service]

ExecStart=/usr/local/bin/buildkitd --oci-worker=false --containerd-worker=true

[Install]

WantedBy=multi-user.target

systemctl daemon-reload

systemctl enable buildkit

systemctl start buildkitk

开始构建

前面都就绪后，准备一个跨平台的镜像吧，例如 alpine或者基于它的2048。

nerdctl build --platform=amd64,arm64 --output type=image,name=cnrock/2048:v2024,push=true .

1	nerdctl build --platform=amd64,arm64 --output type=image,name=cnrock/2048:v2024,push=true .

如果是自签证书的本地harbor，可以加 --insecure-registry

nerdctl build --platform=amd64,arm64 -t harbor.ats.io/etc/2048:v2024 --insecure-registry  -f Dockerfile .

1	nerdctl build --platform=amd64,arm64 -t harbor.ats.io/etc/2048:v2024 --insecure-registry -f Dockerfile .

FROM   alpine:latest
LABEL maintainer="Wan Jie <webmaster@wanjie.info>"

ENV TIMEZONE="Asia/Shanghai" \
    BUILD_DEPS="tzdata \
                sudo \
                nano \
                bash \
                iperf3 \
                nginx \
                stress-ng \
                openssl \
                bind-tools"
RUN sed -i "s/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g" /etc/apk/repositories
RUN apk add --no-cache --repository http://mirrors.aliyun.com/alpine/edge/community stress-ng
RUN apk update && \
    apk add --no-cache ${BUILD_DEPS}                        && \
    cp -rfv  /usr/share/zoneinfo/${TIMEZONE} /etc/localtime && \
    echo ${TIMEZONE} > /etc/timezone                        && \
    apk del tzdata                                          && \
    apk add --no-cache curl ca-certificates && \
    touch /etc/ca-certificates.conf && \
    rm -rf /var/cache/apk/*
COPY 2048 /usr/share/nginx/html
COPY 2048/default.conf /etc/nginx/http.d/default.conf
COPY Dockerfile /usr/share/nginx/html

EXPOSE 80

CMD ["nginx", "-g", "daemon off;"]

FROM alpine:latest

LABEL maintainer="Wan Jie <[email protected]>"

ENV TIMEZONE="Asia/Shanghai" \

BUILD_DEPS="tzdata \

sudo \

nano \

bash \

iperf3 \

nginx \

stress-ng \

openssl \

bind-tools"

RUN sed -i "s/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g" /etc/apk/repositories

RUN apk add --no-cache --repository http://mirrors.aliyun.com/alpine/edge/community stress-ng

RUN apk update && \

apk add --no-cache ${BUILD_DEPS} && \

cp -rfv /usr/share/zoneinfo/${TIMEZONE} /etc/localtime && \

echo ${TIMEZONE} > /etc/timezone && \

apk del tzdata && \

apk add --no-cache curl ca-certificates && \

touch /etc/ca-certificates.conf && \

rm -rf /var/cache/apk/*

COPY 2048 /usr/share/nginx/html

COPY 2048/default.conf /etc/nginx/http.d/default.conf

COPY Dockerfile /usr/share/nginx/html

EXPOSE 80

CMD ["nginx", "-g", "daemon off;"]

构建过程截图

harbor 上看看, 多架构正常

可能问题

1，如果系统内核版本过低，加载 binfmt 时会遇到这个报错, 除x86和x64外，其他都不支持，解决办法就是升级系统内核。

 nerdctl run --privileged --rm tonistiigi/binfmt --install all
installing: arm cannot register "/usr/bin/qemu-arm" to /proc/sys/fs/binfmt_misc/register: write /proc/sys/fs/binfmt_misc/register: invalid argument
installing: ppc64le cannot register "/usr/bin/qemu-ppc64le" to /proc/sys/fs/binfmt_misc/register: write /proc/sys/fs/binfmt_misc/register: invalid argument
installing: riscv64 cannot register "/usr/bin/qemu-riscv64" to /proc/sys/fs/binfmt_misc/register: write /proc/sys/fs/binfmt_misc/register: invalid argument
installing: mips64 cannot register "/usr/bin/qemu-mips64" to /proc/sys/fs/binfmt_misc/register: write /proc/sys/fs/binfmt_misc/register: invalid argument
installing: arm64 cannot register "/usr/bin/qemu-aarch64" to /proc/sys/fs/binfmt_misc/register: write /proc/sys/fs/binfmt_misc/register: invalid argument
installing: s390x cannot register "/usr/bin/qemu-s390x" to /proc/sys/fs/binfmt_misc/register: write /proc/sys/fs/binfmt_misc/register: invalid argument
installing: mips64le cannot register "/usr/bin/qemu-mips64el" to /proc/sys/fs/binfmt_misc/register: write /proc/sys/fs/binfmt_misc/register: invalid argument
{
"supported": [
"linux/amd64",
"linux/386"
],
"emulators": null
}

nerdctl run --privileged --rm tonistiigi/binfmt --install all

installing: arm cannot register "/usr/bin/qemu-arm" to /proc/sys/fs/binfmt_misc/register: write /proc/sys/fs/binfmt_misc/register: invalid argument

installing: ppc64le cannot register "/usr/bin/qemu-ppc64le" to /proc/sys/fs/binfmt_misc/register: write /proc/sys/fs/binfmt_misc/register: invalid argument

installing: riscv64 cannot register "/usr/bin/qemu-riscv64" to /proc/sys/fs/binfmt_misc/register: write /proc/sys/fs/binfmt_misc/register: invalid argument

installing: mips64 cannot register "/usr/bin/qemu-mips64" to /proc/sys/fs/binfmt_misc/register: write /proc/sys/fs/binfmt_misc/register: invalid argument

installing: arm64 cannot register "/usr/bin/qemu-aarch64" to /proc/sys/fs/binfmt_misc/register: write /proc/sys/fs/binfmt_misc/register: invalid argument

installing: s390x cannot register "/usr/bin/qemu-s390x" to /proc/sys/fs/binfmt_misc/register: write /proc/sys/fs/binfmt_misc/register: invalid argument

installing: mips64le cannot register "/usr/bin/qemu-mips64el" to /proc/sys/fs/binfmt_misc/register: write /proc/sys/fs/binfmt_misc/register: invalid argument

{

"supported": [

"linux/amd64",

"linux/386"

"emulators": null

}

2，如果没有安装好buildkit , build时可能会如下报错

ERRO[0000] buildctl needs to be installed and buildkitd needs to be running, see https://github.com/moby/buildkit  error="failed to ping to host unix:///run/buildkit-k8s.io/buildkitd.sock: exec: "buildctl": executable file not found in $PATH\nfailed to ping to host unix:///run/buildkit-default/buildkitd.sock: exec: "buildctl": executable file not found in $PATH\nfailed to ping to host unix:///run/buildkit/buildkitd.sock: exec: "buildctl": executable file not found in $PATH"
FATA[0000] no buildkit host is available, tried 3 candidates: failed to ping to host unix:///run/buildkit-k8s.io/buildkitd.sock: exec: "buildctl": executable file not found in $PATH
failed to ping to host unix:///run/buildkit-default/buildkitd.sock: exec: "buildctl": executable file not found in $PATH
failed to ping to host unix:///run/buildkit/buildkitd.sock: exec: "buildctl": executable file not found in $PATH

ERRO[0000] buildctl needs to be installed and buildkitd needs to be running, see https://github.com/moby/buildkit error="failed to ping to host unix:///run/buildkit-k8s.io/buildkitd.sock: exec: "buildctl": executable file not found in $PATH\nfailed to ping to host unix:///run/buildkit-default/buildkitd.sock: exec: "buildctl": executable file not found in $PATH\nfailed to ping to host unix:///run/buildkit/buildkitd.sock: exec: "buildctl": executable file not found in $PATH"

FATA[0000] no buildkit host is available, tried 3 candidates: failed to ping to host unix:///run/buildkit-k8s.io/buildkitd.sock: exec: "buildctl": executable file not found in $PATH

failed to ping to host unix:///run/buildkit-default/buildkitd.sock: exec: "buildctl": executable file not found in $PATH

failed to ping to host unix:///run/buildkit/buildkitd.sock: exec: "buildctl": executable file not found in $PATH