有个 k8s 项目点火节点使用了docker 仓库 https://docs.docker.com/registry/
，流水线的镜像会存放到这里，简单易用嘛，当然正式环境大家都推荐用harobr 仓库了，直到有一天磁盘满了，于是准备清理，发现了一系列问题，补记一些，实际上发生在11月1号的折腾记录。
|
1 |
./gc1.sh -h "https://fakename-boot.io" -u"admin:changeme" -r "myapp/private-placement" -p "count=15" -d |
有个小提示manifest_unknow oci mainfest found ,but accept header does not support oci manifest，没理会，进容器，执行gc
|
1 |
registry garbage-collect /etc/docker/registry/config.yml -m |
发现没啥效果
找到仓库的讨论贴
https://github.com/distribution/distribution/issues/2914
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 |
#!/bin/bash set -euo pipefail CMD=$0 function usage { cat <<EOU Usage: $CMD REGISTRY_BASE_URL ACTION [OPTIONS..] Actions: - list list repos - list REPO list tags for repo - delete REPO TAG delete tag for repo Example: List all repos /$ $CMD https://fakename-boot.io list List tags for one repo /$ $CMD https://fakename-boot.io list some-repo Delete tag for a repo /$ $CMD https://fakename-boot.io delete some-repo EOU exit 1 } [ $# -lt 2 ] && usage set +e PROTO="$(echo $1 | grep :// | sed -e's,^\(.*://\).*,\1,g')" set -e [ -z "$PROTO" ] && >&2 echo "ERROR: Must have protocol in registry url" && usage # remove the protocol REG="$(echo ${1/$PROTO/})" shift ACTION="$1" shift case "$ACTION" in list) if [ $# -eq 1 ]; then repo=${1} if [ -n "$repo" ]; then curl -k -s $PROTO$REG/v2/$repo/tags/list | python -m json.tool fi else curl -k -s $PROTO$REG/v2/_catalog?n=4096 | python -m json.tool fi ;; delete) repo=$1 tag=$2 echo $repo echo $tag digest=$(curl -k -v -s -H "Accept:application/vnd.docker.distribution.manifest.v2+json" $PROTO$REG/v2/$repo/manifests/$tag 2>&1 |grep "< Docker-Content-Digest:"|awk '{print $3}') digest=${digest//[$'\t\r\n']} echo "DIGEST: $digest" result=$(curl -k -s -o /dev/null -w "%{http_code}" -H "Accept:application/vnd.docker.distribution.manifest.v2+json" -X DELETE "$PROTO$REG/v2/$repo/manifests/$digest") if [ $result -eq 202 ]; then echo "Successfully deleted" exit 0 else echo $result echo "Failed to delete" exit 3 fi ;; esac |
查看仓库repo正常
./gc.sh https://fakename-boot.io list demo/friday-nginx
发现还是无法清理，准备继续换工具。
找到这个工具
https://github.com/solyard/docker-registry-v2-cleaner
执行清理命令发现证书过期
|
1 |
./docker-registry-v2-cleaner-linux-amd64 -glob "myapp/jar-test" -username "admin" -password "changeme" -url "https://10.1.5.96" |
|
1 2 3 4 |
ls -l /etc/ssl/docker/ total 0 lrwxrwxrwx 1 root 1000 14 Sep 28 09:59 tls.crt -> ..data/tls.crt lrwxrwxrwx 1 root 1000 14 Sep 28 09:59 tls.key -> ..data/tls.key |
|
1 2 3 4 5 6 7 8 |
REGISTRY_HTTP_TLS_CERTIFICATE: /etc/ssl/docker/tls.crt REGISTRY_HTTP_TLS_KEY: /etc/ssl/docker/tls.key REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /var/lib/registry Mounts: /etc/docker/registry from registry-docker-registry-config (rw) /etc/ssl/docker from tls-cert (ro) /var/lib/registry/ from data (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gv4fh (ro) |
|
1 2 3 4 5 6 |
证书是 tls-cert ，类型 secret ,名字 registry-tls kubectl get secret -A |grep registry NAMESPACE NAME TYPE DATA AGE registry-system registry-docker-registry-secret Opaque 3 43d registry-system registry-tls kubernetes.io/tls 2 43d registry-system sh.helm.release.v1.registry.v1 helm.sh/release.v1 1 43d |
|
1 2 |
kubectl -n registry-system get secret registry-tls kubectl -n registry-system get secret registry-tls -oyaml |
|
1 |
openssl x509 -in /tmp/c3.pem -text -noout |grep After -C3 |
自己重新生成一个吧，来一个10年证书。
|
1 2 3 4 |
openssl req -x509 -new -nodes -sha256 -days 3650 \ -key ca.key -out ca.crt \ -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=fakename-boot.io" \ -addext "subjectAltName = IP:10.1.5.96, DNS:fakename-boot.io" |
然后podman 写回去
|
1 |
podman exec -i --privileged my-cluster-installer-control-plane bash -c 'openssl req -x509 -nodes -newkey rsa:2048 -days 3650 -keyout /tmp/tls.key -out /tmp/tls.crt -addext "subjectAltName=IP:10.1.5.96,DNS:boot.io" -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=boot.io"' |
|
1 |
kubectl -n registry-system create secret tls registry-tls --key=/tmp/tls.key --cert=/tmp/tls.crt |
发现这个
|
1 2 |
./docker-registry-v2-cleaner-linux-amd64 -glob "myapp/jar-test" -username "admin" -password "changeme" -url "https://10.1.5.96" [2023/11/01 09:47:52] [application] [FATAL] [storage-helper.go:61,httpGetBody] Get "https://10.1.5.96/v2/_catalog": x509: certificate signed by unknown authority |
|
1 2 3 4 5 |
- Copy the tls.crt to the ca-trust directory and trust it - cd /etc/pki/ca-trust/source/anchors/ - cp /tmp/tls.crt . - cp /tmp/tls.key . - update-ca-trust |
|
1 |
./docker-registry-v2-cleaner-linux-amd64 -glob "jar-test" -username "admin" -password "changeme" -url "https://10.1.5.96" |
脚本略，因为依然无效
|
1 |
./gc-ls-registry-prefix.sh -h "https://fakename-boot.io" -u "root:fakepasswd" -p "count=5" -d -r "fakecloud/dao-2048" -a delete |
继续折腾，找到这个工具
https://www.github.com/Joxit/docker-registry-ui
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
cat docker-compose.yaml version: '3.8' services: registry-ui: image: joxit/docker-registry-ui:main restart: always ports: - 8087:80 environment: - SINGLE_REGISTRY=true - REGISTRY_TITLE=Docker Registry UI - DELETE_IMAGES=true - SHOW_CONTENT_DIGEST=true - NGINX_PROXY_PASS_URL=https://10.1.5.96:443 - SHOW_CATALOG_NB_TAGS=true - CATALOG_MIN_BRANCHES=1 - CATALOG_MAX_BRANCHES=1 - TAGLIST_PAGE_SIZE=100 - REGISTRY_SECURED=false - CATALOG_ELEMENTS_LIMIT=1000 container_name: registry-ui |
图形化web页面出来了，
点击清理看看，发现返回 "The operation is unsupported."
日志看看，返回405 ，似乎和之前的脚本差不多。
查看 registry 容器的yaml (/etc/docker/registry/config.yml)，发现少了这个变量
wanjie
这个人很懒，什么都没留下
文章评论