记录一次客户私有镜像下载遭遇504的排障记录，简单脱敏。

1. 缘起

客户反馈：新构建的镜像都无法部署，拉取镜像时会出现 502 或者 504现象，无法更新应用。

我这边通过多轮测试，发现国内线路正常，日本、美国、香港均异常，报错如下：

docker pull...

error pulling image configuration: received unexpected HTTP status: 504 Gateway Timeout（偶尔502）

2. 排查

2.1. 排查第一波

几个环境通过 curl 命令发现海外指向的ip是A252 ，通过同事D 沟通，这些服务对某云上名字registry-outer ，也就是某云的负载均衡机器上，国内指向 A224，名字 registry-inner。

2.2. 排查第二波

登录某云前端页面和服务器终端

海外流量访问公网IP A252, 有一个80端口，它后面是某云内网A230 里面的5000端口

2.2.1. compose部分

A230里面有一个docker-compose启动的nginx和registry，compose.yml配置如下：
root@A230:~ #cat registry_china_prod/docker-compose.yml version: '2' services: registry: container_name: hub_registry restart: always image: xyz/registry:v1.1 network_mode: host volumes: - "./key-pair:/key-pair" - "./registry-china.yml:/etc/docker/registry/config.yml" logging: driver: gelf options: gelf-address: udp://A172:12202 nginx: container_name: hub_nginx restart: always image: xyz/nginx:1.11-alpine volumes: - ./nginx.conf:/etc/nginx/nginx.conf network_mode: host logging: driver: gelf options: gelf-address: udp://A172:12202

2.2.2. registry部分

root@A230:~# cat registry_china_prod/registry-china.yml

registry使用了6000端口，主要是指向一个某云的镜像存放文件地址 file对应的bucket
version: 0.1 log: level: info formatter: text storage: maintenance: uploadpurging: enabled: false delete: enabled: false file: driver: file publickey: 略 privatekey: 略 bucket: 略 uploadendpoint: 略 redirect: disable: false cache: blobdescriptor: inmemory reporting: newrelic: licensekey: 略 name: Hub-Registry http: addr: :6000 host: https://xyz secret: thesecrethahah headers: X-Content-Type-Options: [nosniff] compatibility: schema1: signingkeyfile: /etc/docker/registry/key.json auth: token: realm: https://xyz/auth service: xyz issuer: hub rootcertbundle: /key-pair/public.cert notifications: endpoints: - name: A_notify url: http://A239:4000/event timeout: 5s backoff: 1s ignoredmediatypes: - application/octet-stream

2.2.3. nginx.conf配置部分

nginx使用了 5000 端口，这个自身A230 的 5000端口与前面 A252 的 80 对应上了。

里面还有一个 location v2对应的 proxy_pass http://A.15:9002 代理设置，也就是说下一步指向 A.15

root@A230:~# cat registry_china_prod/nginx.conf

worker_processes auto; worker_rlimit_nofile 65535; events { use epoll; worker_connections 65535; } http { include mime.types; default_type application/octet-stream; sendfile on; keepalive_timeout 900; gzip on; upstream registry { server 127.0.0.1:6000; } log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $request_time $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"' '"$upstream_addr" "$upstream_status" "$upstream_response_time" "$request_time"'; access_log /var/log/nginx/access.log main; server { listen 5000; server_name xyz; chunked_transfer_encoding on; client_max_body_size 0; client_body_timeout 9000; client_header_timeout 9000; location ~ /v2/(.*)/(.*)/tags/list { rewrite /v2/(.*)/(.*)/tags/list /wrapper/$1/$2/tags/list break; proxy_pass http://A15:9002; } location ~ /v2/ { proxy_pass http://registry; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_read_timeout 9000; proxy_connect_timeout 9000; proxy_send_timeout 9000; send_timeout 9000; } location ^~ /v1/ { proxy_pass http://registry; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_read_timeout 9000; proxy_connect_timeout 9000; proxy_send_timeout 9000; send_timeout 9000; } location /ping { add_header Content-Type "text/plain;charset=utf-8"; return 200 "OK"; } location / { rewrite ^/(.*) https://xyz/$1 permanent; } } }