1. 问题描述

某客户反馈他们的 Azure 环境，部署应用时出现ip地址不足的提示

Failed create pod sandbox: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-enveter***" network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses

2. 原因分析

以前客户环境遇到过类似现象，说是 azure-cni 拿到网络内的 ip，需要手动给节点分配辅助 IP 的，这个集群应该是没有那么多IP，后来不清楚客户是否联系了Azure 管理员，恢复了，今天再次遇到。

2.1. 尝试定位

根据现象，找到了一个issue https://github.com/Azure/acs-engine/issues/2845

里面有一个评论

And looking at InUse in /var/run/azure-vnet.json, definitely at least one of these nodes is very close to the limit, and that is also the node showing the "No available addresses" error:

请收藏这个有用的命令

1	请收藏这个有用的命令

kubectl get nodes -o name | cut -d / -f 2 | xargs -I{} -n1 ssh {} 'echo 'node ' {}; grep InUse /var/run/azure-vnet-ipam.json | sort | uniq -c'

可正是这个原本可以直接定位的结果，让我产生了怀疑

我把关注点放到了后半段，工作节点 ip 资源是足够的，忽视了前面3个管理节点没有多余 ip 可用了。

2.2. 定位

同事看到上面到图后提醒我，用户可能把容器部署到管理节点了。

检查 kubelet 日志，果然如此。

Nov 25 21:07:01 azure-uat-mgt01 kubelet[5499]: E1125 21:07:01.607523 5499 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-eventer-5ddc7f9f7f-9lr9z_kube-system" network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses Nov 25 21:07:01 azure-uat-mgt01 kubelet[5499]: E1125 21:07:01.607662 5499 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-eventer-5ddc7f9f7f-9lr9z_kube-system(7aa4e25d-2f1d-11eb-b8cc-0242ac120003)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-eventer-5ddc7f9f7f-9lr9z_kube-system" network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses Nov 25 21:07:01 azure-uat-mgt01 kubelet[5499]: E1125 21:07:01.607687 5499 kuberuntime_manager.go:646] createPodSandbox for pod "kube-eventer-5ddc7f9f7f-9lr9z_kube-system(7aa4e25d-2f1d-11eb-b8cc-0242ac120003)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-eventer-5ddc7f9f7f-9lr9z_kube-system" network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses Nov 25 21:07:01 azure-uat-mgt01 kubelet[5499]: E1125 21:07:01.607793 5499 pod_workers.go:186] Error syncing pod 7aa4e25d-2f1d-11eb-b8cc-0242ac120003 ("kube-eventer-5ddc7f9f7f-9lr9z_kube-system(7aa4e25d-2f1d-11eb-b8cc-0242ac120003)"), skipping: failed to "CreatePodSandbox" for "kube-eventer-5ddc7f9f7f-9lr9z_kube-system(7aa4e25d-2f1d-11eb-b8cc-0242ac120003)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-eventer-5ddc7f9f7f-9lr9z_kube-system(7aa4e25d-2f1d-11eb-b8cc-0242ac120003)\" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod \"kube-eventer-5ddc7f9f7f-9lr9z_kube-system\" network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses" Nov 25 21:07:04 azure-uat-mgt01 kubelet[5499]: E1125 21:07:04.457888 5499 cni.go:259] Error adding network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses Nov 25 21:07:04 azure-uat-mgt01 kubelet[5499]: E1125 21:07:04.457927 5499 cni.go:227] Error while adding to cni network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses