1. 问题描述

某客户反馈他们的 Azure 环境，部署应用时出现ip地址不足的提示

Failed create pod sandbox: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-enveter***" network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses

2. 原因分析

以前客户环境遇到过类似现象，说是 azure-cni 拿到网络内的 ip，需要手动给节点分配辅助 IP 的，这个集群应该是没有那么多IP，后来不清楚客户是否联系了Azure 管理员，恢复了，今天再次遇到。

2.1. 尝试定位

根据现象，找到了一个issue https://github.com/Azure/acs-engine/issues/2845

里面有一个评论

And looking at InUse in /var/run/azure-vnet.json, definitely at least one of these nodes is very close to the limit, and that is also the node showing the "No available addresses" error:

请收藏这个有用的命令

kubectl get nodes -o name | cut -d / -f 2 | xargs -I{} -n1 ssh {} 'echo 'node ' {}; grep InUse /var/run/azure-vnet-ipam.json | sort | uniq -c'

可正是这个原本可以直接定位的结果，让我产生了怀疑

我把关注点放到了后半段，工作节点 ip 资源是足够的，忽视了前面3个管理节点没有多余 ip 可用了。

2.2. 定位

同事看到上面到图后提醒我，用户可能把容器部署到管理节点了。

检查 kubelet 日志，果然如此。

Nov 25 21:07:01 azure-uat-mgt01 kubelet[5499]: E1125 21:07:01.607523 5499 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-eventer-5ddc7f9f7f-9lr9z_kube-system" network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses Nov 25 21:07:01 azure-uat-mgt01 kubelet[5499]: E1125 21:07:01.607662 5499 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-eventer-5ddc7f9f7f-9lr9z_kube-system(7aa4e25d-2f1d-11eb-b8cc-0242ac120003)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-eventer-5ddc7f9f7f-9lr9z_kube-system" network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses Nov 25 21:07:01 azure-uat-mgt01 kubelet[5499]: E1125 21:07:01.607687 5499 kuberuntime_manager.go:646] createPodSandbox for pod "kube-eventer-5ddc7f9f7f-9lr9z_kube-system(7aa4e25d-2f1d-11eb-b8cc-0242ac120003)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-eventer-5ddc7f9f7f-9lr9z_kube-system" network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses Nov 25 21:07:01 azure-uat-mgt01 kubelet[5499]: E1125 21:07:01.607793 5499 pod_workers.go:186] Error syncing pod 7aa4e25d-2f1d-11eb-b8cc-0242ac120003 ("kube-eventer-5ddc7f9f7f-9lr9z_kube-system(7aa4e25d-2f1d-11eb-b8cc-0242ac120003)"), skipping: failed to "CreatePodSandbox" for "kube-eventer-5ddc7f9f7f-9lr9z_kube-system(7aa4e25d-2f1d-11eb-b8cc-0242ac120003)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-eventer-5ddc7f9f7f-9lr9z_kube-system(7aa4e25d-2f1d-11eb-b8cc-0242ac120003)\" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod \"kube-eventer-5ddc7f9f7f-9lr9z_kube-system\" network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses" Nov 25 21:07:04 azure-uat-mgt01 kubelet[5499]: E1125 21:07:04.457888 5499 cni.go:259] Error adding network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses Nov 25 21:07:04 azure-uat-mgt01 kubelet[5499]: E1125 21:07:04.457927 5499 cni.go:227] Error while adding to cni network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses

3. 解决方法

请客户部署到其它有空余 ip 的工作节点，成功部署，问题解决。

4. 后续改进

检查日志时多看一些环节
了解Azure cni ip的分配机制

Azure cni No available addresses

1. 问题描述

2. 原因分析

2.1. 尝试定位

2.2. 定位

3. 解决方法

4. 后续改进

文章评论

Azure cni No available addresses

1. 问题描述

2. 原因分析

2.1. 尝试定位

2.2. 定位

3. 解决方法

4. 后续改进

Related posts:

文章评论