一个伪linux粉丝的blog
  1. 首页
  2. network
  3. 正文

Azure cni No available addresses

2020年11月28日 928点热度 0人点赞 0条评论

1. 问题描述

某客户反馈他们的 Azure 环境,部署应用时出现ip地址不足的提示

Failed create pod sandbox: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-enveter***" network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses

2. 原因分析

以前客户环境遇到过类似现象,说是 azure-cni 拿到网络内的 ip,需要手动给节点分配辅助 IP 的,这个集群应该是没有那么多IP,后来不清楚客户是否联系了Azure 管理员,恢复了,今天再次遇到。

2.1. 尝试定位

根据现象,找到了一个issue https://github.com/Azure/acs-engine/issues/2845

里面有一个评论

And looking at InUse in /var/run/azure-vnet.json, definitely at least one of these nodes is very close to the limit, and that is also the node showing the "No available addresses" error:

请收藏这个有用的命令

kubectl get nodes -o name | cut -d / -f 2 | xargs -I{} -n1 ssh {} 'echo 'node ' {}; grep InUse /var/run/azure-vnet-ipam.json | sort | uniq -c'

可正是这个原本可以直接定位的结果,让我产生了怀疑

我把关注点放到了后半段,工作节点 ip 资源是足够的,忽视了前面3个管理节点没有多余 ip 可用了。

2.2. 定位

同事看到上面到图后提醒我,用户可能把容器部署到管理节点了。

检查 kubelet 日志,果然如此。


Nov 25 21:07:01 azure-uat-mgt01 kubelet[5499]: E1125 21:07:01.607523 5499 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-eventer-5ddc7f9f7f-9lr9z_kube-system" network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses
Nov 25 21:07:01 azure-uat-mgt01 kubelet[5499]: E1125 21:07:01.607662 5499 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-eventer-5ddc7f9f7f-9lr9z_kube-system(7aa4e25d-2f1d-11eb-b8cc-0242ac120003)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-eventer-5ddc7f9f7f-9lr9z_kube-system" network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses
Nov 25 21:07:01 azure-uat-mgt01 kubelet[5499]: E1125 21:07:01.607687 5499 kuberuntime_manager.go:646] createPodSandbox for pod "kube-eventer-5ddc7f9f7f-9lr9z_kube-system(7aa4e25d-2f1d-11eb-b8cc-0242ac120003)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-eventer-5ddc7f9f7f-9lr9z_kube-system" network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses
Nov 25 21:07:01 azure-uat-mgt01 kubelet[5499]: E1125 21:07:01.607793 5499 pod_workers.go:186] Error syncing pod 7aa4e25d-2f1d-11eb-b8cc-0242ac120003 ("kube-eventer-5ddc7f9f7f-9lr9z_kube-system(7aa4e25d-2f1d-11eb-b8cc-0242ac120003)"), skipping: failed to "CreatePodSandbox" for "kube-eventer-5ddc7f9f7f-9lr9z_kube-system(7aa4e25d-2f1d-11eb-b8cc-0242ac120003)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-eventer-5ddc7f9f7f-9lr9z_kube-system(7aa4e25d-2f1d-11eb-b8cc-0242ac120003)\" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod \"kube-eventer-5ddc7f9f7f-9lr9z_kube-system\" network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses"
Nov 25 21:07:04 azure-uat-mgt01 kubelet[5499]: E1125 21:07:04.457888 5499 cni.go:259] Error adding network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses
Nov 25 21:07:04 azure-uat-mgt01 kubelet[5499]: E1125 21:07:04.457927 5499 cni.go:227] Error while adding to cni network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses

3. 解决方法

请客户部署到其它有空余 ip 的工作节点,成功部署,问题解决。

4. 后续改进

  •  检查日志时多看一些环节
  • 了解Azure cni ip的分配机制

Related posts:

  1. Non existing device docker--vg-docker--pool
  2. 见识google dns的暴强Ip地址
  3. 遭遇Dreamhost网络瘫痪
  4. 站点启用https
标签: azure
最后更新:2020年12月01日

WanJie

正经人谁写博客啊?你写博客吗?写出来的那能叫博客?

点赞
< 上一篇
下一篇 >

文章评论

取消回复

分类
  • network
  • Uncategorized
  • unix/linux
  • 业界资讯
  • 公司杂事
  • 数码影像
  • 美剧
  • 美图共赏
  • 英语学习

COPYRIGHT © 2008-2021 一个伪linux粉丝的blog. ALL RIGHTS RESERVED.

THEME KRATOS MADE BY VTROIS