一个伪linux粉丝的blog

  1. 首页
  2. network
  3. 正文

Azure cni No available addresses

11月 28, 2020 965点热度 0人点赞 0条评论

1. 问题描述

某客户反馈他们的 Azure 环境,部署应用时出现ip地址不足的提示

Failed create pod sandbox: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-enveter***" network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses

2. 原因分析

以前客户环境遇到过类似现象,说是 azure-cni 拿到网络内的 ip,需要手动给节点分配辅助 IP 的,这个集群应该是没有那么多IP,后来不清楚客户是否联系了Azure 管理员,恢复了,今天再次遇到。

2.1. 尝试定位

根据现象,找到了一个issue https://github.com/Azure/acs-engine/issues/2845

里面有一个评论

And looking at InUse in /var/run/azure-vnet.json, definitely at least one of these nodes is very close to the limit, and that is also the node showing the "No available addresses" error:

1
请收藏这个有用的命令

kubectl get nodes -o name | cut -d / -f 2 | xargs -I{} -n1 ssh {} 'echo 'node ' {}; grep InUse /var/run/azure-vnet-ipam.json | sort | uniq -c'

可正是这个原本可以直接定位的结果,让我产生了怀疑

我把关注点放到了后半段,工作节点 ip 资源是足够的,忽视了前面3个管理节点没有多余 ip 可用了。

2.2. 定位

同事看到上面到图后提醒我,用户可能把容器部署到管理节点了。

检查 kubelet 日志,果然如此。


Nov 25 21:07:01 azure-uat-mgt01 kubelet[5499]: E1125 21:07:01.607523 5499 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-eventer-5ddc7f9f7f-9lr9z_kube-system" network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses
Nov 25 21:07:01 azure-uat-mgt01 kubelet[5499]: E1125 21:07:01.607662 5499 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-eventer-5ddc7f9f7f-9lr9z_kube-system(7aa4e25d-2f1d-11eb-b8cc-0242ac120003)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-eventer-5ddc7f9f7f-9lr9z_kube-system" network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses
Nov 25 21:07:01 azure-uat-mgt01 kubelet[5499]: E1125 21:07:01.607687 5499 kuberuntime_manager.go:646] createPodSandbox for pod "kube-eventer-5ddc7f9f7f-9lr9z_kube-system(7aa4e25d-2f1d-11eb-b8cc-0242ac120003)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-eventer-5ddc7f9f7f-9lr9z_kube-system" network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses
Nov 25 21:07:01 azure-uat-mgt01 kubelet[5499]: E1125 21:07:01.607793 5499 pod_workers.go:186] Error syncing pod 7aa4e25d-2f1d-11eb-b8cc-0242ac120003 ("kube-eventer-5ddc7f9f7f-9lr9z_kube-system(7aa4e25d-2f1d-11eb-b8cc-0242ac120003)"), skipping: failed to "CreatePodSandbox" for "kube-eventer-5ddc7f9f7f-9lr9z_kube-system(7aa4e25d-2f1d-11eb-b8cc-0242ac120003)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-eventer-5ddc7f9f7f-9lr9z_kube-system(7aa4e25d-2f1d-11eb-b8cc-0242ac120003)\" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod \"kube-eventer-5ddc7f9f7f-9lr9z_kube-system\" network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses"
Nov 25 21:07:04 azure-uat-mgt01 kubelet[5499]: E1125 21:07:04.457888 5499 cni.go:259] Error adding network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses
Nov 25 21:07:04 azure-uat-mgt01 kubelet[5499]: E1125 21:07:04.457927 5499 cni.go:227] Error while adding to cni network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses

3. 解决方法

请客户部署到其它有空余 ip 的工作节点,成功部署,问题解决。

4. 后续改进

  •  检查日志时多看一些环节
  • 了解Azure cni ip的分配机制

相关文章:

  1. Non existing device docker--vg-docker--pool
  2. 见识google dns的暴强Ip地址
  3. 遭遇Dreamhost网络瘫痪
  4. 站点启用https
标签: azure
最后更新:12月 1, 2020

wanjie

这个人很懒,什么都没留下

点赞
< 上一篇
下一篇 >

文章评论

取消回复

此站点使用Akismet来减少垃圾评论。了解我们如何处理您的评论数据。

归档
分类
  • network / 322篇
  • Uncategorized / 114篇
  • unix/linux / 110篇
  • 业界资讯 / 38篇
  • 公司杂事 / 11篇
  • 数码影像 / 12篇
  • 美剧 / 3篇
  • 美图共赏 / 20篇
  • 英语学习 / 3篇
标签聚合
wget Google Voice iMac d90 dreamhost空间 google-chrome 刷机 Google Nginx 虚拟主机 kernel debian ldap 网站运营 nexus 黑莓8700 unveiled today 泰国 VPS 天翼live Ubuntu k8s jira gitlab dreamhost 职责 docker Linux 邮件归档 Android

COPYRIGHT © 2008-2022 wanjie.info. ALL RIGHTS RESERVED.

Theme Kratos Made By Seaton Jiang