一个伪linux粉丝的blog
  1. 首页
  2. unix/linux
  3. 正文

Solving Kubectl “Error from server (InternalError): an error on the server (“”) has prevented the request from succeeding” .

2020年12月03日 555点热度 0人点赞 0条评论

1个k8s 集群报错的可能解决方法,早上遇到一个测试集群无法访问,现象和排错这里记录一下。

错误现象1,

[[email protected] ~]# kubectl get po

发现没有快速返回,等待1-2分钟后才出现  Error from server (InternalError): an error on the server ("") has prevented the request from succeeding

 

看详细日志的命令

kubectl version --v=7

I1203 08:55:22.083493 3691533 loader.go:375] Config loaded from file: /root/.kube/config
I1203 08:55:22.084521 3691533 cert_rotation.go:137] Starting client certificate rotation controller
I1203 08:55:22.084529 3691533 round_trippers.go:420] GET https://127.0.0.1:11081/version?timeout=32s
I1203 08:55:22.084544 3691533 round_trippers.go:427] Request Headers:
I1203 08:55:22.084551 3691533 round_trippers.go:431] Accept: application/json, */*
I1203 08:55:22.084556 3691533 round_trippers.go:431] User-Agent: kubectl/v1.18.6 (linux/amd64) kubernetes/dc73548
I1203 08:55:22.108562 3691533 round_trippers.go:446] Response Status: 200 OK in 23 milliseconds
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dc73548d265", GitTreeState:"clean", BuildDate:"2020-07-28T08:10:15Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"cae93a63867", GitTreeState:"clean", BuildDate:"2020-08-11T08:41:18Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

错误现象2,

管理网页出现

无法访问kubernetesetcd

kubernetes和etcd出了一些小问题,但应用不会因此受影响,您可以尝试重新加载。
如问题仍然存在,请联系客服进行解决。

调试模式看到

message: "HTTPSConnectionPool(host='172.31.0.1', port=443): Max retries exceeded with url: /apis/apps/v1/namespaces/test-system/deployments (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 113] No route to host',))"

错误现象3,

查看节点kubelet日志

12月 03 08:03:15 test-10-23-2-8 kubelet[35086]: E1203 08:03:15.884685 35086 controller.go:136] failed to ensure node lease exists, will retry in 7s, error: an error on the server ("") has prevented the request from succeeding (get leases.coordination.k8s.io test-10-23-2-8)
12月 03 08:03:16 test-10-23-2-8 kubelet[35086]: I1203 08:03:16.404936 35086 prober.go:124] Readiness probe for "calico-node-t2nvp_kube-system(9bb8e18d-2b6a-4996-a87f-e152b7935bb4):calico-node" failed (failure): 2020-12-03 00:03:16.356 [INFO][195255] confd/health.go 180: Number of node(s) with BGP peering established = 0
12月 03 08:03:16 test-10-23-2-8 kubelet[35086]: calico/node is not ready: BIRD is not ready: BGP not established with 10.23.2.21,10.23.2.19,10.21.2.147
12月 03 08:03:16 test-10-23-2-8 kubelet[35086]: I1203 08:03:16.645775 35086 trace.go:116] Trace[1666794611]: "Reflector ListAndWatch" name:k8s.io/client-go/informers/factory.go:135 (started: 2020-12-03 08:02:46.279600487 +0800 CST m=+56421.330370531) (total time: 30.366144752s):
12月 03 08:03:16 test-10-23-2-8 kubelet[35086]: Trace[1666794611]: [30.366144752s] [30.366144752s] END
12月 03 08:03:16 test-10-23-2-8 kubelet[35086]: E1203 08:03:16.645801 35086 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.CSIDriver: an error on the server ("") has prevented the request from succeeding (get csidrivers.storage.k8s.io)

看到第3点日志其实就差不多应该明白了,有机器无法连上10.23.2.21,10.23.2.19,10.21.2.147

检查发现这3个机器被人关机了 :lol:  ,依次启动,问题解决,所以这个时候不要慌,先看看集群里的其它机器是不是失联了,再从日志里找找线索。

 

Related posts:

  1. Android 4.4.2 cm11 error: Device Offline
  2. Azure cni No available addresses
  3. 普通usb ZC0301 WebCam成功设置为Web Cam Server
  4. Moving WordPress blog to a new server
标签: 暂无
最后更新:2020年12月04日

WanJie

正经人谁写博客啊?你写博客吗?写出来的那能叫博客?

点赞
< 上一篇
下一篇 >

文章评论

取消回复

分类
  • network
  • Uncategorized
  • unix/linux
  • 业界资讯
  • 公司杂事
  • 数码影像
  • 美剧
  • 美图共赏
  • 英语学习

COPYRIGHT © 2008-2021 一个伪linux粉丝的blog. ALL RIGHTS RESERVED.

THEME KRATOS MADE BY VTROIS