kubectl报错Unable to connect to the server net http TLS handshake timeout

1,728 阅读4分钟

开启掘金成长之旅!这是我参与「掘金日新计划 · 12 月更文挑战」的第10天,点击查看活动详情

kubectl报错Unable to connect to the server: net/http: TLS handshake timeout

执行kubectl命令报错如下

[root@k8smaster1 ~]# kubectl get pod -n kube-system -o wide Unable to connect to the server: net/http: TLS handshake timeout

看到报错提示TLS第一时间想到的是证书问题,但是证书也昂申请没多久,肯定不会出现过期的现象

逐个排查k8s 核心组件日志最终报错在kubelet组件产生,报错内容如下

[root@k8smaster1 ~]# docker ps
CONTAINER ID   IMAGE                                               COMMAND                  CREATED              STATUS              PORTS     NAMES
e1c9c7e2b24e   0369cf4303ff                                        "etcd --advertise-cl…"   About a minute ago   Up About a minute             k8s_etcd_etcd-k8smaster1_kube-system_c296f99e18d4de379f83b1948b9f4525_17
307abbd3e777   560dd11d4550                                        "kube-controller-man…"   9 minutes ago        Up 9 minutes                  k8s_kube-controller-manager_kube-controller-manager-k8smaster1_kube-system_20932c375ba090d1b39ad47d962586b0_17
7424296c698d   b93ab2ec4475                                        "kube-scheduler --au…"   36 minutes ago       Up 36 minutes                 k8s_kube-scheduler_kube-scheduler-k8smaster1_kube-system_16980849e13ae2e69581aae0f2d57229_15
21aa4dccb260   bfe3a36ebd25                                        "/coredns -conf /etc…"   45 minutes ago       Up 45 minutes                 k8s_coredns_coredns-7f89b7bc75-7cthw_kube-system_6c0fb50b-bcc4-486c-98c9-398114c04ad7_3
dd106660675d   bfe3a36ebd25                                        "/coredns -conf /etc…"   46 minutes ago       Up 46 minutes                 k8s_coredns_coredns-7f89b7bc75-bfvxr_kube-system_4f07d5a8-529e-4ad9-ae2f-bdd0c3753033_3
01dda2179e76   registry.aliyuncs.com/google_containers/pause:3.2   "/pause"                 46 minutes ago       Up 46 minutes                 k8s_POD_coredns-7f89b7bc75-bfvxr_kube-system_4f07d5a8-529e-4ad9-ae2f-bdd0c3753033_3
275bb8b6710d   registry.aliyuncs.com/google_containers/pause:3.2   "/pause"                 46 minutes ago       Up 46 minutes                 k8s_POD_coredns-7f89b7bc75-7cthw_kube-system_6c0fb50b-bcc4-486c-98c9-398114c04ad7_3
247d873ab541   1470783b1474                                        "start_runit"            46 minutes ago       Up 46 minutes                 k8s_calico-node_calico-node-2w8zm_kube-system_21820301-25d4-4ff8-8180-9f7d5134f45f_2
9390f1ce5498   9a1ebfd8124d                                        "/usr/local/bin/kube…"   47 minutes ago       Up 47 minutes                 k8s_kube-proxy_kube-proxy-wdmx9_kube-system_71bbf1a6-9a9b-4a64-ab42-46ef5c22e211_3
7ce21cb7fe2e   registry.aliyuncs.com/google_containers/pause:3.2   "/pause"                 48 minutes ago       Up 47 minutes                 k8s_POD_calico-node-2w8zm_kube-system_21820301-25d4-4ff8-8180-9f7d5134f45f_2
b20905bf592c   registry.aliyuncs.com/google_containers/pause:3.2   "/pause"                 48 minutes ago       Up 47 minutes                 k8s_POD_kube-proxy-wdmx9_kube-system_71bbf1a6-9a9b-4a64-ab42-46ef5c22e211_3
3ecda3a4a204   registry.aliyuncs.com/google_containers/pause:3.2   "/pause"                 49 minutes ago       Up 49 minutes                 k8s_POD_kube-scheduler-k8smaster1_kube-system_16980849e13ae2e69581aae0f2d57229_3
0c793d400c44   registry.aliyuncs.com/google_containers/pause:3.2   "/pause"                 49 minutes ago       Up 49 minutes                 k8s_POD_kube-controller-manager-k8smaster1_kube-system_20932c375ba090d1b39ad47d962586b0_3
7b2579d60947   registry.aliyuncs.com/google_containers/pause:3.2   "/pause"                 49 minutes ago       Up 49 minutes                 k8s_POD_kube-apiserver-k8smaster1_kube-system_41bd6149373b5886f2f18464f819ca45_3
5d1217ebd570   registry.aliyuncs.com/google_containers/pause:3.2   "/pause"                 49 minutes ago       Up 49 minutes                 k8s_POD_etcd-k8smaster1_kube-system_c296f99e18d4de379f83b1948b9f4525_3

[root@k8smaster1 ~]# docker logs -f e1c9c7e2b24e

图片.png

重启kubelet无果

[root@k8smaster1 ~]# systemctl restart kubelet

开始分析是不是由于etcd导致,观察etcd日志发现好几天没有输出,那么问题可能真出在etcd这里,重启完etcd问题解决

[root@k8smaster1 ~]# docker restart e1c9c7e2b24e 
e1c9c7e2b24e

如果不是etcd的问题,很有可能还是服务器内存不够导致某个服务死掉

图片.png

但是内存也排除掉了

接下来,有其他工作安排,就去忙其他了,回头忙完,发现已经正常了

[root@k8smaster1 ~]# kubectl get nodes
NAME         STATUS   ROLES                  AGE   VERSION
k8smaster1   Ready    control-plane,master   34d   v1.20.6
k8smaster2   Ready    control-plane,master   34d   v1.20.6
k8snode1     Ready    worker                 34d   v1.20.6