开启掘金成长之旅!这是我参与「掘金日新计划 · 12 月更文挑战」的第10天,点击查看活动详情
kubectl报错Unable to connect to the server: net/http: TLS handshake timeout
执行kubectl命令报错如下
[root@k8smaster1 ~]# kubectl get pod -n kube-system -o wide Unable to connect to the server: net/http: TLS handshake timeout
看到报错提示TLS第一时间想到的是证书问题,但是证书也昂申请没多久,肯定不会出现过期的现象
逐个排查k8s 核心组件日志最终报错在kubelet组件产生,报错内容如下
[root@k8smaster1 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e1c9c7e2b24e 0369cf4303ff "etcd --advertise-cl…" About a minute ago Up About a minute k8s_etcd_etcd-k8smaster1_kube-system_c296f99e18d4de379f83b1948b9f4525_17
307abbd3e777 560dd11d4550 "kube-controller-man…" 9 minutes ago Up 9 minutes k8s_kube-controller-manager_kube-controller-manager-k8smaster1_kube-system_20932c375ba090d1b39ad47d962586b0_17
7424296c698d b93ab2ec4475 "kube-scheduler --au…" 36 minutes ago Up 36 minutes k8s_kube-scheduler_kube-scheduler-k8smaster1_kube-system_16980849e13ae2e69581aae0f2d57229_15
21aa4dccb260 bfe3a36ebd25 "/coredns -conf /etc…" 45 minutes ago Up 45 minutes k8s_coredns_coredns-7f89b7bc75-7cthw_kube-system_6c0fb50b-bcc4-486c-98c9-398114c04ad7_3
dd106660675d bfe3a36ebd25 "/coredns -conf /etc…" 46 minutes ago Up 46 minutes k8s_coredns_coredns-7f89b7bc75-bfvxr_kube-system_4f07d5a8-529e-4ad9-ae2f-bdd0c3753033_3
01dda2179e76 registry.aliyuncs.com/google_containers/pause:3.2 "/pause" 46 minutes ago Up 46 minutes k8s_POD_coredns-7f89b7bc75-bfvxr_kube-system_4f07d5a8-529e-4ad9-ae2f-bdd0c3753033_3
275bb8b6710d registry.aliyuncs.com/google_containers/pause:3.2 "/pause" 46 minutes ago Up 46 minutes k8s_POD_coredns-7f89b7bc75-7cthw_kube-system_6c0fb50b-bcc4-486c-98c9-398114c04ad7_3
247d873ab541 1470783b1474 "start_runit" 46 minutes ago Up 46 minutes k8s_calico-node_calico-node-2w8zm_kube-system_21820301-25d4-4ff8-8180-9f7d5134f45f_2
9390f1ce5498 9a1ebfd8124d "/usr/local/bin/kube…" 47 minutes ago Up 47 minutes k8s_kube-proxy_kube-proxy-wdmx9_kube-system_71bbf1a6-9a9b-4a64-ab42-46ef5c22e211_3
7ce21cb7fe2e registry.aliyuncs.com/google_containers/pause:3.2 "/pause" 48 minutes ago Up 47 minutes k8s_POD_calico-node-2w8zm_kube-system_21820301-25d4-4ff8-8180-9f7d5134f45f_2
b20905bf592c registry.aliyuncs.com/google_containers/pause:3.2 "/pause" 48 minutes ago Up 47 minutes k8s_POD_kube-proxy-wdmx9_kube-system_71bbf1a6-9a9b-4a64-ab42-46ef5c22e211_3
3ecda3a4a204 registry.aliyuncs.com/google_containers/pause:3.2 "/pause" 49 minutes ago Up 49 minutes k8s_POD_kube-scheduler-k8smaster1_kube-system_16980849e13ae2e69581aae0f2d57229_3
0c793d400c44 registry.aliyuncs.com/google_containers/pause:3.2 "/pause" 49 minutes ago Up 49 minutes k8s_POD_kube-controller-manager-k8smaster1_kube-system_20932c375ba090d1b39ad47d962586b0_3
7b2579d60947 registry.aliyuncs.com/google_containers/pause:3.2 "/pause" 49 minutes ago Up 49 minutes k8s_POD_kube-apiserver-k8smaster1_kube-system_41bd6149373b5886f2f18464f819ca45_3
5d1217ebd570 registry.aliyuncs.com/google_containers/pause:3.2 "/pause" 49 minutes ago Up 49 minutes k8s_POD_etcd-k8smaster1_kube-system_c296f99e18d4de379f83b1948b9f4525_3
[root@k8smaster1 ~]# docker logs -f e1c9c7e2b24e
重启kubelet无果
[root@k8smaster1 ~]# systemctl restart kubelet
开始分析是不是由于etcd导致,观察etcd日志发现好几天没有输出,那么问题可能真出在etcd这里,重启完etcd问题解决
[root@k8smaster1 ~]# docker restart e1c9c7e2b24e
e1c9c7e2b24e
如果不是etcd的问题,很有可能还是服务器内存不够导致某个服务死掉
但是内存也排除掉了
接下来,有其他工作安排,就去忙其他了,回头忙完,发现已经正常了
[root@k8smaster1 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8smaster1 Ready control-plane,master 34d v1.20.6
k8smaster2 Ready control-plane,master 34d v1.20.6
k8snode1 Ready worker 34d v1.20.6