本文已参与「新人创作礼」活动，一起开启掘金创作之路

kubelet运行在每个worker节点上，接收kube-apiserver发送的请求，管理Pod容器，执行交互式命令，如exec、run、logs等
kubelet启动时自动向kube-apiserver注册节点信息，内置的cadvisor统计和监控节点的资源使用情况
下载地址 www.downloadkubernetes.com
参数说明 kubernetes.io/zh/docs/ref…

一、下载 kubelet

1、获取下载地址

微信图片_20220409193922.png

2、下载 kubelet、kubeadm 并复制到所有节点

[root@master1 ~]# cd /opt/install/
[root@master1 install]# wget https://dl.k8s.io/v1.23.5/bin/linux/amd64/kubelet
[root@master1 install]# wget https://dl.k8s.io/v1.23.5/bin/linux/amd64/kuebadm
[root@master1 install]# chmod +x kubelet
[root@master1 install]# chmod +x kubadm
[root@master1 install]# mv -f kubelet /opt/k8s/bin/
[root@master1 install]# mv -f kubeadm /opt/k8s/bin/
[root@master1 install]# for node_ip in ${ALL_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p /opt/k8s/bin"
    ssh root@${node_ip} "mkdir -p /opt/k8s/kubelet"
    scp /opt/k8s/bin/{kubelet,kubeadm} root@${node_ip}:/opt/k8s/bin/
    ssh root@${node_ip} "chmod +x /opt/k8s/bin/*"
  done
>>> 192.168.66.131
kubelet               100%  116MB 228.8MB/s   00:00
kubeadm               100%   43MB 142.3MB/s   00:00
>>> 192.168.66.132
kubelet               100%  116MB 141.2MB/s   00:00
kubeadm               100%   43MB 166.8MB/s   00:00
>>> 192.168.66.133
kubelet               100%  116MB 149.4MB/s   00:00
kubeadm               100%   43MB 152.1MB/s   00:00
>>> 192.168.66.134
kubelet               100%  116MB 168.8MB/s   00:00
kubeadm               100%   43MB 155.3MB/s   00:00
>>> 192.168.66.135
kubelet               100%  116MB 132.2MB/s   00:00
kubeadm               100%   43MB 136.6MB/s   00:00
>>> 192.168.66.136
kubelet               100%  116MB 188.4MB/s   00:00
kubeadm               100%   43MB 129.5MB/s   00:00
[root@master1 install]#

这里下载单个的二级制文件，也可以一次下载整个版本的二进制文件，即：
wget dl.k8s.io/v1.23.5/kub…

二、创建和分发kubelet bootstrap kubeconfig文件

[root@master1 ~]# cd /opt/install/kubeconfig
[root@master1 kubeconfig]# for node_name in ${ALL_NAMES[@]}
  do
    echo ">>> ${node_name}"

    # 创建 token
    export BOOTSTRAP_TOKEN=$(kubeadm token create \
      --description kubelet-bootstrap-token \
      --groups system:bootstrappers:${node_name} \
      --kubeconfig ~/.kube/config)

    # 设置集群参数
    kubectl config set-cluster k8s-demo \
      --certificate-authority=/opt/install/cert/ca.pem \
      --embed-certs=true \
      --server=${KUBE_APISERVER} \
      --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig

    # 设置客户端认证参数
    kubectl config set-credentials kubelet-bootstrap \
      --token=${BOOTSTRAP_TOKEN} \
      --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig

    # 设置上下文参数
    kubectl config set-context default \
      --cluster=k8s-demo \
      --user=kubelet-bootstrap \
      --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig

    # 设置默认上下文
    kubectl config use-context default --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig
  done

分发到所有节点 3Master+3Node，方便监控每个节点的资源利用率

[root@master1 ~]# cd /opt/install/kubeconfig
[root@master1 kubeconfig]# for node_name in ${ALL_NAMES[@]}
  do
    echo ">>> ${node_name}"
    scp kubelet-bootstrap-${node_name}.kubeconfig root@${node_name}:/opt/k8s/etc/kubelet-bootstrap.kubeconfig
  done

三、创建和分发kubelet的参数配置文件

准备模板 kubelet-config.yaml.template

[root@master1 ~]# cd /opt/install/kubeconfig
[root@master1 kubeconfig]# cat > kubelet-config.yaml.template <<EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: "##NODE_IP##"
staticPodPath: ""
syncFrequency: 1m
fileCheckFrequency: 20s
httpCheckFrequency: 20s
staticPodURL: ""
port: 10250
readOnlyPort: 0
rotateCertificates: true
serverTLSBootstrap: true
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
  x509:
    clientCAFile: "/opt/k8s/etc/cert/ca.pem"
authorization:
  mode: Webhook
registryPullQPS: 0
registryBurst: 20
eventRecordQPS: 0
eventBurst: 20
enableDebuggingHandlers: true
enableContentionProfiling: true
healthzPort: 10248
healthzBindAddress: "##NODE_IP##"
clusterDomain: "${CLUSTER_DNS_DOMAIN}"
clusterDNS:
  - "${CLUSTER_DNS_SVC_IP}"
nodeStatusUpdateFrequency: 10s
nodeStatusReportFrequency: 1m
imageMinimumGCAge: 2m
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
volumeStatsAggPeriod: 1m
kubeletCgroups: ""
systemCgroups: ""
cgroupRoot: ""
cgroupsPerQOS: true
cgroupDriver: cgroupfs
runtimeRequestTimeout: 10m
hairpinMode: promiscuous-bridge
maxPods: 220
podCIDR: "${CLUSTER_CIDR}"
podPidsLimit: -1
resolvConf: /etc/resolv.conf
maxOpenFiles: 1000000
kubeAPIQPS: 1000
kubeAPIBurst: 2000
serializeImagePulls: false
evictionHard:
  memory.available:  "100Mi"
  nodefs.available:  "10%"
  nodefs.inodesFree: "5%"
  imagefs.available: "15%"
evictionSoft: {}
enableControllerAttachDetach: true
failSwapOn: true
containerLogMaxSize: 20Mi
containerLogMaxFiles: 10
systemReserved: {}
kubeReserved: {}
systemReservedCgroup: ""
kubeReservedCgroup: ""
enforceNodeAllocatable: ["pods"]
EOF

ddress：kubelet安全端口（https，10250）监听的地址，不能为127.0.0.1，否则kube-apiserver、heapster等不能调用kubelet的API
readOnlyPort=0：关闭只读端口(默认 10255)，等效为未指定
authentication.anonymous.enabled：设置为false，不允许匿名访问10250端口
authentication.x509.clientCAFile：指定签名客户端证书的CA证书，开启HTTP证书认证
authentication.webhook.enabled=true：开启HTTPS bearer token认证

[root@master1 ~]# cd /opt/install/kubeconfig
[root@master1 kubeconfig]# for node_ip in ${ALL_IPS[@]}
  do 
    echo ">>> ${node_ip}"
    sed -e "s/##NODE_IP##/${node_ip}/" kubelet-config.yaml.template > kubelet-config-${node_ip}.yaml.template
    scp kubelet-config-${node_ip}.yaml.template root@${node_ip}:/opt/k8s/etc/kubelet-config.yaml
  done

四、创建kubelet服务

1、拉取pause镜像并上传到私有仓库（公开项目）

[root@master1 ~]# docker pull rancher/pause:3.6
3.6: Pulling from rancher/pause
Digest: sha256:036d575e82945c112ef84e4585caff3648322a2f9ed4c3a6ce409dd10abc4f34
Status: Downloaded newer image for rancher/pause:3.6
docker.io/rancher/pause:3.6
[root@master1 ~]# docker tag rancher/pause:3.6 harbor.demo/k8s/pause:3.6
[root@master1 ~]# docker push harbor.demo/k8s/pause:3.6
The push refers to repository [harbor.demo/k8s/pause]
1021ef88c797: Layer already exists
3.6: digest: sha256:74bf6fc6be13c4ec53a86a5acf9fdbc6787b176db0693659ad6ac89f115e182c size: 526
[root@master1 ~]#

k8s系统镜像(pause、kube-proxy等)，即使同步到了本地私有镜像仓库harbor，也不要设为私有

2、创建kubelete服务模板文件

[root@master1 ~]# cd /opt/install/service
[root@master1 service]# cat > kubelet.service.template <<EOF
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=${K8S_DIR}/kubelet
ExecStart=/opt/k8s/bin/kubelet \\
  --bootstrap-kubeconfig=/opt/k8s/etc/kubelet-bootstrap.kubeconfig \\
  --cert-dir=/etc/kubernetes/cert \\
  --container-runtime=docker \\
  --container-runtime-endpoint=unix:///var/run/dockershim.sock \\
  --root-dir=${K8S_DIR}/kubelet \\
  --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\
  --config=/etc/kubernetes/kubelet-config.yaml \\
  --network-plugin=cni \\
  --hostname-override=##NODE_NAME## \\
  --pod-infra-container-image=harbor.demo/k8s/pause:3.6 \\
  --v=2
Restart=always
RestartSec=5
StartLimitInterval=0

[Install]
WantedBy=multi-user.target
EOF

--bootstrap-kubeconfig：指向bootstrap kubeconfig文件，kubelet使用该文件中的用户名和token向kube-apiserver发送TLS Bootstrapping请求
k8s approve kubelet的csr请求后，在--cert-dir目录创建证书和私钥文件，然后写入--kubeconfig文件
--pod-infra-container-image string 默认值：k8s.gcr.io/pause:3.2，其他 CRI 实现有自己的配置来设置此镜像
拉取容器镜像 docker pull rancher/pause:3.6 并推送到私有仓库 harbor.demo/k8s/pause-amd64:3.6
如果私有仓库拉取镜像需要用户名和密码，如何配置？？？

3、为所有节点创建kubelet service

[root@master1 ~]# cd /opt/install/service
[root@master1 service]# for node_name in ${ALL_NAMES[@]}
  do 
    echo ">>> ${node_name}"
    sed -e "s/##NODE_NAME##/${node_name}/" kubelet.service.template > kubelet-${node_name}.service
    scp kubelet-${node_name}.service root@${node_name}:/etc/systemd/system/kubelet.service
  done
[root@master1 service]#

4、把私服仓库harbor.demo的用户密码告诉k8s-demo(保存在secret中)

[root@master1 ~]# kubectl delete secret docker-registry harbor-demo-secret -n kube-system
[root@master1 ~]# kubectl create secret docker-registry harbor-demo-secret --docker-server=https://harbor.demo --docker-username=admin --docker-password=Harbor12345678 --docker-email=11461337@qq.com -n kube-system
[root@master1 ~]# kubectl delete secret docker-registry harbor-demo-secret -n kube-system
[root@master1 ~]# kubectl create secret docker-registry harbor-demo-secret --docker-server=https://harbor.demo --docker-username=admin --docker-password=Harbor12345678 --docker-email=11461337@qq.com -n kube-system
[root@master1 ~]#

五、启动kubelet服务

1、授予kube-apiserver访问kubelet API的权限

[root@master1 ~]# kubectl create clusterrolebinding k8s-demo-apiserver-kubelet-apis --clusterrole=system:kubelet-api-admin --user k8s-demo-apiserver
clusterrolebinding.rbac.authorization.k8s.io/kube-apiserver:kubelet-apis created

在执行kubectl exec、run、logs等命令时，apiserver会将请求转发到kubelet的https端口。这里定义RBAC规则，授权apiserver使用的证书（apiserver.pem）用户名（CN：k8s-demo-apiserver）访问kubelet API的权限：

2、Bootstrap Token Auth 和授予权限

kubelet启动时查找--kubeletconfig参数对应的文件是否存在，如果不存在则使用--bootstrap-kubeconfig指定的kubeconfig文件向kube-apiserver发送证书签名请求 (CSR)
kube-apiserver收到CSR请求后，对其中的Token进行认证，认证通过后将请求的user设置为system:bootstrap:<Token ID>，group 设置为system:bootstrappers，这一过程称为Bootstrap Token Auth
默认情况下，这个user和group没有创建CSR的权限，kubelet启动失败，错误日志如下：

[root@master1 ~]# journalctl -u kubelet -a |grep -A 2 'certificatesigningrequests'
May 26 12:13:41 zhangjun-k8s-01 kubelet[128468]: I0526 12:13:41.798230  128468 certificate_manager.go:366] Rotating certificates
May 26 12:13:41 zhangjun-k8s-01 kubelet[128468]: E0526 12:13:41.801997  128468 certificate_manager.go:385] Failed while requesting
a signed certificate from the master: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden:
 User "system:bootstrap:82jfrm" cannot create resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope

解决办法是：创建一个clusterrolebinding，将group system:bootstrappers和clusterrole system:node-bootstrapper绑定：

[root@master1 ~]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --group=system:bootstrappers
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created

3、自动approve CSR请求，生成kubelet client证书

创建三个ClusterRoleBinding，分别授予group system:bootstrappers和group system:nodes进行approve client、renew client、renew server证书的权限

[root@master1 ~]# cd /opt/install/yaml
[root@master1 yaml]# cat > csr-crb.yaml <<EOF
 # Approve all CSRs for the group "system:bootstrappers"
 kind: ClusterRoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
   name: auto-approve-csrs-for-group
 subjects:
 - kind: Group
   name: system:bootstrappers
   apiGroup: rbac.authorization.k8s.io
 roleRef:
   kind: ClusterRole
   name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
   apiGroup: rbac.authorization.k8s.io
---
 # To let a node of the group "system:nodes" renew its own credentials
 kind: ClusterRoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
   name: node-client-cert-renewal
 subjects:
 - kind: Group
   name: system:nodes
   apiGroup: rbac.authorization.k8s.io
 roleRef:
   kind: ClusterRole
   name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
   apiGroup: rbac.authorization.k8s.io
---
# A ClusterRole which instructs the CSR approver to approve a node requesting a
# serving cert matching its client cert.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: approve-node-server-renewal-csr
rules:
- apiGroups: ["certificates.k8s.io"]
  resources: ["certificatesigningrequests/selfnodeserver"]
  verbs: ["create"]
---
 # To let a node of the group "system:nodes" renew its own server credentials
 kind: ClusterRoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
   name: node-server-cert-renewal
 subjects:
 - kind: Group
   name: system:nodes
   apiGroup: rbac.authorization.k8s.io
 roleRef:
   kind: ClusterRole
   name: approve-node-server-renewal-csr
   apiGroup: rbac.authorization.k8s.io
EOF
[root@master1 yaml]# kubectl apply -f csr-crb.yaml
clusterrolebinding.rbac.authorization.k8s.io/auto-approve-csrs-for-group created
clusterrolebinding.rbac.authorization.k8s.io/node-client-cert-renewal created
clusterrole.rbac.authorization.k8s.io/approve-node-server-renewal-csr created
clusterrolebinding.rbac.authorization.k8s.io/node-server-cert-renewal created

auto-approve-csrs-for-group：自动approve node的第一次CSR；注意第一次CSR时，请求的Group为system:bootstrappers
node-client-cert-renewal：自动approve node后续过期的client证书，自动生成的证书Group为system:nodes
node-server-cert-renewal：自动approve node后续过期的server证书，自动生成的证书Group为system:nodes

4、手动approve server cert csr

-CSR approving controllers不会自动approve kubelet server证书签名请求，需要手动 approve

[root@node1 ~]# kubectl get csr | grep Pending | awk '{print $1}' | xargs kubectl certificate approve
[root@node1 ~]# kubectl get csr
NAME        AGE   SIGNERNAME                                    REQUESTOR                 REQUESTEDDURATION   CONDITION
csr-bh7b8   1s    kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:b292ax   <none>              Approved,Issued
csr-l9r9n   2s    kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:71syoy   <none>              Approved,Issued
csr-lkdxj   1s    kubernetes.io/kubelet-serving                 system:node:master2       <none>              Approved,Issued
csr-lz5qm   2s    kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:fb1z2g   <none>              Approved,Issued
csr-rw6pt   3s    kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:bkb8zo   <none>              Approved,Issued
csr-wkj55   2s    kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:dhwpgt   <none>              Approved,Issued
csr-xmlvk   1s    kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:ulskib   <none>              Approved,Issued
csr-zsx8m   1s    kubernetes.io/kubelet-serving                 system:node:master1       <none>              Approved,Issued

六、启动并检查kubelet服务的运行状态

1、启动kubelet服务

[root@master1 ~]# for node_ip in ${ALL_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "/usr/sbin/swapoff -a"
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kubelet && systemctl restart kubelet"
  done

启动服务前必须先创建工作目录
关闭swap分区，否则kubelet会启动失败

2、检查kubelet服务状态

[root@master1 ~]# kubectl get node -o wide
NAME      STATUS   ROLES    AGE     VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION                CONTAINER-RUNTIME
master1   Ready    <none>   3m26s   v1.23.5   192.168.66.131   <none>        CentOS Linux 7 (Core)   5.4.188-1.el7.elrepo.x86_64   docker://20.10.9
master2   Ready    <none>   3m26s   v1.23.5   192.168.66.132   <none>        CentOS Linux 7 (Core)   5.4.188-1.el7.elrepo.x86_64   docker://20.10.9
master3   Ready    <none>   3m26s   v1.23.5   192.168.66.133   <none>        CentOS Linux 7 (Core)   5.4.188-1.el7.elrepo.x86_64   docker://20.10.9
node1     Ready    <none>   3m25s   v1.23.5   192.168.66.134   <none>        CentOS Linux 7 (Core)   5.4.188-1.el7.elrepo.x86_64   docker://20.10.9
node2     Ready    <none>   3m25s   v1.23.5   192.168.66.135   <none>        CentOS Linux 7 (Core)   5.4.188-1.el7.elrepo.x86_64   docker://20.10.9
node3     Ready    <none>   3m25s   v1.23.5   192.168.66.136   <none>        CentOS Linux 7 (Core)   5.4.188-1.el7.elrepo.x86_64   docker://20.10.9

Pending的CSR用于创建kubelet server证书，需要手动approve
这里如果看不到信息，显示No Resources Found，可能以前有以前认证过的配置残留，尝试删除/opt/k8s/etc/kubelet-bootstrap.kubeconfig文件后，重启kubelet即可

3、kube-controller-manager为各node生成了kubeconfig文件和公私钥

[root@node1 ~]# ls -l /opt/k8s/etc/cert/kubelet.kubeconfig
-rw------- 1 root root 2304 4月  14 02:25 /opt/k8s/etc/cert/kubelet.kubeconfig

[root@node1 ~]# ls -l /opt/k8s/etc/cert/kubelet-*
-rw------- 1 root root 1248 4月  14 02:25 /opt/k8s/etc/cert/kubelet-client-2022-04-14-02-25-43.pem
lrwxrwxrwx 1 root root   56 4月  14 02:25 /opt/k8s/etc/cert/kubelet-client-current.pem -> /opt/k8s/etc/cert/kubelet-client-2022-04-14-02-25-43.pem
-rw------- 1 root root 1285 4月  14 02:25 /opt/k8s/etc/cert/kubelet-server-2022-04-14-02-25-45.pem
lrwxrwxrwx 1 root root   56 4月  14 02:25 /opt/k8s/etc/cert/kubelet-server-current.pem -> /opt/k8s/etc/cert/kubelet-server-2022-04-14-02-25-45.pem

4、查看节点状态

[root@node1 ~]# kubectl get csr | grep Pending | awk '{print $1}' | xargs kubectl certificate approve
certificatesigningrequest.certificates.k8s.io/csr-8f4kc approved
certificatesigningrequest.certificates.k8s.io/csr-mfvw8 approved
certificatesigningrequest.certificates.k8s.io/csr-ms8vq approved
certificatesigningrequest.certificates.k8s.io/csr-zbhwd approved
[root@master1 ~]# kubectl get nodes
NAME      STATUS   ROLES    AGE     VERSION
master1   Ready    <none>   7m23s   v1.23.5
master2   Ready    <none>   7m23s   v1.23.5
master3   Ready    <none>   7m23s   v1.23.5
node1     Ready    <none>   7m22s   v1.23.5
node2     Ready    <none>   7m22s   v1.23.5
node3     Ready    <none>   7m22s   v1.23.5
[root@master1 ~]# kubectl describe node master1
Name:               master1
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=master1
                    kubernetes.io/os=linux
Annotations:        node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Thu, 14 Apr 2022 02:25:43 +0800
Taints:             node-role.kubernetes.io/master:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  master1
  AcquireTime:     <unset>
  RenewTime:       Thu, 14 Apr 2022 21:29:40 +0800
... ... ... ... ...

5、给节点加上ROLES标签

[root@master1 ~]# kubectl label node master1 node-role.kubernetes.io/master=
[root@master1 ~]# kubectl label node master2 node-role.kubernetes.io/master=
[root@master1 ~]# kubectl label node master3 node-role.kubernetes.io/master=
[root@master1 ~]# kubectl label node node1 node-role.kubernetes.io/node=
[root@master1 ~]# kubectl label node node2 node-role.kubernetes.io/node=
[root@master1 ~]# kubectl label node node3 node-role.kubernetes.io/node=
[root@master1 ~]# kubectl get nodes -o wide
NAME      STATUS   ROLES    AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION                CONTAINER-RUNTIME
master1   Ready    master   1h   v1.23.5   192.168.66.131   <none>        CentOS Linux 7 (Core)   5.4.188-1.el7.elrepo.x86_64   docker://20.10.9
master2   Ready    master   1h   v1.23.5   192.168.66.132   <none>        CentOS Linux 7 (Core)   5.4.188-1.el7.elrepo.x86_64   docker://20.10.9
master3   Ready    master   1h   v1.23.5   192.168.66.133   <none>        CentOS Linux 7 (Core)   5.4.188-1.el7.elrepo.x86_64   docker://20.10.9
node1     Ready    node     1h   v1.23.5   192.168.66.134   <none>        CentOS Linux 7 (Core)   5.4.188-1.el7.elrepo.x86_64   docker://20.10.9
node2     Ready    node     1h   v1.23.5   192.168.66.135   <none>        CentOS Linux 7 (Core)   5.4.188-1.el7.elrepo.x86_64   docker://20.10.9
node3     Ready    node     1h   v1.23.5   192.168.66.136   <none>        CentOS Linux 7 (Core)   5.4.188-1.el7.elrepo.x86_64   docker://20.10.9
[root@master1 ~]#

如果要删除标签，只需要把“=”改为“-”

6、不允许调度Pod到Master节点

[root@node1 ~]# kubectl taint nodes master1 node-role.kubernetes.io/master=:NoSchedule
[root@node1 ~]# kubectl taint nodes master2 node-role.kubernetes.io/master=:NoSchedule
[root@node1 ~]# kubectl taint nodes master3 node-role.kubernetes.io/master=:NoSchedule

Taints 污点选项有
- NoSchedule: 一定不能被调度
- PreferNoSchedule: 尽量不要调度
- NoExecute: 不仅不会调度, 还会驱逐Node上已有的Pod

七、可能遇到的问题（错误信息）

"node sync has not completed yet"

"Error getting node" err="node "master1" not found"

"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials." lastCertificateAvailabilityTime="2022-04-20 23:01:22.569603407 +0800 CST m=+0.052198514" shutdownThreshold="5m0s"

Failed to contact API server when waiting for CSINode publishing: Unauthorized

"SyncLoop (housekeeping, skipped): sources aren't ready yet"

"Eviction manager: failed to get summary stats" err="failed to get node info: node "master1" not found"

error retrieving resource lock kube-system/kube-controller-manager: leases.coordination.k8s.io "kube-controller-manager" is forbidden: User "k8s-demo-ctrl-mgr" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-system"

1、在日志中找到出错的接口，执行wget命令，检查证书，例如

[root@master1 ~]#  wget https://127.0.0.1:8443/apis/storage.k8s.io/v1/csinodes/master1
--2022-04-25 22:33:27--  https://127.0.0.1:8443/apis/storage.k8s.io/v1/csinodes/master1
正在连接 127.0.0.1:8443... 已连接。
错误: 无法验证 127.0.0.1 的由 “/C=China/ST=GuangDong/L=ShenZhen/O=k8s-demo/OU=jason@vip.qq.com/CN=k8s-demo-ca” 颁发的证书:
  无法本地校验颁发者的权限。
要以不安全的方式连接至 127.0.0.1，使用“--no-check-certificate”。
[root@master1 ~]#

2、检查证书的时间，和内容，重点是 service-account 的证书

[root@master1 ~]# cd /opt/k8s/etc/cert/
[root@master1 cert]# ll
总用量 96
-rw------- 1 root root 1675 4月  25 22:18 aggregator-client-key.pem
-rw-r--r-- 1 root root 1452 4月  25 22:18 aggregator-client.pem
-rw------- 1 root root 1675 4月  25 22:22 apiserver-key.pem
-rw-r--r-- 1 root root 1724 4月  25 22:22 apiserver.pem
-rw-r--r-- 1 root root  472 4月  25 22:18 ca-config.json
-rw------- 1 root root 1679 4月  25 22:18 ca-key.pem
-rw-r--r-- 1 root root 1484 4月  19 00:02 calico-cni.crt
-rw-r--r-- 1 root root 1606 4月  19 00:02 calico-cni.csr
-rw-r--r-- 1 root root 3272 4月  19 00:02 calico-cni.key
-rw-r--r-- 1 root root 1371 4月  25 22:18 ca.pem
-rw------- 1 root root 1675 4月  25 22:22 controller-manager-key.pem
-rw-r--r-- 1 root root 1533 4月  25 22:22 controller-manager.pem
-rw------- 1 root root 1675 4月  25 22:18 kubectl-admin-key.pem
-rw-r--r-- 1 root root 1452 4月  25 22:18 kubectl-admin.pem
-rw------- 1 root root 1248 4月  25 22:36 kubelet-client-2022-04-25-22-36-04.pem
lrwxrwxrwx 1 root root   56 4月  25 22:36 kubelet-client-current.pem -> /opt/k8s/etc/cert/kubelet-client-2022-04-25-22-36-04.pem
-rw------- 1 root root 1285 4月  25 22:36 kubelet-server-2022-04-25-22-36-16.pem
lrwxrwxrwx 1 root root   56 4月  25 22:36 kubelet-server-current.pem -> /opt/k8s/etc/cert/kubelet-server-2022-04-25-22-36-16.pem
-rw------- 1 root root 1675 4月  25 22:18 kube-proxy-key.pem
-rw-r--r-- 1 root root 1452 4月  25 22:18 kube-proxy.pem
-rw------- 1 root root 1675 4月  25 22:35 scheduler-key.pem
-rw-r--r-- 1 root root 1521 4月  25 22:35 scheduler.pem
-rw------- 1 root root 1679 4月  25 22:18 service-account-key.pem
-rw-r--r-- 1 root root 1440 4月  25 22:18 service-account.pem
-rw-r--r-- 1 root root  451 4月  25 22:18 service-account.pub
[root@master1 cert]#  openssl x509  -noout -text -in ./apiserver.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            39:77:79:5a:63:9c:da:5a:01:09:3f:6f:cc:2a:db:51:6c:a2:4f:bb
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=China, ST=GuangDong, L=ShenZhen, O=k8s-demo, OU=jason@vip.qq.com, CN=k8s-demo-ca
        Validity
            Not Before: Apr 25 14:14:00 2022 GMT
            Not After : Apr 22 14:14:00 2032 GMT
        Subject: C=China, ST=GuangDong, L=ShenZhen, O=k8s-demo, OU=jason@vip.qq.com, CN=k8s-demo-apiserver
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:c1:16:10:a9:40:de:8e:97:9a:f4:0d:19:66:f8:
                    da:c0:ed:31:7c:e9:fe:1c:bc:76:33:95:ac:f8:a2:
                    83:08:e7:ca:af:f1:da:d6:36:6d:23:db:ca:79:32:
                    ad:d0:f3:54:57:ae:f1:45:1e:30:e5:0b:09:cc:df:
                    1c:68:81:f2:8a:a0:70:27:00:44:c5:05:73:18:bf:
                    28:92:00:bf:99:2d:2d:34:af:17:e6:6e:fa:77:5c:
                    e8:0b:dc:73:76:66:33:d9:3c:66:25:19:9e:39:28:
                    d8:68:c7:c9:ab:64:33:87:d1:dd:a2:20:ef:4d:a2:
                    d3:dc:68:aa:a9:95:96:2e:e9:12:93:13:04:02:cb:
                    cd:a1:c4:20:9d:8c:03:83:4e:3b:93:c5:7c:94:c2:
                    fe:75:ee:91:8a:6d:00:c4:b2:69:25:b0:ce:13:85:
                    eb:2c:b9:43:c1:29:35:ae:75:4e:d9:2c:8e:7a:38:
                    b2:b1:34:8d:35:df:ac:3d:4c:4c:02:f8:45:2f:48:
                    43:f9:86:f9:b2:ad:a8:6a:a9:ff:4d:93:d9:dd:00:
                    1d:5e:39:f5:77:95:d7:55:f0:19:43:6f:d2:67:86:
                    fc:35:fc:fe:6f:c1:ba:c4:a4:00:2e:5e:15:c4:c5:
                    f6:a5:d1:57:2f:4f:60:4f:6f:70:1a:83:dc:51:71:
                    d7:73
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                D9:44:8F:7C:88:EB:BC:04:E9:1F:AE:B1:F4:B5:64:AB:90:AF:AF:D4
            X509v3 Authority Key Identifier:
                keyid:74:67:F1:DB:4C:C6:08:45:2A:57:77:70:E8:DD:A8:89:19:57:1C:34

            X509v3 Subject Alternative Name:
                DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster, DNS:kubernetes.default.svc.cluster.k8sdemo, IP Address:192.168.66.131, IP Address:192.168.66.132, IP Address:192.168.66.133, IP Address:192.168.66.134, IP Address:192.168.66.135, IP Address:192.168.66.136, IP Address:10.66.0.1, IP Address:127.0.0.1
    Signature Algorithm: sha256WithRSAEncryption
         45:36:b4:d8:4c:81:b4:c6:11:43:93:66:3c:bb:a7:69:3e:b2:
         68:0a:ef:63:13:1c:6e:80:99:35:63:f2:0b:15:a7:c4:27:0c:
         ff:ea:7e:c3:63:24:9c:2e:00:6f:30:67:ad:bd:a8:2b:42:6a:
         ce:52:25:86:5b:1b:6f:0b:04:17:96:f8:6f:a6:6d:19:c5:3d:
         6f:44:76:f8:50:5e:b5:85:a9:24:d3:f3:35:6b:cc:eb:fa:73:
         45:58:44:47:09:c1:19:91:75:36:a6:a3:3b:62:00:37:f9:a2:
         c8:75:11:dd:27:f2:54:be:3d:a8:18:68:bc:0a:c3:31:53:90:
         8e:d3:17:d7:fc:8f:16:67:53:fa:de:35:b9:7d:65:2d:25:16:
         42:8e:7f:e3:00:d4:da:58:5a:29:9a:28:41:64:95:11:c2:4c:
         f0:de:94:a6:8c:99:53:f9:08:89:dd:ec:46:c2:2d:4c:40:3e:
         fb:0d:7c:01:11:8f:da:b0:92:ed:03:4e:6b:9b:a6:0e:50:a1:
         48:fd:04:de:77:28:e4:13:2f:b0:09:9c:b7:75:9c:e1:6b:05:
         1c:fe:b0:03:6d:a1:c6:cc:a8:05:29:9d:0a:42:4c:91:6f:7f:
         b0:47:f4:df:36:65:30:24:02:9c:83:ea:f5:7f:6d:b2:4b:48:
         3e:bc:9f:79
[root@master1 cert]#

3、检查 service-account 证书位置信息是否添加到 kube-apiserver 启动参数中

--service-account-issuer=api
--service-account-key-file=/opt/k8s/etc/cert/service-account.pub
--service-account-signing-key-file=/opt/k8s/etc/cert/service-account-key.pem

参考

先用起来，通过操作实践认识kubernetes(k8s)，积累多了自然就理解了
把理解的知识分享出来，自造福田，自得福缘
追求简单，容易使人理解，知识的上下文也是知识的一部分，例如版本，时间等
欢迎留言交流，也可以提出问题，一般在周末回复和完善文档
Jason@vip.qq.com 2022-4-14

k8s-demo集群搭建详细步骤14：部署ipvs模式的kube-kubelet组件v1.23.5

一、下载 kubelet

1、获取下载地址

2、下载 kubelet、kubeadm 并复制到所有节点

二、创建和分发kubelet bootstrap kubeconfig文件

三、创建和分发kubelet的参数配置文件

四、创建kubelet服务

1、拉取pause镜像并上传到私有仓库（公开项目）

2、创建kubelete服务模板文件

3、为所有节点创建kubelet service

4、把私服仓库harbor.demo的用户密码告诉k8s-demo(保存在secret中)

五、启动kubelet服务

1、授予kube-apiserver访问kubelet API的权限

2、Bootstrap Token Auth 和 授予权限

3、自动approve CSR请求，生成kubelet client证书

4、手动approve server cert csr

六、启动并检查kubelet服务的运行状态

1、启动kubelet服务

2、检查kubelet服务状态

3、kube-controller-manager为各node生成了kubeconfig文件和公私钥

4、查看节点状态

5、给节点加上ROLES标签

6、不允许调度Pod到Master节点

七、可能遇到的问题（错误信息）

1、在日志中找到出错的接口，执行wget命令，检查证书，例如

2、检查证书的时间，和内容，重点是 service-account 的证书

3、检查 service-account 证书位置信息是否添加到 kube-apiserver 启动参数中

参考

2、Bootstrap Token Auth 和授予权限