本文已参与「新人创作礼」活动，一起开启掘金创作之路

在3个Master节点上部署kube-controller-manager，启动后将通过竞争选举机制产生一个leader节点，其它节点为阻塞状态

一、下载并复制kube-controller-manager到3个Master节点

1、获取下载地址

微信图片_20220409193922.png

2、下载 kube-controller-manager 并复制到3个Master节点

[root@master1 ~]# cd /opt/install/
[root@master1 install]# wget https://dl.k8s.io/v1.23.5/bin/linux/amd64/kube-controller-manager
[root@master1 install]# chmod +x kube-controller-manager
[root@master1 install]# mv -f kube-controller-manager /opt/k8s/bin/
[root@master1 install]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp /opt/k8s/bin/kube-controller-manager root@${node_ip}:/opt/k8s/bin/
    ssh root@${node_ip} "chmod +x /opt/k8s/bin/kube-controller-manager"
  done
>>> 192.168.66.131
kube-controller-manager           100%  116MB 228.8MB/s   00:00
>>> 192.168.66.132
kube-controller-manager           100%  116MB 141.2MB/s   00:00
>>> 192.168.66.133
kube-controller-manager           100%  116MB 149.4MB/s   00:00
[root@master1 install]#

二、创建和分发kubeconfig文件

1、设置集群：k8s-demo（server 是apiserver的地址）

[root@master1 ~]# cd /opt/install/kubeconfig
[root@master1 kubeconfig]# kubectl config set-cluster k8s-demo \
  --certificate-authority=/opt/k8s/etc/cert/ca.pem \
  --embed-certs=true \
  --server="https://##NODE_IP##:6443" \
  --kubeconfig=controller-manager.kubeconfig

2、设置用户名：k8s-demo-ctrl-mgr

[root@master1 kubeconfig]# kubectl config set-credentials k8s-demo-ctrl-mgr \
  --client-certificate=/opt/k8s/etc/cert/controller-manager.pem \
  --client-key=/opt/k8s/etc/cert/controller-manager-key.pem \
  --embed-certs=true \
  --kubeconfig=controller-manager.kubeconfig

3、设置上下文（用户组？）：system:kube-controller-manager

[root@master1 kubeconfig]# kubectl config set-context system:kube-controller-manager \
  --cluster=k8s-demo --user=k8s-demo-ctrl-mgr \
  --kubeconfig=controller-manager.kubeconfig

4、切换上下文

[root@master1 kubeconfig]# kubectl config use-context system:kube-controller-manager \
  --kubeconfig=controller-manager.kubeconfig

5、分发controller-manager.kubeconfig到三个Master节点

[root@master1 ~]# cd /opt/install/kubeconfig
[root@master1 kubeconfig]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    sed -e "s/##NODE_IP##/${node_ip}/" controller-manager.kubeconfig > controller-manager-${node_ip}.kubeconfig
    scp controller-manager-${node_ip}.kubeconfig root@${node_ip}:/opt/k8s/etc/controller-manager.kubeconfig
  done
>>> 192.168.66.131
controller-manager-192.168.66.131.kubeconfig      100% 6514     4.3MB/s   00:00
>>> 192.168.66.132
controller-manager-192.168.66.132.kubeconfig      100% 6514     3.5MB/s   00:00
>>> 192.168.66.133
controller-manager-192.168.66.133.kubeconfig      100% 6514     5.6MB/s   00:00
[root@master1 kubeconfig]#

6、用户授权

由于证书中没有用系统默认的用户名 system:kube-controller-manager，而是自定义了一个用户名k8s-demo-ctrl-mgr，所以要为该用户绑定Role和ClusterRole

[root@master1 ~]# kubectl create clusterrolebinding k8s-demo-ctrl-mgr --clusterrole=system:kube-controller-manager --user k8s-demo-ctrl-mgr
clusterrolebinding.rbac.authorization.k8s.io/k8s-demo-ctrl-mgr created
[root@master1 ~]# kubectl create rolebinding k8s-demo-ctrl-mgr-rolebinding --role=system::leader-locking-kube-controller-manager --user k8s-demo-ctrl-mgr --namespace=kube-system
rolebinding.rbac.authorization.k8s.io/k8s-demo-ctrl-mgr-rolebinding created
[root@master1 ~]# kubectl create rolebinding k8s-demo-ctrl-mgr-rolebinding-ext --role=extension-apiserver-authentication-reader --user k8s-demo-ctrl-mgr --namespace=kube-system
rolebinding.rbac.authorization.k8s.io/k8s-demo-ctrl-mgr-rolebinding-ext created

对应的删除命令
kubectl delete clusterrolebinding k8s-demo-ctrl-mgr
kubectl delete rolebinding k8s-demo-ctrl-mgr-rolebinding
kubectl delete rolebinding k8s-demo-ctrl-mgr-rolebinding-ext

三、创建kube-controller-manager systemd unit并分发到3个Master节点

1、编写kube-controller-manager systemd unit模板

参考 kubernetes.io/zh/docs/ref…

[root@master1 ~]# cd /opt/install/service
[root@master1 service]# cat > controller-manager.service.template <<EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
WorkingDirectory=${K8S_DIR}/kube-controller-manager
ExecStart=/opt/k8s/bin/kube-controller-manager \\
  --profiling \\
  --cluster-name=k8s-demo \\
  --controllers=*,bootstrapsigner,tokencleaner \\
  --kube-api-qps=1000 \\
  --kube-api-burst=2000 \\
  --leader-elect=true \\
  --use-service-account-credentials=true \\
  --concurrent-service-syncs=2 \\
  --master=https://127.0.0.1:8443 \\
  --bind-address=##NODE_IP## \\
  --secure-port=10252 \\
  --tls-cert-file=/opt/k8s/etc/cert/controller-manager.pem \\
  --tls-private-key-file=/opt/k8s/etc/cert/controller-manager-key.pem \\
  --authentication-kubeconfig=/opt/k8s/etc/controller-manager.kubeconfig \\
  --authorization-kubeconfig=/opt/k8s/etc/controller-manager.kubeconfig \\
  --kubeconfig=/opt/k8s/etc/controller-manager.kubeconfig \\
  --client-ca-file=/opt/k8s/etc/cert/ca.pem \\
  --requestheader-allowed-names="k8s-demo-aggregator" \\
  --requestheader-client-ca-file=/opt/k8s/etc/cert/ca.pem \\
  --requestheader-extra-headers-prefix="X-Remote-Extra-" \\
  --requestheader-group-headers=X-Remote-Group \\
  --requestheader-username-headers=X-Remote-User \\
  --cluster-signing-cert-file=/opt/k8s/etc/cert/ca.pem \\
  --cluster-signing-key-file=/opt/k8s/etc/cert/ca-key.pem \\
  --cluster-signing-duration=87600h \\
  --horizontal-pod-autoscaler-sync-period=10s \\
  --concurrent-deployment-syncs=10 \\
  --concurrent-gc-syncs=30 \\
  --node-cidr-mask-size=24 \\
  --allocate-node-cidrs=true \\
  --cluster-cidr=${CLUSTER_CIDR} \\
  --service-cluster-ip-range=${SERVICE_CIDR} \\
  --pod-eviction-timeout=6m \\
  --terminated-pod-gc-threshold=10000 \\
  --root-ca-file=/opt/k8s/etc/cert/ca.pem \\
  --service-account-private-key-file=/opt/k8s/etc/cert/ca-key.pem \\
  --cloud-provider= \\
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

2、为每个Master节点生成部署文件

[root@master1 ~]# cd /opt/install/service
[root@master1 service]# for (( i=0; i < 3; i++ ))
  do
    sed -e "s/##NODE_NAME##/${MASTER_NAMES[i]}/" -e "s/##NODE_IP##/${MASTER_IPS[i]}/" controller-manager.service.template > controller-manager-${MASTER_IPS[i]}.service 
  done
[root@master1 service]# ls -l controller-manager*
-rw-r--r-- 1 root root 1828 4月   9 19:31 controller-manager-192.168.66.131.service
-rw-r--r-- 1 root root 1828 4月   9 19:31 controller-manager-192.168.66.132.service
-rw-r--r-- 1 root root 1828 4月   9 19:31 controller-manager-192.168.66.133.service
-rw-r--r-- 1 root root 1825 4月   9 19:28 controller-manager.service.template

3、分发到master节点，

[root@master1 ~]# cd /opt/install/service
[root@master1 service]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp controller-manager-${node_ip}.service root@${node_ip}:/etc/systemd/system/kube-controller-manager.service
  done
>>> 192.168.66.131
controller-manager-192.168.66.131.service         100% 1828     1.8MB/s   00:00
>>> 192.168.66.132
controller-manager-192.168.66.132.service         100% 1828   755.2KB/s   00:00
>>> 192.168.66.133
controller-manager-192.168.66.133.service         100% 1828     1.0MB/s   00:00
[root@master1 service]#

四、启动kube-controller-manager集群服务

1、启动kube-controller-manager服务，端口10252

[root@master1 ~]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p ${K8S_DIR}/kube-controller-manager"
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl restart kube-controller-manager"
  done
>>> 192.168.66.131
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /etc/systemd/system/kube-controller-manager.service.
>>> 192.168.66.132
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /etc/systemd/system/kube-controller-manager.service.
>>> 192.168.66.133
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /etc/systemd/system/kube-controller-manager.service.
[root@master1 ~]# ss -lnpt | grep kube-controller-manager
LISTEN  0   4096   192.168.66.131:10252  *:* users:(("kube-controller",pid=1900,fd=7))

2、检查服务状态

[root@master1 ~]# for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl status kube-controller-manager|grep Active"
  done
>>> 192.168.66.131
   Active: active (running) since 六 2022-04-09 19:48:33 CST; 2min 9s ago
>>> 192.168.66.132
   Active: active (running) since 六 2022-04-09 19:48:34 CST; 2min 8s ago
>>> 192.168.66.133
   Active: active (running) since 六 2022-04-09 19:48:34 CST; 2min 8s ago
[root@master1 service]#

3、健康检查

[root@master1 ~]# curl -s --cacert /opt/k8s/etc/cert/ca.pem --cert /opt/k8s/etc/cert/kubectl-admin.pem --key /opt/k8s/etc/cert/kubectl-admin-key.pem https://192.168.66.131:10252/healthz
# 或者
[root@master1 ~]# wget  https://192.168.66.131:10252/healthz --no-check-certificate

4、遇到异常情况，可以查看日志

[root@master1 ~]# journalctl -u kube-controller-manager

五、查看leader和metrics信息

1、metrics信息

[root@master1 ~]# curl -s --cacert /opt/k8s/etc/cert/ca.pem --cert /opt/install/cert/kubectl-admin.pem  --key /opt/k8s/etc/cert/kubectl-admin-key.pem https://192.168.66.131:10252/metrics |head
# HELP apiserver_audit_event_total [ALPHA] Counter of audit events generated and sent to the audit backend.
# TYPE apiserver_audit_event_total counter
apiserver_audit_event_total 0
# HELP apiserver_audit_requests_rejected_total [ALPHA] Counter of apiserver requests rejected due to an error in audit logging backend.
# TYPE apiserver_audit_requests_rejected_total counter
apiserver_audit_requests_rejected_total 0
# HELP apiserver_client_certificate_expiration_seconds [ALPHA] Distribution of the remaining lifetime on the certificate used to authenticate a request.
# TYPE apiserver_client_certificate_expiration_seconds histogram
apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="1800"} 0
[root@master1 ~]#

2、kube-controller-manager的Leader是哪个？

[root@master1 ~]# kubectl get leases -n kube-system
NAME                      HOLDER                                         AGE
kube-controller-manager   master1_68336263-1f96-4a33-929b-7b997f17002b   27h
kube-scheduler            master1_fad25ad7-2dd2-40ef-ba5f-84315bb9243d   69m

先用起来，通过操作实践认识kubernetes(k8s)，积累多了自然就理解了
把理解的知识分享出来，自造福田，自得福缘
追求简单，容易使人理解，知识的上下文也是知识的一部分，例如版本，时间等
欢迎留言交流，也可以提出问题，一般在周末回复和完善文档
Jason@vip.qq.com 2022-4-12

k8s-demo集群搭建详细步骤12：部署kube-controller-manager v1.23.5高可用集群

一、下载并复制kube-controller-manager到3个Master节点

1、获取下载地址

2、下载 kube-controller-manager 并复制到3个Master节点

二、创建和分发kubeconfig文件

1、设置集群：k8s-demo（server 是apiserver的地址）

2、设置用户名：k8s-demo-ctrl-mgr

3、设置上下文（用户组？）：system:kube-controller-manager

4、切换上下文

5、分发controller-manager.kubeconfig到三个Master节点

6、用户授权

三、创建kube-controller-manager systemd unit并分发到3个Master节点

1、编写kube-controller-manager systemd unit模板

2、为每个Master节点生成部署文件

3、分发到master节点，

四、启动kube-controller-manager集群服务

1、启动kube-controller-manager服务，端口10252

2、检查服务状态

3、健康检查

4、遇到异常情况，可以查看日志

五、查看leader和metrics信息

1、metrics信息

2、kube-controller-manager的Leader是哪个？