008.Kubernetes二进制master节点部署高可用

一 kube-apiserver高可用

1.1 Keepalived实现VIP

Keepalived可以提供kube-apiserver VIP，配合Nginx实现kube-apiserver的高可用。

1.2 Nginx实现反向代理

基于 nginx 代理的 kube-apiserver 高可用方案。

控制节点的 kube-controller-manager、kube-scheduler 是多实例部署，所以只要有一个实例正常，就可以保证高可用；

集群内的 Pod 使用 K8S 服务域名 kubernetes 访问 kube-apiserver， kube-dns 会自动解析出多个 kube-apiserver 节点的 VIP，所以也是高可用的；

在每个节点起一个 nginx 进程，后端对接多个 apiserver 实例，nginx 对它们做健康检查和负载均衡；

kubelet、kube-proxy、controller-manager、scheduler 通过本地的 nginx（监听 172.24.8.100）访问 kube-apiserver，从而实现 kube-apiserver 的高可用；

基于 nginx 4 层透明代理功能实现 K8S 节点( master 节点和 worker 节点)高可用访问 kube-apiserver 。

二 Kubernetes高可用部署

2.1 Keepalived安装

  1 [root@master01 ~]# for master_ip in ${MASTER_IPS[@]}
  2   do
  3     echo ">>> ${master_ip}"
  4     ssh ${master_ip} "mkdir -p /opt/k8s/kube-keepalived/"
  5     ssh ${master_ip} "mkdir -p /etc/keepalived/"
  6   done						#创建keepalived目录
  7 [root@master01 ~]# cd /opt/k8s/work
  8 [root@master01 work]# wget http://down.linuxsb.com:8888/software/keepalived-2.0.20.tar.gz
  9 [root@master01 work]# tar -zxvf keepalived-2.0.20.tar.gz
 10 [root@master01 work]# cd keepalived-2.0.20/ && ./configure --sysconf=/etc --prefix=/opt/k8s/kube-keepalived/ && make && make install

提示：本步骤操作仅需要在master01节点操作。

2.2 分发Keepalived二进制文件

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
  4   do
  5     echo ">>> ${master_ip}"
  6     scp -rp /opt/k8s/kube-keepalived/ root@${master_ip}:/opt/k8s/
  7     scp -rp /usr/lib/systemd/system/keepalived.service  root@${master_ip}:/usr/lib/systemd/system/
  8     ssh ${master_ip} "systemctl daemon-reload && systemctl enable keepalived"
  9   done						#分发Keepalived二进制文件

提示：本步骤操作仅需要在master01节点操作。

2.3 Nginx安装

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# wget http://nginx.org/download/nginx-1.19.0.tar.gz
  3 [root@master01 work]# tar -xzvf nginx-1.19.0.tar.gz
  4 [root@master01 work]# cd /opt/k8s/work/nginx-1.19.0/
  5 [root@master01 nginx-1.19.0]# mkdir nginx-prefix
  6 [root@master01 nginx-1.19.0]# ./configure --with-stream --without-http --prefix=$(pwd)/nginx-prefix --without-http_uwsgi_module --without-http_scgi_module --without-http_fastcgi_module
  7 [root@master01 nginx-1.19.0]# make && make install

解释：

--with-stream：开启 4 层透明转发(TCP Proxy)功能；

--without-xxx：关闭所有其他功能，这样生成的动态链接二进制程序依赖最小。

  1 [root@master01 nginx-1.19.0]# ./nginx-prefix/sbin/nginx -v

提示：本步骤操作仅需要在master01节点操作。

2.4 验证编译后的Nginx

  1 [root@master01 ~]# cd /opt/k8s/work/nginx-1.19.0/
  2 [root@master01 nginx-1.19.0]# ./nginx-prefix/sbin/nginx -v
  3 nginx version: nginx/1.19.0
  4 [root@master01 nginx-1.19.0]# ldd ./nginx-prefix/sbin/nginx	#查看 nginx 动态链接的库
  5         linux-vdso.so.1 =>  (0x00007ffe7f596000)
  6         libdl.so.2 => /lib64/libdl.so.2 (0x00007f1df0fb8000)
  7         libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1df0d9c000)
  8         libc.so.6 => /lib64/libc.so.6 (0x00007f1df09ce000)
  9         /lib64/ld-linux-x86-64.so.2 (0x00007f1df11bc000)

提示：由于只开启了 4 层透明转发功能，所以除了依赖 libc 等操作系统核心 lib 库外，没有对其它 lib 的依赖(如 libz、libssl 等)，以便达到精简编译的目的。

2.5 分发Nginx二进制文件

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
  4   do
  5     echo ">>> ${master_ip}"
  6     ssh root@${master_ip} "mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}"
  7     scp /opt/k8s/work/nginx-1.19.0/nginx-prefix/sbin/nginx root@${master_ip}:/opt/k8s/kube-nginx/sbin/kube-nginx
  8     ssh root@${master_ip} "chmod a+x /opt/k8s/kube-nginx/sbin/*"
  9   done						#分发Nginx二进制文件

提示：本步骤操作仅需要在master01节点操作。

2.6 配置Nginx system

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# cat > kube-nginx.service <<EOF
  4 [Unit]
  5 Description=kube-apiserver nginx proxy
  6 After=network.target
  7 After=network-online.target
  8 Wants=network-online.target
  9 
 10 [Service]
 11 Type=forking
 12 ExecStartPre=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -t
 13 ExecStart=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx
 14 ExecReload=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -s reload
 15 PrivateTmp=true
 16 Restart=always
 17 RestartSec=5
 18 StartLimitInterval=0
 19 LimitNOFILE=65536
 20 
 21 [Install]
 22 WantedBy=multi-user.target
 23 EOF

提示：本步骤操作仅需要在master01节点操作。

2.7 分发Nginx systemd

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
  4   do
  5     echo ">>> ${master_ip}"
  6     scp kube-nginx.service  root@${master_ip}:/etc/systemd/system/
  7     ssh ${master_ip} "systemctl daemon-reload && systemctl enable kube-nginx.service"
  8   done

提示：本步骤操作仅需要在master01节点操作。

2.8 创建配置文件

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# wget http://down.linuxsb.com:8888/binngkek8s.sh	#拉取自动部署脚本
  4 [root@master01 work]# vi binngkek8s.sh  #其他部分保持默认

  1 #!/bin/sh
  2 #****************************************************************#
  3 # ScriptName: ngkek8s.sh
  4 # Author: xhy
  5 # Create Date: 2020-05-13 16:32
  6 # Modify Author: xhy
  7 # Modify Date: 2020-05-30 13:24
  8 # Version: v2
  9 #***************************************************************#
 10 
 11 #######################################
 12 # set variables below to create the config files, all files will create at ./config directory
 13 #######################################
 14 
 15 # master keepalived virtual ip address
 16 export K8SHA_VIP=172.24.8.100
 17 
 18 # master01 ip address
 19 export K8SHA_IP1=172.24.8.71
 20 
 21 # master02 ip address
 22 export K8SHA_IP2=172.24.8.72
 23 
 24 # master03 ip address
 25 export K8SHA_IP3=172.24.8.73
 26 
 27 # master01 hostname
 28 export K8SHA_HOST1=master01
 29 
 30 # master02 hostname
 31 export K8SHA_HOST2=master02
 32 
 33 # master03 hostname
 34 export K8SHA_HOST3=master03
 35 
 36 # master01 network interface name
 37 export K8SHA_NETINF1=eth0
 38 
 39 # master02 network interface name
 40 export K8SHA_NETINF2=eth0
 41 
 42 # master03 network interface name
 43 export K8SHA_NETINF3=eth0
 44 
 45 # keepalived auth_pass config
 46 export K8SHA_KEEPALIVED_AUTH=412f7dc3bfed32194d1600c483e10ad1d
 47 
 48 # kubernetes CIDR pod subnet
 49 export K8SHA_PODCIDR=10.10.0.0
 50 
 51 # kubernetes CIDR svc subnet
 52 export K8SHA_SVCCIDR=10.20.0.0

  1 [root@master01 work]# chmod u+x *.sh
  2 [root@master01 work]# ./binngkek8s.sh

解释：如上仅需Master01节点操作。执行binngkek8s.sh脚本后，会自动生成以下配置文件：

• keepalived：keepalived配置文件，位于各个master节点的/etc/keepalived目录
• nginx-lb：nginx-lb负载均衡配置文件，位于各个master节点的/opt/k8s/kube-nginx/conf/kube-nginx.conf目录

三 启动高可用

3.1 确认配置

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
  4   do
  5     echo ">>> ${master_ip}"
  6     echo ">>>> check check sh"
  7     ssh root@${master_ip} "ls -l /etc/keepalived/check_apiserver.sh"
  8     echo ">>> check Keepalived config"
  9     ssh root@${master_ip} "cat /etc/keepalived/keepalived.conf"
 10     echo ">>> check Nginx config"
 11     ssh root@${master_ip} "cat /opt/k8s/kube-nginx/conf/kube-nginx.conf"
 12   done			#检查高可用相关配置

提示：本步骤操作仅需要在master01节点操作。

3.2 启动服务

  1 [root@master01 ~]# cd /opt/k8s/work
  2 [root@master01 work]# source /root/environment.sh
  3 [root@master01 work]# for master_ip in ${MASTER_IPS[@]}
  4   do
  5     echo ">>> ${master_ip}"
  6     ssh root@${master_ip} "systemctl restart keepalived.service && systemctl enable keepalived.service"
  7     ssh root@${master_ip} "systemctl restart kube-nginx.service && systemctl enable kube-nginx.service"
  8     ssh root@${master_ip} "systemctl status keepalived.service | grep Active"
  9     ssh root@${master_ip} "systemctl status kube-nginx.service | grep Active"
 10     ssh root@${master_ip} "netstat -tlunp | grep 16443"
 11   done

提示：本步骤操作仅需要在master01节点操作。

3.3 确认验证

  1 3.3	确认验证
  2 [root@master01 ~]# cd /opt/k8s/work
  3 [root@master01 ~]# cd /opt/k8s/work
  4 [root@master01 work]# source /root/environment.sh
  5 [root@master01 work]# for all_ip in ${ALL_IPS[@]}
  6   do
  7     echo ">>> ${all_ip}"
  8     ssh root@${all_ip} "ping -c1 172.24.8.100"
  9   done							#等待20s左右执行检查

提示：本步骤操作仅需要在master01节点操作。