k8s-demo集群搭建详细步骤08：通过Nginx 1.21.6实现kube-apiserver服务高可用

Jason2020
790 阅读4分钟

本文已参与「新人创作礼」活动，一起开启掘金创作之路

3个master节点部署了apiserver，只要有一个节点的apiserver实例正常，就可以保证集群apiserver可用
下面在每个节点部署一个nginx进程，后端对接3个Master节点的apiserver实例，nginx对它们做健康检查和负载均衡，即高可用

一、下载和编译Nginx

[root@master1 ~]# yum install -y gcc make openssl-devel
... ... ... ... ...
已安装:
  gcc.x86_64 0:4.8.5-44.el7                                                                  openssl-devel.x86_64 1:1.0.2k-25.el7_9

作为依赖被安装:
  cpp.x86_64 0:4.8.5-44.el7                        glibc-devel.x86_64 0:2.17-325.el7_9          glibc-headers.x86_64 0:2.17-325.el7_9           kernel-headers.x86_64 0:3.10.0-1160.62.1.el7
  keyutils-libs-devel.x86_64 0:1.5.8-3.el7         krb5-devel.x86_64 0:1.15.1-51.el7_9          libcom_err-devel.x86_64 0:1.42.9-19.el7         libkadm5.x86_64 0:1.15.1-51.el7_9
  libmpc.x86_64 0:1.0.1-3.el7                      libselinux-devel.x86_64 0:2.5-15.el7         libsepol-devel.x86_64 0:2.5-10.el7              libverto-devel.x86_64 0:0.2.5-4.el7
  mpfr.x86_64 0:3.1.1-4.el7                        pcre-devel.x86_64 0:8.32-17.el7              zlib-devel.x86_64 0:1.2.7-19.el7_9

完毕！
[root@master1 ~]# cd /opt/install/soft
[root@master1 soft]# wget http://nginx.org/download/nginx-1.21.6.tar.gz
[root@master1 soft]# tar -xzvf nginx-1.21.6.tar.gz
[root@master1 soft]# cd /opt/install/nginx-1.21.6
[root@master1 nginx-1.21.6]# mkdir nginx-prefix
[root@master1 nginx-1.21.6]# ./configure --with-stream --with-stream_ssl_preread_module --with-stream_ssl_module --without-http --without-http_uwsgi_module --without-http_scgi_module --without-http_fastcgi_module --prefix=$(pwd)/nginx-prefix
... ... ... ... ...
creating objs/Makefile

Configuration summary
  + PCRE library is not used
  + using system OpenSSL library
  + zlib library is not used

  nginx path prefix: "/opt/install/soft/nginx-1.21.6/nginx-prefix"
  nginx binary file: "/opt/install/soft/nginx-1.21.6/nginx-prefix/sbin/nginx"
  nginx modules path: "/opt/install/soft/nginx-1.21.6/nginx-prefix/modules"
  nginx configuration prefix: "/opt/install/soft/nginx-1.21.6/nginx-prefix/conf"
  nginx configuration file: "/opt/install/soft/nginx-1.21.6/nginx-prefix/conf/nginx.conf"
  nginx pid file: "/opt/install/soft/nginx-1.21.6/nginx-prefix/logs/nginx.pid"
  nginx error log file: "/opt/install/soft/nginx-1.21.6/nginx-prefix/logs/error.log"
  nginx http access log file: "/opt/install/soft/nginx-1.21.6/nginx-prefix/logs/access.log"
  nginx http client request body temporary files: "client_body_temp"
  nginx http proxy temporary files: "proxy_temp"

[root@master1 nginx-1.18.0]# make && make install  #等待编译完成，验证版本
... ... ... ... ...
cp conf/nginx.conf '/opt/install/soft/nginx-1.21.6/nginx-prefix/conf/nginx.conf.default'
test -d '/opt/install/soft/nginx-1.21.6/nginx-prefix/logs' \
        || mkdir -p '/opt/install/soft/nginx-1.21.6/nginx-prefix/logs'
test -d '/opt/install/soft/nginx-1.21.6/nginx-prefix/logs' \
        || mkdir -p '/opt/install/soft/nginx-1.21.6/nginx-prefix/logs'
test -d '/opt/install/soft/nginx-1.21.6/nginx-prefix/html' \
        || cp -R html '/opt/install/soft/nginx-1.21.6/nginx-prefix'
test -d '/opt/install/soft/nginx-1.21.6/nginx-prefix/logs' \
        || mkdir -p '/opt/install/soft/nginx-1.21.6/nginx-prefix/logs'
make[1]: 离开目录“/opt/install/soft/nginx-1.21.6”
[root@master1 nginx-1.21.6]# ./nginx-prefix/sbin/nginx -v
nginx version: nginx/1.21.6
[root@master1 nginx-1.21.6]# ./nginx-prefix/sbin/nginx -V
nginx version: nginx/1.21.6
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC)
built with OpenSSL 1.0.2k-fips  26 Jan 2017
TLS SNI support enabled
configure arguments: --with-stream --with-stream_ssl_preread_module --with-stream_ssl_module --without-http --without-http_uwsgi_module --without-http_scgi_module --without-http_fastcgi_module --prefix=/opt/install/soft/nginx-1.21.6/nginx-prefix
  • --with-stream：开启四层透明转发(TCP Proxy)功能
  • --with-stream_ssl_preread_module
  • --with-stream_ssl_module
  • --without-http：关闭HTTP代理

二、安装和部署nginx

1、nginx配置

[root@master1 nginx-1.21.6]# cat > kube-nginx.conf << \EOF
worker_processes 1;

events {
    worker_connections  1024;
}

stream {
    upstream apiserver{
        hash $remote_addr consistent;
        server 192.168.66.131:6443        max_fails=3 fail_timeout=30s;
        server 192.168.66.132:6443        max_fails=3 fail_timeout=30s;
        server 192.168.66.133:6443        max_fails=3 fail_timeout=30s;
    }

    server {
        listen 127.0.0.1:8443;
        proxy_connect_timeout 1s;
        proxy_pass apiserver;
    }
    
    upstream scheduler {
        hash $remote_addr consistent;
        server 192.168.66.131:10251        max_fails=3 fail_timeout=30s;
        server 192.168.66.132:10251        max_fails=3 fail_timeout=30s;
        server 192.168.66.133:10251        max_fails=3 fail_timeout=30s;
    }

    server {
        listen 127.0.0.1:10259;
        proxy_connect_timeout 1s;
        proxy_pass scheduler;
    }
    
    upstream controller {
        hash $remote_addr consistent;
        server 192.168.66.131:10252        max_fails=3 fail_timeout=30s;
        server 192.168.66.132:10252        max_fails=3 fail_timeout=30s;
        server 192.168.66.133:10252        max_fails=3 fail_timeout=30s;
    }

    server {
        listen 127.0.0.1:10257;
        proxy_connect_timeout 1s;
        proxy_pass controller;
        proxy_ssl on;
        proxy_ssl_certificate     /opt/k8s/etc/cert/kubectl-admin.pem;
        proxy_ssl_certificate_key /opt/k8s/etc/cert/kubectl-admin-key.pem;
    }
}
EOF

2、分发可执行程序到所有节点

[root@master1 ~]# for node_ip in ${ALL_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}"
    scp /opt/install/soft/nginx-1.21.6/nginx-prefix/sbin/nginx  root@${node_ip}:/opt/k8s/kube-nginx/sbin/kube-nginx
    scp /opt/install/soft/nginx-1.21.6/kube-nginx.conf  root@${node_ip}:/opt/k8s/kube-nginx/conf/kube-nginx.conf
    ssh root@${node_ip} "chmod a+x /opt/k8s/kube-nginx/sbin/*"
  done
>>> 192.168.66.131
nginx                     100% 3242KB 214.1MB/s   00:00
kube-nginx.conf           100% 1461   872.0KB/s   00:00
>>> 192.168.66.132
nginx                     100% 3242KB 122.0MB/s   00:00
kube-nginx.conf           100% 1461     1.4MB/s   00:00
>>> 192.168.66.133
nginx                     100% 3242KB 129.4MB/s   00:00
kube-nginx.conf           100% 1461   504.9KB/s   00:00
>>> 192.168.66.134
nginx                     100% 3242KB 116.6MB/s   00:00
kube-nginx.conf           100% 1461   728.9KB/s   00:00
>>> 192.168.66.135
nginx                     100% 3242KB  99.6MB/s   00:00
kube-nginx.conf           100% 1461     1.3MB/s   00:00
>>> 192.168.66.136
nginx                     100% 3242KB 102.3MB/s   00:00
kube-nginx.conf           100% 1461   552.0KB/s   00:00
[root@master1 nginx-1.21.6]#

三、配置nginx服务

1、配置Kube-nginx服务的systemd unit文件

[root@master1 ~]# cd /opt/install/service
[root@master1 service]# cat > kube-nginx.service <<EOF
[Unit]
Description=kube-apiserver nginx proxy
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=forking
ExecStartPre=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -t
ExecStart=/opt/k8s/kube-nginx/sbin/kube-nginx    -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx
ExecReload=/opt/k8s/kube-nginx/sbin/kube-nginx   -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -s reload
PrivateTmp=true
Restart=always
RestartSec=5
StartLimitInterval=0
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

2、分发systemd unit文件

[root@master1 ~]# cd /opt/install/service
[root@master1 service]# for node_ip in ${ALL_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp kube-nginx.service  root@${node_ip}:/etc/systemd/system/
  done
>>> 192.168.66.131
kube-nginx.service          100%  629   812.8KB/s   00:00
>>> 192.168.66.132
kube-nginx.service          100%  629   292.5KB/s   00:00
>>> 192.168.66.133
kube-nginx.service          100%  629   531.6KB/s   00:00
>>> 192.168.66.134
kube-nginx.service          100%  629   524.5KB/s   00:00
>>> 192.168.66.135
kube-nginx.service          100%  629   303.7KB/s   00:00
>>> 192.168.66.136
kube-nginx.service          100%  629   236.0KB/s   00:00
[root@master1 service]#

3、启动kube-nginx服务

[root@master1 ~]# for node_ip in ${ALL_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-nginx && systemctl restart kube-nginx"
  done
>>> 192.168.66.131
>>> 192.168.66.132
>>> 192.168.66.133
>>> 192.168.66.134
>>> 192.168.66.135
>>> 192.168.66.136

4、检查服务状态

[root@master1 ~]# for node_ip in ${ALL_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl status kube-nginx |grep 'Active:'"
  done
>>> 192.168.66.131
   Active: active (running) since 六 2022-04-09 12:55:20 CST; 22s ago
>>> 192.168.66.132
   Active: active (running) since 六 2022-04-09 12:55:21 CST; 21s ago
>>> 192.168.66.133
   Active: active (running) since 六 2022-04-09 12:55:21 CST; 21s ago
>>> 192.168.66.134
   Active: active (running) since 六 2022-04-09 12:55:21 CST; 21s ago
>>> 192.168.66.135
   Active: active (running) since 六 2022-04-09 12:55:21 CST; 21s ago
>>> 192.168.66.136
   Active: active (running) since 六 2022-04-09 12:55:21 CST; 21s ago
[root@master1 cert]#

5、查看kube-nginx服务日志

[root@master1 ~]# journalctl -u kube-nginx
-- Logs begin at 六 2022-04-09 09:01:02 CST, end at 六 2022-04-09 12:55:43 CST. --
4月 09 12:46:31 master1 systemd[1]: Starting kube-apiserver nginx proxy...
4月 09 12:46:31 master1 kube-nginx[8018]: nginx: the configuration file /opt/k8s/kube-nginx/conf/kube-nginx.conf syntax is ok
4月 09 12:46:31 master1 kube-nginx[8018]: nginx: configuration file /opt/k8s/kube-nginx/conf/kube-nginx.conf test is successful
4月 09 12:46:31 master1 systemd[1]: Started kube-apiserver nginx proxy.
4月 09 12:48:34 master1 systemd[1]: Stopping kube-apiserver nginx proxy...
4月 09 12:48:34 master1 systemd[1]: Stopped kube-apiserver nginx proxy.
4月 09 12:48:34 master1 systemd[1]: Starting kube-apiserver nginx proxy...
4月 09 12:48:34 master1 kube-nginx[8074]: nginx: the configuration file /opt/k8s/kube-nginx/conf/kube-nginx.conf syntax is ok
4月 09 12:48:34 master1 kube-nginx[8074]: nginx: configuration file /opt/k8s/kube-nginx/conf/kube-nginx.conf test is successful
4月 09 12:48:34 master1 systemd[1]: Started kube-apiserver nginx proxy.
4月 09 12:55:20 master1 systemd[1]: Stopping kube-apiserver nginx proxy...
4月 09 12:55:20 master1 systemd[1]: Stopped kube-apiserver nginx proxy.
4月 09 12:55:20 master1 systemd[1]: Starting kube-apiserver nginx proxy...
4月 09 12:55:20 master1 kube-nginx[8153]: nginx: the configuration file /opt/k8s/kube-nginx/conf/kube-nginx.conf syntax is ok
4月 09 12:55:20 master1 kube-nginx[8153]: nginx: configuration file /opt/k8s/kube-nginx/conf/kube-nginx.conf test is successful
4月 09 12:55:20 master1 systemd[1]: Started kube-apiserver nginx proxy.
[root@master1 ~]#
  • 先用起来，通过操作实践认识kubernetes(k8s)，积累多了自然就理解了
  • 把理解的知识分享出来，自造福田，自得福缘
  • 追求简单，容易使人理解，知识的上下文也是知识的一部分，例如版本，时间等
  • 欢迎留言交流，也可以提出问题，一般在周末回复和完善文档
  • Jason@vip.qq.com 2022-4-9
avatar
文章
阅读
粉丝