k8s-demo集群搭建详细步骤13：部署ipvs模式的kube-proxy组件v1.23.5

Jason2020
182 阅读4分钟

本文已参与「新人创作礼」活动，一起开启掘金创作之路

kube-proxy运行在所有worker节点上，它监听apiserver中service和endpoint的变化情况，创建路由规则以提供服务IP和负载均衡功能
参数说明 kubernetes.io/zh/docs/ref…

一、下载 kube-proxy

1、获取下载地址

微信图片_20220409193922.png

2、下载 kube-proxy 并复制到所有节点

[root@master1 ~]# cd /opt/install/
[root@master1 install]# wget https://dl.k8s.io/v1.23.5/bin/linux/amd64/kube-proxy
--2022-04-13 23:57:44--  https://dl.k8s.io/v1.23.5/bin/linux/amd64/kube-proxy
正在解析主机 dl.k8s.io (dl.k8s.io)... 34.107.204.206, 2600:1901:0:26f3::
正在连接 dl.k8s.io (dl.k8s.io)|34.107.204.206|:443... 已连接。
已发出 HTTP 请求，正在等待回应... 302 Moved Temporarily
位置：https://storage.googleapis.com/kubernetes-release/release/v1.23.5/bin/linux/amd64/kube-proxy [跟随至新的 URL]
--2022-04-13 23:57:47--  https://storage.googleapis.com/kubernetes-release/release/v1.23.5/bin/linux/amd64/kube-proxy
正在解析主机 storage.googleapis.com (storage.googleapis.com)... 142.251.42.240, 172.217.163.48, 172.217.160.80, ...
正在连接 storage.googleapis.com (storage.googleapis.com)|142.251.42.240|:443... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度：44163072 (42M) [application/octet-stream]
正在保存至: “kube-proxy”
100%[=================================>] 44,163,072  10.3MB/s 用时 17s

2022-04-13 23:58:07 (2.44 MB/s) - 已保存 “kube-proxy” [44163072/44163072])

[root@master1 install]# chmod +x kube-proxy
[root@master1 install]# mv -f kube-proxy /opt/k8s/bin/
[root@master1 install]# for node_ip in ${ALL_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp /opt/k8s/bin/kube-proxy root@${node_ip}:/opt/k8s/bin/
    ssh root@${node_ip} "chmod +x /opt/k8s/bin/kube-proxy"
    ssh root@${node_ip} "mkdir -p /opt/k8s/kube-proxy"
  done
>>> 192.168.66.131
kube-proxy                100%  116MB 228.8MB/s   00:00
>>> 192.168.66.132
kube-proxy                100%  116MB 141.2MB/s   00:00
>>> 192.168.66.133
kube-proxy                100%  116MB 149.4MB/s   00:00
[root@master1 install]#

二、创建和分发kube-proxy的kubeconfig文件

[root@master1 ~]# cd /opt/install/kubeconfig
[root@master1 kubeconfig]# kubectl config set-cluster k8s-demo \
  --certificate-authority=/opt/install/cert/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-proxy.kubeconfig

[root@master1 kubeconfig]# kubectl config set-credentials k8s-demo-kube-proxy \
  --client-certificate=/opt/install/cert/kube-proxy.pem \
  --client-key=/opt/install/cert/kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig
  
[root@master1 kubeconfig]# kubectl config set-context system:kube-proxy \
  --cluster=k8s-demo --user=k8s-demo-kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig

[root@master1 kubeconfig]# kubectl config use-context system:kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig

[root@master1 kubeconfig]# for node_name in ${ALL_NAMES[@]}
  do
    echo ">>> ${node_name}"
    scp kube-proxy.kubeconfig root@${node_name}:/opt/k8s/etc/
  done

三、创建kube-proxy参数配置文件

1、创建模板文件kube-proxy-config.yaml.template

[root@master1 ~]# cd /opt/install/kubeconfig
[root@master1 kubeconfig]# cat > kube-proxy-config.yaml.template <<EOF
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1beta2
clientConnection:
  burst: 200
  kubeconfig: "/opt/k8s/etc/kube-proxy.kubeconfig"
  qps: 100
bindAddress: ##NODE_IP##
healthzBindAddress: ##NODE_IP##:10256
metricsBindAddress: ##NODE_IP##:10249
enableProfiling: true
clusterCIDR: ${CLUSTER_CIDR}
hostnameOverride: ##NODE_NAME##
mode: "ipvs"
portRange: ""
iptables:
  masqueradeAll: false
ipvs:
  scheduler: rr
  excludeCIDRs: []
EOF
  • hostnameOverride：参数值必须与kubelet的值一致，否则kube-proxy启动后会找不到该Node，从而不会创建任何ipvs规则
  • clusterCIDR：kube-proxy根据--cluster-cidr判断集群内部和外部流量，指定--cluster-cidr 或 --masquerade-all 选项后 kube-proxy 才会对访问Service IP的请求做SNAT

2、创建和分发kube-proxy参数配置文件

[root@master1 ~]# cd /opt/install/kubeconfig
[root@master1 kubeconfig]# for (( i=0; i < 6; i++ ))
  do 
    echo ">>> ${ALL_NAMES[i]}"
    sed -e "s/##NODE_NAME##/${ALL_NAMES[i]}/" -e "s/##NODE_IP##/${ALL_IPS[i]}/" kube-proxy-config.yaml.template > kube-proxy-config-${ALL_NAMES[i]}.yaml.template
    scp kube-proxy-config-${ALL_NAMES[i]}.yaml.template root@${ALL_NAMES[i]}:/opt/k8s/etc/kube-proxy-config.yaml
  done

四、创建kube-proxy服务

[root@master1 ~]# cd /opt/install/service
[root@master1 service]# cat > kube-proxy.service <<EOF
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
WorkingDirectory=${K8S_DIR}/kube-proxy
ExecStart=/opt/k8s/bin/kube-proxy \\
  --config=/opt/k8s/etc/kube-proxy-config.yaml \\
  --logtostderr=true \\
  --v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

[root@master1 service]# for node_name in ${ALL_NAMES[@]}
  do 
    echo ">>> ${node_name}"
    scp kube-proxy.service root@${node_name}:/etc/systemd/system/
  done

五、启动并验证各Node节点上的kube-proxy服务

1、创建授权信息，kube-proxy.pem证书中的CN为k8s-demo-kube-proxy

[root@master1 ~]# kubectl create clusterrolebinding k8s-demo-cluster-proxy-binding --clusterrole=system:node-proxier  --user=k8s-demo-kube-proxy

2、启动kube-proxy服务

[root@master1 ~]# for node_ip in ${ALL_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "modprobe ip_vs_rr"
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-proxy && systemctl restart kube-proxy"
  done

3、查看kube-proxy服务状态和端口信息

[root@master1 ~]# for node_ip in ${ALL_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl status kube-proxy|grep Active"
    ssh root@${node_ip} "ss -lnpt | grep kube-proxy"
  done
>>> 192.168.66.131
   Active: active (running) since 四 2022-04-14 00:33:35 CST; 1s ago
LISTEN     0      4096   192.168.66.131:10249   *:*     users:(("kube-proxy",pid=12162,fd=17))
LISTEN     0      4096   192.168.66.131:10256   *:*     users:(("kube-proxy",pid=12162,fd=16))
>>> 192.168.66.132
   Active: active (running) since 四 2022-04-14 00:33:35 CST; 1s ago
LISTEN     0      4096   192.168.66.132:10249   *:*     users:(("kube-proxy",pid=8546,fd=13))
LISTEN     0      4096   192.168.66.132:10256   *:*     users:(("kube-proxy",pid=8546,fd=12))
>>> 192.168.66.133
   Active: active (running) since 四 2022-04-14 00:33:35 CST; 1s ago
LISTEN     0      4096   192.168.66.133:10249   *:*     users:(("kube-proxy",pid=8549,fd=17))
LISTEN     0      4096   192.168.66.133:10256   *:*     users:(("kube-proxy",pid=8549,fd=16))
>>> 192.168.66.134
   Active: active (running) since 四 2022-04-14 00:33:36 CST; 1s ago
LISTEN     0      4096   192.168.66.134:10249   *:*     users:(("kube-proxy",pid=4261,fd=11))
LISTEN     0      4096   192.168.66.134:10256   *:*     users:(("kube-proxy",pid=4261,fd=12))
>>> 192.168.66.135
   Active: active (running) since 四 2022-04-14 00:33:36 CST; 1s ago
LISTEN     0      4096   192.168.66.135:10249   *:*     users:(("kube-proxy",pid=4130,fd=12))
LISTEN     0      4096   192.168.66.135:10256   *:*     users:(("kube-proxy",pid=4130,fd=11))
>>> 192.168.66.136
   Active: active (running) since 四 2022-04-14 00:33:36 CST; 1s ago
LISTEN     0      4096   192.168.66.136:10249   *:*     users:(("kube-proxy",pid=4132,fd=13))
LISTEN     0      4096   192.168.66.136:10256   *:*     users:(("kube-proxy",pid=4132,fd=11))
  • 如果状态不是active (running)，则要查看日志确认原因：
[root@node1 ~]# journalctl -u kube-proxy
  • 查看ipvs路由规则
[root@master1 ~]# for node_ip in ${ALL_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "/usr/sbin/ipvsadm -ln"
  done
>>> 192.168.66.131
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.66.0.1:443 rr
  -> 192.168.66.131:6443          Masq    1      0          0
  -> 192.168.66.132:6443          Masq    1      0          0
  -> 192.168.66.133:6443          Masq    1      0          0
>>> 192.168.66.132
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.66.0.1:443 rr
  -> 192.168.66.131:6443          Masq    1      0          0
  -> 192.168.66.132:6443          Masq    1      0          0
  -> 192.168.66.133:6443          Masq    1      0          0
>>> 192.168.66.133
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.66.0.1:443 rr
  -> 192.168.66.131:6443          Masq    1      0          0
  -> 192.168.66.132:6443          Masq    1      0          0
  -> 192.168.66.133:6443          Masq    1      0          0
>>> 192.168.66.134
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.66.0.1:443 rr
  -> 192.168.66.131:6443          Masq    1      0          0
  -> 192.168.66.132:6443          Masq    1      0          0
  -> 192.168.66.133:6443          Masq    1      0          0
>>> 192.168.66.135
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.66.0.1:443 rr
  -> 192.168.66.131:6443          Masq    1      0          0
  -> 192.168.66.132:6443          Masq    1      0          0
  -> 192.168.66.133:6443          Masq    1      0          0
>>> 192.168.66.136
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.66.0.1:443 rr
  -> 192.168.66.131:6443          Masq    1      0          0
  -> 192.168.66.132:6443          Masq    1      0          0
  -> 192.168.66.133:6443          Masq    1      0          0
[root@master1 ~]#
  • 先用起来，通过操作实践认识kubernetes(k8s)，积累多了自然就理解了
  • 把理解的知识分享出来，自造福田，自得福缘
  • 追求简单，容易使人理解，知识的上下文也是知识的一部分，例如版本，时间等
  • 欢迎留言交流，也可以提出问题，一般在周末回复和完善文档
  • Jason@vip.qq.com 2022-4-13
avatar
文章
阅读
粉丝