静态 Pod 方式部署 kubernetes 控制平面

胜似代码仔
122 阅读5分钟

静态 pod 与 systemd 部署 kubernetes 控制平面的对比

静态 pod 优势

  • 自动重启和健康检查
  • 更容易进行版本升级和回滚
  • 资源隔离性更好
  • 与 Kubernetes 的管理方式保持一致
  • 便于使用容器化的方式管理控制平面组件

systemd 优势

  • 部署和配置更简单直接
  • 与传统的 Linux 系统管理方式一致
  • 故障排查可能更直接
  • 系统级别的服务依赖管理更容易

ssl 证书准备

证书相关的一般放置在 /etc/kubernetes/pki 目录下

  • 创建根证书
  1. 创建 cfssl 根证书请求文件 root-ca-csr.json
{
    "CN": "Root CA",
    "key": {
        "algo": "rsa",
        "size": 4096
    },
    "names": [
        {
            "C": "CN",
            "ST": "Guangdong",
            "L": "Guangzhou",
            "O": "Chillcy",
            "OU": "Technology"
        }
    ],
    "ca": {
        "expiry": "87600h",             
        "pathlen": 1       
    }
}
  1. 根据 root-ca-csr.json 文件生成根证书
sudo cfssl gencert -initca root-ca-csr.json | sudo cfssljson -bare root-ca
  • 签发 kubernetes CA 证书
  1. 创建 cfssl 证书配置文件 config.json
{
    "signing": {
        "default": {
            "expiry": "8760h"
        },
        "profiles": {
            "kubernetes-ca": {
                "usages": [
                    "signing",
                    "digital signature",
                    "key encipherment",
                    "cert sign",
                    "crl sign"
                ],
                "expiry": "43800h",
                "ca_constraint": {
                    "is_ca": true,
                    "max_path_len": 0
                }
            },
            "kubernetes": {
                "usages": [
                    "digital signature",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ],
                "expiry": "8760h"
            }
        }
    }
}
  1. 创建 kubernetes CA 证书请求文件 kubernetes-ca-csr.json
{
    "CN": "Kubernetes CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "Guangdong",
            "L": "Guangzhou",
            "O": "Chillcy",
            "OU": "technology"
        }
    ]
}
  1. 根据 kubernetes-ca-csr.json 生成 kubernetes CA 证书
sudo cfssl genkey -initca kubernetes-ca-csr.json | sudo cfssljson -bare kubernetes-ca
  1. 使用根 CA 证书签发 kubernetes CA 证书
sudo cfssl sign -ca=root-ca.pem -ca-key=root-ca-key.pem -config=config.json -profile=kubernetes-ca kubernetes-ca.csr | sudo cfssljson -bare kubernetes-ca
  • 创建 kubernetes 集群管理员证书
  1. 创建 kubernetes 集群管理员证书请求文件 kubernetes-csr.json
{
    "CN": "kubernetes",
    "hosts": [
        "localhost",
        "127.0.0.1",
        "10.0.30.47",
        "172.16.0.1"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "Guangdong",
            "L": "Guangzhou",
            "O": "system:masters",
            "OU": "technology"
        }
    ]
}
  1. 根据 kubernetes-csr.json 创建 kubernetes 集群管理员证书
sudo cfssl gencert -ca=kubernetes-ca.pem -ca-key=kubernetes-ca-key.pem -config=config.json -profile=kubernetes kubernetes-csr.json | sudo cfssljson -bare kubernetes
  • 创建 CA 证书链
sudo sh -c 'cat kubernetes-ca.pem root-ca.pem > kubernetes-chain.pem'
  • 创建 kubelet 证书
  1. 创建 kubelet 证书请求文件 kubelet-plane-csr.json
{
  "CN": "system:node:srv-k8s-plane",
  "hosts": [
    "srv-k8s-plane",
    "localhost",
    "127.0.0.1",
    "10.0.30.47"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Guangdong",
      "L": "Guangzhou",
      "O": "system:nodes",
      "OU": "technology"
    }
  ]
}
  1. 根据 kubelet-plane-csr.json 创建 kubelet 证书
sudo cfssl gencert -ca=kubernetes-ca.pem -ca-key=kubernetes-ca-key.pem -config=config.json -profile=kubernetes kubelet-plane-csr.json | sudo cfssljson -bare kubelet-plane

kubectl 安装

juejin.cn/post/747304…

kubeconfig 文件准备

sudo kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/kubernetes-chain.pem --embed-certs=true --server=https://10.0.30.47:6443 --kubeconfig=/home/we8/.kube/config

sudo kubectl config set-credentials kubernetes --client-certificate=/etc/kubernetes/pki/kubernetes.pem --client-key=/etc/kubernetes/pki/kubernetes-key.pem --embed-certs=true --kubeconfig=/home/we8/.kube/config

sudo kubectl config set-context kubernetes@kubernetes --cluster=kubernetes --user=kubernetes --kubeconfig=/home/we8/.kube/config

sudo kubectl config use-context kubernetes@kubernetes --kubeconfig=/home/we8/.kube/config

docker 安装

juejin.cn/post/744708…

cri-docker 安装

juejin.cn/post/744744…

kubelet 安装

juejin.cn/post/744750… kubelet 配置文件 /etc/kubernetes/cfg/kubelet.yml

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
cgroupDriver: systemd
clusterDNS:
- 172.16.0.101
clusterDomain: cluster.local
tlsCertFile: /etc/kubernetes/pki/kubelet-plane.pem
tlsPrivateKeyFile: /etc/kubernetes/pki/kubelet-plane-key.pem
containerRuntimeEndpoint: "unix:///var/run/cri-dockerd.sock"
staticPodPath: "/etc/kubernetes/manifests"
registerWithTaints:
- key: node-role.kubernetes.io/control-plane
  effect: NoSchedule

生成 kubelet 专属的 kubeconfig

sudo kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/kubernetes-chain.pem --embed-certs=true  --server=https://10.0.30.47:6443 --kubeconfig=/etc/kubernetes/kubeconfig/kubelet-plane

sudo kubectl config set-credentials system:node:srv-k8s-plane --client-certificate=/etc/kubernetes/pki/kubelet-plane.pem --client-key=/etc/kubernetes/pki/kubelet-plane-key.pem --embed-certs=true  --kubeconfig=/etc/kubernetes/kubeconfig/kubelet-plane

sudo kubectl config set-context kubernetes@system:node:srv-k8s-plane --cluster=kubernetes --user=system:node:srv-k8s-plane --kubeconfig=/etc/kubernetes/kubeconfig/kubelet-plane

sudo kubectl config use-context kubernetes@system:node:srv-k8s-plane --kubeconfig=/etc/kubernetes/kubeconfig/kubelet-plane

控制平面各组件配置清单文件

清单文件一般放在 /etc/kubernetes/manifests 目录下

etcd.yaml

apiVersion: v1
kind: Pod
metadata:
  name: etcd
  namespace: kube-system
spec:
  hostNetwork: true
  containers:
  - name: etcd
    image: k8s.gcr.io/etcd:3.5.15-0
    command:
    - etcd
    - --advertise-client-urls=https://10.0.30.47:2379
    - --cert-file=/etc/kubernetes/pki/kubernetes.pem
    - --client-cert-auth=true
    - --data-dir=/var/lib/etcd
    - --initial-advertise-peer-urls=https://10.0.30.47:2380
    - --initial-cluster=master=https://10.0.30.47:2380
    - --key-file=/etc/kubernetes/pki/kubernetes-key.pem
    - --listen-client-urls=https://10.0.30.47:2379
    - --listen-peer-urls=https://10.0.30.47:2380
    - --name=master
    - --peer-cert-file=/etc/kubernetes/pki/kubernetes.pem
    - --peer-client-cert-auth=true
    - --peer-key-file=/etc/kubernetes/pki/kubernetes-key.pem
    - --peer-trusted-ca-file=/etc/kubernetes/pki/kubernetes-chain.pem
    - --trusted-ca-file=/etc/kubernetes/pki/kubernetes-chain.pem
    volumeMounts:
    - mountPath: /var/lib/etcd
      name: etcd-data
    - mountPath: /etc/kubernetes/pki
      name: etcd-certs
  volumes:
  - hostPath:
      path: /var/lib/etcd
      type: DirectoryOrCreate
    name: etcd-data
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: etcd-certs

kube-apiserver.yaml

apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  namespace: kube-system
spec:
  hostNetwork: true   # 添加这一行
  containers:
  - name: kube-apiserver
    image: k8s.gcr.io/kube-apiserver:v1.31.1
    command:
    - kube-apiserver
    - --advertise-address=10.0.30.47
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/kubernetes-chain.pem
    - --etcd-cafile=/etc/kubernetes/pki/kubernetes-chain.pem
    - --etcd-certfile=/etc/kubernetes/pki/kubernetes.pem
    - --etcd-keyfile=/etc/kubernetes/pki/kubernetes-key.pem
    - --etcd-servers=https://10.0.30.47:2379
    - --kubelet-client-certificate=/etc/kubernetes/pki/kubernetes.pem
    - --kubelet-client-key=/etc/kubernetes/pki/kubernetes-key.pem
    - --tls-cert-file=/etc/kubernetes/pki/kubernetes.pem
    - --tls-private-key-file=/etc/kubernetes/pki/kubernetes-key.pem
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local
    - --service-account-key-file=/etc/kubernetes/pki/kubernetes.pem
    - --service-account-signing-key-file=/etc/kubernetes/pki/kubernetes-key.pem
    - --service-cluster-ip-range=172.16.0.0/16
    - --requestheader-client-ca-file=/etc/kubernetes/pki/kubernetes-chain.pem
    - --proxy-client-cert-file=/etc/kubernetes/pki/kubernetes.pem
    - --proxy-client-key-file=/etc/kubernetes/pki/kubernetes-key.pem
    - --v=2
    volumeMounts:
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs

kube-controller-manager.yaml

apiVersion: v1
kind: Pod
metadata:
  name: kube-controller-manager
  namespace: kube-system
spec:
  hostNetwork: true
  containers:
  - name: kube-controller-manager
    image: k8s.gcr.io/kube-controller-manager:v1.31.1
    command:
    - kube-controller-manager
    - --authentication-kubeconfig=/home/we8/.kube/config
    - --authorization-kubeconfig=/home/we8/.kube/config
    - --bind-address=127.0.0.1
    - --client-ca-file=/etc/kubernetes/pki/kubernetes-chain.pem
    - --cluster-signing-cert-file=/etc/kubernetes/pki/kubernetes.pem
    - --cluster-signing-key-file=/etc/kubernetes/pki/kubernetes-key.pem
    - --kubeconfig=/home/we8/.kube/config
    - --root-ca-file=/etc/kubernetes/pki/kubernetes-chain.pem
    - --service-account-private-key-file=/etc/kubernetes/pki/kubernetes-key.pem
    - --allocate-node-cidrs=true
    - --cluster-cidr=10.244.0.0/16
    - --node-cidr-mask-size=24
    - --service-cluster-ip-range=10.96.0.0/12
    - --leader-elect=true
    - --requestheader-client-ca-file=/etc/kubernetes/pki/kubernetes-chain.pem
    volumeMounts:
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
    - mountPath: /home/we8/.kube
      name: kubeconfig
      readOnly: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /home/we8/.kube
      type: DirectoryOrCreate
    name: kubeconfig

kube-scheduler.yaml

apiVersion: v1
kind: Pod
metadata:
  name: kube-scheduler
  namespace: kube-system
spec:
  hostNetwork: true
  containers:
  - name: kube-scheduler
    image: k8s.gcr.io/kube-scheduler:v1.31.1
    command:
    - kube-scheduler
    - --authentication-kubeconfig=/home/we8/.kube/config
    - --authorization-kubeconfig=/home/we8/.kube/config
    - --bind-address=127.0.0.1
    - --kubeconfig=/home/we8/.kube/config
    volumeMounts:
    - mountPath: /home/we8/.kube
      name: kubeconfig
      readOnly: true
  volumes:
  - hostPath:
      path: /home/we8/.kube
      type: DirectoryOrCreate
    name: kubeconfig

开始部署

kubelet 会自动监听 staticPod 指定的目录下的配置文件，但是为了立即生效，我们还是重启 kubelet

sudo systemctl restart kubelet

查看部署结果

kubectl get pods -A
kube-system   etcd-srv-k8s-plane                      1/1     Running   8          19m
kube-system   kube-apiserver-srv-k8s-plane            1/1     Running   89         18m
kube-system   kube-controller-manager-srv-k8s-plane   1/1     Running   0          2m58s
kube-system   kube-scheduler-srv-k8s-plane            1/1     Running   0          2m40s
kubectl get nodes -A
NAME            STATUS     ROLES    AGE   VERSION
srv-k8s-plane   NotReady   <none>   19m   v1.31.3

各组件正常运行，但是节点状态是 NotReady，因为集群 CNI 插件未正常运行，接下来进行集群 CNI 插件的安装

avatar
文章
阅读
粉丝