ubuntu 22.04安装k8s完整流程参考： Kubernetes 官网文档：Kubernetes 官网文档（很重

参考：

Kubernetes 官网文档：Kubernetes 官网文档（很重要，里面更完整也有演示）

www.cnblogs.com/wangguishe/…

www.cnblogs.com/guangdelw/p…

1. 集群基础设置

1.1. 版本配置

OS： ubuntu 22.04

kubernetes： 1.28.3

Container Runtime：Docker 24.0.7

CRI: cri-docker v0.3.7

Kubernetes自v1.24移除了对docker-shim的支持，而Docker Engine默认又不支持CRI规范，因而二者将无法直接完成整合。

为此，Mirantis和Docker联合创建了cri-dockerd项目，用于为Docker Engine提供一个能够支持到CRI规范的垫片，从而能够让Kubernetes基于CRI控制Docker 。

1.2. 集群规划

服务器IP	主机名	配置
192.168.11.201	k8s-master
192.168.11.202	k8s-node

1.3. 网络规划

pod网络：10.200.0.0/16

service网络： 10.100.0.0/16

节点网络：192.168.11.0/24

1.4. 时间同步

apt -y install chrony
systemctl enable chrony && systemctl start chrony
chronyc sources -v

1.5. 设置主机名

# 设置主机名
hostnamectl set-hostname k8s-master
hostnamectl set-hostname k8s-node-1

# 更新
bash

1.6. 设置 hosts

cat >> /etc/hosts << EOF
192.168.11.201 k8s-master
192.168.11.202 k8s-node-1
EOF

1.7. 禁用 swap

sudo swapoff -a && sed -i '/swap/s/^/#/' /etc/fstab

systemctl disable --now swap.img.swap
systemctl mask swap.target

1.8. 禁用防火墙

ufw disable
ufw status

1.9. 内核参数调整

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

设置所需的 sysctl 参数，参数在重新启动后保持不变

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

应用 sysctl 参数而不重新启动

sudo sysctl --system

通过运行以下指令确认 br_netfilter 和 overlay 模块被加载

lsmod | grep br_netfilter
lsmod | grep overlay

通过运行以下指令确认 net.bridge.bridge-nf-call-iptables、net.bridge.bridge-nf-call-ip6tables 和 net.ipv4.ip_forward 系统变量在你的 sysctl 配置中被设置为 1

sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward

出现：

net.bridge.bridge-nf-call-iptables = 1

net.bridge.bridge-nf-call-ip6tables = 1

net.ipv4.ip_forward = 1

1.10. 开启 ipvs

安装 ipvs

apt install -y ipset ipvsadm

内核加载 ipvs

cat <<EOF | sudo tee /etc/modules-load.d/ipvs.conf
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
EOF

sudo modprobe ip_vs
sudo modprobe ip_vs_rr
sudo modprobe ip_vs_wrr
sudo modprobe ip_vs_sh
sudo modprobe nf_conntrack

确认ipvs模块加载

lsmod |grep -e ip_vs -e nf_conntrack

出现：

ip_vs_sh 16384 0

ip_vs_wrr 16384 0

ip_vs_rr 16384 0

ip_vs 176128 6 ip_vs_rr,ip_vs_sh,ip_vs_wrr

nf_conntrack_netlink 49152 0

nf_conntrack 172032 6 xt_conntrack,nf_nat,xt_nat,nf_conntrack_netlink,xt_MASQUERADE,ip_vs

nf_defrag_ipv6 24576 2 nf_conntrack,ip_vs

nf_defrag_ipv4 16384 1 nf_conntrack

nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables

libcrc32c 16384 6 nf_conntrack,nf_nat,btrfs,nf_tables,raid456,ip_vs

1.11. 开放端口

2. 2 安装 Kubernetes

2.1. 安装 K8s 三大件

安装 kubelet、kebeadm、kubectl；注意版本（1.28.3）


# 安装依赖
apt-get update && apt-get install -y apt-transport-https

# 添加并信任APT证书
curl -fsSL https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.28/deb/Release.key |
    gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
    
# 添加源地址
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.28/deb/ /" |
    tee /etc/apt/sources.list.d/kubernetes.list
apt-get update
# 查看 kubelet、kubeadm 和 kubectl版本
apt-cache madison kubeadm

apt-get install -y kubelet=1.28.3-1.1 kubeadm=1.28.3-1.1 kubectl=1.28.3-1.1

# 确认 kubectl 版本信息
kubectl version

# 锁定版本
sudo apt-mark hold kubelet kubeadm kubectl

# 运行服务
systemctl enable --now kubelet

# 查看 kubelet 状态：一会停止 一会运行。 这个状态是对的，kubelet 等待 kubeadm 发号指令。
systemctl status kubelet

# kubectl 命令补全
apt install -y bash-completion
kubectl completion bash > /etc/profile.d/kubectl_completion.sh
. /etc/profile.d/kubectl_completion.sh

kubernetes源配置方法

developer.aliyun.com/mirror/kube…

2.2. 准备镜像

查看镜像版本

 kubeadm config images list
 # 阿里云镜像
 kubeadm config images list --image-repository registry.cn-hangzhou.aliyuncs.com/google_containers

出现：

registry.k8s.io/kube-apiserver:v1.28.12

registry.k8s.io/kube-controller-manager:v1.28.12

registry.k8s.io/kube-scheduler:v1.28.12

registry.k8s.io/kube-proxy:v1.28.12

registry.k8s.io/pause:3.9

registry.k8s.io/etcd:3.5.9-0

registry.k8s.io/coredns/coredns:v1.10.1

2.3. 下载镜像

所有节点执行下载操作，也可导出导入操作。

docker image save `docker image ls --format "{{.Repository}}:{{.Tag}}"` -o k8s-images-v1.28.3.tar

或者：

# 下载阿里云镜像
#区别于1.23及之前的版本，一定要加--cri-socket unix:///run/cri-dockerd.sock, 指定容器运行时
kubeadm config images pull --kubernetes-version=v1.28.3 --image-repository registry.cn-hangzhou.aliyuncs.com/google_containers --cri-socket unix:///run/cri-dockerd.sock

出现：

[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.28.3

[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.28.3

[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.28.3

[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.28.3

[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9

[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.9-0

[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.10.1

2.4. 生成kubeadm init的配置文件

kubeadm-init-demo.yaml

apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: wgs001.com3yjucgqr276rf # 可以自定义，正则([a-z0-9]{6}).([a-z0-9]{16})
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: ${K8S_API_ADDVERTISE_IP} # 修改成节点ip
  bindPort: 6443
nodeRegistration:
 criSocket: unix:///var/run/cri-dockerd.sock
  imagePullPolicy: IfNotPresent
  name: ${K8S_HOSTNAME} # 节点的hostname
  taints: 
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
  - effect: NoSchedule
    key: node-role.kubernetes.io/control-plane
---
apiServer:
  timeoutForControlPlane: 4m0s
  certSANs: # 主节点IP
  - ${K8S_API_ENDPOINT}
  - ${K8S_API_ADDVERTISE_IP}
apiVersion: kubeadm.k8s.io/v1beta3
controlPlaneEndpoint: "${K8S_API_ENDPOINT}:6443" # 设置高可用地址
certificatesDir: /etc/kubernetes/pki
clusterName: ${K8S_CLUSTER_NAME}
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers # 国内源
kind: ClusterConfiguration
kubernetesVersion: v${K8S_VERSION} # 指定版本
networking:
  dnsDomain: ${K8S_DNS_DOMAIN}
  podSubnet: ${K8S_POD_SUBNET} # 增加指定pod的网段
  serviceSubnet: ${K8S_SERVICE_SUBNET}
scheduler: {}
---
# 用于配置kube-proxy上为Service指定的代理模式: ipvs or iptables
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "${K8S_SERVICE_MODE}"
---
# 指定cgroup
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: "${K8S_CGROUP_DRIVER}"

generate-init-config.sh

#!/bin/bash
# export addresses and other vars
set -a
K8S_API_ENDPOINT=192.168.11.201        # 高可用地址
K8S_API_ADDVERTISE_IP=192.168.11.201   # 节点ip
K8S_VERSION=1.28.3
K8S_CLUSTER_NAME=kubernetes
K8S_SERVICE_MODE=ipvs
K8S_POD_SUBNET="10.200.0.0/16"
K8S_SERVICE_SUBNET="10.100.0.0/16"
K8S_DNS_DOMAIN="wgs.local"
K8S_CGROUP_DRIVER="systemd"
K8S_HOSTNAME="k8s-master"
set +a

envsubst < kubeadm-init-demo.yaml > kubeadm-init-config.yaml

生成配置文件

bash generate-init-config.sh

2.5. 初始化主节点

kubeadm init --config ./kubeadm-init-config.yaml  --upload-certs

遇到问题：

解决

# 安装
# 下载镜像时添加criSocket: unix:///var/run/cri-dockerd.sock

# 重新配置
kubeadm reset
kubeadm init --config ./kubeadm-init-config.yaml  --upload-certs

成功出现:


Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join 192.168.11.201:6443 --token wgs001.com3yjucgqr276rf \
        --discovery-token-ca-cert-hash sha256:e4e5f9cb5fcf3c86eb3c1343d0a563527e7bf2317f5f38af9edfe58487d68304 \
        --control-plane --certificate-key fba1e729d9fa7055d6b612d751985ac84bf3fef5deda3d31a4edf5cfd1621921

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.11.201:6443 --token wgs001.com3yjucgqr276rf \
        --discovery-token-ca-cert-hash sha256:e4e5f9cb5fcf3c86eb3c1343d0a563527e7bf2317f5f38af9edfe58487d68304

按照提示要求，创建配置文件目录以及复制配置文件

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

查看节点状态

kubectl  get node

2.6. 安装网络组件

下载 tigera-operator

wget https://raw.githubusercontent.com/projectcalico/calico/v3.26.3/manifests/tigera-operator.yaml

下载 custom-resources

wget https://raw.githubusercontent.com/projectcalico/calico/v3.26.3/manifests/custom-resources.yaml

修改 custom-resources.yaml

# This section includes base Calico installation configuration.
# For more information, see: https://projectcalico.docs.tigera.io/master/reference/installation/api#operator.tigera.io/v1.Installation
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  # Configures Calico networking.
  calicoNetwork:
    # Note: The ipPools section cannot be modified post-install.
    ipPools:
    - blockSize: 26
      cidr: 10.200.0.0/16    # 和 --pod-network-cidr  保持一致
      encapsulation: VXLANCrossSubnet  # VXLAN 跨子网模式
      natOutgoing: Enabled
      nodeSelector: all()
  #registry: myregistry.com     # 定义hub地址
---

# This section configures the Calico API server.
# For more information, see: https://projectcalico.docs.tigera.io/master/reference/installation/api#operator.tigera.io/v1.APIServer
apiVersion: operator.tigera.io/v1
kind: APIServer
metadata:
  name: default
spec: {}

创建 tigera-operator 资源

kubectl create  -f tigera-operator.yaml

创建 custom-resources

kubectl create -f custom-resources.yaml

查看 calico 资源

watch kubectl get pods -n calico-system

出現：

calico-kube-controllers-5b9898d88d-zm95k 1/1 Running 0 2m33s

calico-node-bx78n 1/1 Running 0 2m33s

calico-typha-5b474fb786-dj6q5 1/1 Running 0 2m33s

csi-node-driver-f9jqn 2/2 Running 0 2m33s

查看节点状态

kubectl get nodes

出现

NAME STATUS ROLES AGE VERSION

k8s-master Ready control-plane 89m v1.28.3

2.7. 工作节点加入到kubernetes集群

要使用初始化节点生成的命令

一定要加上--cri-socket unix:///run/cri-dockerd.sock，指定容器运行时

kubeadm join 192.168.11.201:6443 \
--token wgs001.com3yjucgqr276rf \
--discovery-token-ca-cert-hash \
sha256:e4e5f9cb5fcf3c86eb3c1343d0a563527e7bf2317f5f38af9edfe58487d68304 \
--cri-socket unix:///run/cri-dockerd.sock

2.8. 添加master节点

因为这里只是单master集群，所以不需要加入master节点，如果有需要可以按照如下的方法加入

在其他两个master节点上，接入主节点

用的是第一个master节点初始化时输出的另一个证书,也就是这部分

kubeadm join 192.168.11.201:6443 --token wgs001.com3yjucgqr276rf \
--discovery-token-ca-cert-hash sha256:e4e5f9cb5fcf3c86eb3c1343d0a563527e7bf2317f5f38af9edfe58487d68304 \
--control-plane --certificate-key fba1e729d9fa7055d6b612d751985ac84bf3fef5deda3d31a4edf5cfd1621921

2.8.1. 令牌过期

如果令牌过期了，重新获取令牌。（在 master 服务器中执行）

# 重新获取令牌
kubeadm token create --print-join-command

2.8.2. 节点启动不起来，如下

解决：为节点服务器配置docker镜像

2.9. 集群自我修复测试

集群自我修复能力测试（需要 docker 启动的前提下，systemctl start docker）

3台服务器分别重启

# 重启
reboot

集群还是好好的。 nodes 的 STATUS 都是 Ready，查看 pods STATUS 过一会都是 Running。

2.10. 删除集群节点

驱逐节点上的pod

kubectl drain k8s-node-03 --delete-emptydir-data --force --ignore-daemonsets

重置节点

kubeadm reset -f --cri-socket unix:///run/cri-dockerd.sock

删除节点

kubectl delete node k8s-node

删除配置信息

rm -rf /etc/kubernetes/ /var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni /etc/cni/net.d /var/lib/etcd /run/flannel/   $HOME/.kube/config

2.11. 部署 Dashboard

部署 dashboard（可视化页面），kubernetes 官方提供的可视化界面

2.11.1. 运行 pod（创建资源）(只在 master 服务器中执行)

# 根据 在线配置文件 创建资源
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.3.1/aio/deploy/recommended.yaml

2.11.2. 设置访问端口

# 修改配置文件 找到 type，将 ClusterIP 改成 NodePort, k8s端口暴漏
kubectl edit svc kubernetes-dashboard -n kubernetes-dashboard

# 找到端口，在安全组放行
kubectl get svc -A |grep kubernetes-dashboard

2.11.3. 访问页面

https://192.168.11.201:32493（要注意是 https，port 是映射的端口，在配置文件查看）

2.11.4. 创建访问账号

# 创建 dash-usr.yaml，加入下面配置
vi dash-usr.yaml

# 创建访问账号，准备一个yaml文件，加入下面配置
apiVersion: v1
kind: ServiceAccount
metadata:
  name: dashboard-admin
  namespace: kube-system
#创建角色绑定关系
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount 
  name: dashboard-admin
  namespace: kube-system
---
#创建secret(重点是这个创建secret指定类型为token)
apiVersion: v1
kind: Secret
metadata:
  name: dashboard-admin-secret
  namespace: kube-system
  annotations:
    kubernetes.io/service-account.name: "dashboard-admin" 
type: kubernetes.io/service-account-token

  
# 在 k8s 集群中创建资源
kubectl apply -f dash-usr.yaml

#关联token证书
kubectl patch sa dashboard-admin -p '{"secrets": [{"name": "dashboard-admin-secret"}]}' -n kube-system
#查看token
kubectl get secret -n kube-system $(kubectl get serviceaccount dashboard-admin -n kube-system -o jsonpath='{.secrets[0].name}') -o go-template='{{ .data.token | base64decode }}'

ubuntu 22.04安装k8s完整流程