参考学习
zhuanlan.zhihu.com/p/560557852
blog.csdn.net/weixin_4447…
k8s对应 calico版本
blog.csdn.net/qq_32596527…
k8s下载国内镜像源
www.cnblogs.com/-abm/p/1663…
集成方案
k8s版本：1.23.6
Docker : 20.10+
所有节点关闭 SELinux
setenforce 0
sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux
所有节点关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
关闭 swap
# 临时关闭；关闭swap主要是为了性能考虑
swapoff -a
# 可以通过这个命令查看swap是否关闭了
free
# 永久关闭
sed -ri 's/.*swap.*/#&/' /etc/fstab
在master节点添加host
vim /etc/hosts
192.168.10.11 k8s-slave1
192.168.10.12 k8s-slave2
192.168.10.13 k8s-slave3
192.168.10.10 k8s-master
允许 iptables 检查桥接流量
sudo modprobe br_netfilter
lsmod | grep br_netfilter
Linux 节点的 iptables 能够正确查看桥接流量，请确认 sysctl 配置中的 net.bridge.bridge-nf-call-iptables 设置为 1
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter
# 设置所需的 sysctl 参数，参数在重新启动后保持不变
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
# 应用 sysctl 参数而不重新启动
sudo sysctl --system
时间同步
yum install chrony -y
systemctl start chronyd
systemctl enable chronyd
chronyc sources
安装docker
# step 1: 安装必要的一些系统工具
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
# Step 2: 添加软件源信息
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
sudo yum install docker-ce-20.10.12 docker-ce-cli-20.10.12 containerd.io-1.5.11
# 启动docker
systemctl start docker
# 设置开机启动
systemctl enable docker
# 配置docker，修改cgroup、log、镜像地址
cat > /etc/docker/daemon.json << EOF
{
"registry-mirrors": ["https://82m9ar63.mirror.aliyuncs.com"],
"exec-opts": ["native.cgroupdriver=systemd"],
"insecure-registries": [
"192.168.10.13:8070"
],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
# 重启docker服务
systemctl restart docker
kubeadm、kubelet、kubectl安装
# kubernetes镜像切换成国内源，baseurl中注意架构类型
# x86架构：baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
# arm架构：baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-aarch64/
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF
# 先清空
yum clean all
yum makecache
# 安装指定版本 kubeadm、kubelet、kubectl
yum install -y kubelet-1.23.6 kubeadm-1.23.6 kubectl-1.23.6 --disableexcludes=kubernetes
systemctl enable kubelet
初始化主节点（仅master节点上操作）
这里我的网络集群使用 calico 所以 pod-network-cidr
# 初始化之前 先把主节点分别同步到各节点上
echo "192.168.10.10 k8s-master" >> /etc/hosts
# 这里的image-repository可以换成registry.cn-hangzhou.aliyuncs.com/google_containers 就不用转换了docker
kubeadm init \
--apiserver-advertise-address=192.168.10.10 \
--control-plane-endpoint=k8s-master \
--image-repository registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images \
--kubernetes-version v1.23.6 \
--service-cidr=10.96.0.0/16 \
--pod-network-cidr=192.168.0.0/16
--image-repository：选择用于拉取镜像的镜像仓库（默认为“k8s.gcr.io” ）
--kubernetes-version：选择特定的Kubernetes版本（默认为“stable-1”）
--service-cidr：为服务的VIP指定使用的IP地址范围（默认为“10.96.0.0/12”）
--pod-network-cidr：指定Pod网络的IP地址范围。如果设置，则将自动为每个节点分配CIDR。
master节点初始化成功返回内容
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:
kubeadm join k8s-master:6443 --token y4nxkv.3c43pgity1rlgn3d \
--discovery-token-ca-cert-hash sha256:b1bf1ce549a977fcc965d77dce2a8e586a3b7d8d5f1209c7cb6fb5ea31a01b6d \
--control-plane
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join k8s-master:6443 --token y4nxkv.3c43pgity1rlgn3d \
--discovery-token-ca-cert-hash sha256:b1bf1ce549a977fcc965d77dce2a8e586a3b7d8d5f1209c7cb6fb5ea31a01b6d
在master节点执行regular user
可以发现在成功的时候需要执行以下句子(makdir开始)
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
子节点加入
kubeadm join k8s-master:6443 --token y4nxkv.3c43pgity1rlgn3d \
--discovery-token-ca-cert-hash sha256:b1bf1ce549a977fcc965d77dce2a8e586a3b7d8d5f1209c7cb6fb5ea31a01b6d
加入成功说明
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
# 在master节点执行kubectl get nodes 查看具体加入的node
kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master NotReady control-plane,master 33m v1.23.6
k8s-slave1 NotReady <none> 59s v1.23.6
k8s-slave2 NotReady <none> 52s v1.23.6
k8s-slave3 NotReady <none> 48s v1.23.6
执行 calico网络
可以观察参考，calico需要对应的版本，v1.23.6 需要的版本为 3.23
wget projectcalico.docs.tigera.io/archive/v3.…
# 在master节点执行
kubectl apply -f calico.yaml
# 在master节点查看具体安装情况 -w ：持续观察 可以发现执行成功
kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-54756b744f-9trlk 1/1 Running 0 17m
calico-node-7db7j 1/1 Running 0 17m
calico-node-d87gw 1/1 Running 0 17m
calico-node-x9ptp 1/1 Running 0 17m
calico-node-xbkbb 1/1 Running 0 17m
coredns-5597cb95b7-c4vlc 1/1 Running 0 164m
coredns-5597cb95b7-nw4j5 1/1 Running 0 164m
etcd-k8s-master 1/1 Running 0 165m
kube-apiserver-k8s-master 1/1 Running 0 165m
kube-controller-manager-k8s-master 1/1 Running 0 165m
kube-proxy-2mkc5 1/1 Running 0 132m
kube-proxy-c87qg 1/1 Running 0 132m
kube-proxy-j6bq2 1/1 Running 0 132m
kube-proxy-q5qq6 1/1 Running 0 164m
kube-scheduler-k8s-master 1/1 Running 0 165m
# 查看node节点 发现已经正常启动
kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master Ready control-plane,master 165m v1.23.6
k8s-slave1 Ready <none> 132m v1.23.6
k8s-slave2 Ready <none> 132m v1.23.6
k8s-slave3 Ready <none> 132m v1.23.6
问题总结
主要的问题来源是镜像拉取失败，需要拉取国内镜像，去转换k8s需要的镜像源，改个名字就可以了
问题描述
问题一：kubeadm init 拉取镜像失败
报错情况
[ERROR ImagePull]: failed to pull image registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/etcd:3.5.1-0: output: Error response from daemon: manifest for registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/etcd:3.5.1-0 not found: manifest unknown: manifest unknown
, error: exit status 1
解决方案 参考：k8s下载国内镜像源
# 查看 k8s需要下载 版本需要的列表
kubeadm config images list --kubernetes-version=v1.23.6
k8s.gcr.io/kube-apiserver:v1.23.6
k8s.gcr.io/kube-controller-manager:v1.23.6
k8s.gcr.io/kube-scheduler:v1.23.6
k8s.gcr.io/kube-proxy:v1.23.6
k8s.gcr.io/pause:3.6
k8s.gcr.io/etcd:3.5.1-0
k8s.gcr.io/coredns/coredns:v1.8.6
# 下载国内镜像源
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.23.6
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.23.6
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.23.6
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.23.6
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.1-0
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.6
# 改成对应的下载镜像源（可以查看报错自己需要的镜像源，转换名字就可以）
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.23.6 registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/kube-apiserver:v1.23.6
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.23.6 registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/kube-controller-manager:v1.23.6
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.23.6 registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/kube-scheduler:v1.23.6
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.23.6 registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/kube-proxy:v1.23.6
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6 registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/pause:3.6
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.1-0 registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/etcd:3.5.1-0
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.6 registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/coredns:v1.8.6
# 在进行Init则可成功
问题二：calico 执行kubectl apply -f 初始化失败
通过下面报错可知拉取镜像失败，和问题一是一样的，需要再各个子节点进行问题一的docker pull和docker tag操作就可以解决这个问题
Warning FailedCreatePodSandBox 6m37s (x26 over 12m) kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed pulling image "registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/pause:3.6": Error response from daemon: manifest for registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/pause:3.6 not found: manifest unknown: manifest unknown
Normal Pulling 2m11s kubelet Pulling image "docker.io/calico/cni:v3.23.5"
