1 机器准备
| 主机 | ip | 角色 |
|---|---|---|
| kubesphere | 192.168.137.7 | master节点 |
| work1 | 192.168.137.6 | work节点 |
| work2 | 192.168.137.8 | work节点 |
2 系统准备
查看系统版本
[root@kubesphere ~]# cat /etc/centos-release
CentOS Stream release 8
配置主机名
[root@kubesphere ~]# hostnamectl
Static hostname: kubesphere
Icon name: computer-vm
Chassis: vm
Machine ID: 73bbc5c10c9c4f6f8d7fad01e5dbe954
Boot ID: e40f8d33ea244ae9acdd283e8a25a466
Virtualization: microsoft
Operating System: CentOS Stream 8
CPE OS Name: cpe:/o:centos:centos:8
Kernel: Linux 4.18.0-365.el8.x86_64
Architecture: x86-64
关闭swap
[root@kubesphere ~]# swapoff -a
配置内核参数，将桥接的IPv4流量传递到iptables的链
[root@kubesphere ~]# cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
3 安装常用包
[root@kubesphere ~]# yum install bash-completion net-tools gcc -y
4 安装docker
curl -sSL https://get.daocloud.io/docker | sh
Docker 镜像加速
[root@kubesphere ~]# mkdir -p /etc/docker
[root@kubesphere ~]# tee /etc/docker/daemon.json <<-'EOF'
{
"registry-mirrors": ["https://{xxxx}.mirror.aliyuncs.com/"]
}
EOF
[root@kubesphere ~]# systemctl daemon-reload
[root@kubesphere ~]# systemctl restart docker
{xxxx}替换成可用的
5 安装kubectl、kubelet、kubeadm
添加阿里kubernetes源
[root@kubesphere ~]# cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
安装
[root@kubesphere ~]# yum install kubectl kubelet kubeadm
[root@kubesphere ~]# systemctl enable kubelet
以上操作需要在所有机器执行
6 初始化k8s集群（在kubesphere主节点执行）
[root@kubesphere ~]# kubeadm init --kubernetes-version=1.23.4 \
--apiserver-advertise-address=192.168.137.7 \
--image-repository registry.aliyuncs.com/google_containers \
--service-cidr=10.88.0.0/16 --pod-network-cidr=10.8.0.0/16
pod的网段为：10.8.0.0/16
rvice网段：10.88.0.0/16
apiserver： master本机IP
image-repository：这一步很关键，由于kubeadm 默认从官网k8s.grc.io下载所需镜像，国内无法访问，因此需要通过–image-repository指定阿里云镜像仓库地址
如果出现一下错误
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
[root@kubesphere ~]# tail /var/log/messages
Mar 9 22:02:43 kubesphere kubelet[12627]: I0309 22:02:43.052619 12627 docker_service.go:571] "Hairpin mode is set but kubenet is not enabled, falling back to HairpinVeth" hairpinMode=promiscuous-bridge
Mar 9 22:02:43 kubesphere kubelet[12627]: I0309 22:02:43.052655 12627 docker_service.go:243] "Hairpin mode is set" hairpinMode=hairpin-veth
Mar 9 22:02:43 kubesphere kubelet[12627]: I0309 22:02:43.052712 12627 cni.go:240] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
Mar 9 22:02:43 kubesphere kubelet[12627]: I0309 22:02:43.055341 12627 cni.go:240] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
Mar 9 22:02:43 kubesphere kubelet[12627]: I0309 22:02:43.055407 12627 docker_service.go:258] "Docker cri networking managed by the network plugin" networkPluginName="cni"
Mar 9 22:02:43 kubesphere kubelet[12627]: I0309 22:02:43.055439 12627 cni.go:240] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
Mar 9 22:02:43 kubesphere kubelet[12627]: I0309 22:02:43.062146 12627 docker_service.go:264] "Docker Info" dockerInfo=&{ID:O7WV:ARF4:ZH45:DBG3:L7LF:ZOGI:V2KR:5NB4:XQIN:454K:J37R:DV6S Containers:2 ContainersRunning:0 ContainersPaused:0 ContainersStopped:2 Images:54 Driver:overlay2 DriverStatus:[[Backing Filesystem xfs] [Supports d_type true] [Native Overlay Diff true] [userxattr false]] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:true KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:27 OomKillDisable:true NGoroutines:34 SystemTime:2022-03-09T22:02:43.055960906+08:00 LoggingDriver:json-file CgroupDriver:cgroupfs CgroupVersion:1 NEventsListener:0 KernelVersion:4.18.0-365.el8.x86_64 OperatingSystem:CentOS Stream 8 OSVersion:8 OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc0002764d0 NCPU:4 MemTotal:2838216704 GenericResources:[] DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:kubesphere Labels:[] ExperimentalBuild:false ServerVersion:20.10.12 ClusterStore: ClusterAdvertise: Runtimes:map[io.containerd.runc.v2:{Path:runc Args:[] Shim:<nil>} io.containerd.runtime.v1.linux:{Path:runc Args:[] Shim:<nil>} runc:{Path:runc Args:[] Shim:<nil>}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:<nil> Warnings:[]} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:7b11cfaabd73bb80907dd23182b9347b4245eb5d Expected:7b11cfaabd73bb80907dd23182b9347b4245eb5d} RuncCommit:{ID:v1.0.2-0-g52b36a2 Expected:v1.0.2-0-g52b36a2} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=seccomp,profile=default] ProductLicense: DefaultAddressPools:[] Warnings:[]}
Mar 9 22:02:43 kubesphere kubelet[12627]: E0309 22:02:43.062191 12627 server.go:302] "Failed to run kubelet" err="failed to run Kubelet: misconfiguration: kubelet cgroup driver: \"systemd\" is different from docker cgroup driver: \"cgroupfs\""
Mar 9 22:02:43 kubesphere systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Mar 9 22:02:43 kubesphere systemd[1]: kubelet.service: Failed with result 'exit-code'.
解决方案步骤如下：
1、先修改docker的Cgroup Driver修改/etc/docker/daemon.json文件
{"registry-mirrors":["https://{xxxx}.mirror.aliyuncs.com/"],"exec-opts": ["native.cgroupdriver=systemd"]}
当出现下面的输出说明安装成功了
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.137.7:6443 --token j3znq8.25azqp0r62ity41o \
--discovery-token-ca-cert-hash sha256:50abe2477ed8e84ad3898907f6d5547a94cfd942a73916a8ad860d6108803032
根据提示操作
[root@kubesphere ~]# mkdir -p $HOME/.kube
[root@kubesphere ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@kubesphere ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
[root@kubesphere ~]# export KUBECONFIG=/etc/kubernetes/admin.conf
[root@kubesphere ~]# kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
使kubectl可以自动补充
[root@kubesphere ~]# source <(kubectl completion bash)
安装网络插件calico
curl https://docs.projectcalico.org/manifests/calico.yaml -O
kubectl apply -f calico.yaml
7 从节点加入集群(work节点执行)
[root@work2 ~]# kubeadm join 192.168.137.7:6443 --token j3znq8.25azqp0r62ity41o \
> --discovery-token-ca-cert-hash sha256:50abe2477ed8e84ad3898907f6d5547a94cfd942a73916a8ad860d6108803032
[preflight] Running pre-flight checks
[WARNING FileExisting-tc]: tc not found in system path
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
在master节点查看是否加入成功
[root@kubesphere ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
kubesphere Ready control-plane,master 12m v1.23.4
work2 NotReady <none> 2m32s v1.23.4
work3 NotReady <none> 78s v1.23.4
8 安装kubernetes-dashboard
官方部署dashboard的服务没使用nodeport，将yaml文件下载到本地，在service里添加nodeport
[root@kubesphere ~]# wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-rc7/aio/deploy/recommended.yaml
[root@kubesphere ~]# vim recommended.yaml
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
type: NodePort
ports:
- port: 443
targetPort: 8443
nodePort: 30000
selector:
k8s-app: kubernetes-dashboard
[root@kubesphere ~]# kubectl create -f recommended.yaml
[root@kubesphere k8s]# kubectl get svc -n kubernetes-dashboard
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
dashboard-metrics-scraper ClusterIP 10.88.211.92 <none> 8000/TCP 14s
kubernetes-dashboard NodePort 10.88.14.69 <none> 443:30000/TCP 14s
创建用户dashboard-adminuser.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kubernetes-dashboard
获取token
kubectl -n kubernetes-dashboard get secret $(kubectl -n kubernetes-dashboard get sa/admin-user -o jsonpath="{.secrets[0].name}") -o go-template="{{.data.token | base64decode}}"
eyJhbGciOiJSUzI1NiIsImtpZCI6IkxaY2lvZEtwTlF5M3lyMDdIRVdpZTBUQXZkSDhub3o0T2d0dU1TRGVtNncifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLTJuMno4Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI4MjAzM2Q5MS1mMjYzLTRlNGQtYjFiZC1iNjcxYzQwNjBmM2YiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQ6YWRtaW4tdXNlciJ9.yqss--E8gCWQW6YWel_ZjWarlXyjBqXVnyjsMPVQ5rbCEfy0Li1HdYZjT8dzO2Kk74FUQBNSbIvUTo08ap1HZfey8vY8Zq5At3DRyMAurr9xQG8QPI7IlL9blEQ7XbH-AYgquhf2p1UmhyyGivYjX-p02CQE6iSnQ5Xog_sflXltgfxgkoBV58qb6VliCOIkzb1lrlPqIjEW6tsgKcTy6dJ3nHvfhI9BiSpXRW_h00YFPPFhXon73w-p7ZjczKmH2JTiG7k_kOzoYp53zjG9tG8QoE-8nE756IXDgylIlJ-wJVNeVItdN2vK4zbqpIzfo0FsRWzHaaZYaUa-q1zdww
访问https://192.168.137.7:30000/
注意：需要https
