自动发布结构图

基础环境

服务器	内存	CPU	系统	内网ip
k8s-master	4G	2核	Centos 7.6	192.168.0.6
k8s-node1	4G	2核	Centos 7.6	192.168.0.47
k8s-node2	4G	2核	Centos 7.6	192.168.0.154
k8s-node3`jenkins`	8G	4核	Centos 7.6	192.168.0.72

涉及到的环境安装:

1.Docker
2.Kubernetes
3.K8s界面管理工具(Kuboard)
4.Jenkins
5.Git
6.Maven

1.Docker安装

卸载旧版docker

sudo yum remove docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-selinux \
docker-engine-selinux \
docker-engine

安装依赖包

sudo yum install -y yum-utils

添加阿里云yum软件源

sudo yum-config-manager \
    --add-repo \
    https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
    
sudo sed -i 's/download.docker.com/mirrors.aliyun.com\/docker-ce/g' /etc/yum.repos.d/docker-ce.repo

安装docker并启动

yum install -y docker-ce-19.03.8 docker-ce-cli-19.03.8 containerd.io
systemctl enable docker
systemctl start docker

关闭防火墙,SeLinux,swap

# 关闭 防火墙
systemctl stop firewalld
systemctl disable firewalld

# 关闭 SeLinux
setenforce 0
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

# 关闭 swap
swapoff -a
yes | cp /etc/fstab /etc/fstab_bak
cat /etc/fstab_bak |grep -v swap > /etc/fstab

2.安装kubernetes

修改 /etc/sysctl.conf

echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
echo "net.bridge.bridge-nf-call-ip6tables = 1" >> /etc/sysctl.conf
echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.all.disable_ipv6 = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.default.disable_ipv6 = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.lo.disable_ipv6 = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.all.forwarding = 1"  >> /etc/sysctl.conf

生效配置

sysctl -p

添加Kubernetes yum源

 vi /etc/yum.repos.d/kubernetes.repo
 name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
       http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

卸载旧版本

yum remove -y kubelet kubeadm kubectl

安装kubelet,kubeadm,kubectl版本1.18.9

yum install -y kubelet-1.18.9 kubeadm-1.18.9 kubectl-1.18.9

修改docker Cgroup Driver为systemd

# # 将/usr/lib/systemd/system/docker.service文件中的这一行 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
# # 修改为 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd
# 如果不修改，在添加 worker 节点时可能会碰到如下错误
# [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". 
# Please follow the guide at https://kubernetes.io/docs/setup/cri/

sed -i "s#^ExecStart=/usr/bin/dockerd.*#ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd#g" /usr/lib/systemd/system/docker.service

配置docker镜像加速

export REGISTRY_MIRROR=https://registry.cn-hangzhou.aliyuncs.com
curl -sSL https://kuboard.cn/install-script/set_mirror.sh | sh -s ${REGISTRY_MIRROR}

重启docker 并启动kubelet

crictl config runtime-endpoint /run/containerd/containerd.sock

# 重启 docker，并启动 kubelet
systemctl daemon-reload
systemctl restart docker
systemctl enable kubelet && systemctl start kubelet

containerd --version
kubelet --version
docker version

初始化Master
编辑kubeadm-config.yaml

mkdir /usr/local/k8s
vi kubeadm-config.yaml

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.18.9
imageRepository: registry.aliyuncs.com/k8sxio
controlPlaneEndpoint: "k8s-master:6443"
networking:
  serviceSubnet: "10.96.0.0/16"
  podSubnet: "10.100.0.1/16"
  dnsDomain: "cluster.local"

kubeadm init

kubeadm init --config=kubeadm-config.yaml --upload-certs

配置 kubectl

rm -rf /root/.kube/
mkdir /root/.kube/
cp -i /etc/kubernetes/admin.conf /root/.kube/config

安装calico

wget https://kuboard.cn/install-script/calico/calico-3.13.1.yaml
kubectl apply -f calico-3.13.1.yaml

安装过程中错误排查
命令:journalctl -xeu kubelet

Feb 16 16:32:31 k8s-master kubelet[5138]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Feb 16 16:32:31 k8s-master kubelet[5138]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Feb 16 16:32:31 k8s-master kubelet[5138]: I0216 16:32:31.980708    5138 server.go:417] Version: v1.18.9
Feb 16 16:32:31 k8s-master kubelet[5138]: I0216 16:32:31.981288    5138 plugins.go:100] No cloud provider specified.
Feb 16 16:32:31 k8s-master kubelet[5138]: I0216 16:32:31.981315    5138 server.go:838] Client rotation is on, will bootstrap in background
Feb 16 16:32:31 k8s-master kubelet[5138]: I0216 16:32:31.986158    5138 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Feb 16 16:32:31 k8s-master kubelet[5138]: I0216 16:32:31.987324    5138 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/pki/ca.crt
Feb 16 16:32:32 k8s-master kubelet[5138]: I0216 16:32:32.059614    5138 server.go:647] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
Feb 16 16:32:32 k8s-master kubelet[5138]: I0216 16:32:32.059869    5138 container_manager_linux.go:266] container manager verified user specified cgroup-root exists: []
Feb 16 16:32:32 k8s-master kubelet[5138]: I0216 16:32:32.059879    5138 container_manager_linux.go:271] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:systemd KubeletRootDir:/var/lib/kubelet Pro
Feb 16 16:32:32 k8s-master kubelet[5138]: I0216 16:32:32.059975    5138 topology_manager.go:126] [topologymanager] Creating topology manager with none policy
Feb 16 16:32:32 k8s-master kubelet[5138]: I0216 16:32:32.059981    5138 container_manager_linux.go:301] [topologymanager] Initializing Topology Manager with none policy
Feb 16 16:32:32 k8s-master kubelet[5138]: I0216 16:32:32.059985    5138 container_manager_linux.go:306] Creating device plugin manager: true
Feb 16 16:32:32 k8s-master kubelet[5138]: I0216 16:32:32.060033    5138 client.go:75] Connecting to docker on unix:///var/run/docker.sock
Feb 16 16:32:32 k8s-master kubelet[5138]: I0216 16:32:32.060043    5138 client.go:92] Start docker client with request timeout=2m0s
Feb 16 16:32:32 k8s-master kubelet[5138]: W0216 16:32:32.068209    5138 docker_service.go:562] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
Feb 16 16:32:32 k8s-master kubelet[5138]: I0216 16:32:32.068230    5138 docker_service.go:238] Hairpin mode set to "hairpin-veth"
Feb 16 16:32:32 k8s-master kubelet[5138]: W0216 16:32:32.068312    5138 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
Feb 16 16:32:32 k8s-master kubelet[5138]: W0216 16:32:32.070804    5138 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
Feb 16 16:32:32 k8s-master kubelet[5138]: I0216 16:32:32.070850    5138 docker_service.go:253] Docker cri networking managed by cni
Feb 16 16:32:32 k8s-master kubelet[5138]: W0216 16:32:32.070917    5138 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
Feb 16 16:32:32 k8s-master kubelet[5138]: I0216 16:32:32.079574    5138 docker_service.go:258] Docker Info: &{ID:3OHS:SKBB:HKWB:SO4R:6CSH:6WAP:OILN:OYPM:46KD:65MT:PPN2:QDTI Containers:0 ContainersRunning:0 ContainersPaused:0 ContainersStopped:0 Images:13 Driver:overlay2 DriverStatus:[[Backing Filesystem <unknown>] [Supports d_type
Feb 16 16:32:32 k8s-master kubelet[5138]: I0216 16:32:32.079682    5138 docker_service.go:271] Setting cgroupDriver to systemd
Feb 16 16:32:32 k8s-master kubelet[5138]: F0216 16:32:32.080178    5138 docker_service.go:413] Streaming server stopped unexpectedly: listen tcp [::1]:0: bind: cannot assign requested address
Feb 16 16:32:32 k8s-master systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
Feb 16 16:32:32 k8s-master systemd[1]: Unit kubelet.service entered failed state.
Feb 16 16:32:32 k8s-master systemd[1]: kubelet.service failed.

问题点。 Streaming server stopped unexpectedly: listen tcp [::1]:0: bind: cannot assign requested address 看到[::1]:0:bind 好像在hosts文件中看到过我的完整hosts文件第二行::1 系统初始状态是存在的。被我误删除掉了加上重新安装就可以正常进行启动了

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.0.6	k8s-master
192.168.0.47	k8s-node1
192.168.0.154	k8s-node2
127.0.0.1 k8s-master

初始化worker节点
- 标题Docker安装内容在worker节点重装一次
- 在Master节点执行 kubeadm token create --print-join-command 会得到 kubeadm join apiserver.k8s-master:6443 --token mpfjma.4vjjg8flqihor4vt --discovery-token-ca-cert-hash sha256:6f7a8e40a810323672de5eee6f4d19aa2dbdb38411845a1bf5dd63485c43d303
- 针对所有worker节点只在 worker 节点执行

# 替换 x.x.x.x 为 master 节点的内网 IP
export MASTER_IP=x.x.x.x
# 替换 k8s-master 为初始化 master 节点时所使用的 APISERVER_NAME
export APISERVER_NAME=k8s-master
echo "${MASTER_IP}    ${APISERVER_NAME}" >> /etc/hosts

kubeadm join k8s-master:6443 --token ke1n3g.9eld59l95zio84ly     --discovery-token-ca-cert-hash sha256:2a707ea190ba499e22fd5d12ea5e2db31bcad3b25110810677da159dbe6a5f48

检查初始化结果 kubectl get nodes -o wide
安装IgressController

kubectl apply -f https://kuboard.cn/install-script/v1.18.x/nginx-ingress.yaml

3.安装Kuboard

kubectl apply -f https://kuboard.cn/install-script/kuboard.yaml
kubectl apply -f https://addons.kuboard.cn/metrics-server/0.3.7/metrics-server.yaml

访问地址: 开放安全组端口 32567
浏览器输入: http://公网ip:32567 公网ip为任意子节点这里使用NodePort方式部署
Master节点输入:echo $(kubectl -n kube-system get secret $(kubectl -n kube-system get secret | grep kuboard-user | awk '{print $1}') -o go-template='{{.data.token}}' | base64 -d) 获取管理员token
登录后会看到如下界面
绑定Https域名，证书可以去华为云免费获取(console.huaweicloud.com/ccm) 没有的话可以使用http不勾选https
绑定成功后访问地址为: kuboard.supersindox.com/

只读用户token: token.supersindox.com/

使用gitlab/github单点登录见: kuboard.cn/learning/k8…

4.git安装

yum install git

5.Maven安装

下载地址: maven.apache.org/download.cg…
环境变量

export MAVEN_HOME=/usr/local/maven3
export PATH=$MAVEN_HOME/bin:$PATH

6.jenkins安装

提前环境jdk1.8
下载jenkins war包 updates.jenkins-ci.org/download/wa…

教程选用的版本 2.280

启动jenkins
- nohup java -jar /usr/local/jenkins/jenkins.war &
获取登录密码
- 通过nohup中国的启动日志可以看到密码在 This may also be found at: /root/.jenkins/secrets/initialAdminPassword
- 账号: admin 密码：xxxxx
- 默认端口:8080 使用云服务器开放安全组策略8080，ip:8080 即可进入登录界面
选择不装插件后续我们通过插件管理自定义安装就可以看到这个界面
安装的插件有

apache-httpcomponents-client-4-api  
 bootstrap4-api  
 bouncycastle-api  
 branch-api  
 checks-api  
 cloudbees-folder  
 command-launcher  
 credentials  
 display-url-api  
 echarts-api  
 font-awesome-api  
 git-client  
 gitee  
 github-api  
 github  
 github-pullrequest  
 git  
 git-parameter  
 icon-shim  
 jackson2-api  
 javadoc  
 jdk-tool  
 jquery3-api  
 jquery-detached  
 jquery  
 jsch  
 junit  
 mailer  
 matrix-project  
 maven-plugin  
 okhttp-api  
 plain-credentials  
 plugin-util-api  
 popper-api  
 scm-api  
 script-security  
 snakeyaml-api  
 ssh-credentials  
 structs  
 token-macro  
 trilead-api  
 versionnumber  
 webhook-step  
 workflow-api  
 workflow-cps  
 workflow-job  
 workflow-multibranch  
 workflow-scm-step  
 workflow-step-api

jenkins 项目配置

7.Gitee 配置

可以用公司的Gitlab，后者github(github会经常出现拉代码超时情况),个人服务器资源有限，安装gitlab发现占用资源过高，所以这里采用gitee作为代码管理，gitee访问速度快，同样支持webhook。

注意: 新增webhook时，jenkins密码和jenkins项目配置下截图的WebHook密码一致 钉钉机器人添加的是密钥：在钉钉群组新增机器人会获取到

新增钉钉机器人的方式: 见 blog.csdn.net/keep_learn/… 选择自定义机器人

Kubernetes、Jenkins、Docker实现CI/CD

自动发布结构图

基础环境

涉及到的环境安装:

1.Docker安装

2.安装kubernetes

3.安装Kuboard

4.git安装

5.Maven安装

6.jenkins安装

7.Gitee 配置

备注: 上面jenkins打码的CI/CD脚本在Kuboard上Copy过去就可以了

Kuboard官网: kuboard.cn/

Docker学习使用: yeasy.gitbook.io/docker_prac…

Kubuertes学习使用: docs.kubernetes.org.cn/

熊猫笔记邮箱 `panda_nodes@163.com`

Kubernetes、Jenkins、Docker实现CI/CD

自动发布结构图

基础环境

涉及到的环境安装:

1.Docker安装

2.安装kubernetes

3.安装Kuboard

4.git安装

5.Maven安装

6.jenkins安装

7.Gitee 配置

备注: 上面jenkins打码的CI/CD脚本在Kuboard上Copy过去就可以了

Kuboard官网: kuboard.cn/

Docker学习使用: yeasy.gitbook.io/docker_prac…

Kubuertes学习使用: docs.kubernetes.org.cn/

熊猫笔记邮箱 panda_nodes@163.com

熊猫笔记邮箱 `panda_nodes@163.com`