Jenkins通过Pipeline部署项目至k8s
前情提要
如果对docker、harbor、k8s不熟悉的兄弟们，请先行阅读如下几篇文章：
-
k8s集群搭建教程(特别注意，本篇文章基于此文章的集群环境搭建)
内容摘要
本文通过jenkins相关插件通过pipeline定义部署步骤部署springboot项目到k8s集群。
1、环境准备
1.1、插件SSH Pipeline Steps安装
为了让jenkins在pipeline中能够远程执行脚本及传输文件，我们需要安装插件SSH Pipeline Steps，在插件管理中心中进行安装，后重启jenkins即可。
1.2、项目代码
随便准备一个springboot项目jenkins-demo,然后写一个接口，此处我用的是：jenkins-demo,此项目为公开项目，仅供学习使用。
接口信息：
@GetMapping("sayHello")
public JSONObject hello2() throws Exception{
JSONObject jsonObject = new JSONObject();
jsonObject.put("code","000000");
jsonObject.put("message","交易成功");
JSONObject data = new JSONObject();
data.put("localIp", InetUtils.getLocalIP());
data.put("hostname", InetUtils.getLocalHostName());
jsonObject.put("data",data);
return jsonObject;
}
1.3、Pipeline脚本说明
我们准备将服务jenkins-demo服务部署两个节点pod，然后创建service对外提供服务。然后我们编写部署脚本，按照pipeline规范进行编写，需要准备好如下相关信息，详细脚本信息如下：
- git仓库地址信息
- Dockerfie脚本
- harbor镜像仓库信息
仓库地址：harbor.jianjang.com:1443
- k8s服务器集群
| 节点 | ip | 备注 |
|---|---|---|
| k8s-master | 192.168.2.2 | |
| k8s-node1 | 192.168.2.3 | |
| k83-node2 | 192.168.2.4 |
- k8s中pod部署规约脚本
kind: Deployment
apiVersion: apps/v1
metadata:
namespace: ${k8sNameSpace}
name: uat-${serviceName}-app
spec:
selector:
matchLabels:
app: uat-${serviceName}-app
replicas: ${replicasNum}
template:
metadata:
labels:
app: uat-${serviceName}-app
spec:
imagePullSecrets:
- name: ${k8sHarborTokenId}
containers:
- name: uat-${serviceName}-app
image: ${harborAddress}/${harborNameSpace}/${imageName}:${imageVersion}
imagePullPolicy: IfNotPresent
ports:
- containerPort: ${containerPort}
- k8s中部署服务规约脚本
kind: Service
apiVersion: v1
metadata:
namespace: ${k8sNameSpace}
name: uat-${serviceName}-app
spec:
selector:
app: uat-${serviceName}-app
type: NodePort
ports:
- protocol: TCP
port: ${port}
targetPort: ${containerPort}
nodePort: ${k8sServicePort}
相关最后部署的Pipeline步骤及脚本如下：
部署脚本分如下几个步骤
- 拉取代码
- 构建项目
- 创建Dockerfile
- docker镜像构建
- docker镜像上传仓库
- 生成pod创建文件
- 上传pod创建文件
- 创建或更新pod
- 生成service创建文件
- 上传service创建或文件
- 部署service服务
Pipeline部署脚本为本篇文章的重点,部署步骤及每个步骤完成的工作都在里面，所以很重要，脚本如下
def createVersion() {
// 定义一个版本号作为当次构建的版本，输出结果 20220510175842_69
return new Date().format('yyyyMMddHHmmss') + "_${env.BUILD_ID}"
}
pipeline {
agent any
environment {
//Git URL
gitUrl='git@gitlabs.jianjang.com:zhangjian_sh/jenkins-demo.git'
//Git 凭据，在jenkins后台创建的凭据ID
gitToken='git-ssh-key'
//程序包名称
serviceName = "jenkins-demo"
//包类型
packageType = "jar"
//镜像名称
imageName = "jenkins-demo"
imageVersion=createVersion()
registryLoginName="zhangjian_sh"
registryLoginPassword="zwx123456"
//程序端口
port = 8081
//jar程序启动指定环境配置文件参数
env = "uat"
//uat环境镜像仓库
harborNameSpace = "jianjang"
//仓库地址
harborAddress = "harbor.jianjang.com:1443"
//k8s上deploy名称
deployment = "jenkins-demo-app"
//namespace
k8sNameSpace = "my-namespace"
//pod数量
replicasNum = 2
//pod端口
containerPort = 8081
//jvm 内存
MEM_LIMIT=512
k8sHarborTokenId="my-ns-login"
k8sServicePort=30008
}
stages {
stage('拉取代码') {
steps {
echo "拉取项目[${serviceName}]代码[${gitUrl}] 开始"
git credentialsId: "${gitToken}", url: "${gitUrl}"
echo "拉取项目[${serviceName}]代码[${gitUrl}] 结束"
}
}
stage('构建项目') {
steps {
sh "mvn clean package -Dmaven.test.skip=true"
}
}
stage('创建docker镜像Dockerfile和程序启动脚本') {
steps {
sh '''start_command="java -Xms${MEM_LIMIT}m -Xmx${MEM_LIMIT}m -jar $serviceName.$packageType --debug=false --server.port=$port"
docker_image="java:8"
echo -e "#!/bin/bash\\n${start_command}" > docker-entrypoint.sh
echo "${start_command}" > startsvc
cat << EOF > Dockerfile
FROM ${docker_image}
MAINTAINER JianJang
WORKDIR /data
COPY ${JENKINS_DIR}/target/$serviceName.$packageType /data
COPY docker-entrypoint.sh /usr/local/bin
ENV TZ Asia/Shanghai
ENV LANG en_US.UTF-8
RUN chmod +x /usr/local/bin/docker-entrypoint.sh && ln -s /usr/local/bin/docker-entrypoint.sh
ENTRYPOINT ["docker-entrypoint.sh"]
EOF'''
}
}
//这里需要修改，改成自己的路径
stage('docker镜像构建') {
steps{
sh "sudo docker build -f Dockerfile -t ${imageName}:${imageVersion} ."
echo 'docker镜像构建成功'
}
}
stage('docker镜像上传') {
steps{
sh "sudo docker login --username=${registryLoginName} --password=${registryLoginPassword} ${harborAddress}"
sh "sudo docker tag ${imageName}:${imageVersion} ${harborAddress}/${harborNameSpace}/${imageName}:${imageVersion}"
sh "sudo docker push ${harborAddress}/${harborNameSpace}/${imageName}:${imageVersion}"
sh "sudo docker logout ${harborAddress}"
sh "sudo docker rmi -f ${imageName}:${imageVersion}"
echo "docker镜像上传成功"
}
}
stage('远程执行镜像更新部署pod') {
steps {
script {
def remote = [:]
remote.name = 'uat'
remote.host = '192.168.2.2'
remote.user = 'root'
remote.password = 'zwx123456'
remote.allowAnyHosts = true
stage('生成并上传k8s部署文件') {
writeFile file: "${serviceName}-deployment.yaml", text: """kind: Deployment
apiVersion: apps/v1
metadata:
namespace: ${k8sNameSpace}
name: uat-${serviceName}-app
spec:
selector:
matchLabels:
app: uat-${serviceName}-app
replicas: ${replicasNum}
template:
metadata:
labels:
app: uat-${serviceName}-app
spec:
imagePullSecrets:
- name: ${k8sHarborTokenId}
containers:
- name: uat-${serviceName}-app
image: ${harborAddress}/${harborNameSpace}/${imageName}:${imageVersion}
imagePullPolicy: IfNotPresent
ports:
- containerPort: ${containerPort}"""
//将文件上传到k8s服务器
sshPut remote: remote, from: "${serviceName}-deployment.yaml", into: '/tmp'
}
stage('远程执行kubectl更新部署pod') {
sshCommand remote: remote, command: "kubectl apply -f /tmp/${serviceName}-deployment.yaml"
}
}
}
}
stage('远程执行镜像更新部署Service') {
steps {
script {
def remote = [:]
remote.name = 'uat'
remote.host = '192.168.2.2'
remote.user = 'root'
remote.password = 'zwx123456'
remote.allowAnyHosts = true
stage('生成并上传k8s部署文件') {
writeFile file: "${serviceName}-svc.yaml", text: """kind: Service
apiVersion: v1
metadata:
namespace: ${k8sNameSpace}
name: uat-${serviceName}-app
spec:
selector:
app: uat-${serviceName}-app
type: NodePort
ports:
- protocol: TCP
port: ${port}
targetPort: ${containerPort}
nodePort: ${k8sServicePort}"""
//将文件上传到k8s服务器
sshPut remote: remote, from: "${serviceName}-svc.yaml", into: '/tmp'
}
stage('远程执行kubectl更新部署service') {
sshCommand remote: remote, command: "kubectl apply -f /tmp/${serviceName}-svc.yaml"
}
}
}
}
}
}
1.4、创建Pipeline任务
点击任务重的配置，将脚本copy到pipeline流水线中。
1.5、构建任务
创建完成后点击【Build Now】按钮进行离开构建和部署，构建过程如下图：
构建日志
部署日志如下：
Started by user 张健
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /root/.jenkins/workspace/jenkins-deploy-k8s-pipeline
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (拉取代码)
[Pipeline] echo
拉取项目[jenkins-demo]代码[git@gitlabs.jianjang.com:zhangjian_sh/jenkins-demo.git] 开始
[Pipeline] git
Selected Git installation does not exist. Using Default
The recommended git tool is: NONE
using credential git-ssh-key
> /usr/bin/git rev-parse --resolve-git-dir /root/.jenkins/workspace/jenkins-deploy-k8s-pipeline/.git # timeout=10
Fetching changes from the remote Git repository
> /usr/bin/git config remote.origin.url git@gitlabs.jianjang.com:zhangjian_sh/jenkins-demo.git # timeout=10
Fetching upstream changes from git@gitlabs.jianjang.com:zhangjian_sh/jenkins-demo.git
> /usr/bin/git --version # timeout=10
> git --version # 'git version 1.8.3.1'
using GIT_SSH to set credentials git仓库ssh秘钥
[INFO] Currently running in a labeled security context
[INFO] Currently SELinux is 'enforcing' on the host
> /usr/bin/chcon --type=ssh_home_t /root/.jenkins/workspace/jenkins-deploy-k8s-pipeline@tmp/jenkins-gitclient-ssh8647914336046518610.key
> /usr/bin/git fetch --tags --progress git@gitlabs.jianjang.com:zhangjian_sh/jenkins-demo.git +refs/heads/*:refs/remotes/origin/* # timeout=10
> /usr/bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=10
Checking out Revision a36715ad0a50b84275212fb59d3cf3e030dfc953 (refs/remotes/origin/master)
> /usr/bin/git config core.sparsecheckout # timeout=10
> /usr/bin/git checkout -f a36715ad0a50b84275212fb59d3cf3e030dfc953 # timeout=10
> /usr/bin/git branch -a -v --no-abbrev # timeout=10
> /usr/bin/git branch -D master # timeout=10
> /usr/bin/git checkout -b master a36715ad0a50b84275212fb59d3cf3e030dfc953 # timeout=10
Commit message: "feat: 测试类"
> /usr/bin/git rev-list --no-walk a36715ad0a50b84275212fb59d3cf3e030dfc953 # timeout=10
[Pipeline] echo
拉取项目[jenkins-demo]代码[git@gitlabs.jianjang.com:zhangjian_sh/jenkins-demo.git] 结束
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (构建项目)
[Pipeline] sh
+ mvn clean package -Dmaven.test.skip=true
[INFO] Scanning for projects...
[INFO]
[INFO] -------------------< com.jianjang.demo:jenkins-demo >-------------------
[INFO] Building jenkins-demo 1.0.1
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ jenkins-demo ---
[INFO] Deleting /root/.jenkins/workspace/jenkins-deploy-k8s-pipeline/target
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ jenkins-demo ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 0 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ jenkins-demo ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 3 source files to /root/.jenkins/workspace/jenkins-deploy-k8s-pipeline/target/classes
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ jenkins-demo ---
[INFO] Not copying test resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ jenkins-demo ---
[INFO] Not compiling test sources
[INFO]
[INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ jenkins-demo ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:3.2.0:jar (default-jar) @ jenkins-demo ---
[INFO] Building jar: /root/.jenkins/workspace/jenkins-deploy-k8s-pipeline/target/jenkins-demo.jar
[INFO]
[INFO] --- spring-boot-maven-plugin:2.3.1.RELEASE:repackage (repackage) @ jenkins-demo ---
[INFO] Replacing main artifact with repackaged archive
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 16.888 s
[INFO] Finished at: 2022-05-15T18:42:14+08:00
[INFO] ------------------------------------------------------------------------
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (创建docker镜像Dockerfile和程序启动脚本)
[Pipeline] sh
+ start_command='java -Xms512m -Xmx512m -jar jenkins-demo.jar --debug=false --server.port=8081'
+ docker_image=java:8
+ echo -e '#!/bin/bash\njava -Xms512m -Xmx512m -jar jenkins-demo.jar --debug=false --server.port=8081'
+ echo 'java -Xms512m -Xmx512m -jar jenkins-demo.jar --debug=false --server.port=8081'
+ cat
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (docker镜像构建)
[Pipeline] sh
+ sudo docker build -f Dockerfile -t jenkins-demo:20220515184147_2 .
Sending build context to Docker daemon 17.51MB
Step 1/9 : FROM java:8
---> d23bdf5b1b1b
Step 2/9 : MAINTAINER JianJang
---> Using cache
---> 2ec11b101878
Step 3/9 : WORKDIR /data
---> Using cache
---> 99f0538d561b
Step 4/9 : COPY /target/jenkins-demo.jar /data
---> 3315a8ea4027
Step 5/9 : COPY docker-entrypoint.sh /usr/local/bin
---> 92e98d0c7db2
Step 6/9 : ENV TZ Asia/Shanghai
---> [Warning] IPv4 forwarding is disabled. Networking will not work.
---> Running in 107e29fdce6c
Removing intermediate container 107e29fdce6c
---> b88cd2287c59
Step 7/9 : ENV LANG en_US.UTF-8
---> [Warning] IPv4 forwarding is disabled. Networking will not work.
---> Running in 8f4b32982ed1
Removing intermediate container 8f4b32982ed1
---> f471135b2dad
Step 8/9 : RUN chmod +x /usr/local/bin/docker-entrypoint.sh && ln -s /usr/local/bin/docker-entrypoint.sh
---> [Warning] IPv4 forwarding is disabled. Networking will not work.
---> Running in d5863ee7887d
Removing intermediate container d5863ee7887d
---> 0793875298cb
Step 9/9 : ENTRYPOINT ["docker-entrypoint.sh"]
---> [Warning] IPv4 forwarding is disabled. Networking will not work.
---> Running in 0c9ce0e6eea6
Removing intermediate container 0c9ce0e6eea6
---> 3ed0bcdcf2b9
Successfully built 3ed0bcdcf2b9
Successfully tagged jenkins-demo:20220515184147_2
[Pipeline] echo
docker镜像构建成功
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (docker镜像上传)
[Pipeline] sh
+ sudo docker login --username=zhangjian_sh --password=zwx123456ZJ.. harbor.jianjang.com:1443
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
[Pipeline] sh
+ sudo docker tag jenkins-demo:20220515184147_2 harbor.jianjang.com:1443/jianjang/jenkins-demo:20220515184147_2
[Pipeline] sh
+ sudo docker push harbor.jianjang.com:1443/jianjang/jenkins-demo:20220515184147_2
The push refers to repository [harbor.jianjang.com:1443/jianjang/jenkins-demo]
1dcfa6972eec: Preparing
0218d112b7bb: Preparing
c8de0441cbcc: Preparing
c780ad03ea84: Preparing
35c20f26d188: Preparing
c3fe59dd9556: Preparing
6ed1a81ba5b6: Preparing
a3483ce177ce: Preparing
ce6c8756685b: Preparing
30339f20ced0: Preparing
0eb22bfb707d: Preparing
a2ae92ffcd29: Preparing
ce6c8756685b: Waiting
c3fe59dd9556: Waiting
6ed1a81ba5b6: Waiting
a3483ce177ce: Waiting
30339f20ced0: Waiting
0eb22bfb707d: Waiting
a2ae92ffcd29: Waiting
c780ad03ea84: Layer already exists
35c20f26d188: Layer already exists
6ed1a81ba5b6: Layer already exists
c3fe59dd9556: Layer already exists
ce6c8756685b: Layer already exists
a3483ce177ce: Layer already exists
0eb22bfb707d: Layer already exists
30339f20ced0: Layer already exists
1dcfa6972eec: Pushed
0218d112b7bb: Pushed
a2ae92ffcd29: Layer already exists
c8de0441cbcc: Pushed
20220515184147_2: digest: sha256:e1d1a36721df7cddd4d15a8ac80636b35fef3c287f25bbe40b8760f01c29df99 size: 2833
[Pipeline] sh
+ sudo docker logout harbor.jianjang.com:1443
Removing login credentials for harbor.jianjang.com:1443
[Pipeline] sh
+ sudo docker rmi -f jenkins-demo:20220515184147_2
Untagged: jenkins-demo:20220515184147_2
[Pipeline] echo
docker镜像上传成功
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (远程执行镜像更新部署pod)
[Pipeline] script
[Pipeline] {
[Pipeline] stage
[Pipeline] { (生成并上传k8s部署文件)
[Pipeline] writeFile
[Pipeline] sshPut
Sending a file/directory to uat[192.168.2.2]: from: /root/.jenkins/workspace/jenkins-deploy-k8s-pipeline/jenkins-demo-deployment.yaml into: /tmp
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (远程执行kubectl更新部署pod)
[Pipeline] sshCommand
Executing command on uat[192.168.2.2]: kubectl apply -f /tmp/jenkins-demo-deployment.yaml sudo: false
deployment.apps/uat-jenkins-demo-app configured
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (远程执行镜像更新部署Service)
[Pipeline] script
[Pipeline] {
[Pipeline] stage
[Pipeline] { (生成并上传k8s部署文件)
[Pipeline] writeFile
[Pipeline] sshPut
Sending a file/directory to uat[192.168.2.2]: from: /root/.jenkins/workspace/jenkins-deploy-k8s-pipeline/jenkins-demo-svc.yaml into: /tmp
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (远程执行kubectl更新部署service)
[Pipeline] sshCommand
Executing command on uat[192.168.2.2]: kubectl apply -f /tmp/jenkins-demo-svc.yaml sudo: false
service/uat-jenkins-demo-app unchanged
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
1.6、服务验证
登录k8s管理后台，查看服务部署情况，两个节点启动成功，如下图:
节点信息
服务信息
| 节点类型 | IP | 端口 |
|---|---|---|
| NodePort | 10.10.225.223 | 30008 |
| pod | 10.122.169.178 | 8081 |
| pod | 10.122.36.111 | 8081 |
请求服务接口验证,负载均衡效果
curl 192.168.2.2:30008
以上jenkins部署到k8s服务过程完毕。
