使用Docker创建Jenkins容器
# 主要是在容器中挂载docker程序
docker run -d -u 0 -p 8099:8080 \
-p 50099:50000 \
-v /usr/local/jenkins:/var/jenkins_home \
-v /usr/bin/docker:/usr/bin/docker \
-v /var/run/docker.sock:/var/run/docker.sock \
--restart always \
--name jenkins jenkins/jenkins:centos7-jdk11
K8s创建ServiceAccount并获取token
首先创建一个命名空间
# 创建namespace.yaml文件，后面的操作在改名称空间下
cat > namespace.yaml << EOF
apiVersion: v1
kind: Namespace
metadata:
name: demoapp
labels:
name: demoapp
EOF
# 应用yaml文件
kubectl apply -f namespace.yaml
创建ServiceAccount获取token
创建yaml文件
# serviceAccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: jenkins
name: jenkins
namespace: demoapp
---
# 注意apiGroups要写apps，写""默认为核心apiGroup
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: jenkins
rules:
- apiGroups: ["apps"]
resources: ["namespaces","deployments"]
verbs: ["create","delete","get","list","patch","update","watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: jenkins
namespace: demoapp
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: jenkins
subjects:
- kind: ServiceAccount
name: jenkins
namespace: demoapp
---
# k8s v1.24.0之后创建ServiceAccount之后不会自动创建Secret，需要手动创建
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
name: jenkins
namespace: demoapp
annotations:
kubernetes.io/service-account.name: "jenkins"
应用yaml文件
[root@master demoapp]# kubectl apply -f serviceAccount.yaml
serviceaccount/jenkins created
role.rbac.authorization.k8s.io/jenkins created
rolebinding.rbac.authorization.k8s.io/jenkins created
secret/jenkins created
获取token
# 获取k8s token
kubectl get secret jenkins -n demoapp -o jsonpath={".data.token"} | base64 -d
将刚才获取的K8s的token添加到Jenkins中，该token用于后面向k8s发送http请求时候使用
类型选择Secret-text
在Jenkins中创建Gitlab的token
首先获取Gitlab的token
将获取的Gitlab的token添加到Jenkins中，该token用于后面向Gitlab发送http请求使用
同时在Jenkins中创建gitlab的用户密码类型的凭据，用于后面拉取代码
Jenkins安装需要用到的插件
- Generic Webhook Trigger
- GitLab
- Pipeline
- HTTP Request ： 用于发送http请求
- Docker Pipeline： 可以在pipeline脚本中使用docker
- Pipeline Utility Steps：该插件有一些常用的工具，例如readYaml等
首先先实现自动打包Docker镜像并推送到Harbor
在jenkins创建一个名称为k8s-devops流水线的项目
配置webhook
配置Jenkins中的webhook
选择Generic Webhook Trigger
设置该Trigger
在Gitlab设置对应的webhook，当有代码提交的时候会自动触发jenkins构建
添加变量
结果测试
- 在该pipeline中添加脚本，内容如下
- 往Gitlab推送代码后，可以看到Jenkins的构建控制台打印输出了该变量
编译代码，并生成docker镜像推送到Harbor
再添加一个变量，用于获取gitlab项目请求地址
编写pipeline脚本自动拉取代码
pipeline脚本如下
pipeline {
agent any
stages {
// 拉取代码
stage('pullCode') {
steps {
script {
branchName = ref - 'refs/heads/' // 获取分支名
// 拉取分支的语法可以用jenkins自动生成，然后修改下就可以
checkout scmGit(branches: [[name: branchName]], extensions: [], userRemoteConfigs: [[credentialsId: 'gitlab-auth', url: gitUrl]])
}
}
}
}
}
结果测试
打包构建镜像推送到Harbor
首先在Jenkins中添加Harbor的凭证
修改pipeline的脚本如下
pipeline {
agent any
stages {
// 拉取代码
stage('pullCode') {
steps {
script {
branchName = ref - 'refs/heads/'
checkout scmGit(branches: [[name: "${branchName}"]], extensions: [], userRemoteConfigs: [[credentialsId: 'gitlab-auth', url: "${gitUrl}"]])
}
}
}
// 打包
stage('goBuild') {
steps {
script {
docker.image('golang:bullseye').inside {
sh """
ls -al
go version
go build -o main
"""
}
}
}
}
// 构建镜像
stage('BuildImage') {
steps {
script {
withCredentials([usernamePassword(credentialsId: 'harbor', passwordVariable: 'harborPass', usernameVariable: 'harborUser')]) {
sh """
# 登陆harbor
docker login -u ${harborUser} -p ${harborPass} www.mysite.top
# 构建镜像,docker会使用go项目中写好的docerfile文件构建镜像
docker build -t www.mysite.top/web/demo-go-app:${branchName} .
sleep 1
# 上传镜像
docker push www.mysite.top/web/demo-go-app:${branchName}
sleep 1
docker rmi www.mysite.top/web/demo-go-app:${branchName}
"""
}
}
}
}
}
}
查看结果
Harbor中已推送成功镜像
编写Jenkins的library用于请求
jenkins的library作用是将一些通用的代码封装在一起，以后在不同的流水线中都可以调用
项目目录结构
➜ jenkinslib git:(main) ✗ tree
.
├── README.md
└── src
└── org
└── devops
├── gitlab.groovy
└── kubernetes.groovy
3 directories, 3 files
创建K8s请求服务使用的library
// kubernetes.groovy文件
package org.devops
// 封装http请求
def httpReq(reqType, reqUrl, reqBody) {
def result
// k8s的api可以参考https://kubernetes.io/zh-cn/docs/reference/using-api/api-concepts/
def apiServer = 'https://192.168.2.24:6443/apis/apps/v1'
withCredentials([string(credentialsId: 'k8s-token', variable: 'k8stoken')]) {
result = httpRequest consoleLogResponseBody: true,
customHeaders: [[maskValue: true, name: 'Authorization', value: "Bearer ${k8stoken}"], [maskValue: false, name: 'Content-Type', value: 'application/yaml'], [maskValue: false, name: 'Accept', value: 'application/yaml']],
httpMode: reqType,
ignoreSslErrors: true,
requestBody: reqBody,
url: "${apiServer}/${reqUrl}",
wrapAsMultipart: false
}
return result
}
// 获取某一个deployment
def getDeployment(nameSpace, deployName) {
apiUrl = "namespaces/${nameSpace}/deployments/${deployName}"
response = httpReq('GET', apiUrl, '')
return response
}
// 更新deployment
def updateDeployment(nameSpace, deployName, deployBody) {
apiUrl = "namespaces/${nameSpace}/deployments/${deployName}"
response = httpReq('PATCH', apiUrl, deployBody)
println(response)
}
// 获取所有的deployments
def getAllDeployment(nameSpace) {
apiUrl = "namespaces/${nameSpace}/deployments"
response = httpReq('GET', apiUrl, '')
return response.content
}
// 创建一个deployment
def createDeployment(nameSpace, deployBody) {
apiUrl = "namespaces/${nameSpace}/deployments"
response = httpReq('POST', apiUrl, deployBody)
return response.content
}
// 删除一个deployment
def deleteDeployment(nameSpace, deployName) {
apiUrl = "namespaces/${nameSpace}/deployments/${deployName}"
response = httpReq('DELETE', apiUrl, '')
return response.content
}
创建Gitlab请求所使用的library
// gitlab.groovy文件
package org.devops
// 封装http请求
def httpReq(reqType, reqUrl, reqBody) {
def apiServer = 'http://172.157.38.2/api/v4'
withCredentials([string(credentialsId: 'gitlab-token', variable: 'gitlabtoken')]) {
result = httpRequest consoleLogResponseBody: true,
contentType: 'APPLICATION_JSON',
customHeaders: [[maskValue: true, name: 'PRIVATE-TOKEN', value: "${gitlabtoken}"]],
httpMode: reqType,
ignoreSslErrors: true,
requestBody: reqBody,
url: "${apiServer}/${reqUrl}",
wrapAsMultipart: false
}
return result
}
// 获取文件内容
def getRepoFile(projectId, filePath) {
apiUrl = "projects/${projectId}/repository/files/${filePath}/raw"
response = httpReq('GET', apiUrl, '')
return response.content
}
// 更新文件内容
def updateRepoFile(projectId, filePath, fileContent) {
apiUrl = "projects/${projectId}/repository/files/${filePath}"
reqBody = """
{"branch": "main", "encoding": "base64", "content": "${fileContent}", "commit_message": "Update a new file"}
"""
response = httpReq('PUT', apiUrl, reqBody)
println(response.content)
}
使用Gitlab的api，获取yaml文件文件
在Jenkins中添加上面创建的共享库
使用K8s部署项目
K8s上创建一个Docker登陆使用的Secret
kubectl -n demoapp create secret docker-registry registry-key \
--docker-server=www.mysite.top \
--docker-username=root \
--docker-password=Harbor12345
创建一个仓库，用于单独保存K8s部署使用的yaml文件
以后有新的代码提交的时候，都会先从gitlab中拉取该yaml文件，然后将该yaml文件中的镜像替换为新的镜像，从而得到一个新的yaml文件
# 目录结构如下，仓库名称为k8s-info-service
➜ k8s-info-service git:(main) tree
.
├── README.md
└── uat
└── deploy.yaml
# deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: demo-go-app
spec:
selector:
matchLabels:
app: demo-go-app
template:
metadata:
labels:
app: demo-go-app
spec:
imagePullSecrets:
- name: registry-key # 拉取镜像时候使用指定的key
containers:
- name: demo-go-app
image: www.mysite.top/web/demo-go-app:release-v0.0.1
imagePullPolicy: Always
K8s自动部署的pipeline脚本文件
// 引用共享库
@Library('jenkinslib@main') _
// 使用java包里面的Base64，因为groovy兼容java
import java.util.Base64
def gitlab = new org.devops.gitlab()
def k8s = new org.devops.kubernetes()
pipeline {
agent any
stages {
// 拉取代码
stage('pullCode') {
steps {
script {
branchName = ref - 'refs/heads/'
checkout scmGit(branches: [[name: "${branchName}"]], extensions: [], userRemoteConfigs: [[credentialsId: 'gitlab-auth', url: "${gitUrl}"]])
}
}
}
// 构建代码
stage('goBuild') {
steps {
script {
docker.image('golang:bullseye').inside {
sh """
ls -al
go version
go build -o main
"""
}
}
}
}
// 生成镜像
stage('BuildImage') {
steps {
script {
withCredentials([usernamePassword(credentialsId: 'harbor', passwordVariable: 'harborPass', usernameVariable: 'harborUser')]) {
sh """
# 登陆harbor
docker login -u ${harborUser} -p ${harborPass} www.mysite.top
# 构建镜像
docker build -t www.mysite.top/web/demo-go-app:${branchName} .
sleep 1
# 上传镜像
docker push www.mysite.top/web/demo-go-app:${branchName}
sleep 1
docker rmi www.mysite.top/web/demo-go-app:${branchName}
"""
}
}
}
}
// 部署到k8s
stage('Deployment') {
steps {
script {
dockerImage = "www.mysite.top/web/demo-go-app:${branchName}"
// 下载版本库文件
// 使用前面创建的k8s-info-service项目，查看gitlab该项目的id为5,所以第一个参数传5
response = gitlab.getRepoFile(5, "uat%2fdeploy.yaml")
// 替换文件的内容（替换镜像)
fileData = readYaml text: """${response}"""
fileData["spec"]["template"]["spec"]["containers"][0]["image"] = dockerImage
String ret = writeYaml returnText: true, data: fileData // 将yaml转换为字符串
String base64encodedString = Base64.getEncoder().encodeToString(ret.getBytes("utf-8")); // 将字符串进行base64编码
println(base64encodedString)
// 更新gitlab中的内容
gitlab.updateRepoFile(5, 'uat%2fdeploy.yaml', base64encodedString)
//获取部署的deployment状态
deployment = k8s.getAllDeployment("demoapp")
// 如果demo-go-app不存在，部署
yamlData = readYaml text: deployment
// 定义一个包含所有的deployemt的数组
deploymentList = []
// 遍历，将demoapp名称空间下的deployment都取出来
for (item in yamlData.items) {
deploymentList.add(item.metadata.name)
}
if (deploymentList.contains('demo-go-app')) {
// 如果已经包含了这个deployment，则先删除，再创建
println("如果已经包含了这个deployment")
k8s.deleteDeployment("demoapp", "demo-go-app")
sleep 1
k8s.createDeployment("demoapp", ret)
} else {
// 没有包含这个deployment，创建
println("没有包含这个deployment，创建")
k8s.createDeployment("demoapp", ret)
}
}
}
}
}
}
创建一个新的分支
git push origin release-v0.0.2:release-v0.0.2
可以看到jenkins开始打包构建了
查看结果
[root@master demoapp]# kubectl get pods -n demoapp
NAME READY STATUS RESTARTS AGE
demo-go-app-54fc9b7596-pv747 0/1 ContainerCreating 0 18s
创建一个Service
apiVersion: v1
kind: Service
metadata:
name: demo-go-app
namespace: demoapp
spec:
selector:
app: demo-go-app
type: NodePort
ports:
- protocol: TCP
port: 9090
targetPort: 9090
nodePort: 31111
应用后访问测试，可以正常访问
➜ ~ curl 192.168.3.192:31111/hello
{"code":200,"success":true}%
其他内容
本次测试使用的dockerfile
FROM ubuntu:latest
COPY main /main
RUN chmod +x /main
EXPOSE 9090
ENTRYPOINT ["/main"]
