基于tekton实现cicd流水线自动发布
案例的配置接近生产环境：使用tekton定义运行任务，tekton-trigger监听gitlab事件，触发cicd流水线
此处未写gitlab部署，相信看到这里的小伙伴也知道怎么部署gitlab了
1）部署harbor
#harbor生成tls证书
mkdir -p /etc/harbor/certs/
cd !$
wget https://files.cnblogs.com/files/blogs/731344/cert.sh
sh cert.sh
#下载harbor安装程序
export https_proxy=http://frp1.freefrp.net:16324
wget https://github.com/goharbor/harbor/releases/download/v2.10.0/harbor-offline-installer-v2.10.0.tgz
unset https_proxy
tar xf harbor-offline-installer-v2.10.0.tgz
#配置harbor
cd harbor/
cp harbor.yml.tmpl harbor.yml
vim harbor.yml
#安装
./install.sh
cp /usr/libexec/docker/cli-plugins/docker-compose /usr/local/bin/
#测试访问harbor
curl -ILk 127.0.0.1
生成证书，ip可以是自己的ip，我这里自签证书，且是测试使用，没有域名解析，所以加个ip方便识别
配置harbor，指定证书路径
配置harbor登录密码
2）docker配置harbor信任
#为docker配置harbor的证书
mkdir -p /etc/docker/certs.d/harbor.hj.com
\cp /etc/harbor/certs/* !$
cd !$
ln -s ca.crt ca.cert
ln -s harbor.hj.com.crt harbor.hj.com.cert
echo 2.2.2.67 harbor.hj.com >> /etc/hosts
#这里简便写了，直接不验证，不然每个都要创建对应目录复制证书，比较麻烦，如果要验证证书，参考上面的步骤即可
cat > /etc/docker/daemon.json <<eof
{
"insecure-registries": [
"2.2.2.67",
"2.2.2.67:80",
"2.2.2.67:443"
]
}
eof
#登录
docker login -u admin harbor.hj.com -p 123456
docker login -u admin 2.2.2.67 -p 123456
docker login -u admin 2.2.2.67:80 -p 123456
docker login -u admin 2.2.2.67:443 -p 123456
docker完成校验证书登录，走https
走http登录
3）containerd配置harbor信任
cri信任
修改：/etc/containerd/config.toml，新增以下内容
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."2.2.2.67".tls]
insecure_skip_verify = false
ca_file = "/etc/harbor/certs/ca.crt"
[plugins."io.containerd.grpc.v1.cri".registry.configs."2.2.2.67:80".tls]
insecure_skip_verify = false
ca_file = "/etc/harbor/certs/ca.crt"
[plugins."io.containerd.grpc.v1.cri".registry.configs."2.2.2.67:443".tls]
insecure_skip_verify = false
ca_file = "/etc/harbor/certs/ca.crt"
#配置直接为containerd配置账号登录，这种做法不安全，建议还是使用k8s的secret保存信息，这里配置后，这台主机上面拉2.2.2.67:80仓库都可直接访问
#[plugins."io.containerd.grpc.v1.cri".registry.configs."2.2.2.67:80".auth]
# username = "admin"
# password = "123456"
#这一步必须配置，上面的配置在未开启https时，都可以不配置，下面的必须配置后containerd（cri）才能访问harbor仓库
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."2.2.2.67"]
endpoint = ["http://2.2.2.67"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."2.2.2.67:80"]
endpoint = ["http://2.2.2.67:80"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."2.2.2.67:443"]
endpoint = ["https://2.2.2.67:443"]
ctr信任
新版本配置方式，官方文档中说这个是新配置方式，用于替代直接修改/etc/containerd/config.toml文件，但使用后发现这设计太反人类，并不能做到完全替代上面cri的信任方法，这里添加后，任需要在主配置文件中加入这段配置
不过此方法相比于修改主配置文件，好处在于不用重启containerd服务
注：配置有问题，暂未找出原因，下面的配置http时ctr可以使用，https时不能正常工作
#指定跳过对：2.2.2.67:80/镜像名 的https验证（这对ctr、cri都生效）
mkdir /etc/containerd/certs.d/2.2.2.67:80/
cat > /etc/containerd/certs.d/2.2.2.67:80/hosts.toml <<eof
server = "http://2.2.2.67:80"
[host."http://2.2.2.67:80"]
capabilities = ["pull", "resolve", "push"]
ca = ["/path/to/ca.crt"]
skip_verify = true
eof
测试
crictl pull --creds admin:123456 2.2.2.67:80/spring-boot-helloworld/spring-boot-helloworld
ctr image pull --plain-http=true -u admin:123456 2.2.2.67:80/spring-boot-helloworld/spring-boot-helloworld:latest
4）创建相关依赖secret
#创建docker认证凭据，后面使用kaniko构建镜像，需要挂载此文件才能访问harbor
kubectl create secret generic docker-config --from-file=/root/.docker/config.json
#创建harbor的证书，同样也是kaniko依赖的
kubectl create secret generic harbor-cert --from-file /etc/harbor/certs/harbor.hj.com.crt
#创建barbor拉镜像的凭据，执行部署任务时，要调用此凭据从harbor获取构建完成的镜像
#由于secret与pod模板的image都是base64编码，2者的harbor仓库地址有一点区别都不行，比如image指定了端口，secret没指定端口就不匹配
kubectl create secret docker-registry harbor-http-secret --docker-server=2.2.2.67 --docker-username=admin --docker-password=123456
5）创建gitlab webhook的回调token
#token要手动生成，生成命令：openssl rand -base64 12
token=`openssl rand -base64 32`
kubectl apply -f - <<eof
apiVersion: v1
kind: Secret
metadata:
name: gitlab-webhook-token
type: Opaque
stringData:
webhookToken: "$token"
eof
#查看一下token，保存下，后面需要用到
echo $token | tee token
6）配置tekton-trigger的rbac授权
允许事件监听器访问tekton-trigger的相关组件，确保有权限访问
apiVersion: v1
kind: ServiceAccount
metadata:
name: tekton-triggers-gitlab-sa
secrets:
- name: gitlab-webhook-token
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: tekton-triggers-gitlab-minimal
rules:
# Permissions for every EventListener deployment to function
- apiGroups: ["triggers.tekton.dev"]
resources:
- eventlisteners
- triggerbindings
- interceptors
- triggertemplates
- triggers
verbs:
- get
- list
- watch
- apiGroups: [""]
# secrets are only needed for Github/Gitlab interceptors, serviceaccounts only for per trigger authorization
resources:
- "configmaps"
- "secrets"
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- serviceaccounts
verbs:
- impersonate
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
# Permissions to create resources in associated TriggerTemplates
- apiGroups: ["tekton.dev"]
resources:
- pipelineruns
- pipelineresources
- taskruns
verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: tekton-triggers-gitlab-binding
subjects:
- kind: ServiceAccount
name: tekton-triggers-gitlab-sa
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: tekton-triggers-gitlab-minimal
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: tekton-triggers-gitlab-minimal
rules:
- apiGroups:
- triggers.tekton.dev
resources:
- clustertriggerbindings
- clusterinterceptors
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: tekton-triggers-gitlab-binding
subjects:
- kind: ServiceAccount
name: tekton-triggers-gitlab-sa
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: tekton-triggers-gitlab-minimal
7）创建业务应用运行的sa
apiVersion: v1
kind: ServiceAccount
metadata:
name: helloworld-admin
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: helloworld-admin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: helloworld-admin
namespace: default
8）申请pvc
用于缓存打包文件，加速maven项目构建速度
注：需要使用存储类，图方便的话可以参考之前写的关于nfs-csi的文章：nfs-csi驱动部署
kubectl apply -f - <<eof
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mvn-cache
spec:
storageClassName: nfs-csi
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
eof
9）创建cicd任务
apiVersion: tekton.dev/v1
kind: Task
metadata:
name: git-clone
spec:
description: Clone the code repository to the workspace.
params:
- name: url
type: string
default: ""
- name: branch
type: string
default: "main"
workspaces:
- name: source
steps:
- name: git-clone
image: alpine/git:2.40.1
script: |
sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories
date
apk add tzdata bash
cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
echo "Asia/Shanghai" > /etc/timezone
cat > tmp.sh <<eof
#!/bin/bash
if [[ \$(params.branch) =~ ^refs/heads/(.*)$ ]] ;then
echo "\${BASH_REMATCH[1]}"
else
echo main
fi
eof
branch=`bash -x tmp.sh`
echo "$branch -- $(params.url) -- $(workspaces.source.path)"
git clone -v -b $branch $(params.url) $(workspaces.source.path)
ls -hltr
---
# 打包
apiVersion: tekton.dev/v1
kind: Task
metadata:
name: build-to-pkg
spec:
description: build application and package the files to image
workspaces:
- name: source
steps:
- name: build
image: maven:3.8.7-openjdk-18-slim
workingDir: $(workspaces.source.path)
script: |
sed -i -e '/\/mirrors/i\ <mirror>' \
-e '/\/mirrors/i\ <id>aliyunmaven</id>' \
-e '/\/mirrors/i\ <mirrorOf>*</mirrorOf>' \
-e '/\/mirrors/i\ <name>阿里云公共仓库</name>' \
-e '/\/mirrors/i\ <url>https://maven.aliyun.com/repository/public</url>' \
-e '/\/mirrors/i\ </mirror>' \
/usr/share/maven/conf/settings.xml
mvn clean install
volumeMounts:
- name: mvn-cache
mountPath: /root/.m2
volumes:
- name: mvn-cache
persistentVolumeClaim:
claimName: mvn-cache
---
# 生成镜像版本id
apiVersion: tekton.dev/v1
kind: Task
metadata:
name: generate-build-id
spec:
params:
- name: version
type: string
default: latest
results:
- name: datetime
- name: buildId
steps:
- name: generate-datetime
image: busybox
script: |
datetime=`date +%Y%m%d-%H%M%S`
echo -n ${datetime} | tee $(results.datetime.path)
- name: generate-buildid
image: alpine
script: |
sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories
date
apk add tzdata bash
cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
echo "Asia/Shanghai" > /etc/timezone
cat > tmp.sh <<eof
#!/bin/bash
if [[ \$(params.version) =~ ^refs/tags/(.*)$ ]] ;then
echo "\${BASH_REMATCH[1]}"
else
echo latest
fi
eof
tag=`bash -x tmp.sh`
buildDatetime=`cat $(results.datetime.path)`
buildId="${tag}-${buildDatetime}"
[ `echo $tag |grep latest` ] && buildId="${tag}"
echo -n ${buildId} |tee $(results.buildId.path)
---
# 镜像构建上传
apiVersion: tekton.dev/v1
kind: Task
metadata:
name: img-build
spec:
description: package the application files to image
params:
- name: dockerfile
default: Dockerfile
- name: image-url
- name: image-tag
default: latest
workspaces:
- name: source
- name: dockerconfig
#将docker账号信息，挂载到kaniko容器中，用于推送镜像到仓库
mountPath: /kaniko/.docker
- name: harbor-cert
mountPath: /tmp/cert
steps:
- name: build-and-push-img
image: m.daocloud.io/gcr.io/kaniko-project/executor:debug
securityContext:
runAsUser: 0
command:
- /kaniko/executor
args:
#选项解释：https://github.com/GoogleContainerTools/kaniko#additional-flags
- -f=$(params.dockerfile) #dockefile文件名
- -c=$(workspaces.source.path) #构建镜像时的工作目录
- -d=$(params.image-url):$(params.image-tag) #镜像上传路径与标签
- --cache=true
- --cache-repo
- --cache-run-layers
#缓存7天，默认2周，单位h
- --cache-ttl=168h
#- --insecure
#- --insecure-registry=2.2.2.67 #http访问的harbor，允许多次指定
#- --insecure-registry=harbor.hj.com
- --push-retry=3
- --image-download-retry=2
- --registry-certificate=2.2.2.67=$(workspaces.source.path)/harbor.hj.com.crt #格式：仓库url=证书路径
#- --registry-certificate=2.2.2.67=$(workspaces.source.path)/ca.crt
#- --skip-tls-verify #所有push/pull都跳过tls验证
#允许多次指定，不验证tls的仓库
- --skip-tls-verify-registry=2.2.2.67
- --skip-tls-verify-registry=harbor.hj.com
#跳过push权限检验，可以提高访问速度
- --skip-push-permission-check
---
# kubectl部署到集群
apiVersion: tekton.dev/v1
kind: Task
metadata:
name: deploy-using-kubectl
spec:
workspaces:
- name: source
description: The git repo
params:
- name: deploy-config-file
- name: docker-secret
- name: image-url
- name: image-tag
steps:
- name: update-yaml
image: alpine:3.16
script: |
set -x
sed -i -e "s@__IMAGE__@$(params.image-url):$(params.image-tag)@g" \
-e "s#__harbor_secret__#$(params.docker-secret)#g" \
"$(workspaces.source.path)/deploy/$(params.deploy-config-file)"
- name: run-kubectl
image: lachlanevenson/k8s-kubectl
command: ["kubectl"]
args:
#当前部署的集群与kubectl相同集群，所以不需要传递kubeconfig
- "apply"
- "-f"
- "$(workspaces.source.path)/deploy/$(params.deploy-config-file)"
10）创建流水线，编排任务
apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
name: source-to-img
spec:
params:
- name: git-url
- name: git-revision
type: string
- name: img-build-context
default: .
- name: image-url
- name: deploy-config-file
default: all-in-one.yaml
- name: version
type: string
default: "latest"
- name: docker-secret
#default: docker-config
workspaces:
- name: codebase
- name: docker-config
- name: harbor-cert
tasks:
- name: git-clone
taskRef:
name: git-clone
params:
- name: url
value: "$(params.git-url)"
- name: branch
value: "$(params.version)"
workspaces:
- name: source
workspace: codebase
- name: build-to-pkg
taskRef:
name: build-to-pkg
workspaces:
- name: source
workspace: codebase
runAfter:
- git-clone
- name: generate-build-id
taskRef:
name: generate-build-id
params:
- name: version
value: "$(params.version)"
runAfter:
- git-clone
- name: img-build-push
taskRef:
name: img-build
params:
- name: image-url
value: "$(params.image-url)"
- name: image-tag
value: "$(tasks.generate-build-id.results.buildId)"
workspaces:
- name: source
workspace: codebase
- name: dockerconfig
workspace: docker-config
- name: harbor-cert
workspace: harbor-cert
runAfter:
- build-to-pkg
- generate-build-id
- name: deploy-to-cluster
taskRef:
name: deploy-using-kubectl
workspaces:
- name: source
workspace: codebase
params:
- name: deploy-config-file
value: "$(params.deploy-config-file)"
- name: docker-secret
value: "$(params.docker-secret)"
- name: image-url
value: "$(params.image-url)"
- name: image-tag
value: "$(tasks.generate-build-id.results.buildId)"
runAfter:
- img-build-push
11）创建触发器运行任务模板
apiVersion: triggers.tekton.dev/v1beta1
kind: TriggerTemplate
metadata:
name: gitlab-trigger-tt
spec:
params: # 定义参数
- name: git-revision
- name: git-repo-url
- name: image-url
- name: version
- name: docker-secret
resourcetemplates:
- apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
generateName: gitlab-trigger-run- # PipelineRun 名称前缀
spec:
serviceAccountName: default
pipelineRef:
name: source-to-img
taskRunSpecs:
- pipelineTaskName: deploy-to-cluster
taskServiceAccountName: helloworld-admin
params:
- name: git-url
value: $(tt.params.git-repo-url)
- name: git-revision
value: $(tt.params.git-revision)
- name: image-url
value: $(tt.params.image-url)
- name: version
value: $(tt.params.version)
- name: docker-secret
value: $(tt.params.docker-secret)
workspaces:
- name: codebase
volumeClaimTemplate:
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 100Mi
storageClassName: nfs-csi
- name: docker-config
secret:
secretName: docker-config
- name: harbor-cert
secret:
secretName: harbor-cert
12) 事件数据获取触发器
apiVersion: triggers.tekton.dev/v1beta1
kind: TriggerBinding
metadata:
name: gitlab-binding
spec:
params:
- name: git-revision
value: $(body.checkout_sha)
- name: git-repo-url
value: $(body.repository.git_http_url)
- name: image-url
value: 2.2.2.67/spring-boot-helloworld/spring-boot-helloworld
- name: version
value: $(body.ref)
#底层cri拉镜像时，登录harbor仓库的账号密码，harbor名称必须和params.git-repo-url的名称对应，否则会拉镜像失败
- name: docker-secret
value: harbor-http-secret
13）创建事件监听器
apiVersion: triggers.tekton.dev/v1beta1
kind: EventListener
metadata:
name: gitlab-event-listener
spec:
serviceAccountName: tekton-triggers-gitlab-sa
triggers:
- name: gitlab-push-events-trigger
interceptors:
- ref:
name: "gitlab"
params:
- name: "secretRef"
value:
secretName: gitlab-webhook-token
secretKey: webhookToken
- name: "eventTypes"
value:
- "Push Hook"
- "Tag Push Hook"
- "Merge Request Hook"
bindings:
- ref: gitlab-binding
template:
ref: gitlab-trigger-tt
14）解析gitlab webhook地址
由于测试环境，没有ingress，也没有公网域名，这里直接暴露事件监听器的端口，方便gitlab识别
##### k8s主机操作
kubectl port-forward deploy/el-gitlab-event-listener --address 0.0.0.0 8080
#为tekton配置一个虚拟ip
ip link a vip0 type dummy
ip add a 2.2.2.67/32 dev vip0
#查看事件监听器的访问地址，后面要将这个添加到gitlab webhook
kubectl get svc -l app.kubernetes.io/managed-by=EventListener
##### gitlab主机操作
#gitlab主机中添加hosts解析，解析tekton的事件监听器地址
echo 2.2.2.67 el-gitlab-event-listener.default.svc.cluster.local >> /etc/hosts
curl el-gitlab-event-listener.default.svc.cluster.local:8080
在自己的本地gitlab中导入案例git仓库，地址：gitee.com/mageedu/spr…
https、ssh协议自选，并配置为公开项目（此处是测试，所以配置方便访问，生产环境一般是私有，则需要配置访问gitlab的secret）
文件如下
手动在仓库中创建webhook，我这里手动生成的token是：DXeqvozMlTA67aQB 订阅事件：推送、标签、合并请求
关闭tls验证
15）测试
#创建标签后，查看流水线运行情况
tkn pipelinerun ls
tkn pipelinerun logs -f gitlab-trigger-run-mnfpb
新建一个标签，测试发布
事件监听器监听到tag事件，开始自动运行流水线
图形界面可以看到，任务都执行完成，mvn也用到了缓存，克隆打包到发布总共用41秒
获取到最新tag是8.0
harbor仓库也有了最新的镜像
此时，pod的镜像版本是最新的8.0
