## 1. 准备工作 1. 首先你得拥有一个阿里云个人或企业级的账号，并存在可支配余额； 2. 购置一台阿里云ECS，作为我们的部署发布机器，并申请一个可用的域名； 3. 注意后续我们会购置阿里云k8s集群，并且还将附带购置多个ECS运行节点。 > 部署机推荐配置：4 vCPU 8 GiB CentOS 7.8 64位 100Mbps 带宽按量付费 ## 2. 安装 jdk-1.8 ```bash yum install java-1.8.0-openjdk.x86_64 -y yum install -y java-1.8.0-openjdk-devel.x86_64 java -version # openjdk version "1.8.0_372" # OpenJDK Runtime Environment (build 1.8.0_372-b07) # OpenJDK 64-Bit Server VM (build 25.372-b07, mixed mode) ``` ## 3. 安装 nexus ```bash cd /usr/local mkdir -p soft/nexus cd soft/nexus wget --no-check-certificate https://download.sonatype.com/nexus/3/nexus-3.56.0-01-unix.tar.gz tar zxvf nexus-3.56.0-01-unix.tar.gz ``` 配置 nexus 的环境变量： ```bash vim /etc/profile ## 将以下内容复制到 profile文件最后 # nexus export NEXUS_HOME=/usr/local/soft/nexus/nexus-3.56.0-01 export PATH=$PATH:$NEXUS_HOME/bin source /etc/profile ``` 修改 nexus 的端口配置并启动（阿里云安全组开放8084端口）： ```bash vim nexus-3.56.0-01/etc/nexus-default.properties ## 将以下内容替换原文件中的内容 # Jetty section application-port=8084 application-host=0.0.0.0 cd nexus-3.56.0-01/bin ./nexus start # 其他命令 (nexus stop/restart/status) ``` 浏览器中打开 http://{你的nexus地址:端口} 默认账号：admin
默认密码：使用如下命令查看 ```bash cat /usr/local/soft/nexus/sonatype-work/nexus3/admin.password ``` 添加阿里云的资源库到 maven-pubic： ![image.png](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/d8f7e395a2d9452db8d7d0a43f0bd034~tplv-k3u1fbpfcp-watermark.image?) 选择 maven2（proxy）： ![image.png](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/de4ce0ad35dd455093f4afeee53b65f1~tplv-k3u1fbpfcp-watermark.image?) 填写名称 aliyun-public 和地址 https://maven.aliyun.com/repository/public ![1688627442844.jpg](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/81c87fffc9d142a1b8709f5b7fec55a2~tplv-k3u1fbpfcp-watermark.image?) 选择 maven-public 导入刚才的资源库： ![1688627559080.jpg](https://p6-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/bc061345041045e18ad15e8d73625a92~tplv-k3u1fbpfcp-watermark.image?) ![1688627617259.jpg](https://p6-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/9cb4f8d2f0524e7b9cc87be576aed1bf~tplv-k3u1fbpfcp-watermark.image?) ## 4. 安装 maven ```bash cd /usr/local mkdir -p soft/maven cd soft/maven wget --no-check-certificate https://dlcdn.apache.org/maven/maven-3/3.9.3/binaries/apache-maven-3.9.3-bin.tar.gz tar zxvf apache-maven-3.9.3-bin.tar.gz ``` 配置 maven 的环境变量： ```bash vim /etc/profile ## 将以下内容复制到 profile文件最后 # maven export MAVEN_HOME=/usr/local/soft/maven/apache-maven-3.9.3 export PATH=$PATH:$MAVEN_HOME/bin source /etc/profile mvn --version # Java version: 1.8.0_372, vendor: Red Hat, Inc., runtime: /usr/lib/jvm/java-1.8.0-# openjdk-1.8.0.372.b07-1.el7_9.x86_64/jre # Default locale: en_US, platform encoding: UTF-8 # OS name: "linux", version: "3.10.0-1127.19.1.el7.x86_64", arch: "amd64", family: "unix" ``` 修改 maven 的配置文件 settings.xml，指向我们自己搭建的 nexus 服务： ```bash vim apache-maven-3.9.3/conf/settings.xml ``` settings.xml 文件：注意替换 {你的nexus密码} 和 {你的nexus地址:端口} ```xml nexus admin ${你的nexus密码} maven-releases admin ${你的nexus密码} maven-snapshots admin ${你的nexus密码} nexus * http://${你的nexus地址:端口}/repository/maven-public/ alimaven central https://maven.aliyun.com/repository/public/ alimaven_central central http://maven.aliyun.com/nexus/content/repositories/central/ nexus central http://central true true central http://central true true jdk1.8 true 1.8 UTF-8 1.8 1.8 1.8 nexus jdk1.8 ``` ## 5. 安装 gitlab ```bash vi /etc/yum.repos.d/gitlab-ce.repo ## 将以下内容写入文件 [gitlab-ce] name=Gitlab CE Repository baseurl=https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el$releasever/ gpgcheck=0 enabled=1 yum makecache yum clean all yum install -y gitlab-ce ``` 修改 gitlab 的端口配置并重启： ```bash vi /etc/gitlab/gitlab.rb ## 将以下内容写入文件，阿里云安全组开放8081端口 gitlab_rails['time_zone'] = 'Asia/Shanghai' external_url 'http://物理机ECS的公网IP:8081' puma['port'] = 8066 nginx['listen_port'] = 8081 gitlab-ctl reconfigure gitlab-ctl restart # 其他命令 (gitlab-ctl stop/start) systemctl enable gitlab-runsvdir.service ``` 浏览器中打开 http://{你的gitlab地址:端口} 默认账号：root
默认密码：使用如下命令查看 ```bash cat /etc/gitlab/initial_root_password ``` ![1688628361497.jpg](https://p1-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/a98847709cdb4e0faf04d5d87f077291~tplv-k3u1fbpfcp-watermark.image?) 后续自己新建账号权限等，这里不再赘述，应该是一个程序员的基本技能。 ## 6. 安装 gitlab-runner ```bash ## 回到根目录或找个自己喜欢的目录 wget --no-check-certificate https://mirrors.tuna.tsinghua.edu.cn/gitlab-runner/yum/el7/gitlab-runner-16.0.2-1.x86_64.rpm yum -y install gitlab-runner-16.0.2-1.x86_64.rpm systemctl status gitlab-runner # ● gitlab-runner.service - GitLab Runner # Loaded: loaded (/etc/systemd/system/gitlab-runner.service; enabled; vendor preset: disabled) # Active: active (running) since Wed 2023-07-05 18:06:25 CST; 21h ago # Main PID: 2045 (gitlab-runner) # Tasks: 10 # Memory: 21.7M # CGroup: /system.slice/gitlab-runner.service # └─2045 /usr/bin/gitlab-runner run --working-directory /home/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --user root ``` 将 gitlab-runner 的用户设置为 root： ```bash sudo gitlab-runner uninstall gitlab-runner install --working-directory /home/gitlab-runner --user root systemctl restart gitlab-runner.service ``` 将 gitlab-runner 注册到 gitlab，起名 tag 为 runner-k8s : ![1688628885558.jpg](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/e611011d58094c1084488f16544ed032~tplv-k3u1fbpfcp-watermark.image?) 注意人家说的版本最低要求，有些设置后续也可以再进来设置： ![1688628976781.jpg](https://p6-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/7e78ccb138f04acabb5f74c60900afb2~tplv-k3u1fbpfcp-watermark.image?) ![1688629071774.jpg](https://p9-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/9435d3b1ae8f48139d0a6876719af610~tplv-k3u1fbpfcp-watermark.image?) ![1688629192664.jpg](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/dcd4327f759f423a9db721b56d9ac0fb~tplv-k3u1fbpfcp-watermark.image?) 执行 gitlab 要求给你的 token 命令就行： ```bash gitlab-runner register --url http://{你的gitlab地址:端口} --token glrt-xkke7cBL2HDux-o9ttAz ``` 命令执行后，出来的表单按要求步骤分别填写： 1. http://{你的gitlab地址:端口} 2. runner-k8s（这里填写runner的tag后续有用） 3. shell（这里选择shell直接在机器上执行命令即可） ![1688629331426.jpg](https://p6-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/81c459c2fe614ea5855b50d38927c8df~tplv-k3u1fbpfcp-watermark.image?) ## 7. 安装 docker ```bash yum -y install net-tools yum-utils yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo yum -y install docker-ce docker-ce-cli containerd.io systemctl enable docker systemctl start docker docker --version # Docker version 24.0.2, build cb74dfc ``` 在阿里云搜索“容器镜像服务”，创建个人实例： ![1688629813224.jpg](https://p6-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/1063eeca498e4be897ed0cf8883f4e98~tplv-k3u1fbpfcp-watermark.image?) 配置上我们自己的阿里云镜像加速地址： ![1688629879792.jpg](https://p1-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/7c6d289c5dfa45f9bbc659254cbfcc98~tplv-k3u1fbpfcp-watermark.image?) ```bash mkdir -p /etc/docker sudo tee /etc/docker/daemon.json <<-'EOF' { "registry-mirrors": ["https://你的加速地址.mirror.aliyuncs.com"] } EOF sudo systemctl daemon-reload sudo systemctl restart docker ``` 进入个人实例，选择公网IP和你的阿里云账号登录： ```bash sudo docker login --username=你的阿里云账号 registry.ap-southeast-1.aliyuncs.com # 执行后填写密码即可登录 ``` ![1688630021519.jpg](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/a45163ffc4e340728b3736eafd6762c7~tplv-k3u1fbpfcp-watermark.image?) 创建一个命名空间 belife 后续有用，公开安全性低但是无需 docker login，私有需要在每台 ECS 机器上 docker login，这个可以自己选择（我们方便演示直接选择公开）： ![1688630270628.jpg](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/583f0e48618c43499deeac4591961fca~tplv-k3u1fbpfcp-watermark.image?) ## 8. 购置阿里云 k8s 集群在阿里云搜索“容器服务 Kubernetes 版”，这里贴出推荐的购买配置，关于 k8s 的前置知识，可以去作者本人的专栏学习： [Kubernetes 原理与实战专栏 --> https://juejin.cn/column/7174618899512557624](https://juejin.cn/column/7174618899512557624) 版本选择 1.22，因为 1.24 版本及以上不支持 docker 容器会比较麻烦： ![1688630820269.jpg](https://p9-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/14b0d23c155f493faf7d957c8d559cde~tplv-k3u1fbpfcp-watermark.image?) ![1688630924345.jpg](https://p6-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/8c8fd237d4bb49ddbadff4ea75459891~tplv-k3u1fbpfcp-watermark.image?) ![1688631028460.jpg](https://p6-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/4dabf9cdd0474081acb5832b7e52ab6e~tplv-k3u1fbpfcp-watermark.image?) 节点配置至少选择 4核 8G，否则无法满足最低运行条件： ![image.png](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/defccd7f5a1c44dca183c034ba407536~tplv-k3u1fbpfcp-watermark.image?) ![1688631258977.jpg](https://p6-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/b22ac2e65469405c8d35443c094676bb~tplv-k3u1fbpfcp-watermark.image?) 我们选择 nginx-ingress，阿里云购买后会自动建立 SLB 负载均衡等必备基础设施： ![image.png](https://p9-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/619d3a8e7923417589a00405ca987c01~tplv-k3u1fbpfcp-watermark.image?) 刚才我们选择的所有的配置清单，大致的服务费用计费方式： ![1688631496918.jpg](https://p6-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/ee1f78a3380c4c83925d6b4ebcc324ca~tplv-k3u1fbpfcp-watermark.image?) 购买完成后等待一段时间，进入我们自己创建的集群查看资源等一些信息： ![1688631853491.jpg](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/c4fbf4635a40420490b371454505ec0a~tplv-k3u1fbpfcp-watermark.image?) 购置新节点，节点扩容，按照我们刚才的配置，至少2个以上节点才能支撑整个集群： ![1688634625611.jpg](https://p6-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/06c7dcc8535642be8138d2ca45bf7979~tplv-k3u1fbpfcp-watermark.image?) 阿里云会自动按照我们上边预设的ECS配置购买节点，如想变更可以在这里设置： ![1688634744919.jpg](https://p9-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/6b0f931a951c45ddaeecffab17b0c64f~tplv-k3u1fbpfcp-watermark.image?) ## 9. 安装 kubectrl 我们回到自己的发布部署机器，回到根目录： ```bash curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" curl -LO "https://dl.k8s.io/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl.sha256" echo "$(cat kubectl.sha256) kubectl" | sha256sum --check sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl kubectl version --short # Client Version: v1.27.3 # Kustomize Version: v5.0.1 # Server Version: v1.22.15-aliyun.1 # WARNING: version difference between client (1.27) and server (1.22) exceeds the supported minor version skew of +/-1 ``` 进入集群查看连接信息，将阿里云黑色部分信息（公网或内网）存贮到我们的部署机器： ```bash mkdir -p /root/.kube cd /root/.kube vim config # 黑色部分的内容复制进去 kubectl cluster-info # Kubernetes control plane is running at https://172.16.214.227:6443 # metrics-server is running at https://172.16.214.227:6443/api/v1/namespaces/kube-system/services/heapster/proxy # KubeDNS is running at https://172.16.214.227:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy # To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. ``` ![1688631994129.jpg](https://p1-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/458d3e1f6bbc47979be9d08de732d5d2~tplv-k3u1fbpfcp-watermark.image?) ## 10. 新建一个 SpringBoot 项目注意项目的 maven 配置，和刚才机器上的一样，连接至我们搭建好的 nexus 私服： ![image.png](https://p9-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/cda7477db0ae41f4bd9b022c9225115b~tplv-k3u1fbpfcp-watermark.image?) pom.xml ```xml 4.0.0 org.springframework.boot spring-boot-starter-parent 2.7.0 com.belife belife-web 1.0.0-SNAPSHOT UTF-8 1.8 1.8 org.springframework.boot spring-boot-starter org.springframework.boot spring-boot-starter-web org.springframework.boot spring-boot-starter-test test belife-web org.springframework.boot spring-boot-maven-plugin org.apache.maven.plugins maven-compiler-plugin ${maven.compiler.source} ${maven.compiler.target} ``` application.yml ```yml server: port: 8080 spring: application: name: belife-web profiles: active: dev ``` BelifeWebApplication.java ```java package com.belife; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; @SpringBootApplication public class BelifeWebApplication { public static void main(String[] args) { SpringApplication.run(BelifeWebApplication.class, args); } } ``` TestController.java ```java package com.belife.api; import org.springframework.beans.factory.annotation.Value; import org.springframework.web.bind.annotation.RequestMapping; import org.springframework.web.bind.annotation.RequestMethod; import org.springframework.web.bind.annotation.RestController; @RestController @RequestMapping("/v1") public class TestController { @Value("${spring.profiles.active}") private String env; @RequestMapping(value = "/heartbeat", method = {RequestMethod.GET}) public String heartbeat() { return "OK"; } @RequestMapping(value = "/app/test", method = {RequestMethod.GET, RequestMethod.POST}) public String test() { return "This is a test, env: " + env; } } ``` 在 gitlab 上新建一个 belife 分组，将项目提交到我们的自己搭建的 gitlab 上边： ![1688632808886.jpg](https://p1-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/4937520e00cf4c75b1a25d6668b54ffc~tplv-k3u1fbpfcp-watermark.image?) ## 11. 编写 CI/CD 相关的脚本 Dockerfile（这里的内容不做过多解释，感觉应该是程序员的必备技能） ``` FROM azul/zulu-openjdk-centos:8 ENV LANG en_US.UTF-8 ENV LC_ALL en_US.UTF-8 ENV LANGUAGE en_US:en ENV TZ=Asia/Shanghai RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone ADD ./target/belife-web.jar /srv WORKDIR /srv CMD java ${JAVA_OPTS} -jar /srv/belife-web.jar ``` .gitlab-ci.yml 这里我们定义了3个流程步骤： 1. **package 主要是 maven 打包相关**； 2. **build 主要是 docker 镜像打包 push 相关**； 3. **deploy 主要是发布到阿里云 k8s 集群，涉及的 yaml 资源清单后续会给出**； ``` stages: - package - build - deploy variables: NAMESPACE: belife-${CI_COMMIT_BRANCH} NODE_LABELS: deploy-${CI_COMMIT_BRANCH} REGISTRY_NAME: "registry.ap-southeast-1.aliyuncs.com/belife" REGISTRY_USERNAME: belife REGISTRY_PASSWORD: 7e8086e81b9d5804b4b7d68c97207bb9 PROJECT_NAME: ${CI_PROJECT_NAME}-${CI_COMMIT_BRANCH} IMAGE_NAME: ${REGISTRY_NAME}/${PROJECT_NAME} DEPLOY_NAME: ${PROJECT_NAME} SERVICE_NAME: service-${PROJECT_NAME} INGRESS_NAME: ingress-${PROJECT_NAME} LOG_STORE_OUT: aliyun_logs_${PROJECT_NAME} LOG_STORE_INIT: aliyun_logs_${PROJECT_NAME}_logstore PORT: "80" HEALTH_URL: "/v1/heartbeat" JAVA_OPTS: "-Dspring.profiles.active=${CI_COMMIT_BRANCH} -Dserver.port=${PORT} \ -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfile.encoding=utf8 \ -Xms1024m -Xmx1024m -XX:MetaspaceSize=256M -XX:MaxNewSize=512m -XX:MaxMetaspaceSize=512m" cache: &global_cache key: ${CI_COMMIT_BRANCH} paths: - "target/${CI_PROJECT_NAME}.jar" maven-package: stage: package tags: - runner-k8s only: - test@belife/belife-web - gray@belife/belife-web - prod@belife/belife-web cache: <<: *global_cache policy: push script: - mvn clean package -Dmaven.test.skip=true docker-build: stage: build tags: - runner-k8s only: - test@belife/belife-web - gray@belife/belife-web - prod@belife/belife-web cache: <<: *global_cache policy: pull script: - docker login -u ${REGISTRY_USERNAME} -p ${REGISTRY_PASSWORD} ${REGISTRY_NAME} - docker build -t ${IMAGE_NAME}:$CI_COMMIT_SHORT_SHA . - docker push ${IMAGE_NAME}:$CI_COMMIT_SHORT_SHA k8s-deploy-test: stage: deploy tags: - runner-k8s only: - test@belife/belife-web cache: {} script: - envsubst < deploy.yaml | cat - - envsubst < deploy.yaml | kubectl apply -f - - envsubst < ingress-test.yaml | cat - - envsubst < ingress-test.yaml | kubectl apply -f - - kubectl rollout status deployment/$DEPLOY_NAME -n $NAMESPACE variables: REPLICAS: 1 k8s-deploy-gray: stage: deploy tags: - runner-k8s only: - gray@belife/belife-web cache: {} script: - envsubst < deploy.yaml | cat - - envsubst < deploy.yaml | kubectl apply -f - - envsubst < ingress-gray.yaml | cat - - envsubst < ingress-gray.yaml | kubectl apply -f - - kubectl rollout status deployment/$DEPLOY_NAME -n $NAMESPACE variables: REPLICAS: 1 NAMESPACE: belife-prod k8s-deploy-prod: stage: deploy tags: - runner-k8s only: - prod@belife/belife-web cache: {} script: - envsubst < deploy.yaml | cat - - envsubst < deploy.yaml | kubectl apply -f - - envsubst < ingress-prod.yaml | cat - - envsubst < ingress-prod.yaml | kubectl apply -f - - kubectl rollout status deployment/$DEPLOY_NAME -n $NAMESPACE variables: REPLICAS: 2 when: manual ``` 我们通过 variables 定义了顶层的环境变量，这些环境变量不仅能在 .gitlab-ci.yml 中使用，还能在 k8s 相关的 yaml 资源清单文件中使用；tag 就是我们刚才设置的 gitlab-runner 的 tag 叫 runner-k8s 来执行； {CI_PROJECT_NAME} 和 {CI_COMMIT_BRANCH} 是 gitlab-runner 的内置变量，{CI_PROJECT_NAME} 就是 belife-web，{CI_COMMIT_BRANCH} 就是分支名称，我们通过 test，gray，prod 分支的提交，触发对应环境的 deploy 发布动作；gray，prod 灰度和生产必须位于同一个 namespace，test 和 gray 我们只启动 1 个容器实例，prod 我们设置启动 2 个容器实例，并且设置 deploy 动作为 manual 手动执行。 deploy.yaml（这里主要是 Pod 和 Service 的资源清单） ```yml apiVersion: apps/v1 kind: Deployment metadata: name: $DEPLOY_NAME # Pod名称称 namespace: $NAMESPACE # 命名空间 spec: replicas: $REPLICAS # 分片个数 revisionHistoryLimit: 3 selector: matchLabels: app: $PROJECT_NAME template: metadata: labels: app: $PROJECT_NAME spec: restartPolicy: Always containers: - name: $CI_PROJECT_NAME image: ${IMAGE_NAME}:$CI_COMMIT_SHORT_SHA # 镜像名称 imagePullPolicy: Always # 总是拉取镜像 resources: limits: # 资源限制 cpu: 500m memory: 2000Mi requests: # 所需资源 cpu: 200m memory: 1200Mi ports: - name: http containerPort: $PORT # 端口设置 env: - name: JAVA_OPTS # 环境变量 value: $JAVA_OPTS - name: $LOG_STORE_OUT # 阿里云SLS相关 value: stdout - name: $LOG_STORE_INIT # 阿里云SLS相关 value: $PROJECT_NAME startupProbe: # 启动检查探针 httpGet: path: $HEALTH_URL port: $PORT scheme: HTTP initialDelaySeconds: 5 timeoutSeconds: 1 periodSeconds: 5 successThreshold: 1 failureThreshold: 60 livenessProbe: # 存活检查探针 httpGet: path: $HEALTH_URL port: $PORT scheme: HTTP initialDelaySeconds: 5 timeoutSeconds: 2 periodSeconds: 5 successThreshold: 1 failureThreshold: 3 readinessProbe: # 就绪检查探针 httpGet: path: $HEALTH_URL port: $PORT scheme: HTTP timeoutSeconds: 2 periodSeconds: 5 successThreshold: 1 failureThreshold: 3 affinity: nodeAffinity: # 节点亲和性，软亲和性 preferredDuringSchedulingIgnoredDuringExecution: - weight: 1 preference: matchExpressions: - key: node.labels.deploy operator: NotIn # 防止分片启动到同一个节点 values: - $NODE_LABELS --- apiVersion: v1 kind: Service metadata: name: $SERVICE_NAME namespace: $NAMESPACE spec: selector: app: $PROJECT_NAME type: ClusterIP ports: - port: $PORT targetPort: $PORT ``` 截取发布后的几个图，方便大家对照注释理解： ![1688634313910.jpg](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/249ed6b1af6e4b68887de390ebfc1c70~tplv-k3u1fbpfcp-watermark.image?) ![1688634350880.jpg](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/4d52e2eaf2c64ce890515a661f1d2ae4~tplv-k3u1fbpfcp-watermark.image?) ![1688634371380.jpg](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/d375f5b16e93461c9552ecee21af7cba~tplv-k3u1fbpfcp-watermark.image?) ![1688634425606.jpg](https://p6-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/d7cb415b67ad4947889712b536ce8a9a~tplv-k3u1fbpfcp-watermark.image?) ![1688634456680.jpg](https://p6-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/5347a8197c0749bfba1a0b22cfbb08d0~tplv-k3u1fbpfcp-watermark.image?) ![1688634485591.jpg](https://p9-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/570706fda3f14842a591051c844e1441~tplv-k3u1fbpfcp-watermark.image?) ![1688634923974.jpg](https://p6-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/234d2732fda84766b509dc628001b22d~tplv-k3u1fbpfcp-watermark.image?) 不同环境的 nginx-ingress 路由配置，这里我们专门做了灰度发布的教程，注意 gray 和 prod 不仅位于同一个命名空间，路由配置也是同一个，发布后会动态覆盖实现不同效果，后文会具体阐述； ingress-test.yaml ```yml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: $INGRESS_NAME namespace: $NAMESPACE annotations: kubernetes.io/ingress.class: "nginx" spec: rules: - host: test.home.belifeapp.net http: paths: - path: /v1 pathType: Prefix backend: service: name: ${SERVICE_NAME} port: number: $PORT ``` ingress-prod.yaml ```yml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-${CI_PROJECT_NAME}-gray-prod namespace: $NAMESPACE annotations: kubernetes.io/ingress.class: "nginx" spec: rules: - host: home.belifeapp.net http: paths: - path: /v1 pathType: Prefix backend: service: name: ${SERVICE_NAME} port: number: $PORT ``` ingress-gray.yaml ```yml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-${CI_PROJECT_NAME}-gray-prod namespace: $NAMESPACE annotations: kubernetes.io/ingress.class: "nginx" nginx.ingress.kubernetes.io/service-weight: | service-${CI_PROJECT_NAME}-gray: 30, service-${CI_PROJECT_NAME}-prod: 70 spec: rules: - host: home.belifeapp.net http: paths: - path: /v1 pathType: Prefix backend: service: name: service-${CI_PROJECT_NAME}-gray port: number: $PORT - path: /v1 pathType: Prefix backend: service: name: service-${CI_PROJECT_NAME}-prod port: number: $PORT ``` ## 12. 发布到测试环境首先去阿里云的 k8s 集群建立名为 belife-test 的命名空间： ![1688635627624.png](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/b5d0ba44a9724bbbac18c86dc3a5da4f~tplv-k3u1fbpfcp-watermark.image?) idea 新建一个名为 test 的分支提交或者合并，即可触发开始发布流程： ![1688635627627.jpg](https://p6-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/156f48cdee2646378933c8648ccfa35d~tplv-k3u1fbpfcp-watermark.image?) 我们进入 gitlab，选择自己的项目 Build -> Pipelines 即可看到执行过程和日志： ![1688635699056.jpg](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/96323daafcf848828dc77a971d95bf7b~tplv-k3u1fbpfcp-watermark.image?) 我们看到部署完成后，去集群命名空间里看下是否 OK，点击编辑就能看到上边的 Pod 截图信息： ![1688636103744.jpg](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/cbba8fa7b13448f9a7cbdbf635f01a89~tplv-k3u1fbpfcp-watermark.image?) 点击 Pod 名称 belife-web-test 进入，我们就能看到容器的日志信息： ![1688697575049.jpg](https://p9-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/dcfbc649be3a494688e58f6bef1558da~tplv-k3u1fbpfcp-watermark.image?) 我们还可以从集群的资源列表直接点击进入阿里云 SLS 日志库，我们配置了自动建库： ![1688697674348.jpg](https://p9-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/0d697e259171448f9407bb59d897257c~tplv-k3u1fbpfcp-watermark.image?) ![image.png](https://p1-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/9f3dbdc714864d76b17799e65858bb76~tplv-k3u1fbpfcp-watermark.image?) 查看阿里云购置 k8s 集群自动为我们配置的 SLB 负载均衡服务，可以从我们的集群资源列表直接点击链接进入，另一个是 Api-Server 的： ![image.png](https://p1-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/6cc567c5a3b04f06b0976046840a5648~tplv-k3u1fbpfcp-watermark.image?) ![1688635850427.jpg](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/103f36bf68814aeeae709bbcab843aa8~tplv-k3u1fbpfcp-watermark.image?) 在阿里云搜索“域名服务”，配置域名解析到这个公网 IP，公网 IP 我图中涂掉了： ![1688636019074.jpg](https://p6-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/6810f6c751d44c118ac3662d1bfe56ec~tplv-k3u1fbpfcp-watermark.image?) 浏览器试试地址：[http://test.home.belifeapp.net/v1/app/test](http://test.home.belifeapp.net/v1/app/test) ``` This is a test, env: test ``` ## 13. 发布到生产环境生产环境的发布和测试环境类似，发布完成后我们看下是否 OK： ![image.png](https://p1-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/292f55add4764b44b92d968a5fb607a8~tplv-k3u1fbpfcp-watermark.image?) ![1688636367264.jpg](https://p1-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/8a7d099af33a4c72a10d6c0032ffd74d~tplv-k3u1fbpfcp-watermark.image?) 浏览器试试地址：[http://home.belifeapp.net/v1/app/test](http://home.belifeapp.net/v1/app/test) ``` This is a test, env: prod ``` ## 14. 灰度发布（蓝绿发布）我们会将 belife-web-gray 和 belife-web-prod 发布到同一个命名空间 belife-prod，通过 nginx-ingress 动态切换流量； ![1688636483526.jpg](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/3ceaa3fc12754fd181ef6051018b32f4~tplv-k3u1fbpfcp-watermark.image?) 不知道各位是否忘记上边的 ingress-gray.yaml 中有下边几条配置： ```yml annotations: kubernetes.io/ingress.class: "nginx" nginx.ingress.kubernetes.io/service-weight: | service-${CI_PROJECT_NAME}-gray: 30, service-${CI_PROJECT_NAME}-prod: 70 ``` 进入集群的流量切分三七开，30% 流向 belife-web-gray 灰度环境，70% 流向 belife-web-prod 线上环境，当然百分比你自己设置；浏览器试试地址：[http://home.belifeapp.net/v1/app/test](http://home.belifeapp.net/v1/app/test) ``` This is a test, env: gray # 30%概率 This is a test, env: prod # 70%概率 ``` ### Nginx Ingress Controller 的 Annotation 注解说明 nginx.ingress.kubernetes.io/service-match 该注解用来配置新版本服务的路由匹配规则。 ```bash nginx.ingress.kubernetes.io/service-match: | : # 参数说明： # service-name：服务名称，满足match-rule的请求会被路由到该服务中。 # match-rule：路由匹配规则 # # 路由匹配规则： # 1. 支持的匹配类型 # - header：基于请求头，支持正则匹配和完整匹配。 # - cookie：基于cookie，支持正则匹配和完整匹配。 # - query：基于请求参数，支持正则匹配和完整匹配。 # # 2. 匹配方式 # - 正则匹配格式：/{regular expression}/，//表明采用正则方式匹配。 # - 完整匹配格式："{exact expression}"，""表明采用完整方式匹配。 ``` nginx.ingress.kubernetes.io/service-weight 该注解用来配置新旧版本服务的流量权重。 ```bash nginx.ingress.kubernetes.io/service-weight: | :, : 参数说明： new-svc-name：新版本服务名称 new-svc-weight：新版本服务权重 old-svc-name：旧版本服务名称 old-svc-weight：旧版本服务权重 ``` ### 灰度验证完成，再次发布生产环境细心的各位是否已经发现，在上边 ingress-gray.yaml 和 ingress-prod.yaml 中，ingress 路由名字都是同一个 name: ingress-belife-web-gray-prod，这意味着什么？再来看下 ingress-prod.yaml ```yml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-${CI_PROJECT_NAME}-gray-prod namespace: $NAMESPACE annotations: kubernetes.io/ingress.class: "nginx" spec: rules: - host: home.belifeapp.net http: paths: - path: /v1 pathType: Prefix backend: service: name: ${SERVICE_NAME} port: number: $PORT ``` 这代表着，如果我再次发布生产环境，原先灰度的名为 ingress-belife-web-gray-prod 的路由规则会被覆盖，因为本身灰度和生产用的同一个路由配置名，kubectrl apply 命令会覆盖更新；当测试验证完灰度环境后，所有的流向将 100% 切换到 belife-web-prod 线上环境（如果你想把灰度环境资源也当作线上环境利用起来可以不切换，但是要保证两个环境的镜像也就是代码功能是一致的）。 ### 根据用户ID，精准切换流量到灰度环境存在这样一种发布场景，公司开发了几个大型的新功能，想优先拉一部分用户进行体验（当然也包括本公司测试同学），看看效果和反响，还想通过这种方式发现一些 BUG 和问题。下面给出一种方案，我们以用户登录 oauth 2.0 为例子来看： 1. 首先服务端应存在一张配置表，可以动态将某个用户 ID 设置为灰度用户； 2. 在用户登录成功颁发 access_token 时，根据服务端的配置，可以在 access_token 上做一个区分，如正式环境的 `access_token = 3EiKJrXQlWWQrngDhmW6FlYC8vW4MpcA`，颁发给灰度的可以为 `access_token = gray-3EiKJrXQlWWQrngDhmW6FlYC8vW4MpcA`； 3. 在 nginx-ingress 中进行头配置区分流量，判断 access_token 以 `gray` 开头则切换灰度环境； 4. 如果用户已经登录过，这时候动态切换环境，服务端可以在下次客户端通过 refresh_token 更换 access_token 时，颁发给客户端新的 access_token 来区分环境；假如你必须想达到实时生效，那就要另外探索服务端实时消息通知客户端的方案。 ```yml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-${CI_PROJECT_NAME}-gray-prod namespace: $NAMESPACE annotations: kubernetes.io/ingress.class: "nginx" nginx.ingress.kubernetes.io/service-match: | service-${CI_PROJECT_NAME}-gray: header("access_token", /^gray-.+$/) ## 流量切换规则 spec: rules: - host: home.belifeapp.net http: paths: - path: /v1 pathType: Prefix backend: service: name: service-${CI_PROJECT_NAME}-gray port: number: $PORT - path: /v1 pathType: Prefix backend: service: name: service-${CI_PROJECT_NAME}-prod port: number: $PORT ``` 从上边的配置来看，只要 access_token 的正则满足 `^gray-.+$` 即以 `gray-` 开头就将流量动态切换至灰度环境。 ```yml annotations: kubernetes.io/ingress.class: "nginx" nginx.ingress.kubernetes.io/service-match: | service-${CI_PROJECT_NAME}-gray: header("access_token", /^gray-.+$/) ``` 当然灰度发布的做法很多，小伙伴们可以自己探索，作者这里只是简单的给出一个演示案例。 ## 15. 配置 https 证书和 http 跳转在阿里云搜索“数字证书管理服务”，下载我们买的证书选择 Apache 的 crt/key： ![1688701145300.jpg](https://p6-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/3539e3e753314d2e8c48f88af2aa3bdd~tplv-k3u1fbpfcp-watermark.image?) 进入集群管理的配置管理->保密字典，将下载的证书内容写入，注意下载下来的证书有三个文件，我们只使用名字类似 "belifeapp.net.key" 和 "belifeapp.net_public.crt" 的文件： ![1688701400362.jpg](https://p1-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/916e9730403b4b6a95c4266510c8129f~tplv-k3u1fbpfcp-watermark.image?) 如图，我们配置的保密字典的名字是 "belife-tls"： ![1688701593969.jpg](https://p1-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/327de4e65bf1496bb700f26156536bbe~tplv-k3u1fbpfcp-watermark.image?) 最后我们修改 nginx-ingress 的配置，加入`nginx.ingress.kubernetes.io/force-ssl-redirect: "true"` 表示强制 http 跳转 https，加入 `tls: ... secretName: belife-tls` 配置我们的证书： ```yml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-${CI_PROJECT_NAME}-gray-prod namespace: $NAMESPACE annotations: kubernetes.io/ingress.class: "nginx" nginx.ingress.kubernetes.io/force-ssl-redirect: "true" ## 强制跳转https nginx.ingress.kubernetes.io/service-weight: | service-${CI_PROJECT_NAME}-gray: 30, service-${CI_PROJECT_NAME}-prod: 70 spec: tls: - hosts: - home.belifeapp.net secretName: belife-tls ## 配置tls的证书 rules: - host: home.belifeapp.net http: paths: - path: /v1 pathType: Prefix backend: service: name: service-${CI_PROJECT_NAME}-gray port: number: $PORT - path: /v1 pathType: Prefix backend: service: name: service-${CI_PROJECT_NAME}-prod port: number: $PORT ``` 浏览器测试地址 https：[https://home.belifeapp.net/v1/app/test](https://home.belifeapp.net/v1/app/test) 浏览器测试地址 http 会自动跳转到 https：[http://home.belifeapp.net/v1/app/test](http://home.belifeapp.net/v1/app/test)