K3S + Argo CD 初创公司云原生利器

摘要：本文主要介绍k3s的简单安装配置和argo-cd的配置使用，k3s做为容器平台，argo-cd做为ci cd工具，极大的方便我们交付的版本管理，更加贴合开发人员工作，让效率更高。

k3s安装

国内加速地址安装

curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn sh -

配置镜像加速

编辑/etc/rancher/k3s/registries.yaml文件；https://docker.m.daocloud.io有白名单限制了，不建议用了，下面还配置了私服的地址和密码。

mirrors:
  docker.io:
    endpoint:
      - "https://docker.1panel.live"
      - "https://docker.m.daocloud.io"
  192.168.137.100:5000:
    endpoint:
      - "http://192.168.137.100:5000"
configs:
  "192.168.137.100:5000":
    auth:
      username: registry
      password: ui

重启k3s

systemctl restart k3s

结果

root@u-151:/# kubectl get pod --all-namespaces
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   coredns-7b98449c4-nzzg9                   1/1     Running     0          43m
kube-system   helm-install-traefik-crd-dbc5x            0/1     Completed   0          43m
kube-system   helm-install-traefik-vwpfb                0/1     Completed   1          43m
kube-system   local-path-provisioner-6795b5f9d8-h6g7w   1/1     Running     0          43m
kube-system   metrics-server-cdcc87586-7477l            1/1     Running     0          43m
kube-system   svclb-traefik-3544762d-8ggdj              2/2     Running     0          11m
kube-system   traefik-67f6c94c47-chvrh                  1/1     Running     0          11m

registry简易docker仓库安装

方便后面镜像的更新使用，临时搞一个仓。

• htpasswd文件

对应的用户名和密码是registry:ui

registry:$2y$11$1bmuJLK8HrQl5ACS/WeqRuJLUArUZfUcP2R23asmozEpfN76.pCHy

• config.yml

配置文件

version: 0.1
log:
  fields:
    service: registry
storage:
  delete:
    enabled: true
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /var/lib/registry
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
    Access-Control-Allow-Origin: ['*']
    Access-Control-Allow-Methods: ['HEAD', 'GET', 'OPTIONS', 'DELETE']
    Access-Control-Allow-Headers: ['Authorization', 'Accept']
    Access-Control-Max-Age: [1728000]
    Access-Control-Allow-Credentials: [true]
    Access-Control-Expose-Headers: ['Docker-Content-Digest']
auth:
  htpasswd:
    realm: basic-realm
    path: /etc/docker/registry/htpasswd

• docker-compose.yml

version: '3'
services:
  registry:
    image: registry:2
    volumes:
      - ./registry-data:/var/lib/registry
      - ./config.yml:/etc/docker/registry/config.yml
      - ./htpasswd:/etc/docker/registry/htpasswd
    networks:
      - default

  ui:
    image: joxit/docker-registry-ui:latest
    ports:
      - 5000:80
    environment:
      - REGISTRY_TITLE=My Private Docker Registry
      - NGINX_PROXY_PASS_URL=http://registry:5000
      - SINGLE_REGISTRY=true
    depends_on:
      - registry
    networks:
      - default

networks:
  default:
    external:
      name: nisec

• 启动docker-compose up -d

访问http://localhost:5000

k3s私服地址配置

编辑/etc/rancher/k3s/registries.yaml文件

mirrors:
  docker.io:
    endpoint:
      - "https://docker.1panel.live"
      - "https://docker.m.daocloud.io"
  192.168.137.100:
    endpoint:
      - "http://192.168.137.100:5000"
configs:
  "http://192.168.137.100:5000":
    auth: 
      username: registry
      password: ui

argo-cd安装

官方安装

kubectl create namespace argocd 
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

开启web访问

# 设置为`NodePort`有端口
kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "NodePort"}}'
# 查看端口
kubectl -n argocd get svc argocd-server
# 查看`admin`密码
kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "NodePort"}}'

• 访问：https://192.168.137.151:32589

用argo-cd部署应用

镜像准备

我使用的是java镜像，用springboot最简单的controller

@RestController
public class ArgoCdController {

    @GetMapping(value = {"","/index"})
    public String index() throws Exception{
        return InetAddress.getLocalHost().getHostAddress() + "argo-cd v1.0.3";
    }

    @GetMapping(value = "/ready")
    public String ready(){
        return "ok";
    }

    @GetMapping(value = "healthz")
    public String healthz(){
        return "ok";
    }
}

Dockerfile

FROM eclipse-temurin:8u402-b06-jdk
LABEL org.opencontainers.image.authors="nisec@nisec.com"
ADD app.jar /app.jar
ENTRYPOINT ["/bin/bash", "-c", "java $JAVA_OPTS -jar app.jar"]

构建镜像

docker build命令构建

docker build -t 192.168.137.100:5000/argo-cd-java:v1.0.0 .
docker push 192.168.137.100:5000/argo-cd-java:v1.0.0

nerdctl命令构建

需要先启动buildkit.service，执行systemctl start buildkit.service，并配置私服地址

• 私服地址配置

在/etc/containerd/certs.d创建192.168.137.100:5000文件夹，编辑hosts.toml文件，内容如下

server = "http://192.168.137.100:5000"

[host."http://192.168.137.100:5000"]
capabilities = ["pull", "push", "resolve"]
skip_verify = true
auth = "registry:ui"

• 构建进行

nerdctl build -t 192.168.137.100:5000/argo-cd-java:v1.0.0 . 

• 推送镜像

nerdctl push 192.168.137.100:5000/argo-cd-java:v1.0.0

buildctl构建

配置/etc/buildkit/buildkitd.toml

配置了加速地址和私服地址

[registry."docker.io"]
  mirrors = ["https://docker.1panel.live"]
  insecure = true
[registry."192.168.137.100:5000"]
  http = true

配置私服地址密码~/.docker/config.json

密码是base64(用户名:密码)，仓库没有密码不用配置

{
  "auths":{
    "192.168.137.100:5000":{
      "auth": "cmVnaXN0cnk6dWk="
    }
  }
}

构建命令

buildctl build --frontend dockerfile.v0 --local context=. --local dockerfile=. --output type=image,name=192.168.137.100:5000/argo-cd-java:v1.0.0,push=true

创建一个git工程来控制部署内容

注意一定要配置readinessProbe，这样svc才会等到服务启动完成后才把流量路由到这个新的pod，否则更新过程中会导致服务中断。

k3s-java.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: nisec
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: nisec
  name: argo-cd-java-deplolyment
spec:
  selector:
    matchLabels:
      app: argo-cd-java
  replicas: 3
  template:
    metadata:
      labels:
        app: argo-cd-java
    spec:
      containers:
        - name: argo-cd-java
          image: 192.168.137.100:5000/argo-cd-java:v1.0.3
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 15
            failureThreshold: 30
            periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  namespace: nisec
  name: argo-cd-java-service
spec:
  selector:
    app: argo-cd-java
  type: NodePort
  ports:
    - port: 8080
      targetPort: 8080
      nodePort: 30007

argo-cd配置

新增Project

配置Repository

我用的ssh，没有开启https，且是非标准端口的ssh

配置项目