Kubernetes策略之Kyverno实践

背景

Kubernetes 已经能够允许人们大规模地运行分布式应用程序来彻底改变云原生生态系统。虽然 Kubernetes 是一个功能丰富、健壮的容器编排平台，但它也有自己的一套复杂性。与多个团队一起大规模管理 Kubernetes 并不容易，而且要确保人们做正确的事情并且不越界是很难管理的。

概念

Kyverno 正是解决这个问题的合适工具。它是一个开源的 Kubernetes 原生策略引擎，可以帮助您使用简单的 Kubernetes manifests 定义策略。它可以验证、修改和生成 Kubernetes 资源。因此，它允许组织定义和执行策略，以便开发人员和管理员保持一定的标准。

Kyverno工作原理

Kyverno 在 Kubernetes 集群中作为动态准入控制器运行。该控制器检查您通过 Kubectl 发送到 Kube API 服务端的每个请求。如果请求与策略匹配，Kyverno 就应用它。否则，它将使用已定义的消息拒绝请求。

架构

一个高可用性安装的 Kyverno 可以运行多个副本，每个 Kyverno 副本都有多个控制器，执行不同的功能。Webhook 处理来自 kubernettes API 服务器的 AdmissionReview 请求，它的 Monitor 组件创建和管理必需的配置。PolicyController 监视策略资源，并根据配置的扫描间隔启动后台扫描。GenerateController 管理生成资源的生命周期。

特性

• 检查 CPU 和内存限制。
• 确保用户不更改默认的网络策略。
• 检查资源名称是否与特定模式匹配。
• 确保特定的资源总是包含特定的标签。
• 拒绝对特定资源的删除和更改。
• 如果镜像标签是 latest 将自动更改 imagePullPolicy 为 Always
• 为每个新的命名空间生成一个默认的网络策略。

安装

yaml 安装

• 安装命令

  kubectl create -f https://raw.githubusercontent.com/kyverno/kyverno/main/config/release/install.yaml
  

• 安装结果

  [root@master kyverno]# kubectl apply -f install.yaml 
  namespace/kyverno created
  customresourcedefinition.apiextensions.k8s.io/clusterpolicies.kyverno.io created
  customresourcedefinition.apiextensions.k8s.io/clusterpolicyreports.wgpolicyk8s.io created
  customresourcedefinition.apiextensions.k8s.io/clusterreportchangerequests.kyverno.io created
  customresourcedefinition.apiextensions.k8s.io/generaterequests.kyverno.io created
  customresourcedefinition.apiextensions.k8s.io/policies.kyverno.io created
  customresourcedefinition.apiextensions.k8s.io/policyreports.wgpolicyk8s.io created
  customresourcedefinition.apiextensions.k8s.io/reportchangerequests.kyverno.io created
  serviceaccount/kyverno-service-account created
  clusterrole.rbac.authorization.k8s.io/kyverno:admin-policies created
  clusterrole.rbac.authorization.k8s.io/kyverno:admin-policyreport created
  clusterrole.rbac.authorization.k8s.io/kyverno:admin-reportchangerequest created
  clusterrole.rbac.authorization.k8s.io/kyverno:customresources created
  clusterrole.rbac.authorization.k8s.io/kyverno:generatecontroller created
  clusterrole.rbac.authorization.k8s.io/kyverno:leaderelection created
  clusterrole.rbac.authorization.k8s.io/kyverno:policycontroller created
  clusterrole.rbac.authorization.k8s.io/kyverno:userinfo created
  clusterrole.rbac.authorization.k8s.io/kyverno:webhook created
  clusterrolebinding.rbac.authorization.k8s.io/kyverno:customresources created
  clusterrolebinding.rbac.authorization.k8s.io/kyverno:generatecontroller created
  clusterrolebinding.rbac.authorization.k8s.io/kyverno:leaderelection created
  clusterrolebinding.rbac.authorization.k8s.io/kyverno:policycontroller created
  clusterrolebinding.rbac.authorization.k8s.io/kyverno:userinfo created
  clusterrolebinding.rbac.authorization.k8s.io/kyverno:webhook created
  configmap/kyverno created
  configmap/kyverno-metrics created
  service/kyverno-svc created
  service/kyverno-svc-metrics created
  deployment.apps/kyverno created
  poddisruptionbudget.policy/kyverno created
  

• 查看安装情况

  [root@master kyverno]# kubectl get pods -n kyverno
  NAME                       READY   STATUS    RESTARTS   AGE
  kyverno-7dc7f46bd7-j2nwc   1/1     Running   0          63s
  

helm安装

• 添加helm repository

  [root@master kyverno]# helm repo add kyverno https://kyverno.github.io/kyverno/
  "kyverno" has been added to your repositories
  

• 更新helm

  [root@master kyverno]# helm repo update
  Hang tight while we grab the latest from your chart repositories...
  ...Successfully got an update from the "kyverno" chart repository
  Update Complete. ⎈Happy Helming!⎈
  

• 安装kyverno

  [root@master kyverno]# helm install kyverno --namespace kyverno kyverno/kyverno --create-namespace
  NAME: kyverno
  LAST DEPLOYED: Mon Jan 10 23:31:24 2022
  NAMESPACE: kyverno
  STATUS: deployed
  REVISION: 1
  NOTES:
  Thank you for installing kyverno v2.1.5 😀
  
  Your release is named kyverno, app version v1.5.3
  

• 查看kyverno安装结果

  [root@master kyverno]# helm list -n kyverno
  NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
  kyverno kyverno         1               2022-01-10 23:31:24.313178176 +0800 CST deployed        kyverno-v2.1.5  v1.5.3
  [root@master kyverno]# kubectl get all -n kyverno
  NAME                           READY   STATUS    RESTARTS   AGE
  pod/kyverno-86bc448594-bskrh   1/1     Running   0          4m22s
  
  NAME                          TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
  service/kyverno-svc           ClusterIP   10.1.182.158   <none>        443/TCP    4m22s
  service/kyverno-svc-metrics   ClusterIP   10.1.199.83    <none>        8000/TCP   4m22s
  
  NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
  deployment.apps/kyverno   1/1     1            1           4m22s
  
  NAME                                 DESIRED   CURRENT   READY   AGE
  replicaset.apps/kyverno-86bc448594   1         1         1       4m22s
  

快速尝试

添加一个策略，要求所有的pod都包含一个 demo 的标签

• 添加yaml文件

  • vim kyverno-label.yaml

    apiVersion: kyverno.io/v1
    kind: ClusterPolicy
    metadata:
      name: require-labels
    spec:
      validationFailureAction: enforce
      rules:
      - name: check-for-labels
        match:
          resources:
            kinds:
            - Pod
        validate:
          message: "label 'demo' is required"
          pattern:
            metadata:
              labels:
                demo: "?*"
    

• 添加到Kubernetes集群

  [root@master kyverno]# kubectl apply -f kyverno-label.yaml 
  clusterpolicy.kyverno.io/require-labels created
  

• 查看策略

  [root@master kyverno]# kubectl get clusterpolicies.kyverno.io 
  NAME             BACKGROUND   ACTION    READY
  require-labels   true         enforce   true
  

添加一个没有 demo 标签的应用

• kubectl create deployment nginx --image=nginx

  [root@master kyverno]# kubectl create deployment nginx --image=nginx
  error: failed to create deployment: admission webhook "validate.kyverno.svc-fail" denied the request: 
  
  resource Deployment/default/nginx was blocked due to the following policies
  
  require-labels:
    autogen-check-for-labels: 'validation error: label ''demo'' is required. Rule autogen-check-for-labels
      failed at path /spec/template/metadata/labels/demo/'
  

  可以看到提示，需要一个 demo 标签

• 重新创建这个应用，增加上 demo 标签

  • kubectl run nginx --image nginx --labels demo=nginx

    [root@master kyverno]# kubectl run nginx --image nginx --labels demo=nginx
    pod/nginx created
    [root@master kyverno]# kubectl get pods -n default 
    NAME                     READY   STATUS              RESTARTS   AGE
    kdemo-598fbd6c64-dgr5h   1/1     Running             0          17d
    nginx                    0/1     ContainerCreating   0          5s
    secret-pod               1/1     Running
    

功能说明

验证资源

验证规则可能是您将要使用的最常见和最实用的规则类型，当用户或进程创建新资源时，Kyverno 将根据验证规则检查该资源的属性。如果验证通过，则允许创建资源。如果验证失败，则创建被阻塞。

验证示例

要求：

1. 创建的pod必须包含 environment 标签。
2. 创建的pod必须增加资源限制（CPU，内存限制）。

创建验证规则

• vim rule-lable-source.yaml

  apiVersion: kyverno.io/v1
  # “ClusterPolicy”适用于整个集群。
  kind: ClusterPolicy
  metadata:
    name: require-lable-source
  # "spec" 定义属性的政策。
  spec:
    # “validationFailureAction”告诉Kyverno如果资源应该允许被认可,但报道(“审计”)或阻塞(“执行”)。
    validationFailureAction: enforce
    # "rules" 是一个或多个规则必须是真实的。
    rules:
    - name: require-lable
      # "match" 的语句设置将检查的范围。在这种情况下,它是任何名称空间的资源。
      match:
        resources:
          kinds:
          - Pod
      # "validate" 声明试图积极检查是什么定义。如果声明,与所请求的资源相比,是真的,这是被允许的。如果错误,它被阻塞。
      validate:
        # "message" 是向用户显示的内容如果这个规则验证失败,因此受阻。
        message: "新创建的 POD 必须包含 标签 "environment""
        # "模式" 的对象定义了模式将检查资源。在这种情况下,它正在寻找的元数据.
        pattern:
          metadata:
            labels:
              environment: "?*"
    - name: require-source
          match:
          resources: 
          kinds:
          - Pod
      validate:
          message: "新创建的 POD 必须设置 CPU 和 内存限制"
        pattern: 
          spec: 
                  containers:
            # 选择所有容器的豆荚。这里的“名称”字段是没有特别要求,但作为一个视觉援助用于教学目的
            - name: "*"
                  resources:
                  limits: 
                  memory: "?*"
                  cpu: "?*"
                requests:
                  memory: "?*"
                  cpu: "?*"
  
  

• kubectl apply -f rule-lable-source.yaml

  [root@master kyverno]# kubectl apply -f rule-lable-source.yaml 
  clusterpolicy.kyverno.io/require-lable-source created
  [root@master kyverno]# kubectl get clusterpolicies
  NAME                   BACKGROUND   ACTION    READY
  require-labels         true         enforce   true
  require-lable-source   true         enforce   true
  

• 通配符说明

  • "?" ：匹配零个或多个字母数字字符
  • "*" ：匹配一个字母数字字符

测试规则

• 创建pod

  • kubectl run nginx --image nginx 不设置label

    [root@master kyverno]# kubectl run nginx --image nginx
    Error from server: admission webhook "validate.kyverno.svc-fail" denied the request: 
    
    resource Pod/default/nginx was blocked due to the following policies
    
    require-lable-source:
      require-lable: 'validation error: 新创建的 POD 必须包含 标签 "environment". Rule require-lable
        failed at path /metadata/labels/environment/'
      require-source: 'validation error: 新创建的 POD 必须设置 CPU 和 内存限制. Rule require-source
        failed at path /spec/containers/0/resources/limits/'
    

  • kubectl run nginx --image nginx --labels environment=test 只设置标签

    [root@master kyverno]# kubectl run nginx --image nginx --labels environment=test
    Error from server: admission webhook "validate.kyverno.svc-fail" denied the request: 
    
    resource Pod/default/nginx was blocked due to the following policies
    
    require-lable-source:
      require-source: 'validation error: 新创建的 POD 必须设置 CPU 和 内存限制. Rule require-source
        failed at path /spec/containers/0/resources/limits/'
    

• 设置标签+资源限制

  • vim nginx.yaml

    apiVersion: v1
    kind: Pod
    metadata:
      name: nginx
      labels:
        app: nginx
        environment: test
    spec:
      containers:
        - name: nginx
          image: nginx
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              cpu: "0.5"
              memory: "1024Mi"
            requests:
              cpu: "0.2"
              memory: "256Mi"
      restartPolicy: Always
    

  • kubectl apply -f nginx.yaml

    [root@master kyverno]# kubectl apply -f nginx.yaml 
    pod/nginx created
    [root@master kyverno]# kubectl get pods -n default 
    NAME                     READY   STATUS    RESTARTS   AGE
    kdemo-598fbd6c64-dgr5h   1/1     Running   0          18d
    nginx                    1/1     Running   0          70s
    secret-pod               1/1     Running   0          18d
    

• 更多验证规则可以查看：kyverno.io/docs/writin…

变更

变更规则可以用于修改匹配到规则的资源，可以使用RFC 6902 JSON、战略合并补丁（比如规则设置了metadata字段可以和资源的metadata进行合并），就是根据我们设置的规则来修改对应的资源。

验证示例

要求：

1. 给所有包含 nginx 容器 的 pod 都加上一个标签（type=nginx）

添加验证规则

• vim nginx-type-label.yaml

  apiVersion: kyverno.io/v1
  kind: ClusterPolicy
  metadata:
    name: nginx-type-label
  spec:
    rules:
      - name: nginx-type-label
        match:
          resources:
            kinds:
            - Pod
        mutate:
          patchStrategicMerge:
            metadata:
              labels:
                type: nginx
            spec:
              (containers):
              - (image): "*nginx*"
  

  [root@master kyverno]# kubectl apply -f nginx-type-label.yaml 
  clusterpolicy.kyverno.io/nginx-type-label created
  [root@master kyverno]# kubectl get clusterpolicy.kyverno.io
  NAME               BACKGROUND   ACTION   READY
  nginx-type-label   true         audit    true
  

添加POD验证

• 创建一个pod kubectl run --image=nginx nginx

  [root@master kyverno]# kubectl run --image=nginx nginx
  pod/nginx created
  [root@master kyverno]# kubectl get pods 
  NAME                     READY   STATUS              RESTARTS   AGE
  kdemo-598fbd6c64-dgr5h   1/1     Running             0          18d
  nginx                    0/1     ContainerCreating   0          4s
  secret-pod               1/1     Running
  

• 查看标签 kubectl get pods --show-label

  [root@master kyverno]# kubectl get pods --show-labels
  NAME                     READY   STATUS    RESTARTS   AGE   LABELS
  kdemo-598fbd6c64-dgr5h   1/1     Running   0          18d   app=kdemo,pod-template-hash=598fbd6c64
  nginx                    1/1     Running   0          54s   run=nginx,type=nginx
  secret-pod               1/1     Running   0          18d   <none>
  

• 更多变更规则可以查看：kyverno.io/docs/writin…

生成

生成规则可用于在创建新资源或更新源时创建其他资源。这对于创建支持资源非常有用，例如为命名空间创建新 RoleBindings 或 networkpolicy。

演示示例

要求：

1. 在创建命名空间时，给当前命名空间下增加一个ConfigMap，内容为数据库HOST

添加规则

• vim new-namepase-configmysql.yaml

  apiVersion: kyverno.io/v1
  kind: ClusterPolicy
  metadata:
    name: new-namespace-configmysql
  spec:
    rules:
    - name: new-namespace-configmysql
      match:
        resources:
          kinds:
          - Namespace
      exclude:
        resources:
          namespaces:
          - kube-system
          - default
          - kube-public
          - kyverno
      generate:
        synchronize: true
        kind: ConfigMap
        name: mysql-host
        # generate the resource in the new namespace
        namespace: "{{request.object.metadata.name}}"
        data:
          kind: ConfigMap
          metadata:
            labels:
              somekey: somevalue
          data:
            MYSQL_HOST: "mysql.database"
  

• kubectl apply -f new-namespace-configmysql.yaml

  [root@master kyverno]# kubectl apply -f new-namepase-configmysql.yaml 
  clusterpolicy.kyverno.io/new-namespace-configmysql created
  [root@master kyverno]# kubectl get clusterpolicies.kyverno.io 
  NAME                        BACKGROUND   ACTION   READY
  new-namespace-configmysql   true         audit    true
  nginx-type-label            true         audit    true
  [root@master kyverno]# 
  

测试规则

• 创建一个新的命名空间 kubectl create namespace testconfig

  [root@master kyverno]# kubectl create namespace testconfig
  namespace/testconfig created
  [root@master kyverno]# 
  

• 查看confimap

  [root@master kyverno]# kubectl get configmaps -n testconfig 
  NAME                 DATA   AGE
  istio-ca-root-cert   1      27s
  kube-root-ca.crt     1      27s
  mysql-host           1      26s
  [root@master kyverno]# kubectl describe configmaps -n testconfig mysql-host 
  Name:         mysql-host
  Namespace:    testconfig
  Labels:       app.kubernetes.io/managed-by=kyverno
                kyverno.io/generated-by-kind=Namespace
                kyverno.io/generated-by-name=testconfig
                kyverno.io/generated-by-namespace=
                policy.kyverno.io/gr-name=gr-8hwrf
                policy.kyverno.io/policy-name=new-namespace-configmysql
                policy.kyverno.io/synchronize=enable
                somekey=somevalue
  Annotations:  <none>
  
  Data
  ====
  MYSQL_HOST:
  ----
  mysql.database
  
  BinaryData
  ====
  
  Events:  <none>
  [root@master kyverno]#
  

• 更多生成规则可查看：kyverno.io/docs/writin…

完结