Kubernetes Dynamic Admission Control 思考与实践

一、istio的成功之处：

1、使用Kubernetes声明式api的优势，可以对同一种资源进行path操作

2、借助Kubernetes动态准入控制，对将istio的数据面envoy以sidecar的方式注入到pod中

二、Dynamic Admission Control 在APIServer中的介入点

可用如下图片说明：图片来自:techblog.cisco.com/blog/dark-s…

这里我们主要用的是mututing admission webhooks

三、实现的目标

我们都清楚istio的sidecar是从configmap中读取envoy Container的配置，然后注入到一个pod的中，从而实现流量代理

pod的yaml内容如下所示：

apiVersion: v1
kind: Pod
metadata:
  name: demo-pod
  labels:
    app: demo
spec:
  containers:
  - name: demo-container
    image: busybox
    command: ['sh', '-c', 'echo Hello Kubernetes! && sleep 3600']

configmap的内容如下所示：

apiVersion: v1
kind: ConfigMap
metadata:
  name: envoy-initializer
data:
  config: |
    containers:
      - name: envoy
        image: lyft/envoy:845747db88f102c0fd262ab234308e9e22f693a1
        command: ["/usr/local/bin/envoy"]
        args:
          - "--concurrency 4"
          - "--config-path /etc/envoy/envoy.json"
          - "--mode serve"
        ports:
          - containerPort: 80
            protocol: TCP
        resources:
          limits:
            cpu: "1000m"
            memory: "512Mi"
          requests:
            cpu: "100m"
            memory: "64Mi"
        volumeMounts:
          - name: envoy-conf
            mountPath: /etc/envoy
    volumes:
      - name: envoy-conf
        configMap:
          name: envoy

Dynamic Admission Control的作用把两者合并到一起去，内容如下：

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: demo
  name: demo-pod
  namespace: default
spec:
  containers:
  - command:
    - sh
    - -c
    - echo Hello Kubernetes! && sleep 3600
    image: busybox
    name: myapp-container
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-dl4zl
      readOnly: true
  - args:
    - --concurrency 4
    - --config-path /etc/envoy/envoy.json
    - --mode serve
    command:
    - /usr/local/bin/envoy
    image: lyft/envoy:845747db88f102c0fd262ab234308e9e22f693a1
    imagePullPolicy: IfNotPresent
    name: envoy
    ports:
    - containerPort: 80
      protocol: TCP
    resources:
      limits:
        cpu: "1"
        memory: 512Mi
      requests:
        cpu: 100m
        memory: 64Mi
    volumeMounts:
    - mountPath: /etc/envoy
      name: envoy-conf
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-dl4zl
      readOnly: true

四、具体实现

代码实现：github地址：github.com/wwwhaoxu/Dy…

1、读取configmap（我们是以Incluster的方式部署的，所以需要给pod所关联的serviceaccout进行configmap资源的授权）

// creates the in-cluster config
config, err := rest.InClusterConfig()
if err != nil {
   panic(err.Error())
}
// creates the clientset
clientset, err := kubernetes.NewForConfig(config)
if err != nil {
   panic(err.Error())
}

// Load the Envoy Initializer configuration from a Kubernetes ConfigMap.
cm, err := clientset.CoreV1().ConfigMaps(namespace).Get(ctx, configmap, metav1.GetOptions{})
if err != nil {
   klog.Fatal()
}

c, err := configmapToConfig(cm)
if err != nil {
   klog.Fatal(err)
}

2、从admission.Request获取用户发起请求的pod资源(这里我们称之为老的pod，后面也一样)，反序列化corev1.pod

pod := &corev1.Pod{}
//initializePod := runtime.DeepCopyJSONValue(pod)

err = p.decode.Decode(req, pod)
if err != nil {
   podMutateLog.Error(err, "failed decoder pod")
   return admission.Errored(http.StatusBadRequest, err)
}

3、将configmap中的内容填充到新的pod中去

pod.Spec.Containers = append(pod.Spec.Containers, c.Containers...)
pod.Spec.Volumes = append(pod.Spec.Volumes, c.Volumes...)

4、将新旧两个pod merge到一个pod中，再进行json序列化，最后发起Path请求

marshaledPod, err := json.Marshal(pod)
if err != nil {
   podMutateLog.Error(err, "failed marshal pod")
   return admission.Errored(http.StatusInternalServerError, err)
}
return admission.PatchResponseFromRaw(req.Object.Raw, marshaledPod)

五、部署

1、参考github中把程序打包成image

docker build -t docker build -t admission-webhook-demo:v1 ./ 

2、证书我们借助于cert-manager实现，将cert-manager部署到我们的集群中

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.4.0/cert-manager.yaml

3、yaml文件如下图所示，使用kustomize管理：

由于程序需要访问configmap资源，而默认的default用户是没有权限，所以我们先对default serviceaccout进行授权

kubectl create clusterrolebinding myapp-view-binding --clusterrole=view --serviceaccount=admission-webhook-demo-system:default

kubectl apply -k spec/

4、验证

kubectl apply -f validationYaml/envoycm.yaml

kubectl label ns default pod-admission-webhook-injection=enabled

kubectl apply -f validationYaml/demo.yaml