基于Ingress Nginx的灰度发布

Ingress-Nginx支持配置Ingress Annotations (注解信息)来实现不同场景下的灰度发布和测试

它能够满足:

金丝雀发布
蓝绿部署
A/B测试等不同的业务场景

Annotations介绍

Annotations 这是当作元数据附加在资源之上的元数据
有些软件可以基于Annotations 生成配置信息
传统业务迁移到k8s时很多软件未能按照k8s法则的前提下可以用Annotations

基于Ingress Nginx的Canary规则

Ingress Nginx Annotations支持的Canary规则

策略: 基于用户

通过header标头值进行区分可以将某一部分的用户请求发给Canary版本余下的用户给生产版本

nginx.ingress.kubernetes.io/canary-by-header

基于该Annotation中指定Request Header进行流量切分，适用于灰度发布以及A/B测试

在请求报文中，若存在该Header且其值为always时，请求将会被发送到Canary版本
若存在该Header且其值为never时，请求发给稳定的身缠版本请求将不会被发送至Canary版本
对于任何其它值，将忽略该Annotation指定的Header，并通过优先级将请求与其他金丝雀规则进行优先级的比较

nginx.ingress.kubernetes.io/canary-by-header-value

基于该Annotation中指定的Request Header的值进行流量切分，标头名称则由前一个Annotation进行制定

请求报文中存在指定的标头，且其值与该Annotation的值匹配时，它将被路由到Canary版本
对于任何其它值，将忽略该Annotation

nginx.ingress.kubernetes.io/canary-by-header-pattern

同canary-by-header-value的功能类似，但该Annotation基于正则表达式匹配Request Header的值
若该Annotation与canary-by-header-value同时存在，则该Annotation会被忽略

策略: 基于权重

指定一定的比例将流量发给Canary版本余下的流量给生产版本

nginx.ingress.kubernetes.io/canary-weight

基于服务权重进行流量切分，适用于蓝绿部署，权重范围0-100按百分比将请求路由到Canary Ingress中指定的服务

权重为 0 意味着该金丝雀规则不会向Canary入口的服务发送任何请求
权重为100意味着所有请求都将被发送到 Canary 入

策略: 基于cookie

nginx.ingress.kubernetes.io/canary-by-cookie

基于 cookie 的流量切分，适用于灰度发布与 A/B 测试

cookie的值设置为always时，它将被路由到Canary入口
cookie的值设置为 never时，请求不会被发送到Canary入口
对于任何其他值，将忽略 cookie 并将请求与其他金丝雀规则进行优先级的比较

规则的应用次序

Canary规则会按特定的次序进行评估
次序：canary-by-header -> canary-by-cookie -> canary-weight

配置示例-- 基于标头的金丝雀发布

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "Username"
    nginx.ingress.kubernetes.io/canary-by-header-pattern: "(vip|VIP)_.*"
  name: demoapp-canary-by-header-pattern
spec:
  rules:
  - host: demoapp.magedu.com
    http:
      paths:
      - backend:
          service:
            name: demoapp-v11
            port:
              number: 80
        path: /
        pathType: Prefix

配置示例-- 基于权重的金丝雀发布

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"
  name: demoapp-canary-by-weight
spec:
  rules:
  - host: demoapp.magedu.com
    http:
      paths:
      - backend:
          service:
            name: demoapp-v11
            port:
              number: 80
        path: /
        pathType: Prefix

测试验证1-基于标头策略

前提: ingress-nginx-controller 已经绑定了EXTERNAL-IP 并且配置了demoapp.magedu.com域名解析

step1 先创建2组deployment

demoapp-v10 模拟稳定的生产版本

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: demoapp
  name: demoapp-v10
spec:
  replicas: 1
  selector:
    matchLabels:
      app: demoapp
      version: v1.0
  strategy: {}
  template:
    metadata:
      labels:
        app: demoapp
        version: v1.0
    spec:
      containers:
      - image: ikubernetes/demoapp:v1.0
        name: demoapp
        resources: {}
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: demoapp
  name: demoapp-v10
spec:
  ports:
  - name: http-80
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: demoapp
    version: v1.0
  type: ClusterIP

demoapp-v11 模拟金丝雀版本

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: demoapp
  name: demoapp-v11
spec:
  replicas: 1
  selector:
    matchLabels:
      app: demoapp
      version: v1.1
  strategy: {}
  template:
    metadata:
      labels:
        app: demoapp
        version: v1.1
    spec:
      containers:
      - image: ikubernetes/demoapp:v1.1
        name: demoapp
        resources: {}
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: demoapp
  name: demoapp-v11
spec:
  ports:
  - name: http-80
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: demoapp
    version: v1.1
  type: ClusterIP

step2 创建ingress 将正常的访问流量转到v10

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demoapp
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: demoapp.magedu.com
    http:
      paths:
      - backend:
          service:
            name: demoapp-v10
            port: 
              number: 80
        path: /
        pathType: Prefix

step3 创建ingress 将标头X-Canary 流量转到v11

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "X-Canary"
  name: demoapp-canary-by-header
spec:
  rules:
  - host: demoapp.magedu.com
    http:
      paths:
      - backend:
          service:
            name: demoapp-v11
            port: 
              number: 80
        path: /
        pathType: Prefix

curl demoapp.magedu.com:30635 正常的流量都到了demoapp-v10-7f44bbc8c5-rqxgh的pod

curl demoapp.magedu.com:30635 -H "X-Canary: always"

流量都到了demoapp-v11-5f64448cd5-bslbz

测试验证2-基于标头自定义值策略

创建Ingress规则

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "IsVIP"
    nginx.ingress.kubernetes.io/canary-by-header-value: "false"
  name: demoapp-canary-by-header-value
spec:
  rules:
  - host: demoapp.magedu.com
    http:
      paths:
      - backend:
          service:
            name: demoapp-v11
            port: 
              number: 80
        path: /
        pathType: Prefix

流量发给了demoapp-v11

curl demoapp.magedu.com:30635 -H "IsVIP: false"

流量发给了demoapp-v10

curl demoapp.magedu.com:30635 -H "IsVIP: xxx"

测试验证3-基于标头正则匹配策略

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "Username"
    nginx.ingress.kubernetes.io/canary-by-header-pattern: "(vip|VIP)_.*"
  name: demoapp-canary-by-header-pattern
spec:
  rules:
  - host: demoapp.magedu.com
    http:
      paths:
      - backend:
          service:
            name: demoapp-v11
            port:
              number: 80
        path: /
        pathType: Prefix

流量到了demoapp-v11

curl demoapp.magedu.com:30635 -H "Username: vip_xxx"

流量到了demoapp-v10

curl demoapp.magedu.com:30635 -H "Username: normal_xxx"
curl demoapp.magedu.com:30635

测试验证4-基于权重切割策略

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"
  name: demoapp-canary-by-weight
spec:
  rules:
  - host: demoapp.magedu.com
    http:
      paths:
      - backend:
          service:
            name: demoapp-v11
            port:
              number: 80
        path: /
        pathType: Prefix

可以看到只有很小的比例才能到 demoapp-v11

kubetnetes-ingress灰度发布介绍、金丝雀发布策略、配置示例和模拟测试