在Kubernetes中使用就绪性和有效性探测来检查容器的健康状况

344 阅读4分钟

在这个例子中,我们将使用Kubernetes的准备度和有效性探针来自动检测不健康的容器/应用程序,以便适当地处理请求。它可以帮助我们以最少的停机时间提供尽可能可靠的服务。

准备度探针

指示容器是否 "完全 "准备好接受流量以响应请求。

准备就绪探针是为了让Kubernetes知道你的容器何时准备好提供流量。Kubernetes在允许服务向Pod发送流量之前,会确保准备好的探测器通过。如果准备就绪探测器开始失败,Kubernetes将停止向Pod发送流量,直到它通过。当一个Pod的所有容器都准备好接受连接或请求时,它就被认为是准备好了。如果一个容器不在准备状态,你不想杀死它,但你也不想向它发送请求。例如,一个Pod需要时间来启动/重启,所以在这种情况下,Kubernetes要么停止向它发送流量,要么只是将流量转移到已经运行的Pod(如果存在)。

什么时候应该使用准备就绪探测器?

如果你的容器需要在启动过程中加载大型数据、配置文件、等待外部服务器连接或迁移,请指定一个准备就绪探测器。

有效性探针

指示Container是否正在运行。如果有效性探测失败,kubelet会杀死Container,Container会受到其 "重启策略 "的约束。

有效性探针让Kubernetes知道你的Container是活着还是死了。如果你的应用程序是活的,Kubernetes就不会管它。如果你的容器死了,Kubernetes会移除Pod并启动一个新的Pod来替代它。

什么时候应该使用有效性探针?

一个应用程序可能会崩溃,变得没有反应等。如果你希望你的容器在这种情况下被重新启动,请指定一个有效性探测器。你还需要指定一个AlwaysOnFailurerestartPolicy 。然而,有时应用程序被允许崩溃。在这种情况下,你不需要指定liveness probe。Kubernetes很聪明,会根据restartPolicy 的值执行适当的操作。

配置选项

命令、HTTP和TCP的活度/准备度探针实现的配置是相同的。你可以用下面的选项来控制探针。

命令

在目标容器中执行一个命令,例如:cat /tmp/healthy ,。如果命令成功,则返回0。如果命令返回非零值(失败),则被重新启动。

HTTP端点

向运行在容器中并在指定端口上监听的服务器发送HTTP GET请求--例如:/health 。任何大于或等于200且小于400的响应代码表示成功。任何其他代码表示失败,所以会触发重启。

TCP端口

将尝试在指定的端口上为你的容器打开一个套接字。如果它能建立连接,则认为该容器是健康的,如果它不能,则认为是失败的。对于TCP探测,kubelet在节点上建立探测连接,而不是在Pod中,这意味着你不能在host参数中使用服务名称,因为kubelet无法解决它。

结构

├── Makefile
├── deploy
│   └── k8s
│       ├── deployment.yaml
│       └── service.yaml
├── docker
│   └── dev
│       └── Dockerfile
└── main.go

文件

Makefile

## Build application binary.
.PHONY: build
build:
   go build -race -ldflags "-s -w" -o bin/main main.go

## Build application binary and run it.
.PHONY: run
run: build
   bin/main

## Send a dummy request to the local server.
.PHONY: local-test
local-test:
   curl -I --request GET http://localhost:3000/health

## -------------------------------------------------------------

## Build, tag and push application image to registry then clean up.
.PHONY: push
push:
   docker build -t you/address-finder:latest -f ./docker/dev/Dockerfile .
   docker push you/address-finder:latest
   docker rmi you/address-finder:latest
   docker system prune --volumes --force

## Deploy application to kubernetes cluster.
.PHONY: deploy
deploy:
   kubectl apply -f deploy/k8s/deployment.yaml
   kubectl apply -f deploy/k8s/service.yaml

## Send a dummy request to the exposed pod on kubernetes.
.PHONY: k8s-test
k8s-test:
   curl -I --request GET $(shell minikube service address-finder-service --url)/health

Docker文件

#
# STAGE 1: build
#
FROM golang:1.15-alpine3.12 as build

WORKDIR /source
COPY . .

RUN CGO_ENABLED=0 go build -ldflags "-s -w" -o bin/main main.go

#
# STAGE 2: run
#
FROM alpine:3.12 as run

COPY --from=build /source/bin/main /main

ENTRYPOINT ["/main"]

main.go

package main

import (
   "log"
   "net/http"
   "time"
)

func main() {
   log.Println("app will be read in 10 sec")
   time.Sleep(time.Second*10)
   log.Println("app is ready")

   rtr := http.NewServeMux()
   rtr.HandleFunc("/health", health)

   if err := http.ListenAndServe(":3000", rtr); err != nil {
      log.Fatalln(err)
   }
}

func health(_ http.ResponseWriter, r *http.Request) {
   log.Println("X-Probe:", r.Header.Get("X-Probe"))
}

部署.yaml

apiVersion: apps/v1
kind: Deployment

metadata:
  name: address-finder-deployment
  labels:
    app: address-finder

spec:
  replicas: 1
  selector:
    matchLabels:
      app: address-finder
  template:
    metadata:
      labels:
        app: address-finder
    spec:
      containers:
        - name: golang
          image: you/address-finder:latest
          ports:
            - containerPort: 3000
          readinessProbe:
            httpGet:
              path: /health
              port: 3000
              httpHeaders:
                - name: X-Probe
                  value: readiness
            initialDelaySeconds: 10 # Wait x seconds before initialising probe.
            periodSeconds: 10       # Probe ever x seconds.
            timeoutSeconds: 5       # Time out probe after x seconds.
            successThreshold: 1     # Consider probe to be successful after minimum x consecutive successes.
            failureThreshold: 5     # Try x times before giving up. Pod will be marked as "Unready".
          livenessProbe:
            httpGet:
              path: /health
              port: 3000
              httpHeaders:
                - name: X-Probe
                  value: liveness
            initialDelaySeconds: 10 # Wait x seconds before initialising probe.
            periodSeconds: 10       # Probe ever x seconds.
            timeoutSeconds: 5       # Time out probe after x seconds.
            successThreshold: 1     # Consider probe to be successful after minimum x consecutive successes.
            failureThreshold: 2     # Try x times before giving up. Container will be restarted.

service.yaml

apiVersion: v1
kind: Service

metadata:
  name: address-finder-service

spec:
  type: NodePort
  selector:
    app: address-finder
  ports:
    - protocol: TCP
      port: 80
      targetPort: 3000

测试

运行make pushmake deploy 来最终完成部署。在这里我们验证部署探测器:

$ kubectl describe deployment.apps/address-finder-deployment
…
    Liveness:     http-get http://:3000/health delay=10s timeout=5s period=10s #success=1 #failure=2
    Readiness:    http-get http://:3000/health delay=10s timeout=5s period=10s #success=1 #failure=5

当我们在部署后和10秒前测试时:

$ make k8s-test
curl -I --request GET http://192.168.99.100:32560/health
curl: (7) Failed to connect to 192.168.99.100 port 32560: Connection refused
make: *** [k8s-test] Error 7

当我们在启动后10秒测试时:

$ make k8s-test
curl -I --request GET http://192.168.99.100:32560/health
HTTP/1.1 200 OK
Date: Sun, 13 Dec 2020 17:19:49 GMT
Content-Length: 0

这些是日志:

$ kubectl logs pod/address-finder-deployment-6f5fc66d4d-k7pb2
2020/12/13 17:19:28 app will be read in 10 sec
2020/12/13 17:19:38 app is ready
2020/12/13 17:19:40 X-Probe: liveness
2020/12/13 17:19:46 X-Probe: readiness
2020/12/13 17:19:49 X-Probe: // This is the moment when we got 200 OK while testing, compare the time
2020/12/13 17:19:50 X-Probe: liveness
2020/12/13 17:19:56 X-Probe: readiness
2020/12/13 17:20:00 X-Probe: liveness
2020/12/13 17:20:06 X-Probe: readiness
2020/12/13 17:20:10 X-Probe: liveness
2020/12/13 17:20:16 X-Probe: readiness