Kubenetes排障：Coredns解析DNSSEC异常

义达
419 阅读1分钟

Kubenetes排障：Coredns解析DNSSEC异常

现象

coredns在解析带RRSIG的域名，ttl到期后第一次解析可能会返回DNSSEC（当域名启用了DNSSEC，并且相应的资源记录被数字签名时，才会出现RRSIG记录）

$ dig @10.178.64.64 openapi.dossen.com
A 13 3 600 20231123104824 20231121084824 56113 dossen.com. 2wqQg3nHgmksMGor70wnQK9wHKa1Qn2DzGhgPKJBjvyh25BOIT1z7UIf 1i6rOdner1sGLqyQi3lklfwJ1lmYjQ==
# 正常解析为
openapi.dossen.com.	30	IN	A	39.108.79.209

问题修复

在高版本修复：github.com/coredns/cor…

集群版本是：1.3.1，升级目标版本为1.8.4，kubectl set image deployment/coredns -n kube-system coredns=xxx/coredns-amd64:1.8.4

升级注意事项：

        forward . 10.110.xx.yy 10.110.xx.yy

        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          upstream # 删除
          fallthrough in-addr.arpa ip6.arpa
        }

更新后的问题

因为我们直接跨多个版本升级，更新后查看coredns的日志发现有权限问题

E1129 08:51:12.910891       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.21.1/tools/cache/reflector.go:167: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:serviceaccount:kube-system:coredns" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
E1129 08:51:55.765513       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.21.1/tools/cache/reflector.go:167: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:serviceaccount:kube-system:coredns" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope

我们去github.com/coredns/cor…下载对应的源码，查看notes/coredns-1.8.4.md，我们会发现1.8.4修复了一些dnssec的bug：

The CoreDNS team has released
[CoreDNS-1.8.4](https://github.com/coredns/coredns/releases/tag/v1.8.4). This release includes a
bunch of bugfixes and a few enhancements mostly in the *dnssec* and *kubernetes* plugins, and a new
(small) plugin called *minimal*.

在notes/coredns-1.8.1.md中要求我们赋予endpointslices的list和watch权限

If using the [kubernetes](https://coredns.io/plugins/kubernetes) plugin for a Kubernetes
cluster >= 1.19, CoreDNS must be granted `list` and `watch` access to `endpointslices`.

查看endpointslice资源

$ kubectl explain endpointslice
​
KIND:     EndpointSlice
VERSION:  discovery.k8s.io/v1beta1

添加权限 sudo kubectl get clusterrole system:coredns -o yaml

rules:
- apiGroups:
  - ""
  resources:
  - endpoints
  - services
  - pods
  - namespaces
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
- apiGroups: # 添加
  - "discovery.k8s.io"
  resources:
  - endpointslices
  verbs:
  - list
  - watch
avatar
文章
阅读
粉丝