calico ipstables 是否会干扰 kube-ovn iptables 问题分析

48 阅读55分钟

这个环境中,我之前部署了 calico, 然后把 CNI 切换为 kube-ovn,目前已经清理了集群中所有旧的 calico pod。

问题: 如果这些 calico 的旧的 iptables 规则一直残留,是否会对 kube-ovn 的 pod 造成网络影响?

环境信息


root@k8s-ctrl:~/kubespray1-31# iptables-save
# Generated by iptables-save v1.8.7 on Mon Nov 17 07:40:30 2025
*raw
:PREROUTING ACCEPT [157702:46273812]
:OUTPUT ACCEPT [163654:72133040]
:CILIUM_OUTPUT_raw - [0:0]
:CILIUM_PRE_raw - [0:0]
:cali-OUTPUT - [0:0]
:cali-PREROUTING - [0:0]
:cali-from-host-endpoint - [0:0]
:cali-rpf-skip - [0:0]
:cali-to-host-endpoint - [0:0]
-A PREROUTING -d 169.254.25.10/32 -p udp -m udp --dport 53 -m comment --comment "NodeLocal DNS Cache: skip conntrack" -j NOTRACK
-A PREROUTING -d 169.254.25.10/32 -p tcp -m tcp --dport 53 -m comment --comment "NodeLocal DNS Cache: skip conntrack" -j NOTRACK
-A PREROUTING -m comment --comment "cilium-feeder: CILIUM_PRE_raw" -j CILIUM_PRE_raw
-A PREROUTING -m comment --comment "cali:6gwbT8clXdHdC1b1" -j cali-PREROUTING
-A OUTPUT -s 169.254.25.10/32 -p tcp -m tcp --sport 8080 -m comment --comment "NodeLocal DNS Cache: skip conntrack" -j NOTRACK
-A OUTPUT -d 169.254.25.10/32 -p tcp -m tcp --dport 8080 -m comment --comment "NodeLocal DNS Cache: skip conntrack" -j NOTRACK
-A OUTPUT -d 169.254.25.10/32 -p udp -m udp --dport 53 -m comment --comment "NodeLocal DNS Cache: skip conntrack" -j NOTRACK
-A OUTPUT -d 169.254.25.10/32 -p tcp -m tcp --dport 53 -m comment --comment "NodeLocal DNS Cache: skip conntrack" -j NOTRACK
-A OUTPUT -s 169.254.25.10/32 -p udp -m udp --sport 53 -m comment --comment "NodeLocal DNS Cache: skip conntrack" -j NOTRACK
-A OUTPUT -s 169.254.25.10/32 -p tcp -m tcp --sport 53 -m comment --comment "NodeLocal DNS Cache: skip conntrack" -j NOTRACK
-A OUTPUT -m comment --comment "cilium-feeder: CILIUM_OUTPUT_raw" -j CILIUM_OUTPUT_raw
-A OUTPUT -m comment --comment "cali:tVnHkvAo15HuiPy0" -j cali-OUTPUT
-A CILIUM_OUTPUT_raw -o lxc+ -m mark --mark 0xa00/0xfffffeff -m comment --comment "cilium: NOTRACK for proxy return traffic" -j CT --notrack
-A CILIUM_OUTPUT_raw -o cilium_host -m mark --mark 0xa00/0xfffffeff -m comment --comment "cilium: NOTRACK for proxy return traffic" -j CT --notrack
-A CILIUM_OUTPUT_raw -o lxc+ -m mark --mark 0x800/0xe00 -m comment --comment "cilium: NOTRACK for L7 proxy upstream traffic" -j CT --notrack
-A CILIUM_OUTPUT_raw -o cilium_host -m mark --mark 0x800/0xe00 -m comment --comment "cilium: NOTRACK for L7 proxy upstream traffic" -j CT --notrack
-A CILIUM_PRE_raw -m mark --mark 0x200/0xf00 -m comment --comment "cilium: NOTRACK for proxy traffic" -j CT --notrack
-A cali-OUTPUT -m comment --comment "cali:njdnLwYeGqBJyMxW" -j MARK --set-xmark 0x0/0xf0000
-A cali-OUTPUT -m comment --comment "cali:rz86uTUcEZAfFsh7" -j cali-to-host-endpoint
-A cali-OUTPUT -m comment --comment "cali:pN0F5zD0b8yf9W1Z" -m mark --mark 0x10000/0x10000 -j ACCEPT
-A cali-PREROUTING -m comment --comment "cali:XFX5xbM8B9qR10JG" -j MARK --set-xmark 0x0/0xf0000
-A cali-PREROUTING -i cali+ -m comment --comment "cali:EWMPb0zVROM-woQp" -j MARK --set-xmark 0x40000/0x40000
-A cali-PREROUTING -m comment --comment "cali:PWuxTAIaFCtsg5Qa" -m mark --mark 0x40000/0x40000 -j cali-rpf-skip
-A cali-PREROUTING -m comment --comment "cali:fSSbGND7dgyemWU7" -m mark --mark 0x40000/0x40000 -m rpfilter --validmark --invert -j DROP
-A cali-PREROUTING -m comment --comment "cali:ImU0-4Rl2WoOI9Ou" -m mark --mark 0x0/0x40000 -j cali-from-host-endpoint
-A cali-PREROUTING -m comment --comment "cali:lV4V2MPoMBf0hl9T" -m mark --mark 0x10000/0x10000 -j ACCEPT
COMMIT
# Completed on Mon Nov 17 07:40:30 2025
# Generated by iptables-save v1.8.7 on Mon Nov 17 07:40:30 2025
*nat
:PREROUTING ACCEPT [35:2486]
:INPUT ACCEPT [32:2330]
:OUTPUT ACCEPT [141:8358]
:POSTROUTING ACCEPT [127:7518]
:CILIUM_OUTPUT_nat - [0:0]
:CILIUM_POST_nat - [0:0]
:CILIUM_PRE_nat - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-LOAD-BALANCER - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODE-PORT - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SERVICES - [0:0]
:OVN-MASQUERADE - [0:0]
:OVN-NAT-POLICY - [0:0]
:OVN-POSTROUTING - [0:0]
:OVN-PREROUTING - [0:0]
:cali-OUTPUT - [0:0]
:cali-POSTROUTING - [0:0]
:cali-PREROUTING - [0:0]
:cali-fip-dnat - [0:0]
:cali-fip-snat - [0:0]
:cali-nat-outgoing - [0:0]
-A PREROUTING -m comment --comment "cilium-feeder: CILIUM_PRE_nat" -j CILIUM_PRE_nat
-A PREROUTING -m comment --comment "kube-ovn prerouting rules" -j OVN-PREROUTING
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m comment --comment "cali:6gwbT8clXdHdC1b1" -j cali-PREROUTING
-A OUTPUT -m comment --comment "cilium-feeder: CILIUM_OUTPUT_nat" -j CILIUM_OUTPUT_nat
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m comment --comment "cali:tVnHkvAo15HuiPy0" -j cali-OUTPUT
-A POSTROUTING -m comment --comment "cilium-feeder: CILIUM_POST_nat" -j CILIUM_POST_nat
-A POSTROUTING -m comment --comment "kube-ovn postrouting rules" -j OVN-POSTROUTING
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -m comment --comment "cali:O3lYWMrLQYEMJtB5" -j cali-POSTROUTING
-A KUBE-LOAD-BALANCER -j KUBE-MARK-MASQ
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODE-PORT -p tcp -m comment --comment "Kubernetes nodeport TCP port for masquerade purpose" -m set --match-set KUBE-NODE-PORT-TCP dst -j KUBE-MARK-MASQ
-A KUBE-POSTROUTING -m comment --comment "Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose" -m set --match-set KUBE-LOOP-BACK dst,dst,src -j MASQUERADE
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE --random-fully
-A KUBE-SERVICES -s 127.0.0.0/8 -j RETURN
-A KUBE-SERVICES ! -s 10.222.0.0/16 -m comment --comment "Kubernetes service cluster ip + port for masquerade purpose" -m set --match-set KUBE-CLUSTER-IP dst,dst -j KUBE-MARK-MASQ
-A KUBE-SERVICES -m addrtype --dst-type LOCAL -j KUBE-NODE-PORT
-A KUBE-SERVICES -m set --match-set KUBE-CLUSTER-IP dst,dst -j ACCEPT
-A OVN-MASQUERADE -j MARK --set-xmark 0x0/0xffffffff
-A OVN-MASQUERADE -j MASQUERADE --random-fully
-A OVN-POSTROUTING -m set --match-set ovn40services src -m set --match-set ovn40subnets dst -m mark --mark 0x4000/0x4000 -j SNAT --to-source 172.16.189.100 --random-fully
-A OVN-POSTROUTING -m mark --mark 0x4000/0x4000 -j OVN-MASQUERADE
-A OVN-POSTROUTING -m set --match-set ovn40subnets src -m set --match-set ovn40subnets dst -j OVN-MASQUERADE
-A OVN-POSTROUTING -m mark --mark 0x80000/0x80000 -m set --match-set ovn40subnets-distributed-gw dst -j RETURN
-A OVN-POSTROUTING -m mark --mark 0x80000/0x80000 -j OVN-MASQUERADE
-A OVN-POSTROUTING -p tcp -m tcp --tcp-flags SYN NONE -m conntrack --ctstate NEW -j RETURN
-A OVN-POSTROUTING -m set ! --match-set ovn40subnets src -m set ! --match-set ovn40other-node src -m set --match-set ovn40subnets-nat dst -j RETURN
-A OVN-POSTROUTING -m set --match-set ovn40subnets-nat-policy src -m set ! --match-set ovn40subnets dst -j OVN-NAT-POLICY
-A OVN-POSTROUTING -m mark --mark 0x90001/0x90001 -j OVN-MASQUERADE
-A OVN-POSTROUTING -m mark --mark 0x90002/0x90002 -j RETURN
-A OVN-POSTROUTING -m set --match-set ovn40subnets-nat src -m set ! --match-set ovn40subnets dst -j OVN-MASQUERADE
-A OVN-PREROUTING -i ovn0 -m set --match-set ovn40subnets src -m set --match-set KUBE-CLUSTER-IP dst,dst -j MARK --set-xmark 0x4000/0x4000
-A OVN-PREROUTING -p tcp -m addrtype --dst-type LOCAL -m set --match-set KUBE-NODE-PORT-LOCAL-TCP dst -j MARK --set-xmark 0x80000/0x80000
-A OVN-PREROUTING -p tcp -m set --match-set ovn40other-node src -m set --match-set KUBE-NODE-PORT-LOCAL-TCP dst -j MARK --set-xmark 0x4000/0x4000
-A OVN-PREROUTING -p udp -m addrtype --dst-type LOCAL -m set --match-set KUBE-NODE-PORT-LOCAL-UDP dst -j MARK --set-xmark 0x80000/0x80000
-A OVN-PREROUTING -p udp -m set --match-set ovn40other-node src -m set --match-set KUBE-NODE-PORT-LOCAL-UDP dst -j MARK --set-xmark 0x4000/0x4000
-A cali-OUTPUT -m comment --comment "cali:GBTAv2p5CwevEyJm" -j cali-fip-dnat
-A cali-POSTROUTING -m comment --comment "cali:Z-c7XtVd2Bq7s_hA" -j cali-fip-snat
-A cali-POSTROUTING -m comment --comment "cali:nYKhEzDlr11Jccal" -j cali-nat-outgoing
-A cali-POSTROUTING -o tunl0 -m comment --comment "cali:SXWvdsbh4Mw7wOln" -m addrtype ! --src-type LOCAL --limit-iface-out -m addrtype --src-type LOCAL -j MASQUERADE --random-fully
-A cali-PREROUTING -m comment --comment "cali:r6XmIziWUJsdOK6Z" -j cali-fip-dnat
-A cali-nat-outgoing -m comment --comment "cali:flqWnvo8yq4ULQLa" -m set --match-set cali40masq-ipam-pools src -m set ! --match-set cali40all-ipam-pools dst -j MASQUERADE --random-fully
COMMIT
# Completed on Mon Nov 17 07:40:30 2025
# Generated by iptables-save v1.8.7 on Mon Nov 17 07:40:30 2025
*filter
:INPUT ACCEPT [8833:2382092]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [9348:3904259]
:CILIUM_FORWARD - [0:0]
:CILIUM_INPUT - [0:0]
:CILIUM_OUTPUT - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-FORWARD - [0:0]
:KUBE-IPVS-FILTER - [0:0]
:KUBE-IPVS-OUT-FILTER - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-NODE-PORT - [0:0]
:KUBE-PROXY-FIREWALL - [0:0]
:KUBE-SOURCE-RANGES-FIREWALL - [0:0]
:cali-FORWARD - [0:0]
:cali-INPUT - [0:0]
:cali-OUTPUT - [0:0]
:cali-cidr-block - [0:0]
:cali-forward-check - [0:0]
:cali-forward-endpoint-mark - [0:0]
:cali-from-endpoint-mark - [0:0]
:cali-from-hep-forward - [0:0]
:cali-from-host-endpoint - [0:0]
:cali-from-wl-dispatch - [0:0]
:cali-from-wl-dispatch-3 - [0:0]
:cali-fw-cali14897c537b0 - [0:0]
:cali-fw-cali3782683684d - [0:0]
:cali-fw-cali3c22a7abad0 - [0:0]
:cali-fw-calia56f50b9553 - [0:0]
:cali-fw-calic484cf12723 - [0:0]
:cali-fw-calif3ad7fd4559 - [0:0]
:cali-pri-_8zZ8x7SANlAuYdH3So - [0:0]
:cali-pri-_u2Tn2rSoAPffvE7JO6 - [0:0]
:cali-pri-kns.default - [0:0]
:cali-pri-kns.kube-system - [0:0]
:cali-pri-ksa.default.default - [0:0]
:cali-pro-_8zZ8x7SANlAuYdH3So - [0:0]
:cali-pro-_u2Tn2rSoAPffvE7JO6 - [0:0]
:cali-pro-kns.default - [0:0]
:cali-pro-kns.kube-system - [0:0]
:cali-pro-ksa.default.default - [0:0]
:cali-set-endpoint-mark - [0:0]
:cali-set-endpoint-mark-3 - [0:0]
:cali-sm-cali14897c537b0 - [0:0]
:cali-sm-cali3782683684d - [0:0]
:cali-sm-cali3c22a7abad0 - [0:0]
:cali-sm-calia56f50b9553 - [0:0]
:cali-sm-calic484cf12723 - [0:0]
:cali-sm-calif3ad7fd4559 - [0:0]
:cali-to-hep-forward - [0:0]
:cali-to-host-endpoint - [0:0]
:cali-to-wl-dispatch - [0:0]
:cali-to-wl-dispatch-3 - [0:0]
:cali-tw-cali14897c537b0 - [0:0]
:cali-tw-cali3782683684d - [0:0]
:cali-tw-cali3c22a7abad0 - [0:0]
:cali-tw-calia56f50b9553 - [0:0]
:cali-tw-calic484cf12723 - [0:0]
:cali-tw-calif3ad7fd4559 - [0:0]
:cali-wl-to-host - [0:0]
-A INPUT -d 169.254.25.10/32 -p udp -m udp --dport 53 -m comment --comment "NodeLocal DNS Cache: allow DNS traffic" -j ACCEPT
-A INPUT -d 169.254.25.10/32 -p tcp -m tcp --dport 53 -m comment --comment "NodeLocal DNS Cache: allow DNS traffic" -j ACCEPT
-A INPUT -m comment --comment "cilium-feeder: CILIUM_INPUT" -j CILIUM_INPUT
-A INPUT -p tcp -m mark ! --mark 0x4000/0x4000 -m set --match-set ovn40services dst -m conntrack --ctstate NEW -j REJECT --reject-with icmp-port-unreachable
-A INPUT -m set --match-set ovn40services dst -j ACCEPT
-A INPUT -m set --match-set ovn40services src -j ACCEPT
-A INPUT -m set --match-set ovn40subnets dst -j ACCEPT
-A INPUT -m set --match-set ovn40subnets src -j ACCEPT
-A INPUT -m comment --comment "kubernetes ipvs access filter" -j KUBE-IPVS-FILTER
-A INPUT -m comment --comment "kube-proxy firewall rules" -j KUBE-PROXY-FIREWALL
-A INPUT -m comment --comment "kubernetes health check rules" -j KUBE-NODE-PORT
-A INPUT -j KUBE-FIREWALL
-A INPUT -m comment --comment "cali:Cz_u1IQiXIMmKD4c" -j cali-INPUT
-A FORWARD -m comment --comment "cilium-feeder: CILIUM_FORWARD" -j CILIUM_FORWARD
-A FORWARD -d 10.222.0.0/16 -m comment --comment "ovn-subnet-gateway,ovn-default"
-A FORWARD -s 10.222.0.0/16 -m comment --comment "ovn-subnet-gateway,ovn-default"
-A FORWARD -d 100.64.0.0/16 -m comment --comment "ovn-subnet-gateway,join"
-A FORWARD -s 100.64.0.0/16 -m comment --comment "ovn-subnet-gateway,join"
-A FORWARD -m set --match-set ovn40services dst -j ACCEPT
-A FORWARD -m set --match-set ovn40services src -j ACCEPT
-A FORWARD -m set --match-set ovn40subnets dst -j ACCEPT
-A FORWARD -m set --match-set ovn40subnets src -j ACCEPT
-A FORWARD -m comment --comment "kube-proxy firewall rules" -j KUBE-PROXY-FIREWALL
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -m comment --comment "cali:wUHhoiAYhphO9Mso" -j cali-FORWARD
-A FORWARD -m comment --comment "cali:S93hcgKJrXEqnTfs" -m comment --comment "Policy explicitly accepted packet." -m mark --mark 0x10000/0x10000 -j ACCEPT
-A FORWARD -m comment --comment "cali:mp77cMpurHhyjLrM" -j MARK --set-xmark 0x10000/0x10000
-A OUTPUT -s 169.254.25.10/32 -p udp -m udp --sport 53 -m comment --comment "NodeLocal DNS Cache: allow DNS traffic" -j ACCEPT
-A OUTPUT -s 169.254.25.10/32 -p tcp -m tcp --sport 53 -m comment --comment "NodeLocal DNS Cache: allow DNS traffic" -j ACCEPT
-A OUTPUT -m comment --comment "cilium-feeder: CILIUM_OUTPUT" -j CILIUM_OUTPUT
-A OUTPUT -p tcp -m mark ! --mark 0x4000/0x4000 -m set --match-set ovn40services dst -m conntrack --ctstate NEW -j REJECT --reject-with icmp-port-unreachable
-A OUTPUT -p udp -m udp --dport 4789 -j MARK --set-xmark 0x0/0xffffffff
-A OUTPUT -p udp -m udp --dport 6081 -j MARK --set-xmark 0x0/0xffffffff
-A OUTPUT -m comment --comment "kubernetes ipvs access filter" -j KUBE-IPVS-OUT-FILTER
-A OUTPUT -j KUBE-FIREWALL
-A OUTPUT -m comment --comment "cali:tVnHkvAo15HuiPy0" -j cali-OUTPUT
-A CILIUM_FORWARD -o cilium_host -m comment --comment "cilium: any->cluster on cilium_host forward accept" -j ACCEPT
-A CILIUM_FORWARD -i cilium_host -m comment --comment "cilium: cluster->any on cilium_host forward accept (nodeport)" -j ACCEPT
-A CILIUM_FORWARD -i lxc+ -m comment --comment "cilium: cluster->any on lxc+ forward accept" -j ACCEPT
-A CILIUM_FORWARD -i cilium_net -m comment --comment "cilium: cluster->any on cilium_net forward accept (nodeport)" -j ACCEPT
-A CILIUM_INPUT -m mark --mark 0x200/0xf00 -m comment --comment "cilium: ACCEPT for proxy traffic" -j ACCEPT
-A CILIUM_OUTPUT -m mark --mark 0xa00/0xe00 -m comment --comment "cilium: ACCEPT for proxy traffic" -j ACCEPT
-A CILIUM_OUTPUT -m mark --mark 0x800/0xe00 -m comment --comment "cilium: ACCEPT for l7 proxy upstream traffic" -j ACCEPT
-A CILIUM_OUTPUT -m mark ! --mark 0xe00/0xf00 -m mark ! --mark 0xd00/0xf00 -m mark ! --mark 0x400/0xf00 -m mark ! --mark 0xa00/0xe00 -m mark ! --mark 0x800/0xe00 -m mark ! --mark 0xf00/0xf00 -m comment --comment "cilium: host->any mark as from host" -j MARK --set-xmark 0xc00/0xf00
-A KUBE-FIREWALL ! -s 127.0.0.0/8 -d 127.0.0.0/8 -m comment --comment "block incoming localnet connections" -m conntrack ! --ctstate RELATED,ESTABLISHED,DNAT -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-IPVS-FILTER -m set --match-set KUBE-LOAD-BALANCER dst,dst -j RETURN
-A KUBE-IPVS-FILTER -m set --match-set KUBE-CLUSTER-IP dst,dst -j RETURN
-A KUBE-IPVS-FILTER -m set --match-set KUBE-EXTERNAL-IP dst,dst -j RETURN
-A KUBE-IPVS-FILTER -m set --match-set KUBE-EXTERNAL-IP-LOCAL dst,dst -j RETURN
-A KUBE-IPVS-FILTER -m set --match-set KUBE-HEALTH-CHECK-NODE-PORT dst -j RETURN
-A KUBE-IPVS-FILTER -m conntrack --ctstate NEW -m set --match-set KUBE-IPVS-IPS dst -j REJECT --reject-with icmp-port-unreachable
-A KUBE-NODE-PORT -m comment --comment "Kubernetes health check node port" -m set --match-set KUBE-HEALTH-CHECK-NODE-PORT dst -j ACCEPT
-A KUBE-SOURCE-RANGES-FIREWALL -j DROP
-A cali-FORWARD -m comment --comment "cali:vjrMJCRpqwy5oRoX" -j MARK --set-xmark 0x0/0xe0000
-A cali-FORWARD -m comment --comment "cali:A_sPAO0mcxbT9mOV" -m mark --mark 0x0/0x10000 -j cali-from-hep-forward
-A cali-FORWARD -i cali+ -m comment --comment "cali:8ZoYfO5HKXWbB3pk" -j cali-from-wl-dispatch
-A cali-FORWARD -o cali+ -m comment --comment "cali:jdEuaPBe14V2hutn" -j cali-to-wl-dispatch
-A cali-FORWARD -m comment --comment "cali:12bc6HljsMKsmfr-" -j cali-to-hep-forward
-A cali-FORWARD -m comment --comment "cali:NOSxoaGx8OIstr1z" -j cali-cidr-block
-A cali-INPUT -p ipencap -m comment --comment "cali:PajejrV4aFdkZojI" -m comment --comment "Allow IPIP packets from Calico hosts" -m set --match-set cali40all-hosts-net src -m addrtype --dst-type LOCAL -j ACCEPT
-A cali-INPUT -p ipencap -m comment --comment "cali:_wjq-Yrma8Ly1Svo" -m comment --comment "Drop IPIP packets from non-Calico hosts" -j DROP
-A cali-INPUT -m comment --comment "cali:ss8lEMQsXi-s6qYT" -j MARK --set-xmark 0x0/0xfff00000
-A cali-INPUT -m comment --comment "cali:PgIW-V0nEjwPhF_8" -j cali-forward-check
-A cali-INPUT -m comment --comment "cali:QMJlDwlS0OjHyfMN" -m mark ! --mark 0x0/0xfff00000 -j RETURN
-A cali-INPUT -i cali+ -m comment --comment "cali:nDRe73txrna-aZjG" -g cali-wl-to-host
-A cali-INPUT -m comment --comment "cali:iX2AYvqGXaVqwkro" -m mark --mark 0x10000/0x10000 -j ACCEPT
-A cali-INPUT -m comment --comment "cali:bhpnxD5IRtBP8KW0" -j MARK --set-xmark 0x0/0xf0000
-A cali-INPUT -m comment --comment "cali:H5_bccAbHV0sooVy" -j cali-from-host-endpoint
-A cali-INPUT -m comment --comment "cali:inBL01YlfurT0dbI" -m comment --comment "Host endpoint policy accepted packet." -m mark --mark 0x10000/0x10000 -j ACCEPT
-A cali-OUTPUT -m comment --comment "cali:Mq1_rAdXXH3YkrzW" -m mark --mark 0x10000/0x10000 -j ACCEPT
-A cali-OUTPUT -m comment --comment "cali:5Z67OUUpTOM7Xa1a" -m mark ! --mark 0x0/0xfff00000 -g cali-forward-endpoint-mark
-A cali-OUTPUT -o cali+ -m comment --comment "cali:M2Wf0OehNdig8MHR" -j RETURN
-A cali-OUTPUT -p ipencap -m comment --comment "cali:AJBkLho_0Qd8LNr3" -m comment --comment "Allow IPIP packets to other Calico hosts" -m set --match-set cali40all-hosts-net dst -m addrtype --src-type LOCAL -j ACCEPT
-A cali-OUTPUT -m comment --comment "cali:iz2RWXlXJDUfsLpe" -j MARK --set-xmark 0x0/0xf0000
-A cali-OUTPUT -m comment --comment "cali:xQqLi8S0sxbiyvjR" -m conntrack ! --ctstate DNAT -j cali-to-host-endpoint
-A cali-OUTPUT -m comment --comment "cali:aSnsxZdmhxm_ilRZ" -m comment --comment "Host endpoint policy accepted packet." -m mark --mark 0x10000/0x10000 -j ACCEPT
-A cali-forward-check -m comment --comment "cali:Pbldlb4FaULvpdD8" -m conntrack --ctstate RELATED,ESTABLISHED -j RETURN
-A cali-forward-check -p tcp -m comment --comment "cali:ZD-6UxuUtGW-xtzg" -m comment --comment "To kubernetes NodePort service" -m multiport --dports 30000:32767 -m set --match-set cali40this-host dst -g cali-set-endpoint-mark
-A cali-forward-check -p udp -m comment --comment "cali:CbPfUajQ2bFVnDq4" -m comment --comment "To kubernetes NodePort service" -m multiport --dports 30000:32767 -m set --match-set cali40this-host dst -g cali-set-endpoint-mark
-A cali-forward-check -m comment --comment "cali:jmhU0ODogX-Zfe5g" -m comment --comment "To kubernetes service" -m set ! --match-set cali40this-host dst -j cali-set-endpoint-mark
-A cali-forward-endpoint-mark -m comment --comment "cali:O0SmFDrnm7KggWqW" -m mark ! --mark 0x100000/0xfff00000 -j cali-from-endpoint-mark
-A cali-forward-endpoint-mark -o cali+ -m comment --comment "cali:aFl0WFKRxDqj8oA6" -j cali-to-wl-dispatch
-A cali-forward-endpoint-mark -m comment --comment "cali:AZKVrO3i_8cLai5f" -j cali-to-hep-forward
-A cali-forward-endpoint-mark -m comment --comment "cali:96HaP1sFtb-NYoYA" -j MARK --set-xmark 0x0/0xfff00000
-A cali-forward-endpoint-mark -m comment --comment "cali:VxO6hyNWz62YEtul" -m comment --comment "Policy explicitly accepted packet." -m mark --mark 0x10000/0x10000 -j ACCEPT
-A cali-from-endpoint-mark -m comment --comment "cali:qmW2JjmgM5CEjDsa" -m mark --mark 0x60d00000/0xfff00000 -g cali-fw-cali14897c537b0
-A cali-from-endpoint-mark -m comment --comment "cali:yZ2sqRg9xTYsCr3y" -m mark --mark 0x57700000/0xfff00000 -g cali-fw-cali3782683684d
-A cali-from-endpoint-mark -m comment --comment "cali:oDjdup0MENsU2T0V" -m mark --mark 0xe9400000/0xfff00000 -g cali-fw-cali3c22a7abad0
-A cali-from-endpoint-mark -m comment --comment "cali:ftH1A1yJTN7c7gi9" -m mark --mark 0xd7f00000/0xfff00000 -g cali-fw-calia56f50b9553
-A cali-from-endpoint-mark -m comment --comment "cali:Utbl-RKrkRaAe_7u" -m mark --mark 0x8ab00000/0xfff00000 -g cali-fw-calic484cf12723
-A cali-from-endpoint-mark -m comment --comment "cali:BLzUJThjBoxaGbtq" -m mark --mark 0x17800000/0xfff00000 -g cali-fw-calif3ad7fd4559
-A cali-from-endpoint-mark -m comment --comment "cali:ykb-GgFF7201bamA" -m comment --comment "Unknown interface" -j DROP
-A cali-from-wl-dispatch -i cali14897c537b0 -m comment --comment "cali:3UiA-1aScl48cevq" -g cali-fw-cali14897c537b0
-A cali-from-wl-dispatch -i cali3+ -m comment --comment "cali:JQnUig9H7Zn4lzxa" -g cali-from-wl-dispatch-3
-A cali-from-wl-dispatch -i calia56f50b9553 -m comment --comment "cali:oui77L_EIg7fVy_s" -g cali-fw-calia56f50b9553
-A cali-from-wl-dispatch -i calic484cf12723 -m comment --comment "cali:iQvLQqtZfcQwG5KO" -g cali-fw-calic484cf12723
-A cali-from-wl-dispatch -i calif3ad7fd4559 -m comment --comment "cali:t0obBsS80qXXQps-" -g cali-fw-calif3ad7fd4559
-A cali-from-wl-dispatch -m comment --comment "cali:Fe5JAVmiKS23OMdf" -m comment --comment "Unknown interface" -j DROP
-A cali-from-wl-dispatch-3 -i cali3782683684d -m comment --comment "cali:oFBhRxLcaoHrlhpy" -g cali-fw-cali3782683684d
-A cali-from-wl-dispatch-3 -i cali3c22a7abad0 -m comment --comment "cali:Y4Wi--MwOhiuXTo8" -g cali-fw-cali3c22a7abad0
-A cali-from-wl-dispatch-3 -m comment --comment "cali:QYcOBIjDHWzpDROX" -m comment --comment "Unknown interface" -j DROP
-A cali-fw-cali14897c537b0 -m comment --comment "cali:YK-zTO5v-W4LKEIj" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A cali-fw-cali14897c537b0 -m comment --comment "cali:WnBOo4dZI8thON-M" -m conntrack --ctstate INVALID -j DROP
-A cali-fw-cali14897c537b0 -m comment --comment "cali:vZ12X4Tzs8jtDEyH" -j MARK --set-xmark 0x0/0x30000
-A cali-fw-cali14897c537b0 -p udp -m comment --comment "cali:aedjecVU_EumAarc" -m comment --comment "Drop VXLAN encapped packets originating in workloads" -m multiport --dports 4789 -j DROP
-A cali-fw-cali14897c537b0 -p ipencap -m comment --comment "cali:tx5PtvfQIPHVbzsY" -m comment --comment "Drop IPinIP encapped packets originating in workloads" -j DROP
-A cali-fw-cali14897c537b0 -m comment --comment "cali:zkSGiFDpz0FogFWd" -j cali-pro-kns.kube-system
-A cali-fw-cali14897c537b0 -m comment --comment "cali:ABTwfnYwRJF94c6a" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-fw-cali14897c537b0 -m comment --comment "cali:5Z_9ZvBe2cREizG0" -j cali-pro-_u2Tn2rSoAPffvE7JO6
-A cali-fw-cali14897c537b0 -m comment --comment "cali:bzj56OoudHWhuCpU" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-fw-cali14897c537b0 -m comment --comment "cali:QbxYSoJ281PL7HcM" -m comment --comment "Drop if no profiles matched" -j DROP
-A cali-fw-cali3782683684d -m comment --comment "cali:sDcHykBCLad6vbyL" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A cali-fw-cali3782683684d -m comment --comment "cali:qkhv4-A30CA0x6ar" -m conntrack --ctstate INVALID -j DROP
-A cali-fw-cali3782683684d -m comment --comment "cali:lLwvW6_Rn_wScHLf" -j MARK --set-xmark 0x0/0x30000
-A cali-fw-cali3782683684d -p udp -m comment --comment "cali:eXpCyHKTL5e0n5I4" -m comment --comment "Drop VXLAN encapped packets originating in workloads" -m multiport --dports 4789 -j DROP
-A cali-fw-cali3782683684d -p ipencap -m comment --comment "cali:zdY6NxUi-HRlEqUr" -m comment --comment "Drop IPinIP encapped packets originating in workloads" -j DROP
-A cali-fw-cali3782683684d -m comment --comment "cali:U9QTIibrJ1L6y1RS" -j cali-pro-kns.default
-A cali-fw-cali3782683684d -m comment --comment "cali:ZAMupusXTAOtXAbs" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-fw-cali3782683684d -m comment --comment "cali:3lWfT5cXm3hjbfC_" -j cali-pro-_8zZ8x7SANlAuYdH3So
-A cali-fw-cali3782683684d -m comment --comment "cali:UwF4mgDJgY_65aoG" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-fw-cali3782683684d -m comment --comment "cali:D9Hgi4gDTmen9GSC" -m comment --comment "Drop if no profiles matched" -j DROP
-A cali-fw-cali3c22a7abad0 -m comment --comment "cali:WH6A4imwzPXSdozv" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A cali-fw-cali3c22a7abad0 -m comment --comment "cali:Lcf1da9zSkACZfXU" -m conntrack --ctstate INVALID -j DROP
-A cali-fw-cali3c22a7abad0 -m comment --comment "cali:LMla1HHBgT-s5VcG" -j MARK --set-xmark 0x0/0x30000
-A cali-fw-cali3c22a7abad0 -p udp -m comment --comment "cali:p37tYpkfYjlHLWMV" -m comment --comment "Drop VXLAN encapped packets originating in workloads" -m multiport --dports 4789 -j DROP
-A cali-fw-cali3c22a7abad0 -p ipencap -m comment --comment "cali:z3JgP_rou_bzukCU" -m comment --comment "Drop IPinIP encapped packets originating in workloads" -j DROP
-A cali-fw-cali3c22a7abad0 -m comment --comment "cali:qYo8S9hciWTSADvo" -j cali-pro-kns.default
-A cali-fw-cali3c22a7abad0 -m comment --comment "cali:syrSUbqgTpGc-JB_" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-fw-cali3c22a7abad0 -m comment --comment "cali:RL6yFAbzu-CtLHCn" -j cali-pro-ksa.default.default
-A cali-fw-cali3c22a7abad0 -m comment --comment "cali:Eh_Ffiu5Kg2xasNt" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-fw-cali3c22a7abad0 -m comment --comment "cali:q6GQh-ifzX-5TVdD" -m comment --comment "Drop if no profiles matched" -j DROP
-A cali-fw-calia56f50b9553 -m comment --comment "cali:ShScJ9vaOGUtvyBk" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A cali-fw-calia56f50b9553 -m comment --comment "cali:yZH7bvkO3FGnAawv" -m conntrack --ctstate INVALID -j DROP
-A cali-fw-calia56f50b9553 -m comment --comment "cali:pJUbweOS31xhatc5" -j MARK --set-xmark 0x0/0x30000
-A cali-fw-calia56f50b9553 -p udp -m comment --comment "cali:OFKiHFLYEYSnmsxo" -m comment --comment "Drop VXLAN encapped packets originating in workloads" -m multiport --dports 4789 -j DROP
-A cali-fw-calia56f50b9553 -p ipencap -m comment --comment "cali:ZxbIcpM2YQug7WWZ" -m comment --comment "Drop IPinIP encapped packets originating in workloads" -j DROP
-A cali-fw-calia56f50b9553 -m comment --comment "cali:y0vb_A5vOCGJU0XA" -j cali-pro-kns.default
-A cali-fw-calia56f50b9553 -m comment --comment "cali:vvIekqlIAImloAue" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-fw-calia56f50b9553 -m comment --comment "cali:IHKQO2fvj1v6wq6t" -j cali-pro-_8zZ8x7SANlAuYdH3So
-A cali-fw-calia56f50b9553 -m comment --comment "cali:cu3XF_Ey2Qg4n_tu" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-fw-calia56f50b9553 -m comment --comment "cali:eJK91sf6Tj8snGCm" -m comment --comment "Drop if no profiles matched" -j DROP
-A cali-fw-calic484cf12723 -m comment --comment "cali:lgfP-PpVj-5uN1KY" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A cali-fw-calic484cf12723 -m comment --comment "cali:FcOMKTr9rIUF-rUz" -m conntrack --ctstate INVALID -j DROP
-A cali-fw-calic484cf12723 -m comment --comment "cali:wbHJAy8pkcjlldM3" -j MARK --set-xmark 0x0/0x30000
-A cali-fw-calic484cf12723 -p udp -m comment --comment "cali:bNk-4M9stRUy7TXV" -m comment --comment "Drop VXLAN encapped packets originating in workloads" -m multiport --dports 4789 -j DROP
-A cali-fw-calic484cf12723 -p ipencap -m comment --comment "cali:bo7DJM0EqPutELuW" -m comment --comment "Drop IPinIP encapped packets originating in workloads" -j DROP
-A cali-fw-calic484cf12723 -m comment --comment "cali:bClYDWdWtyY6LsDp" -j cali-pro-kns.default
-A cali-fw-calic484cf12723 -m comment --comment "cali:CYR5gfLwk7oVi3Rr" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-fw-calic484cf12723 -m comment --comment "cali:kMnQPWEViLEBnxCj" -j cali-pro-ksa.default.default
-A cali-fw-calic484cf12723 -m comment --comment "cali:wps3bBgQFZ9bEr_l" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-fw-calic484cf12723 -m comment --comment "cali:1NFo5yVsO54jLtDE" -m comment --comment "Drop if no profiles matched" -j DROP
-A cali-fw-calif3ad7fd4559 -m comment --comment "cali:d30FdXxFpoCXbFri" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A cali-fw-calif3ad7fd4559 -m comment --comment "cali:xbqJsPxv0NLpgrUh" -m conntrack --ctstate INVALID -j DROP
-A cali-fw-calif3ad7fd4559 -m comment --comment "cali:FHqeNY2orFLxUIJo" -j MARK --set-xmark 0x0/0x30000
-A cali-fw-calif3ad7fd4559 -p udp -m comment --comment "cali:cibFnrg-wOz0h_JP" -m comment --comment "Drop VXLAN encapped packets originating in workloads" -m multiport --dports 4789 -j DROP
-A cali-fw-calif3ad7fd4559 -p ipencap -m comment --comment "cali:okiOHlEu7nccFzo2" -m comment --comment "Drop IPinIP encapped packets originating in workloads" -j DROP
-A cali-fw-calif3ad7fd4559 -m comment --comment "cali:b9Abk_LETA9nGBIa" -j cali-pro-kns.default
-A cali-fw-calif3ad7fd4559 -m comment --comment "cali:1M7I9u1VkakBn-VX" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-fw-calif3ad7fd4559 -m comment --comment "cali:u9aXvTQaMSUutkbC" -j cali-pro-_8zZ8x7SANlAuYdH3So
-A cali-fw-calif3ad7fd4559 -m comment --comment "cali:krnMBRV_hS8AmQSl" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-fw-calif3ad7fd4559 -m comment --comment "cali:lROFNTFpKlp64iPQ" -m comment --comment "Drop if no profiles matched" -j DROP
-A cali-pri-_8zZ8x7SANlAuYdH3So -m comment --comment "cali:Tdrw0r_LsS7B0-Nw" -m comment --comment "Profile ksa.default.kube-pinger ingress"
-A cali-pri-_u2Tn2rSoAPffvE7JO6 -m comment --comment "cali:WqgznqAQ-uYV0oBx" -m comment --comment "Profile ksa.kube-system.coredns ingress"
-A cali-pri-kns.default -m comment --comment "cali:WMSw8BmYOknRHfsz" -m comment --comment "Profile kns.default ingress" -j MARK --set-xmark 0x10000/0x10000
-A cali-pri-kns.kube-system -m comment --comment "cali:J1TyxtHWd0qaBGK-" -m comment --comment "Profile kns.kube-system ingress" -j MARK --set-xmark 0x10000/0x10000
-A cali-pri-ksa.default.default -m comment --comment "cali:PrckJA84jX_kGp99" -m comment --comment "Profile ksa.default.default ingress"
-A cali-pro-_8zZ8x7SANlAuYdH3So -m comment --comment "cali:FZ08FWNRvmlApB-h" -m comment --comment "Profile ksa.default.kube-pinger egress"
-A cali-pro-_u2Tn2rSoAPffvE7JO6 -m comment --comment "cali:0-_UPh39dt5XfhmJ" -m comment --comment "Profile ksa.kube-system.coredns egress"
-A cali-pro-kns.default -m comment --comment "cali:Vr81boRqq4V77Sg8" -m comment --comment "Profile kns.default egress" -j MARK --set-xmark 0x10000/0x10000
-A cali-pro-kns.kube-system -m comment --comment "cali:tgOR2S8DVHZW3F1M" -m comment --comment "Profile kns.kube-system egress" -j MARK --set-xmark 0x10000/0x10000
-A cali-pro-ksa.default.default -m comment --comment "cali:bUZzZcietq9v5Ybq" -m comment --comment "Profile ksa.default.default egress"
-A cali-set-endpoint-mark -i cali14897c537b0 -m comment --comment "cali:3h0SDopjyKO2Tbaw" -g cali-sm-cali14897c537b0
-A cali-set-endpoint-mark -i cali3+ -m comment --comment "cali:7mFPxdNGeV2lLVvw" -g cali-set-endpoint-mark-3
-A cali-set-endpoint-mark -i calia56f50b9553 -m comment --comment "cali:4neSsAtU2B6Snnko" -g cali-sm-calia56f50b9553
-A cali-set-endpoint-mark -i calic484cf12723 -m comment --comment "cali:AEZAaYQzQRQxHzyp" -g cali-sm-calic484cf12723
-A cali-set-endpoint-mark -i calif3ad7fd4559 -m comment --comment "cali:8a79KAY2_zh4gwC_" -g cali-sm-calif3ad7fd4559
-A cali-set-endpoint-mark -i cali+ -m comment --comment "cali:-4C8KmShAMrdZKuB" -m comment --comment "Unknown endpoint" -j DROP
-A cali-set-endpoint-mark -m comment --comment "cali:zkVgTdrfQeVpVrHN" -m comment --comment "Non-Cali endpoint mark" -j MARK --set-xmark 0x100000/0xfff00000
-A cali-set-endpoint-mark-3 -i cali3782683684d -m comment --comment "cali:11DVj7xUCoaEYIKZ" -g cali-sm-cali3782683684d
-A cali-set-endpoint-mark-3 -i cali3c22a7abad0 -m comment --comment "cali:CIsEGV4B56ji6bjb" -g cali-sm-cali3c22a7abad0
-A cali-sm-cali14897c537b0 -m comment --comment "cali:17NIOjoTQFnHdqtR" -j MARK --set-xmark 0x60d00000/0xfff00000
-A cali-sm-cali3782683684d -m comment --comment "cali:kVHS1G1mgiVkbrgD" -j MARK --set-xmark 0x57700000/0xfff00000
-A cali-sm-cali3c22a7abad0 -m comment --comment "cali:vfvDUgvyNPl6rsr_" -j MARK --set-xmark 0xe9400000/0xfff00000
-A cali-sm-calia56f50b9553 -m comment --comment "cali:WWPplTP01n04jrxF" -j MARK --set-xmark 0xd7f00000/0xfff00000
-A cali-sm-calic484cf12723 -m comment --comment "cali:rxsyGJP8iGFAN9qY" -j MARK --set-xmark 0x8ab00000/0xfff00000
-A cali-sm-calif3ad7fd4559 -m comment --comment "cali:zgSzkzM6ePg93X3d" -j MARK --set-xmark 0x17800000/0xfff00000
-A cali-to-wl-dispatch -o cali14897c537b0 -m comment --comment "cali:ZzTRO-Gs5DKU2bx9" -g cali-tw-cali14897c537b0
-A cali-to-wl-dispatch -o cali3+ -m comment --comment "cali:xfOMOxFhy_4SCjja" -g cali-to-wl-dispatch-3
-A cali-to-wl-dispatch -o calia56f50b9553 -m comment --comment "cali:e0jK-n2wBLGst2OJ" -g cali-tw-calia56f50b9553
-A cali-to-wl-dispatch -o calic484cf12723 -m comment --comment "cali:Xx1zY83WlmfifTm_" -g cali-tw-calic484cf12723
-A cali-to-wl-dispatch -o calif3ad7fd4559 -m comment --comment "cali:soXEadAlqUoReMJA" -g cali-tw-calif3ad7fd4559
-A cali-to-wl-dispatch -m comment --comment "cali:93n5ITDKtV6wGhBd" -m comment --comment "Unknown interface" -j DROP
-A cali-to-wl-dispatch-3 -o cali3782683684d -m comment --comment "cali:ZFsQl-BnfaYJYAy0" -g cali-tw-cali3782683684d
-A cali-to-wl-dispatch-3 -o cali3c22a7abad0 -m comment --comment "cali:4s8VQxEQIawCj4ZW" -g cali-tw-cali3c22a7abad0
-A cali-to-wl-dispatch-3 -m comment --comment "cali:JF8Q99iT6ID2w2Ag" -m comment --comment "Unknown interface" -j DROP
-A cali-tw-cali14897c537b0 -m comment --comment "cali:MtKfELE8lpEqaeal" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A cali-tw-cali14897c537b0 -m comment --comment "cali:4u2ZdUBeYgIkGwIE" -m conntrack --ctstate INVALID -j DROP
-A cali-tw-cali14897c537b0 -m comment --comment "cali:1zgO1vgJ4Ye9PX79" -j MARK --set-xmark 0x0/0x30000
-A cali-tw-cali14897c537b0 -m comment --comment "cali:clcsWBhHhTVikgNS" -j cali-pri-kns.kube-system
-A cali-tw-cali14897c537b0 -m comment --comment "cali:Gwy41HUAts3qjJUH" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-tw-cali14897c537b0 -m comment --comment "cali:BGV6kM2WeDdWHaEP" -j cali-pri-_u2Tn2rSoAPffvE7JO6
-A cali-tw-cali14897c537b0 -m comment --comment "cali:h0M9H7UEXXyGk3l0" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-tw-cali14897c537b0 -m comment --comment "cali:iF1YfDDZgI-e7zD0" -m comment --comment "Drop if no profiles matched" -j DROP
-A cali-tw-cali3782683684d -m comment --comment "cali:uhF4KX6WuO7Jfllp" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A cali-tw-cali3782683684d -m comment --comment "cali:z2n2dRGutZO2X4_M" -m conntrack --ctstate INVALID -j DROP
-A cali-tw-cali3782683684d -m comment --comment "cali:SRKZ0aQFOc6Nel4h" -j MARK --set-xmark 0x0/0x30000
-A cali-tw-cali3782683684d -m comment --comment "cali:LIZ-l9lmV18JK-YE" -j cali-pri-kns.default
-A cali-tw-cali3782683684d -m comment --comment "cali:8IOJrscLkQMGuPP-" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-tw-cali3782683684d -m comment --comment "cali:ePAPUW5N1-XaMQpz" -j cali-pri-_8zZ8x7SANlAuYdH3So
-A cali-tw-cali3782683684d -m comment --comment "cali:K0jvFOaD-C933wZb" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-tw-cali3782683684d -m comment --comment "cali:BhuZ9B27Lpml2qhH" -m comment --comment "Drop if no profiles matched" -j DROP
-A cali-tw-cali3c22a7abad0 -m comment --comment "cali:D4hNEz3Vrr1R4MGw" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A cali-tw-cali3c22a7abad0 -m comment --comment "cali:9ws8CGSW6wpZi5Ab" -m conntrack --ctstate INVALID -j DROP
-A cali-tw-cali3c22a7abad0 -m comment --comment "cali:Yh12Rnu4dgrzhrAf" -j MARK --set-xmark 0x0/0x30000
-A cali-tw-cali3c22a7abad0 -m comment --comment "cali:YXTl54241Pnj-fVl" -j cali-pri-kns.default
-A cali-tw-cali3c22a7abad0 -m comment --comment "cali:4znOgJ9VDKrKT7IB" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-tw-cali3c22a7abad0 -m comment --comment "cali:_xBvXFFVqboH3drr" -j cali-pri-ksa.default.default
-A cali-tw-cali3c22a7abad0 -m comment --comment "cali:IE6ugvAUq-eIQ4QX" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-tw-cali3c22a7abad0 -m comment --comment "cali:rfaq-LJjZMWw5ncR" -m comment --comment "Drop if no profiles matched" -j DROP
-A cali-tw-calia56f50b9553 -m comment --comment "cali:-0uHcfxdsI74m823" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A cali-tw-calia56f50b9553 -m comment --comment "cali:cIFZDVye0e0VSbju" -m conntrack --ctstate INVALID -j DROP
-A cali-tw-calia56f50b9553 -m comment --comment "cali:StAzhOOw2mF1f10f" -j MARK --set-xmark 0x0/0x30000
-A cali-tw-calia56f50b9553 -m comment --comment "cali:-TdBda3j7IGSYKls" -j cali-pri-kns.default
-A cali-tw-calia56f50b9553 -m comment --comment "cali:jwIQr09LIW2Nhmn0" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-tw-calia56f50b9553 -m comment --comment "cali:pf2331s5NLqVU-Nd" -j cali-pri-_8zZ8x7SANlAuYdH3So
-A cali-tw-calia56f50b9553 -m comment --comment "cali:vJgR1wWSAo8U8WU1" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-tw-calia56f50b9553 -m comment --comment "cali:Xg88Vj7zdlmZNVT4" -m comment --comment "Drop if no profiles matched" -j DROP
-A cali-tw-calic484cf12723 -m comment --comment "cali:ZhOcmXNaXLjBnIVf" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A cali-tw-calic484cf12723 -m comment --comment "cali:h2XOX2DNFTc68Mua" -m conntrack --ctstate INVALID -j DROP
-A cali-tw-calic484cf12723 -m comment --comment "cali:kAGa8LkUj9PXyCpY" -j MARK --set-xmark 0x0/0x30000
-A cali-tw-calic484cf12723 -m comment --comment "cali:dVJTsu7Q4ajCsr6r" -j cali-pri-kns.default
-A cali-tw-calic484cf12723 -m comment --comment "cali:6B8Mm6xXq0BRsbgQ" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-tw-calic484cf12723 -m comment --comment "cali:FN5IR6D2lzb8Tawj" -j cali-pri-ksa.default.default
-A cali-tw-calic484cf12723 -m comment --comment "cali:7bkO8P2mhNbMnMes" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-tw-calic484cf12723 -m comment --comment "cali:G6II9PKe1aAqM80I" -m comment --comment "Drop if no profiles matched" -j DROP
-A cali-tw-calif3ad7fd4559 -m comment --comment "cali:XI9josiu0p8_2njg" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A cali-tw-calif3ad7fd4559 -m comment --comment "cali:J2Lo7744AMr9awyg" -m conntrack --ctstate INVALID -j DROP
-A cali-tw-calif3ad7fd4559 -m comment --comment "cali:0LslewBVSoW5onht" -j MARK --set-xmark 0x0/0x30000
-A cali-tw-calif3ad7fd4559 -m comment --comment "cali:tmPWG1cXvwfpC2dh" -j cali-pri-kns.default
-A cali-tw-calif3ad7fd4559 -m comment --comment "cali:s0bUJlXNGgcnCAGm" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-tw-calif3ad7fd4559 -m comment --comment "cali:9OYJC404D4Y2WKMR" -j cali-pri-_8zZ8x7SANlAuYdH3So
-A cali-tw-calif3ad7fd4559 -m comment --comment "cali:GkC0zhpXrOh32gLq" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-tw-calif3ad7fd4559 -m comment --comment "cali:qAJQKKWJZkPJD-1i" -m comment --comment "Drop if no profiles matched" -j DROP
-A cali-wl-to-host -m comment --comment "cali:Ee9Sbo10IpVujdIY" -j cali-from-wl-dispatch
-A cali-wl-to-host -m comment --comment "cali:sO1YJiY1b553biDi" -m comment --comment "Configured DefaultEndpointToHostAction" -j RETURN
COMMIT
# Completed on Mon Nov 17 07:40:30 2025
# Generated by iptables-save v1.8.7 on Mon Nov 17 07:40:30 2025
*mangle
:PREROUTING ACCEPT [7690:3289240]
:INPUT ACCEPT [193867:63016041]
:FORWARD ACCEPT [238:12376]
:OUTPUT ACCEPT [200513:93936296]
:POSTROUTING ACCEPT [200554:93935611]
:CILIUM_POST_mangle - [0:0]
:CILIUM_PRE_mangle - [0:0]
:KUBE-IPTABLES-HINT - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:OVN-OUTPUT - [0:0]
:OVN-POSTROUTING - [0:0]
:OVN-PREROUTING - [0:0]
:cali-POSTROUTING - [0:0]
:cali-PREROUTING - [0:0]
:cali-from-host-endpoint - [0:0]
:cali-to-host-endpoint - [0:0]
-A PREROUTING -m comment --comment "cilium-feeder: CILIUM_PRE_mangle" -j CILIUM_PRE_mangle
-A PREROUTING -m comment --comment "cali:6gwbT8clXdHdC1b1" -j cali-PREROUTING
-A POSTROUTING -m comment --comment "cilium-feeder: CILIUM_POST_mangle" -j CILIUM_POST_mangle
-A POSTROUTING -m comment --comment "kube-ovn postrouting rules" -j OVN-POSTROUTING
-A POSTROUTING -m comment --comment "cali:O3lYWMrLQYEMJtB5" -j cali-POSTROUTING
-A CILIUM_PRE_mangle ! -o lo -m socket --transparent -m mark ! --mark 0xe00/0xf00 -m mark ! --mark 0x800/0xf00 -m comment --comment "cilium: any->pod redirect proxied traffic to host proxy" -j MARK --set-xmark 0x200/0xffffffff
-A CILIUM_PRE_mangle -p tcp -m mark --mark 0x97800200 -m comment --comment "cilium: TPROXY to host cilium-dns-egress proxy" -j TPROXY --on-port 32919 --on-ip 127.0.0.1 --tproxy-mark 0x200/0xffffffff
-A CILIUM_PRE_mangle -p udp -m mark --mark 0x97800200 -m comment --comment "cilium: TPROXY to host cilium-dns-egress proxy" -j TPROXY --on-port 32919 --on-ip 127.0.0.1 --tproxy-mark 0x200/0xffffffff
-A OVN-POSTROUTING -p tcp -m set --match-set ovn40subnets src -m tcp --tcp-flags RST RST -m state --state INVALID -j DROP
-A cali-POSTROUTING -m comment --comment "cali:NX-7roTexQ3fGRfU" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-POSTROUTING -m comment --comment "cali:qaajsWArU1ku9saf" -m mark ! --mark 0x0/0xfff00000 -j RETURN
-A cali-POSTROUTING -m comment --comment "cali:N2faOPfc4DVQAfQj" -j MARK --set-xmark 0x0/0xf0000
-A cali-POSTROUTING -m comment --comment "cali:IR1ghU6yHNWsaaJF" -m conntrack --ctstate DNAT -j cali-to-host-endpoint
-A cali-POSTROUTING -m comment --comment "cali:fcjhvOBNywbfCkS2" -m comment --comment "Host endpoint policy accepted packet." -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-PREROUTING -m comment --comment "cali:6BJqBjBC7crtA-7-" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A cali-PREROUTING -m comment --comment "cali:KX7AGNd6rMcDUai6" -m mark --mark 0x10000/0x10000 -j ACCEPT
-A cali-PREROUTING -m comment --comment "cali:wNH7KsA3ILKJBsY9" -j cali-from-host-endpoint
-A cali-PREROUTING -m comment --comment "cali:Cg96MgVuoPm7UMRo" -m comment --comment "Host endpoint policy accepted packet." -m mark --mark 0x10000/0x10000 -j ACCEPT
COMMIT
# Completed on Mon Nov 17 07:40:30 2025 
root@k8s-ctrl:~/kubespray1-31# ipset list | grep cali
Name: cali40this-host
Name: cali40all-ipam-pools
Name: cali40masq-ipam-pools
Name: cali40all-hosts-net
root@k8s-ctrl:~/kubespray1-31#
root@k8s-ctrl:~/kubespray1-31#
root@k8s-ctrl:~/kubespray1-31#
root@k8s-ctrl:~/kubespray1-31# ipset list cali40this-host
Name: cali40this-host
Type: hash:ip
Revision: 5
Header: family inet hashsize 1024 maxelem 1048576 bucketsize 12 initval 0x8140768f
Size in memory: 440
References: 3
Number of entries: 6
Members:
169.254.25.10
127.0.0.1
10.222.12.0
192.168.41.150
127.0.0.0
172.16.189.100
root@k8s-ctrl:~/kubespray1-31# ipset list cali40all-ipam-pools
Name: cali40all-ipam-pools
Type: hash:net
Revision: 7
Header: family inet hashsize 1024 maxelem 1048576 bucketsize 12 initval 0x07b19799
Size in memory: 552
References: 1
Number of entries: 2
Members:
10.222.0.0/16
10.198.0.0/16
root@k8s-ctrl:~/kubespray1-31# ipset list cali40masq-ipam-pools
Name: cali40masq-ipam-pools
Type: hash:net
Revision: 7
Header: family inet hashsize 1024 maxelem 1048576 bucketsize 12 initval 0x83772cbd
Size in memory: 552
References: 1
Number of entries: 2
Members:
10.198.0.0/16
10.222.0.0/16
root@k8s-ctrl:~/kubespray1-31# ipset list cali40all-hosts-net
Name: cali40all-hosts-net
Type: hash:net
Revision: 7
Header: family inet hashsize 1024 maxelem 1048576 bucketsize 12 initval 0xc19e9fba
Size in memory: 600
References: 2
Number of entries: 3
Members:
172.16.189.101
172.16.189.100
172.16.189.102

比如: 是否会导致如下问题:

  • NAT 规则冲突

  • PREROUTING/POSTROUTING 优先级错乱

  • 同一流量被多个 CNI 设置 mark

  • 反向路径过滤(RPFilter)和策略路由被不同 CNI 修改

  • NodePort、ClusterIP、Pod-to-Pod 流量不可预测

  • Cilium、Calico、OVN 在 iptables/nftables 上互相覆盖链(明显)

你的 iptables 中同时存在:

Cilium 链(大量 CILIUM_xxx)

例如:

CILIUM_OUTPUT_raw
CILIUM_PRE_raw
CILIUM_INPUT
CILIUM_OUTPUT
CILIUM_FORWARD

Calico 链(大量 cali-xxx)

例如:

cali-FORWARD
cali-nat-outgoing
cali-from-wl-dispatch
cali-fw-caliXXXX
...

OVN-Kubernetes 链

例如:

OVN-POSTROUTING
OVN-PREROUTING
ovn40subnets
ovn40services

Kube-proxy (iptables mode) 也在里面产生链:

KUBE-SERVICES
KUBE-NODE-PORT
KUBE-POSTROUTING

这说明:

这台节点曾经运行过 Cilium、Calico、OVN-Kubernetes,并且清理不干净,三家规则共存。

❗ 这种状态几乎必然造成流量错乱

例如:

1. Cilium mark 规则与 Calico 的 mark 冲突

  • Cilium 使用 0x200/0xf000xa00/0xe00
  • Calico 使用 0x10000/0x100000x40000/0x40000
  • OVN 使用 0x4000/0x4000, 0x80000/0x80000

三套系统都在用 mark 做策略路由 → 破坏彼此行为。

2. Cilium 的 RAW 表 NOTRACK 与 Calico/OVN 的 NAT 冲突

  • Cilium:大量 --notrack
  • Calico:需要 conntrack 才能做 NAT
  • OVN:也在依赖 conntrack 做 EIP、DGP(Distributed Gateway Port)的 SNAT

导致:

  • 流量被 Cilium 设置了 notrack → Calico NAT 失效
  • 或反过来 conntrack 被 Calico 修改导致 cilium_host 行为异常

3. OVN 的 NodePort 处理与 kube-proxy/Calico 同时存在

OVN 会在 PREROUTING 修改 NodePort mark,不经过 kube-proxy
而 kube-proxy 也在抢 NodePort 流量
Calico 也在做 host endpoint policy

三套 NodePort 行为叠加,极易导致某些端口访问失败。

4. IPIP/VXLAN 被多个 CNI 锁定

你有:

  • Calico 的 IPIP / VXLAN 防护(DROP workload-originating 4789)
  • Cilium 的 VXLAN / Geneve(若是 tunnel 模式)
  • OVN 的 Geneve(6081)

这三者的封装可能被 Calico DROP。

GPT 判定一定会出问题

2. 继续分析 kubeovn + chanin cilium 模式


# 环境信息


目前这个环境被切换为 kubeovn chain cilium 模式: # cat /tmp/cilium-values.yml
routingMode: native
tunnelProtocol: ""
kubeProxyReplacement: true
devices: "bond+ eth+ ens+ ovn0 genev_sys_6081"
forceDeviceDetection: true
loadBalancer:
  algorithm: maglev
  mode: snat
cni:
  chainingMode: "generic-veth"
  chainingTarget: "kube-ovn"
  configMap: cni-configuration
  customConf: true
  exclusive: false
  uninstall: false
bpf:
  masquerade: false
  hostLegacyRouting: false
enableIPv4Masquerade: false
enableIPv6Masquerade: false

ipam:
  mode: cluster-pool
  operator:
    clusterPoolIPv4MaskSize: 24
    clusterPoolIPv6MaskSize: 120
    clusterPoolIPv4PodCIDRList:
      - 10.222.12.0/24
      - 10.222.219.0/24
      - 10.222.160.0/24
    clusterPoolIPv6PodCIDRList:

# Enable SCTP support. NOTE: Currently, SCTP support does not support rewriting ports or multihoming.
sctp:
  enabled: true
policyEnforcementMode: never

k8s:
  apiServerURLs: "https://172.16.189.100:6443"
image:
  tag: "v1.18.3"
  pullPolicy: "IfNotPresent"
  useDigest: false

operator:
  image:
    tag: "v1.18.3"
    pullPolicy: "IfNotPresent"
    useDigest: false
  unmanagedPodWatcher:
    restart: false
root@k8s-ctrl:~/kubespray1-31#

好的,我已经确认你现在这个集群的状态是:

Kube-OVN + Cilium(generic-veth chaining)模式(无隧道、无 NAT)

并且启用了:

  • kubeProxyReplacement: true(Cilium 接管服务转发)
  • Cilium chaining to kube-ovn(generic-veth)
  • Cilium routingMode=native(无 cilium_vxlan、无 geneve)
  • Cilium bpf.masquerade=false
  • Cilium 不做 SNAT(LB 也是 SNAT mode)
  • IPAM cluster-pool 模式(给 kube-ovn pod 分配二段网段)

这是一个非常特别的 Cilium 作为数据面增强、Kube-OVN 作为主 CNI 的组合结构。

📌 1. 现在 Cilium 不负责 Pod 接入(eth0)

因为你启用了:

cni.chainingMode: "generic-veth"
cni.chainingTarget: "kube-ovn"
exclusive: false

这意味着:

  • kube-ovn CNI 负责创建 veth pair、分配 IP
  • Cilium 只在 host 侧接受被链式调用,加载 BPF(ingress/egress)
  • kube-ovn 管理路由、Geneve、策略路由

👉 Pod 的 eth0 是 kube-ovn 的,不是 cilium_host 的 “lxc+ / cali+”。


📌 2. 现在 Cilium 不应该创建隧道接口

你已经设置:

routingMode: native
tunnelProtocol: ""

所以:

  • 不会创建 cilium_vxlan
  • 不会创建 cilium_geneve
  • 仍然会创建 cilium_host(BPF 需要)
  • 仍然会创建 cilium_net(native routing 需要)

你的 iptables dump 显示确实 没有 cilium_vxlan / cilium_geneve

这是符合预期的。


📌 3. 你仍然看到大量 Calico 链,是因为旧的 Calico 没有被清理

Cilium chaining 本身不会创建 Calico 链,
而且你明确设置:

uninstall: false

所以 Helm 不清理旧链。

你的节点中同时出现:

✔ Cilium 链

✔ OVN 链

❗ 大量 Calico 链

✔ kube-proxy 链(但你又启用 kubeProxyReplacement=true)

这证明:

  • 旧 Calico 没卸载干净
  • kube-proxy 可能仍然在运行(但 Cilium 也在抢 iptables)

📌 4. 你的 kube-proxy 实际上应该被禁用

由于你启用了:

kubeProxyReplacement: true
loadBalancer.mode: snat

kube-proxy 应该被 disable:

  • 不能让 Cilium 与 kube-proxy 共享 KUBE-SERVICES
  • Cilium 会接管 NodePort/ClusterIP
  • kube-proxy 残留会导致所有 svc PREROUTING 冲突

你现在 dump 出来的链显示 kube-proxy 规则仍在使用。

→ 这会导致 svc、NodePort 多路径冲突。


📌 5. 你现在是 kube-ovn 主导路由,Cilium 主导服务 + 安全

路径如下:

Pod 出栈

kube-ovn eth0 → cilium cgroup egress BPF → host routing → kube-ovn geneve

Pod 入栈

node geneve → kube-ovn → pod veth → cilium ingress BPF(chaining)

流量都正常,但前提是:iptables 只保留 kube-ovn + cilium


⚠ 6. 你现在最大的问题就是:Calico + kube-proxy 残留链破坏行为

你配置是正确的,
但节点的 iptables 现状是不干净的!

会出现:

  • NodePort 随机不通
  • svc DNAT 一会儿走 kube-proxy、一会儿走 Cilium
  • Pod 流量可能部分被 Calico DROP
  • OVN 的 geneve 4789/6081 可能被 Calico DROP
  • 外部 SNAT/DSR 行为混乱
  • conntrack 状态被多个 CNI 改写

🧨 必须清理的内容

❌ 删除所有 Calico 规则

(cali-*, calico-xxxx, FELIX-*, cali-nat-outgoing, etc)

❌ 删除所有 kube-proxy 规则

你启用了 kubeProxyReplacement=true,所以:

  • disable kube-proxy deployment
  • 删除 KUBE-* 链

到现在 GPT 还警告一定要删除,其实我目前真的不想处理它,准备到最后迁移完毕后,再一次性处理掉。

继续分析并分类考虑 iptables 规则:

  • 基于 ipset 匹配后处理
  • 基于 cali veth 或者 ovn0 接口匹配后打标签,然后继续处理
  • node 上的通用规则

比如打标签类型的:


root@k8s-ctrl:/etc/cni/net.d# iptables-save | grep -i ovn0
-A OVN-PREROUTING -i ovn0 -m set --match-set ovn40subnets src -m set --match-set KUBE-CLUSTER-IP dst,dst -j MARK --set-xmark 0x4000/0x4000
root@k8s-ctrl:/etc/cni/net.d#
root@k8s-ctrl:/etc/cni/net.d# iptables-save | grep -i 0x4000 | grep -v 40000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A OVN-POSTROUTING -m set --match-set ovn40services src -m set --match-set ovn40subnets dst -m mark --mark 0x4000/0x4000 -j SNAT --to-source 172.16.189.100 --random-fully
-A OVN-POSTROUTING -m mark --mark 0x4000/0x4000 -j OVN-MASQUERADE
-A OVN-PREROUTING -i ovn0 -m set --match-set ovn40subnets src -m set --match-set KUBE-CLUSTER-IP dst,dst -j MARK --set-xmark 0x4000/0x4000
-A OVN-PREROUTING -p tcp -m set --match-set ovn40other-node src -m set --match-set KUBE-NODE-PORT-LOCAL-TCP dst -j MARK --set-xmark 0x4000/0x4000
-A OVN-PREROUTING -p udp -m set --match-set ovn40other-node src -m set --match-set KUBE-NODE-PORT-LOCAL-UDP dst -j MARK --set-xmark 0x4000/0x4000
-A INPUT -p tcp -m mark ! --mark 0x4000/0x4000 -m set --match-set ovn40services dst -m conntrack --ctstate NEW -j REJECT --reject-with icmp-port-unreachable
-A OUTPUT -p tcp -m mark ! --mark 0x4000/0x4000 -m set --match-set ovn40services dst -m conntrack --ctstate NEW -j REJECT --reject-with icmp-port-unreachable
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT

# PREROUTING INPUT OUTPUT POSTROUTING FORWARD

root@k8s-ctrl:/etc/cni/net.d# iptables-save  | grep -i 0x10000
-A cali-OUTPUT -m comment --comment "cali:pN0F5zD0b8yf9W1Z" -m mark --mark 0x10000/0x10000 -j ACCEPT
-A cali-PREROUTING -m comment --comment "cali:lV4V2MPoMBf0hl9T" -m mark --mark 0x10000/0x10000 -j ACCEPT
-A FORWARD -m comment --comment "cali:S93hcgKJrXEqnTfs" -m comment --comment "Policy explicitly accepted packet." -m mark --mark 0x10000/0x10000 -j ACCEPT
-A FORWARD -m comment --comment "cali:mp77cMpurHhyjLrM" -j MARK --set-xmark 0x10000/0x10000
-A cali-FORWARD -m comment --comment "cali:A_sPAO0mcxbT9mOV" -m mark --mark 0x0/0x10000 -j cali-from-hep-forward
-A cali-INPUT -m comment --comment "cali:iX2AYvqGXaVqwkro" -m mark --mark 0x10000/0x10000 -j ACCEPT
-A cali-INPUT -m comment --comment "cali:inBL01YlfurT0dbI" -m comment --comment "Host endpoint policy accepted packet." -m mark --mark 0x10000/0x10000 -j ACCEPT
-A cali-OUTPUT -m comment --comment "cali:Mq1_rAdXXH3YkrzW" -m mark --mark 0x10000/0x10000 -j ACCEPT
-A cali-OUTPUT -m comment --comment "cali:aSnsxZdmhxm_ilRZ" -m comment --comment "Host endpoint policy accepted packet." -m mark --mark 0x10000/0x10000 -j ACCEPT
-A cali-forward-endpoint-mark -m comment --comment "cali:O0SmFDrnm7KggWqW" -m mark ! --mark 0x100000/0xfff00000 -j cali-from-endpoint-mark
-A cali-forward-endpoint-mark -m comment --comment "cali:VxO6hyNWz62YEtul" -m comment --comment "Policy explicitly accepted packet." -m mark --mark 0x10000/0x10000 -j ACCEPT
-A cali-fw-cali14897c537b0 -m comment --comment "cali:ABTwfnYwRJF94c6a" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-fw-cali14897c537b0 -m comment --comment "cali:bzj56OoudHWhuCpU" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-fw-cali3782683684d -m comment --comment "cali:ZAMupusXTAOtXAbs" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-fw-cali3782683684d -m comment --comment "cali:UwF4mgDJgY_65aoG" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-fw-cali3c22a7abad0 -m comment --comment "cali:syrSUbqgTpGc-JB_" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-fw-cali3c22a7abad0 -m comment --comment "cali:Eh_Ffiu5Kg2xasNt" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-fw-calia56f50b9553 -m comment --comment "cali:vvIekqlIAImloAue" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-fw-calia56f50b9553 -m comment --comment "cali:cu3XF_Ey2Qg4n_tu" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-fw-calic484cf12723 -m comment --comment "cali:CYR5gfLwk7oVi3Rr" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-fw-calic484cf12723 -m comment --comment "cali:wps3bBgQFZ9bEr_l" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-fw-calif3ad7fd4559 -m comment --comment "cali:1M7I9u1VkakBn-VX" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-fw-calif3ad7fd4559 -m comment --comment "cali:krnMBRV_hS8AmQSl" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-pri-kns.default -m comment --comment "cali:WMSw8BmYOknRHfsz" -m comment --comment "Profile kns.default ingress" -j MARK --set-xmark 0x10000/0x10000
-A cali-pri-kns.kube-system -m comment --comment "cali:J1TyxtHWd0qaBGK-" -m comment --comment "Profile kns.kube-system ingress" -j MARK --set-xmark 0x10000/0x10000
-A cali-pro-kns.default -m comment --comment "cali:Vr81boRqq4V77Sg8" -m comment --comment "Profile kns.default egress" -j MARK --set-xmark 0x10000/0x10000
-A cali-pro-kns.kube-system -m comment --comment "cali:tgOR2S8DVHZW3F1M" -m comment --comment "Profile kns.kube-system egress" -j MARK --set-xmark 0x10000/0x10000
-A cali-set-endpoint-mark -m comment --comment "cali:zkVgTdrfQeVpVrHN" -m comment --comment "Non-Cali endpoint mark" -j MARK --set-xmark 0x100000/0xfff00000
-A cali-tw-cali14897c537b0 -m comment --comment "cali:Gwy41HUAts3qjJUH" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-tw-cali14897c537b0 -m comment --comment "cali:h0M9H7UEXXyGk3l0" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-tw-cali3782683684d -m comment --comment "cali:8IOJrscLkQMGuPP-" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-tw-cali3782683684d -m comment --comment "cali:K0jvFOaD-C933wZb" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-tw-cali3c22a7abad0 -m comment --comment "cali:4znOgJ9VDKrKT7IB" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-tw-cali3c22a7abad0 -m comment --comment "cali:IE6ugvAUq-eIQ4QX" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-tw-calia56f50b9553 -m comment --comment "cali:jwIQr09LIW2Nhmn0" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-tw-calia56f50b9553 -m comment --comment "cali:vJgR1wWSAo8U8WU1" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-tw-calic484cf12723 -m comment --comment "cali:6B8Mm6xXq0BRsbgQ" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-tw-calic484cf12723 -m comment --comment "cali:7bkO8P2mhNbMnMes" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-tw-calif3ad7fd4559 -m comment --comment "cali:s0bUJlXNGgcnCAGm" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-tw-calif3ad7fd4559 -m comment --comment "cali:GkC0zhpXrOh32gLq" -m comment --comment "Return if profile accepted" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-POSTROUTING -m comment --comment "cali:NX-7roTexQ3fGRfU" -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-POSTROUTING -m comment --comment "cali:fcjhvOBNywbfCkS2" -m comment --comment "Host endpoint policy accepted packet." -m mark --mark 0x10000/0x10000 -j RETURN
-A cali-PREROUTING -m comment --comment "cali:KX7AGNd6rMcDUai6" -m mark --mark 0x10000/0x10000 -j ACCEPT
-A cali-PREROUTING -m comment --comment "cali:Cg96MgVuoPm7UMRo" -m comment --comment "Host endpoint policy accepted packet." -m mark --mark 0x10000/0x10000 -j ACCEPT
root@k8s-ctrl:/etc/cni/net.d#

# PREROUTING INPUT OUTPUT POSTROUTING FORWARD fw(firewall)

ovn 相对简单,和 k8s KUBE-FORWARD 的标签是一致的,从 ovn0 出来就打上标签,一直到 OVN-MASQUERADE 出去

calico 的规则更多,甚至还有基于标签,然后实现 fw 的匹配逻辑。

从标签设计的角度,两者是不会冲突的

ipsec 分类

这些规则通过 -m set --match-set <ipset_name> 来匹配 IP 或网络段,常用于 Calico、OVN、Kube-OVN 或 Kubernetes 内部服务的流量选择。作用通常是快速分类流量并应用策略(如 ACCEPT、DROP、MASQUERADE)。

主要 ipset 和作用

从你输出中看到的 ipset:

ipset 名称类型包含内容规则引用场景
cali40this-hosthash:ip本机 Pod、Host IPcali 链中针对本机 Pod/Host 流量的特殊处理
cali40all-ipam-poolshash:netPod CIDRmasq/SNAT 策略匹配 Pod 源网络
cali40masq-ipam-poolshash:netPod CIDRMASQUERADE / SNAT 针对 Pod CIDR 的流量
cali40all-hosts-nethash:netNode 内部网允许 Node 间 VXLAN/IPIP 封装流量
KUBE-CLUSTER-IP / KUBE-EXTERNAL-IP / KUBE-NODE-PORT 等hash:ipCluster IP / NodePort / External IPKube-proxy 管理服务访问和 SNAT

典型规则示例

  1. Pod-to-Pod / Pod-to-Node 流量(Calico)
-A FORWARD -m set --match-set ovn40subnets dst -j ACCEPT
-A FORWARD -m set --match-set ovn40subnets src -j ACCEPT
-A FORWARD -m set --match-set ovn40services dst -j ACCEPT
-A FORWARD -m set --match-set ovn40services src -j ACCEPT
  • 匹配 OVN 子网或服务 IP
  • 直接 ACCEPT,保证 Pod 间、Pod-to-Service 流量不被其他链阻塞
  1. MASQUERADE / SNAT 流量(Calico / Kube-OVN)
-A cali-nat-outgoing -m set --match-set cali40masq-ipam-pools src -m set ! --match-set cali40all-ipam-pools dst -j MASQUERADE --random-fully
  • 匹配 Pod 源网络属于 masq pool,目标不在 IPAM pool
  • 对 Pod 访问外部网络做 SNAT
  1. Kubernetes Service 流量(Kube-proxy)
-A KUBE-SERVICES ! -s 10.222.0.0/16 -m set --match-set KUBE-CLUSTER-IP dst,dst -j KUBE-MARK-MASQ
  • 匹配 ClusterIP 流量
  • 对非本机 Pod 做 SNAT(MASQUERADE)

2️⃣ 不基于 ipset 匹配的规则

这些规则没有引用 ipset,而是通过接口、协议、端口、mark 或 conntrack 来匹配流量。功能更灵活,可处理标记、封装或特殊主机流量。

常见匹配方式

  • -i <interface> / -o <interface>:匹配输入/输出接口
  • -p tcp/udp:匹配协议
  • --dport/--sport:匹配端口
  • -m mark --mark:匹配 packet mark(Calico / Cilium 用于 policy / proxy / L7 流量)
  • -m conntrack --ctstate:匹配连接状态

典型规则示例

  1. NodeLocal DNS Cache / 代理流量
-A PREROUTING -d 169.254.25.10/32 -p udp --dport 53 -j NOTRACK
-A OUTPUT -d 169.254.25.10/32 -p tcp --dport 53 -j NOTRACK
  • Node 本地 DNS 流量跳过 conntrack
  • 提高性能,避免被 SNAT / Mark 干扰
  1. VXLAN / IPIP 封装安全
-A cali-fw-cali14897c537b0 -p udp --dport 4789 -j DROP
-A cali-fw-cali14897c537b0 -p ipencap -j DROP
  • 丢弃 Pod origin 的 VXLAN/IPIP 封装包,防止内部封装被错误发送到主机
  1. Calico policy / host endpoint / proxy
-A cali-OUTPUT -m mark --mark 0x10000/0x10000 -j ACCEPT
-A cali-from-endpoint-mark -m mark 0x60d00000/0xfff00000 -g cali-fw-cali14897c537b0
  • 使用 packet mark 匹配,区分 Pod / Endpoint / Host 流量
  • 决定是否通过 firewall 链处理或直接 ACCEPT
  1. Cilium BPF / TPROXY
-A CILIUM_PRE_mangle -p tcp -m mark --mark 0x97800200 -j TPROXY --on-port 32919 --on-ip 127.0.0.1
  • 将特定标记流量重定向到本地 BPF 代理(DNS 或 L7)
  • 不依赖 ipset,而是依赖 mark 和协议端口

总结

类别匹配方式功能
基于 ipset-m set --match-setPod CIDR / Host / Service 流量分类、SNAT/MASQ、快速 ACCEPT/DROP
不基于 ipset接口 / 端口 / mark / conntrackHost/Pod 特殊流量、VXLAN/IPIP 安全、L7 proxy、NodeLocal DNS、策略 enforcement

💡 关键点

  • ipset 规则:高速匹配大批量 IP/网络,主要处理 Pod-to-Pod、Pod-to-Service 或 SNAT
  • 非 ipset 规则:处理更复杂的流量,如特定接口、端口、标记流量、安全控制或 BPF 重定向
  • 残留 Calico:会影响第二类规则中 Pod-to-Pod 或 NodePort 流量,如果仍匹配旧 mark 或 ipset

结论: 可以不清理

关键点

  • ipset 规则:高速匹配大批量 IP/网络,主要处理 Pod-to-Pod、Pod-to-Service 或 SNAT
  • 非 ipset 规则:处理更复杂的流量,如特定接口、端口、标记流量、安全控制或 BPF 重定向
  • 残留 Calico:会影响第二类规则中 Pod-to-Pod 或 NodePort 流量,如果仍匹配旧 mark 或 ipset

完全正确,你的测试结果和分析说明了几个关键点:


1️⃣ ipset 规则对现有 Pod 流量影响有限

  • 旧的 Calico ipset(如 cali40this-hostcali40masq-ipam-pools)主要匹配的是 旧 Calico Pod 的 IP 或 Node IP。
  • 现在 OVN Pod 使用的是 VPC 内东西向二层转发和接口(ovn0),所以 OVN 流量不会匹配到这些旧 ipset,SNAT / MASQ 规则不会干扰。

2️⃣ 非 ipset 的规则依赖接口或 mark

  • Calico 的 iptables 规则主要通过 -i cali+ / -o cali+ 接口cali-mark 来区分 Pod 流量。
  • OVN 的 Pod 流量走 ovn0 接口,并且 OVN/Cilium 打的 mark 不会落入 Calico 的规则链。
  • 因此东西向流量在 VPC 内 Pod-to-Pod 或 Pod-to-Service 时完全不触碰旧 Calico 规则。

3️⃣ Pod 重建和 VPC 内流量

  • 即使旧 Calico Pod 在 OVN 网络中重建,它的东西向流量都是 VPC 内通信,OVN 的路由和策略链起作用,不会走 Node SNAT,旧 Calico iptables 规则不会影响。
  • 只有南北向(Pod-to-External / NodePort / Internet)流量才会触及 Node 的 NAT / SNAT 规则。
  • 因为 OVN 的链和 Cilium 的 chaining-mode 正确处理了这些流量,所以 Pod 访问外网或 NodePort 也正常。

4️⃣ 总结

  • 残留 Calico iptables 规则对 OVN 新 Pod 的网络几乎没有影响,因为:

    1. 匹配的 IP 段不同 → ipset 不匹配
    2. 匹配的接口不同 → Calico veth vs OVN ovn0
    3. 标记不同 → Calico mark vs OVN/Cilium mark
  • 当前可以认为 Calico 规则“被隔离”在旧 Pod 流量上,不会干扰新 OVN 网络。


如果你愿意,我可以画一张 OVN Pod、Calico Pod、iptables链和接口的流量隔离示意图,直观展示为什么残留 Calico 规则不会影响 OVN 流量。

最后结论:

下面给你画一张 “IP 重叠但规则隔离的流量路径示意图(TXT 格式)”
重点展示:

  • OVN Pod 与旧 Calico Pod 的 IP 即使重叠
  • 流量仍然不会被 Calico 链干扰
  • 访问 Service(ClusterIP)和访问外网(SNAT)均正常
  • 因为匹配依赖 接口(veth vs ovn0)mark(cali-mark vs ovn-mark)

📌 IP 重叠但规则隔离示意图(TXT)

                        +-------------------------------+
                        |            Kubernetes         |
                        |         Service (ClusterIP)   |
                        +-------------------------------+
                                        ^
                                        |
                                        | (Pod -> SVC 流量)
                                        |
                   +-----------------------------------------------+
                   |               NODE iptables                   |
                   |                                               |
                   |  +-------------------+   +------------------+ |
                   |  |   KUBE-OVN 链     |   |   CILIUM 链      | |
                   |  +-------------------+   +------------------+ |
                   |            ^                    ^             |
                   |            |                    |             |
                   |   (OVN Pod 流量匹配)   (Cilium 代理/BPF)    |
                   |                                               |
                   |   +----------------------------------------+  |
                   |   |         ▼ 不会命中 ▼                   |  |
                   |   |        Calico 链区块 (残留)            |  |
                   |   |                                        |  |
                   |   |   cali-FORWARD                         |  |
                   |   |   cali-OUTPUT                          |  |
                   |   |   cali-POSTROUTING                     |  |
                   |   |   cali-nat-outgoing                    |  |
                   |   |                                        |  |
                   |   |   这些链依赖:                         |  |
                   |   |   - 接口 cali+                         |  |
                   |   |   - mark = 0x1xxxx / 0x4xxxx           |  |
                   |   |   OVN Pod 不满足 → 不会影響流量       |  |
                   |   +----------------------------------------+  |
                   +-----------------------------------------------+
                                        ^
                                        |
                                        |
+------------------+        +-----------+------------+
|   OVN Pod        |        |     Calico Pod (旧)   |
|  IP: 10.222.2.3  |        |  IP: 10.222.2.3       |
|  IF: eth0->ovn0  |        |  IF: eth0->caliXYZ    |
+------------------+        +------------------------+
      |                                 |
      |   Pod-to-Service 流量           |   Pod-to-Service 流量
      +---------------------------------+--------------------------+
                                      (IP 重叠无影响)

📌 南北向(Pod-to-External SNAT)隔离示意图

OVN Pod (IP: 10.222.2.3)
Interface: ovn0
Mark: ovn-mark
     |
     | Pod-to-External
     v
+----------------------------------------+
|              Node iptables             |
|                                        |
|  OVN SNAT/MASQ 链 (有效)               |
|  - OVN-POSTROUTING                     |
|  - KUBE-POSTROUTING                    |
|                                        |
|  Calico SNAT 链 (无效)                 |
|  - cali-nat-outgoing                   |
|                                        |
|  因为 Calico 链需要:                  |
|    IF = cali+                          |
|    mark = cali-mark                    |
|  OVN Pod 都不满足                      |
|  → 不会触发 calico SNAT                |
+----------------------------------------+
     |
     v
   外网 OK

📌 结论可视化

          IP 重叠?  YES
             ↓
   接口是否 Calico?  NO → ovn0
             ↓
   cali-mark 是否匹配? NO
             ↓
   Calico iptables 是否生效?  NO
             ↓
   OVN/Cilium 链接管网络 → 正常

结果:即使 IP 重叠,Calico 规则也不会干扰 OVN 或 Cilium

原因是:

  1. 接口隔离

    • OVN Pod 使用 ovn0
    • Calico Pod 使用 cali+
  2. mark 隔离

    • Calico 用 0x1xxxx / 0x4xxxx
    • OVN/Cilium 用完全不同的 mark
  3. 流量路径隔离

    • OVN Pod → 进 OVN 链
    • Calico Pod → 进 cali 链
    • 即使 IP 重叠也不影响
  4. Service(SVC)流量匹配的是 kube-proxy 或 Cilium → 与 Calico 无关

  5. SNAT 流量:Calico 链不会命中 OVN Pod