安装ipv6网络的k8s集群

ZZ不想说话
2,086 阅读6分钟

1.安装前置条件

1.1、关闭firewalld

 systemctl stop firewalld; systemctl disable firewalld

1.2、关闭SElinux

 setenforce 0; sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

1.3、关闭Swap

 swapoff -a; sed -i "s/\/dev\/mapper\/centos-swap/\#\/dev\/mapper\/centos-swap/g" /etc/fstab 
如果不想关闭swap，可以跳过检查
vim /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--fail-swap-on=false"
在初始化时加入参数--ignore-preflight-errors=Swap

1.4、使用阿里云yum源：

 wget -O /etc/yum.repos.d/CentOS7-Aliyun.repo http://mirrors.aliyun.com/repo/Centos-7.repo

1.5、修改内核参数

#增加配置
[root@master ~]# vim /etc/sysctl.conf
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.default.disable_ipv6 = 0
net.ipv6.conf.lo.disable_ipv6 = 0
net.ipv6.conf.all.forwarding=1
#使之生效
[root@master ~]# sysctl -p
[root@master ~]# vim /etc/sysconfig/network
#添加
NETWORKING_IPV6=yes
[root@master ~]# vim /etc/sysconfig/network-scripts/ifcfg-eth0
#添加
IPV6INIT=yes
IPV6_AUTOCONF=yes

1.6、配置本地解析

vim /etc/hosts
本机ipv6地址 master

2、安装docker

3.kubernetes集群安装

3.1 使用阿里云的kubernetes源

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

3.2 安装kubeadm kubelet kubectl

yum install -y kubelet kubeadm kubectl

3.3 启动kubelet

systemctl enable kubelet;systemctl start kubelet 　

3.3下载所需镜像

# 查看需要的镜像版本
[root@node1 ~]# kubeadm config images list
k8s.gcr.io/kube-apiserver:v1.20.4
k8s.gcr.io/kube-controller-manager:v1.20.4
k8s.gcr.io/kube-scheduler:v1.20.4
k8s.gcr.io/kube-proxy:v1.20.4
k8s.gcr.io/pause:3.2
k8s.gcr.io/etcd:3.4.13-0
k8s.gcr.io/coredns:1.7.0
# 下载镜像（因为k8s.gcr.io访问限制，现在使用国内镜像下载然后tag改name）
# 使用如下脚本下载

set -o errexit
set -o nounset
set -o pipefail
##这里定义版本
KUBE_VERSION=v1.20.4
KUBE_PAUSE_VERSION=3.2
ETCD_VERSION=3.4.13-0
DNS_VERSION=1.7.0

GCR_URL=k8s.gcr.io
##这里就是写你要使用的仓库
DOCKERHUB_URL=gotok8s
##这里是镜像列表
images=(
kube-proxy:${KUBE_VERSION}
kube-scheduler:${KUBE_VERSION}
kube-controller-manager:${KUBE_VERSION}
kube-apiserver:${KUBE_VERSION}
pause:${KUBE_PAUSE_VERSION}
etcd:${ETCD_VERSION}
coredns:${DNS_VERSION}
)
##这里是拉取和改名的循环语句
for imageName in ${images[@]} ; do
  docker pull $DOCKERHUB_URL/$imageName
  docker tag $DOCKERHUB_URL/$imageName $GCR_URL/$imageName
  docker rmi $DOCKERHUB_URL/$imageName
done

3.4 修改初始化配置文件

# 查看配置信息
[root@node1 ~]# kubeadm config print init-defaults
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 1.2.3.4
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: node1
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.20.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
scheduler: {}

## 修改为ipv6模式的配置文件
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 2003:ac18::30a:1
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: node1
  kubeletExtraArgs:
    node-ip: 2003:ac18::30a:1
---
apiServer:
  extraArgs:
    advertise-address: 2003:ac18::30a:1
    bind-address: '::'
    etcd-servers: https://[2003:ac18::30a:1]:2379
    service-cluster-ip-range: fd00:10:96::/112
apiVersion: kubeadm.k8s.io/v1beta2
controllerManager:
  extraArgs:
    allocate-node-cidrs: 'true'
    bind-address: '::'
    cluster-cidr: fd00:10:16::/64
    node-cidr-mask-size: '64'
    service-cluster-ip-range: fd00:10:96::/112
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
    extraArgs:
      name: master
      advertise-client-urls: https://[2003:ac18::30a:1]:2379
      initial-advertise-peer-urls: https://[2003:ac18::30a:1]:2380
      initial-cluster: master=https://[2003:ac18::30a:1]:2380
      listen-client-urls: https://[2003:ac18::30a:1]:2379
      listen-peer-urls: https://[2003:ac18::30a:1]:2380
kind: ClusterConfiguration
networking:
  dnsDomain: cluster.local
  serviceSubnet: fd00:10:96::/112
scheduler:
  extraArgs:
    bind-address: '::'
kubernetesVersion: v1.20.0
---
apiVersion: kubelet.config.k8s.io/v1beta1
failSwapOn: false
nodeIp: 2003:ac18::30a:1
address: '::'
clusterDNS:
- fd00:10:96::a
healthzBindAddress: '::1'
healthzPort: 10248
kind: KubeletConfiguration
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clusterCIDR: fd00:10:16::/64
mode: ipvs
ipvs:
  minSyncPeriod: 5s
  syncPeriod: 5s
  scheduler: "wrr"

3.5 创建集群

kubeadm init --config=init-config-ipv6.yaml 等待集群创建完成查看pod、sevice是否创建成功分配ipv6地址 kubectl get pod -A -owide

NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE     IP                 NODE    
kube-system   coredns-74ff55c5b-54vlp                0/1     Running   0          2m41s   fd00:10:16::6      node1   
kube-system   coredns-74ff55c5b-csw9f                0/1     Running   0          2m41s   fd00:10:16::5      node1  
kube-system   etcd-node1                             1/1     Running   0          2m41s   2003:ac18::30a:1   node1   
kube-system   kube-apiserver-node1                   1/1     Running   0          2m41s   2003:ac18::30a:1   node1   
kube-system   kube-controller-manager-node1          1/1     Running   0          2m41s   2003:ac18::30a:1   node1       
kube-system   kube-proxy-9fbb6                       1/1     Running   0          2m41s   2003:ac18::30a:1   node1   
kube-system   kube-scheduler-node1                   1/1     Running   0          2m41s   2003:ac18::30a:1   node1

3.6 安装cni插件

我使用的是kube-ovn 更改安装脚本支持ipv6模式vim install.sh IPv6=${IPv6:-true}

然后执行安装文件sh install.sh 等待安装完成查看状态kubectl get pod -A -owide

NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE     IP                 NODE    NOMINATED NODE   READINESS GATES
default       hello-deployment-6d48f47cf9-rrt8w      1/1     Running   0          3h23m   fd00:10:16::8      node1   <none>           <none>
default       hello-deployment-6d48f47cf9-v4rsz      1/1     Running   0          3h23m   fd00:10:16::9      node1   <none>           <none>
kube-system   coredns-74ff55c5b-54vlp                1/1     Running   0          3h25m   fd00:10:16::6      node1   <none>           <none>
kube-system   coredns-74ff55c5b-csw9f                1/1     Running   0          3h25m   fd00:10:16::5      node1   <none>           <none>
kube-system   etcd-node1                             1/1     Running   0          3h29m   2003:ac18::30a:1   node1   <none>           <none>
kube-system   kube-apiserver-node1                   1/1     Running   0          3h29m   2003:ac18::30a:1   node1   <none>           <none>
kube-system   kube-controller-manager-node1          1/1     Running   0          3h29m   2003:ac18::30a:1   node1   <none>           <none>
kube-system   kube-ovn-cni-jbb7f                     1/1     Running   0          3h26m   2003:ac18::30a:1   node1   <none>           <none>
kube-system   kube-ovn-controller-6554f7b67d-jfqrv   1/1     Running   0          3h26m   2003:ac18::30a:1   node1   <none>           <none>
kube-system   kube-ovn-pinger-wfx8j                  1/1     Running   0          3h25m   fd00:10:16::7      node1   <none>           <none>
kube-system   kube-proxy-9fbb6                       1/1     Running   0          3h29m   2003:ac18::30a:1   node1   <none>           <none>
kube-system   kube-scheduler-node1                   1/1     Running   0          3h29m   2003:ac18::30a:1   node1   <none>           <none>
kube-system   ovn-central-55f5b55587-jqqbt           2/2     Running   0          3h26m   2003:ac18::30a:1   node1   <none>           <none>
kube-system   ovs-ovn-rqnpj                          1/1     Running   0          3h26m   2003:ac18::30a:1   node1   <none>           <none>

4. 安装过程中可能遇到的问题

!!#ff0000 1.kubelet无法正常启动,安装失败最后输出如下信息!! 解决： 1.systemctl status kubelet查看kubelet是否正常运行如不正常运行 journallctl -xeu kubelet查看日志 kubelet 和docker的cgroupdriver不同，修改为一致的cgroupfs或者systemd即可 2. kubelet正常启动，然后查看etcd docker是否正常启动docker ps -a|grep etcd 然后查看容器报错日志 etcd的配置信息需要修改 name: master(hostname) initial-cluster: master=https://[2003:ac18::30a:1]:2380

!!#ff0000 2. kube-ovn一直安装不成功!! 所有service无法访问 查看kube-proxy日志

[root@node1 ~]# kubectl logs -n kube-system kube-proxy-7fxwz
I0225 01:42:06.301557       1 node.go:172] Successfully retrieved node IP: 2004:ac18::30a:1
I0225 01:42:06.301616       1 server_others.go:139] kube-proxy node IP is an IPv6 address (2004:ac18::30a:1), assume IPv6 operation
W0225 01:42:06.316024       1 server_others.go:578] Unknown proxy mode "", assuming iptables proxy
I0225 01:42:06.316110       1 server_others.go:185] Using iptables Proxier.
W0225 01:42:06.316125       1 server_others.go:455] detect-local-mode set to ClusterCIDR, but no cluster CIDR defined
I0225 01:42:06.316129       1 server_others.go:466] detect-local-mode: ClusterCIDR , defaulting to no-op detect-local
I0225 01:42:06.317685       1 server.go:650] Version: v1.20.4
I0225 01:42:06.318164       1 conntrack.go:52] Setting nf_conntrack_max to 262144
I0225 01:42:06.318451       1 config.go:315] Starting service config controller
I0225 01:42:06.318469       1 shared_informer.go:240] Waiting for caches to sync for service config
I0225 01:42:06.318506       1 config.go:224] Starting endpoint slice config controller
I0225 01:42:06.318539       1 shared_informer.go:240] Waiting for caches to sync for endpoint slice config
I0225 01:42:06.418631       1 shared_informer.go:247] Caches are synced for endpoint slice config
I0225 01:42:06.418647       1 shared_informer.go:247] Caches are synced for service config

可以看到有如下信息但并不是error detect-local-mode set to ClusterCIDR, but no cluster CIDR defined 原因是kube-proxy没有配置cluster CIDR信息 查看kube-proxy的configmap发现clusterCIDR="" kubeadm输出的默认配置没有kube-proxy相关的配置

解决：需要在初始化配置中添加如下配置

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clusterCIDR: fd00:10:16::/64
mode: ipvs
ipvs:
  minSyncPeriod: 5s
  syncPeriod: 5s
  scheduler: "wrr"

执行 kubeadm init --config , 有以下类型可以配置

  • InitConfiguration
  • ClusterConfiguration
  • KubeProxyConfiguration
  • KubeletConfiguration
avatar
文章
阅读
粉丝
相关推荐
minikube搭建K8S集群
648阅读
 · 
1点赞
滴滴崩溃，损失几个亿的k8s 方案
49k阅读
 · 
272点赞
K8s使用外部Ceph集群
765阅读
 · 
0点赞
1. 使用kind搭建k8s集群
2.0k阅读
 · 
0点赞
再次白嫖 ，30分钟搭建自己的 DevOps + K8S 集群
7.6k阅读
 · 
59点赞
K8s 多集群实践思考和探索
2.6k阅读
 · 
19点赞
K8S集群安全机制
1.2k阅读
 · 
1点赞
第5节 K8S介绍和集群环境搭建
3.2k阅读
 · 
24点赞
基于k8s容器化部署Nacos集群
1.7k阅读
 · 
4点赞
k8s部署MySQL集群
3.3k阅读
 · 
3点赞

友情链接：