rancher高可用部署因为K8S的规定，主机名只支持包含 - 和 .(中横线和点)两种特殊符号，并且主机名不能出现重复

1 主机配置

在每台部署机器上执行主机配置

1）主机名配置

因为K8S的规定，主机名只支持包含 - 和 .(中横线和点)两种特殊符号，并且主机名不能出现重复。

2） Hosts配置

配置每台主机的hosts(/etc/hosts),添加host_ip $hostname到/etc/hosts文件中。

3）关闭selinux

sudo sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

4）关闭防火墙

systemctl stop firewalld.service && systemctl disable firewalld.service

5）配置主机时间、时区、系统语言

查看时区
date -R或者timedatectl
修改时区
ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
修改系统语言环境
sudo echo 'LANG="en_US.UTF-8"' >> /etc/profile;source /etc/profile
配置主机NTP时间同步

6） Kernel性能调优

echo "
net.bridge.bridge-nf-call-ip6tables=1
net.bridge.bridge-nf-call-iptables=1
net.ipv4.ip_forward=1
net.ipv4.conf.all.forwarding=1
net.ipv4.neigh.default.gc_thresh1=128
net.ipv4.neigh.default.gc_thresh2=6144
net.ipv4.neigh.default.gc_thresh3=8192
net.ipv4.neigh.default.gc_interval=60
net.ipv4.neigh.default.gc_stale_time=120
kernel.perf_event_paranoid=-1
kernel.softlockup_panic=0
kernel.watchdog_thresh=30
fs.file-max=2097152
fs.inotify.max_user_instances=8192
fs.inotify.max_queued_events=16384
fs.inotify.max_user_watches=524288
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.lo.disable_ipv6=1
kernel.yama.ptrace_scope=0
vm.swappiness=0
kernel.core_uses_pid=1
net.ipv4.conf.default.promote_secondaries=1
net.ipv4.conf.all.promote_secondaries=1
net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce=2
net.ipv4.conf.all.arp_announce=2
net.ipv4.tcp_syncookies=1
net.ipv4.tcp_fin_timeout=30
net.ipv4.tcp_synack_retries=2
net.ipv4.tcp_max_syn_backlog=8096
kernel.sysrq=1
net.ipv4.tcp_tw_recycle=0
" >> /etc/sysctl.conf

数值根据实际环境自行配置，最后执行sysctl -p保存配置。

cat >> /etc/security/limits.conf <<EOF
* soft nofile 65535
* hard nofile 65536
EOF

7）内核模块

以下模块需要在主机上加载

	module_list='br_netfilter ip6_udp_tunnel ip_set ip_set_hash_ip ip_set_hash_net iptable_filter iptable_nat iptable_mangle iptable_raw nf_conntrack_netlink nf_conntrack nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat nf_nat_ipv4 nf_nat_masquerade_ipv4 nfnetlink udp_tunnel veth vxlan x_tables xt_addrtype xt_conntrack xt_comment xt_mark xt_multiport xt_nat xt_recent xt_set xt_statistic xt_tcpudp'

	for module in $module_list;
	do
		  if ! lsmod | grep -q $module; then
				echo "module $module is not present"
				modprobe $module
		  fi
	done;

2 docker 部署

每台部署机器上执行docker安装

1）下载解压docker-18.09.6.tgz

tar -zxvf docker-18.09.6.tgz

2）复制到/usr/bin目录

cp docker/* /usr/bin/

3）配置dockerd启动项

cat > /etc/docker/daemon.json <<EOF
{
    "oom-score-adjust": -1000,
    "log-driver": "json-file",
    "log-opts": {
    "max-size": "100m",
    "max-file": "3"
    },
    "max-concurrent-downloads": 10,
    "max-concurrent-uploads": 10,
    "bip": "192.168.100.1/24",
"registry-mirrors": ["https://harbor01.io"],
"insecure-registries": [
  "harbor01.io"
],
    "storage-driver": "overlay2",
    "storage-opts": [
    "overlay2.override_kernel_check=true"
],
"graph": "/export/docker",
"hosts":["unix:///var/run/docker.sock", "tcp://0.0.0.0:8100"]
}
EOF

4）配置dockerd到systemd

cat > /usr/lib/systemd/system/docker.service <<EOF
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target docker.socket
[Service]
OOMScoreAdjust=-1000
Type=notify
EnvironmentFile=-/run/flannel/docker
WorkingDirectory=/usr/bin
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
ExecStartPost=/usr/sbin/iptables -P FORWARD ACCEPT
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

启动

systemctl daemon-reload
systemctl start docker
systemctl enable docker.service

3 harbor部署

在其中一台机器上部署镜像仓库

1）下载安装docker-compose

mv docker-compose /usr/bin/
chmod +x /usr/bin/docker-compose

下载解压harbor安装包

tar -zxvf harbor-ha-offline-installer-v1.7.0.tgz
cd harbor

配置harbor

vi harbor/harbor.cfg
hostname = harbor01.io
customize_crt = false

注：参数ui_url_protocol, auth_mode, db_host, db_password等信息依据需求选择修改

挂载共享存储glusterfs

mkdir -p /share_storage
mount -t glusterfs <glusterfs_node_ip>:<glusterfs_vol_name> /share_storage
mkdir -p /share_storage/harbor
cp -r -f common /share_storage/harbor/
cp -r -f harbor.cfg /share_storage/harbor/

启动

sh install.sh

检查 docker ps

安装完成后浏览器打开，默认端口80

未修改harbor.cfg默认用户名密码是 admin/Harbor12345

4 rancher 高可用部署

4.1 生成自签名ssl证书

下载create_self-signed-cert.sh，执行

./create_self-signed-cert.sh --ssl-domain=rancher.test.com

4.2 上传rancher镜像

在其中一台服务器执行rancher部署操作

1）下载镜像包rancher-images.tar.gz、rancher-images.txt和脚本rancher-load-images.sh

2）登录harbor页面，创建名为rancher的项目，访问级别公开

3）上传镜像到harbor

执行docker login命令

docker login harbor01.io -u admin

注：harbor镜像仓库默认admin密码为Harbor12345，如有修改，则输入修改后密码

上传镜像 sh rancher-load-images.sh --image-list rancher-images.txt --registry harbor01.io

4.3 安装nginx

编辑/etc/nginx.conf，替换IP_NODE_1、IP_NODE_2、IP_NODE_3为真实IP地址

nginx.conf配置如下：

worker_processes 4;
worker_rlimit_nofile 40000;

events {
    worker_connections 8192;
}

stream {
    upstream rancher_servers_http {
        least_conn;
        server <IP_NODE_1>:80 max_fails=3 fail_timeout=5s;
        server <IP_NODE_2>:80 max_fails=3 fail_timeout=5s;
        server <IP_NODE_3>:80 max_fails=3 fail_timeout=5s;
    }
    server {
        listen     80;
        proxy_pass rancher_servers_http;
    }

    upstream rancher_servers_https {
        least_conn;
        server <IP_NODE_1>:443 max_fails=3 fail_timeout=5s;
        server <IP_NODE_2>:443 max_fails=3 fail_timeout=5s;
        server <IP_NODE_3>:443 max_fails=3 fail_timeout=5s;
    }
    server {
        listen     443;
        proxy_pass rancher_servers_https;
    }
}

容器启动nginx服务

docker run -d --restart=unless-stopped \
  -p 80:80 -p 443:443 \
  -v /etc/nginx.conf:/etc/nginx/nginx.conf \
  harbor01.io/rancher/nginx:1.14

4.4 rke安装k8s

下载rke、helm和kubectl，并将其放入/usr/bin目录下

mv rke kubectl helm /usr/bin/
chmod 777 /usr/bin/rke /usr/bin/kubectl /usr/bin/helm

打通ssh通信每台机器添加docker用户，设置docker用户密码

groupadd docker
useradd docker -g docker
passwd docker

rke执行机器执行

su – docker
ssh-keygen
ssh-copy-id docker@<IP_NODE_1>
ssh-copy-id docker@<IP_NODE_2>
ssh-copy-id docker@<IP_NODE_3>

每台机器上添加/etc/hosts配置

<IP_nginx_node> rancher.gly.com

创建rke配置文件rancher-cluster.yml，替换IP_NODE_1、IP_NODE_2、IP_NODE_3为真实IP地址

nodes:
- address: <IP_NODE_1>
  user: docker
  role: [ "controlplane", "etcd", "worker" ]
  ssh_key_path: /home/docker/.ssh/id_rsa
nodes:
- address: <IP_NODE_2>
  user: docker
  role: [ "controlplane", "etcd", "worker" ]
  ssh_key_path: /home/docker/.ssh/id_rsa
nodes:
- address: <IP_NODE_3>
  user: docker
  role: [ "controlplane", "etcd", "worker" ]
  ssh_key_path: /home/docker/.ssh/id_rsa
private_registries:
- url: harbor01.io # private registry url
  user: admin
  password: "Harbor12345"
  is_default: true

创建kubernetest集群 rke up --config ./rancher-cluster.yml
执行成功后，会在当前目录下生成一个kube_config_rancher-cluster.yml文件，执行 mkdir /root/.kube

mv kube_config_rancher-cluster.yml /root/.kube/config

查看集群状态，执行 kubectl get nodes

4.5 安装rancher

证书tls.crt、tls.key和cacerts.pem为3.3.4.1步骤中生成

kubectl create namespace cattle-system
kubectl -n cattle-system create secret tls tls-rancher-ingress --cert=./tls.crt --key=./tls.key   
kubectl -n cattle-system create secret generic tls-ca --from-file=cacerts.pem
helm install rancher ./rancher \
    --namespace cattle-system \
    --set hostname=rancher.gly.com \
    --set ingress.tls.source=secret \
    --set privateCA=true \
    --set useBundledSystemChart=true \
    --set rancherImage=harbor01.io/rancher/rancher

5 kubernetes安装

1） rancher界面在全局添加集群

访问rancher.test.com，登录rancher界面

点击“添加集群”，选择“添加主机自建Kubernetes集群”，输入集群名称ocean，点击下一步

2） rancher界面在主机处添加服务器说明：

若为生产环境建议集群规格为3个master，2个worker；

若为演示poc可以使用1个master和1个worker的规格。

Master：主机规格中勾选Etcd、Control

Worker: 主机规格中勾选Worker

根据节点规格复制界面上的相应命令去相应主机执行命令