Kubernetes-mysql-nfs动态存储集群部署
背景
目前在高可用分布式集群的架构下要求mysql数据库做读写分离集群部署，来减轻数据库压力，本章节将介绍基于kubernetes，nfs做mysql数据库数据存储官方建议此种方式构建mysql一主多从集群模式
一、环境准备，安装kubernetes集群
当前节点base、master、slave1、slave2
kubernetes安装不在此赘述，请参考参考资料进行安装
kubernetes安装nfs不在此赘述，请参考参考资料进行安装
参考资料
1-Kubernetes基于Centos7构建基础环境（一）
2-Kubernetes基于Centos7构建基础环境（二）
3-Kubernetes基于Centos7构建基础环境（三）
4-Kubernetes-基于Centos7安装面板及监控（四）
1-kubernetes-nfs动态存储部署)
| 集群名称 | 集群域名 | 说明 |
|---|---|---|
| base | base.xincan.cn | 部署harbor、nfs等服务 |
| master | master.xincan.cn | kubernetes主节点，做污点容忍，排除业务资源，nfs客户端等 |
| slave1 | slave1.xincan.cn | kubernetes从节点，nfs客户端等 |
| slave2 | slave2.xincan.cn | kubernetes从节点，nfs客户端等 |
二、总体流程：
- 找一台虚拟机部署harbor、nfs服务，我这里不在赘述，请参照前面的文档；
- 在nfs服务端创建mysql数据存储挂载本地位置；
- 在kubernetes主节点上某个目录下创建mysql-multiple文件夹，也可以自定义，主要保证内部kubernetes资源定义文件一样即可；
- 进入此文件夹之后，复制一下文档中列出的kubernetes资源并保存到此目录下，所有资源如下列表;
[root@master mysql-multiple]# ll
mysql-multiple
├── 1-mysql-namespace.yaml
├── 2-mysql-nfs-rbac.yaml
├── 3-mysql-nfs-provisioner.yaml
├── 4-mysql-nfs-storageclas.yaml
├── 5-mysql-secret.yaml
├── 6-mysql-configmap.yaml
├── 7-mysql-service.yaml
└── 8-mysql-statefulset.yaml
- 分别在kubernetes集群节点上，测试nfs连接情况；
- 资源创建；
- 效果展示；
- 测试主从同步，外部链接；
三、创建mysql数据挂在本地位置
- 前提在base机器上安装好了NFS服务端
- 创建mysql数据库数据将要挂载到本地的位置，并和NFS服务端相连接
- 提示/nfs，/data，/mysql-multiple三个文件夹均要设置777权限
[root@base mysql-multiple]# pwd
/nfs/data/mysql-multiple
[root@base mysql-multiple]# ll
总用量 0
[root@base mysql-multiple]#
四、NFS连接测试
- base虚拟机本地添加路径映射，后面的IP根据个人机器决定一般是网关地址，括号内是操作权限，同步异步等等设置
[root@base mysql-multiple]# vim /etc/exports
/nfs/data/mysql-multiple 172.16.124.1/24(rw,sync,no_root_squash,no_all_squash)
[root@base mysql-multiple]#
- base虚拟机本地地址暴露测试，可以看到刚刚我们创建的文件夹已经暴露（/nfs/data/mysql-multiple）
[root@base mysql-multiple]# exportfs -arv
exporting 172.16.124.1/24:/nfs/data/nginx
exporting 172.16.124.1/24:/nfs/data/mysql-multiple
exporting 172.16.124.1/24:/nfs/data/mysql-single
exporting 172.16.124.1/24:/nfs/data/prometheus
[root@base mysql-multiple]#
- kubernetes集群连接测试，发现/nfs/data/mysql-multiple 172.16.124.1/24已经存在
# 主节点master连接测试
[root@master /]# showmount -e base.xincan.cn
Export list for master.xincan.cn:
/nfs/data/nginx 172.16.124.1/24
/nfs/data/mysql-multiple 172.16.124.1/24
/nfs/data/mysql-single 172.16.124.1/24
/nfs/data/prometheus 172.16.124.1/24
[root@master /]#
# 子节点slave1连接测试
[root@slave1 /]# showmount -e base.xincan.cn
Export list for slave1.xincan.cn:
/nfs/data/nginx 172.16.124.1/24
/nfs/data/mysql-multiple 172.16.124.1/24
/nfs/data/mysql-single 172.16.124.1/24
/nfs/data/prometheus 172.16.124.1/24
[root@master /]#
# 子节点slave2连接测试
[root@slave2 /]# showmount -e base.xincan.cn
Export list for slave2.xincan.cn:
/nfs/data/nginx 172.16.124.1/24
/nfs/data/mysql-multiple 172.16.124.1/24
/nfs/data/mysql-single 172.16.124.1/24
/nfs/data/prometheus 172.16.124.1/24
[root@master /]#
五、资源创建
- 创建mysql的命名空间Namespace，名称为：mysql
- 将所有的资源挂载到此命名空间下
[root@master mysql-multiple]# vim 1-mysql-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: mysql
labels:
app: mysql
- 创建NFS权限配置ServiceAccount、ClusterRole、ClusterRoleBinding、Role、RoleBinding，名称为：mysql-nfs-client-provisioner、mysql-nfs-client-provisioner-runner、run-mysql-nfs-client-provisioner、leader-locking-mysql-nfs-client-provisioner、leader-locking-mysql-nfs-client-provisioner
[root@master mysql-multiple]# vim 2-mysql-nfs-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: mysql-nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: mysql
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mysql-nfs-client-provisioner-runner
namespace: mysql
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: run-mysql-nfs-client-provisioner
namespace: mysql
subjects:
- kind: ServiceAccount
name: mysql-nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: mysql
roleRef:
kind: ClusterRole
name: mysql-nfs-client-provisioner-runner
apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-mysql-nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: mysql
rules:
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-mysql-nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: mysql
subjects:
- kind: ServiceAccount
name: mysql-nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: mysql
roleRef:
kind: Role
name: leader-locking-mysql-nfs-client-provisioner
apiGroup: rbac.authorization.k8s.io
- 创建NFS资源Deployment,名称为：mysql-nfs-client-provisioner
- mountPath: /persistentvolumes 该文件夹是nfs-client-provisioner镜像运行之后容器内部固定的文件夹它会mount到/nfs/data/nginx nfs服务器nginx文件夹下
- value: 172.16.124.130 该地址是对应NFS服务器地址
- path: /nfs/data/mysql-multiple path地址路径则是NFS服务器自己创建挂载点的文件路径，也就是后续数据库存储的位置
[root@master mysql-multiple]# vim 3-mysql-nfs-provisioner.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: mysql-nfs-client-provisioner
labels:
app: mysql-nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: mysql
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: mysql-nfs-client-provisioner
template:
metadata:
labels:
app: mysql-nfs-client-provisioner
spec:
serviceAccountName: mysql-nfs-client-provisioner
containers:
- name: mysql-nfs-client-provisioner
image: base.xincan.cn/library/nfs-client-provisioner:v1.5.2
volumeMounts:
- name: nfs-client-root
# 该文件夹是nfs-client-provisioner运行之后容器内部固定的文件夹它会mount到/nfs/data/nginx nfs服务器nginx文件夹下
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: fuseim.pri/ifs
- name: NFS_SERVER
value: 172.16.124.130
- name: NFS_PATH
value: /nfs/data/mysql-multiple
volumes:
- name: nfs-client-root
nfs:
server: 172.16.124.130
path: /nfs/data/mysql-multiple
-
创建mysql主节点StorageClass，名称为：mysql-nfs-storage
- 后续pv、pvc通过Deployment动态创建
[root@master mysql-multiple]# vim 4-mysql-nfs-storageclas.yaml
apiVersion: v1
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: mysql-nfs-storage
namespace: mysql
provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
archiveOnDelete: "false"
- 创建mysql从节点Secret，名称为：mysql-secret
-
通过echo对数据库密码123456进行
base64加密，默认用户是root，后续用到； -
echo -n "123456" | base64
-
[root@master mysql-multiple]# vim 5-mysql-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: mysql-secret
namespace: mysql
labels:
app: mysql
type: Opaque
data:
password: MTIzNDU2
- 创建mysql的资源配置ConfigMap，名称为：mysql
[root@master mysql-multiple]# vim 6-mysql-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: mysql-config
namespace: mysql
labels:
app: mysql
data:
master.cnf: |
# Mmaster配置
[mysqld]
log-bin=mysqllog
skip-name-resolve
#设置时区和字符集
default-time-zone='+8:00'
character-set-client-handshake=FALSE
character-set-server=utf8mb4
collation-server=utf8mb4_unicode_ci
init_connect='SET NAMES utf8mb4 COLLATE utf8mb4_unicode_ci'
slave.cnf: |
# Slave配置
[mysqld]
super-read-only
skip-name-resolve
log-bin=mysql-bin
replicate-ignore-db=mysql
#设置时区和字符集
default-time-zone='+8:00'
character-set-client-handshake=FALSE
character-set-server=utf8mb4
collation-server=utf8mb4_unicode_ci
init_connect='SET NAMES utf8mb4 COLLATE utf8mb4_unicode_ci'
-
创建mysql服务对外暴露访问资源Service，名称为：mysql-master，mysql-read
-
mysql-master：设置mysql主库对外提供服务名称，用于数据库的增删改
-
mysql-read：设置msyql从库对外提供服务名称，用于数据的读取数据
-
[root@master mysql-multiple]# vim 7-mysql-service.yaml
apiVersion: v1
kind: Service
metadata:
name: mysql-cluster
namespace: mysql
labels:
app: mysql
spec:
ports:
- name: mysql
protocol: TCP
port: 3306
targetPort: 3306
type: ClusterIP
selector:
app: mysql
---
apiVersion: v1
kind: Service
metadata:
name: mysql-master
namespace: mysql
labels:
app: mysql
spec:
selector:
statefulset.kubernetes.io/pod-name: mysql-0
app: mysql
type: NodePort
ports:
- name: mysql
protocol: TCP
port: 3306
targetPort: 3306
nodePort: 31110
---
apiVersion: v1
kind: Service
metadata:
name: mysql-read-1
namespace: mysql
labels:
app: mysql
spec:
selector:
statefulset.kubernetes.io/pod-name: mysql-1
app: mysql
type: NodePort
ports:
- name: mysql
protocol: TCP
port: 3306
targetPort: 3306
nodePort: 31120
---
apiVersion: v1
kind: Service
metadata:
name: mysql-read-2
namespace: mysql
labels:
app: mysql
spec:
selector:
statefulset.kubernetes.io/pod-name: mysql-2
app: mysql
type: NodePort
ports:
- name: mysql
protocol: TCP
port: 3306
targetPort: 3306
nodePort: 31130
- 创建mysql无头服务StatefulSet，名称为：根据metadata.name+“-”+实例序号，序号从0开始,如我的最后结果是：mysql-0，mysql-1
- 通过
xtrabackup做数据库处理、如：增量备份、权量备份等等，在此不多讲 - replicas: 3 最终会生成3分mysql实例
- 通过
[root@master mysql-multiple]# vim 8-mysql-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mysql
namespace: mysql
labels:
app: mysql
spec:
selector:
matchLabels:
app: mysql
serviceName: mysql-master
replicas: 3
template:
metadata:
labels:
app: mysql
spec:
initContainers:
- name: init-mysql
image: base.xincan.cn/library/mysql:v5.7.28
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: password
optional: true
command:
- bash
- "-c"
- |
set -ex
# 从 Pod 的序号，生成 server-id
[[ $(hostname) =~ -([0-9]+)$ ]] || exit 1
ordinal=${BASH_REMATCH[1]}
echo [mysqld] > /mnt/conf.d/server-id.cnf
# 由于 server-id 不能为 0，因此给 ID 加 100 来避开它
echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf
# 如果 Pod 的序号为 0，说明它是 Master 节点，从 ConfigMap 里把 Master 的配置文件拷贝到 /mnt/conf.d 目录下
# 否则，拷贝 ConfigMap 里的 Slave 的配置文件
if [[ ${ordinal} -eq 0 ]]; then
cp /mnt/config-map/master.cnf /mnt/conf.d
else
cp /mnt/config-map/slave.cnf /mnt/conf.d
fi
volumeMounts:
- name: conf
mountPath: /mnt/conf.d
- name: config-map
mountPath: /mnt/config-map
- name: clone-mysql
image: base.xincan.cn/library/xtrabackup:v1.0
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: password
optional: true
command:
- bash
- "-c"
- |
set -ex
# 拷贝操作只需要在第一次启动时进行，所以数据已经存在则跳过
[[ -d /var/lib/mysql/mysql ]] && exit 0
# Master 节点（序号为 0）不需要这个操作
[[ $(hostname) =~ -([0-9]+)$ ]] || exit 1
ordinal=${BASH_REMATCH[1]}
[[ $ordinal == 0 ]] && exit 0
# 使用 ncat 指令，远程地从前一个节点拷贝数据到本地
ncat --recv-only mysql-$(($ordinal-1)).mysql-master 3307 | xbstream -x -C /var/lib/mysql
# 执行 --prepare，这样拷贝来的数据就可以用作恢复了
xtrabackup --prepare --target-dir=/var/lib/mysql
volumeMounts:
- name: data
mountPath: /var/lib/mysql
subPath: mysql
- name: conf
mountPath: /etc/mysql/conf.d
containers:
- name: mysql
image: base.xincan.cn/library/mysql:v5.7.28
env:
# - name: MYSQL_ALLOW_EMPTY_PASSWORD
# value: "1"
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: password
optional: true
ports:
- name: mysql
containerPort: 3306
volumeMounts:
- name: data
mountPath: /var/lib/mysql
subPath: mysql
- name: conf
mountPath: /etc/mysql/conf.d
resources:
requests:
cpu: 500m
memory: 1Gi
livenessProbe:
exec:
command: ["mysqladmin", "ping", "-uroot", "-p${MYSQL_ROOT_PASSWORD}"]
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
readinessProbe:
exec:
command: ["mysqladmin", "ping", "-uroot", "-p${MYSQL_ROOT_PASSWORD}"]
initialDelaySeconds: 5
periodSeconds: 2
timeoutSeconds: 1
- name: xtrabackup
image: base.xincan.cn/library/xtrabackup:v1.0
ports:
- name: xtrabackup
containerPort: 3307
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: password
optional: true
command:
- bash
- "-c"
- |
set -ex
cd /var/lib/mysql
# 从备份信息文件里读取 MASTER_LOG_FILE 和 MASTER_LOG_POS 这 2 个字段的值，用来拼装集群初始化 SQL
if [[ -f xtrabackup_slave_info ]]; then
# 如果 xtrabackup_slave_info 文件存在，说明这个备份数据来自于另一个 Slave 节点
# 这种情况下，XtraBackup 工具在备份的时候，就已经在这个文件里自动生成了 "CHANGE MASTER TO" SQL 语句
# 所以，只需要把这个文件重命名为 change_master_to.sql.in，后面直接使用即可
mv xtrabackup_slave_info change_master_to.sql.in
# 所以，也就用不着 xtrabackup_binlog_info 了
rm -f xtrabackup_binlog_info
elif [[ -f xtrabackup_binlog_info ]]; then
# 如果只是存在 xtrabackup_binlog_info 文件，说明备份来自于 Master 节点，就需要解析这个备份信息文件，读取所需的两个字段的值
[[ $(cat xtrabackup_binlog_info) =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1
rm xtrabackup_binlog_info
# 把两个字段的值拼装成 SQL，写入 change_master_to.sql.in 文件
echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\
MASTER_LOG_POS=${BASH_REMATCH[2]}" > change_master_to.sql.in
fi
# 如果存在 change_master_to.sql.in，就意味着需要做集群初始化工作
if [[ -f change_master_to.sql.in ]]; then
# 但一定要先等 MySQL 容器启动之后才能进行下一步连接 MySQL 的操作
echo "Waiting for mysqld to be ready（accepting connections）"
until mysql -h 127.0.0.1 -uroot -p${MYSQL_ROOT_PASSWORD} -e "SELECT 1"; do sleep 1; done
echo "Initializing replication from clone position"
# 将文件 change_master_to.sql.in 改个名字
# 防止这个 Container 重启的时候，因为又找到了 change_master_to.sql.in，从而重复执行一遍初始化流程
mv change_master_to.sql.in change_master_to.sql.orig
# 使用 change_master_to.sql.orig 的内容，也就是前面拼装的 SQL，组成一个完整的初始化和启动 Slave 的 SQL 语句
mysql -h 127.0.0.1 -uroot -p${MYSQL_ROOT_PASSWORD} << EOF
$(< change_master_to.sql.orig),
MASTER_HOST='mysql-0.mysql-master',
MASTER_USER='root',
MASTER_PASSWORD='${MYSQL_ROOT_PASSWORD}',
MASTER_CONNECT_RETRY=10;
START SLAVE;
EOF
fi
# 使用 ncat 监听 3307 端口。
# 它的作用是，在收到传输请求的时候，直接执行 xtrabackup --backup 命令，备份 MySQL 的数据并发送给请求者
exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \
"xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root --password=${MYSQL_ROOT_PASSWORD}"
volumeMounts:
- name: data
mountPath: /var/lib/mysql
subPath: mysql
- name: conf
mountPath: /etc/mysql/conf.d
volumes:
- name: conf
emptyDir: {}
- name: config-map
configMap:
name: mysql-config
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes:
- "ReadWriteOnce"
storageClassName: mysql-local-storage
resources:
requests:
storage: 2Gi
- 创建资源
- 进入mysql-single执行命令：kubectl apply -f ../mysql-multiple/
- 监听Pod创建命令：kubectl -n mysql get pod -w -o wide
- 撤销进程命令：control+z
- 删除所有资源命令：kubectl delete -f ../mysql-multiple/
[root@master mysql-multiple]# kubectl apply -f ../mysql-multiple/
namespace/mysql created
serviceaccount/mysql-nfs-client-provisioner created
clusterrole.rbac.authorization.k8s.io/mysql-nfs-client-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/run-mysql-nfs-client-provisioner created
role.rbac.authorization.k8s.io/leader-locking-mysql-nfs-client-provisioner created
rolebinding.rbac.authorization.k8s.io/leader-locking-mysql-nfs-client-provisioner created
deployment.apps/mysql-nfs-client-provisioner created
storageclass.storage.k8s.io/mysql-nfs-storage created
secret/mysql-secret created
configmap/mysql created
service/mysql-master created
service/mysql-read created
statefulset.apps/mysql created
[root@master mysql-multiple]#
[root@master mysql-multiple]# kubectl -n mysql get pod -w -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
mysql-0 1/2 Running 0 16s 192.168.34.82 slave2.xincan.cn <none> <none>
mysql-nfs-client-provisioner-6fb45b9598-pmlbg 1/1 Running 0 16s 192.168.226.59 slave1.xincan.cn <none> <none>
mysql-0 2/2 Running 0 16s 192.168.34.82 slave2.xincan.cn <none> <none>
mysql-1 0/2 Pending 0 0s <none> <none> <none> <none>
mysql-1 0/2 Pending 0 0s <none> <none> <none> <none>
mysql-1 0/2 Pending 0 3s <none> slave1.xincan.cn <none> <none>
mysql-1 0/2 Init:0/2 0 3s <none> slave1.xincan.cn <none> <none>
mysql-1 0/2 Init:0/2 0 4s <none> slave1.xincan.cn <none> <none>
mysql-1 0/2 Init:1/2 0 4s 192.168.226.63 slave1.xincan.cn <none> <none>
mysql-1 0/2 Init:Error 0 5s 192.168.226.63 slave1.xincan.cn <none> <none>
mysql-1 0/2 Init:Error 1 6s 192.168.226.63 slave1.xincan.cn <none> <none>
mysql-1 0/2 Init:CrashLoopBackOff 1 7s 192.168.226.63 slave1.xincan.cn <none> <none>
mysql-1 0/2 Init:1/2 2 21s 192.168.226.63 slave1.xincan.cn <none> <none>
mysql-1 0/2 PodInitializing 0 31s 192.168.226.63 slave1.xincan.cn <none> <none>
mysql-1 1/2 Running 0 32s 192.168.226.63 slave1.xincan.cn <none> <none>
mysql-1 1/2 Error 0 33s 192.168.226.63 slave1.xincan.cn <none> <none>
mysql-1 1/2 Running 1 34s 192.168.226.63 slave1.xincan.cn <none> <none>
mysql-1 2/2 Running 1 39s 192.168.226.63 slave1.xincan.cn <none> <none>
mysql-2 0/2 Pending 0 0s <none> <none> <none> <none>
mysql-2 0/2 Pending 0 0s <none> <none> <none> <none>
mysql-2 0/2 Pending 0 3s <none> slave2.xincan.cn <none> <none>
mysql-2 0/2 Init:0/2 0 3s <none> slave2.xincan.cn <none> <none>
mysql-2 0/2 Init:0/2 0 4s <none> slave2.xincan.cn <none> <none>
mysql-2 0/2 Init:1/2 0 5s 192.168.34.90 slave2.xincan.cn <none> <none>
mysql-2 0/2 Init:1/2 0 6s 192.168.34.90 slave2.xincan.cn <none> <none>
mysql-2 0/2 PodInitializing 0 16s 192.168.34.90 slave2.xincan.cn <none> <none>
mysql-2 1/2 Running 0 17s 192.168.34.90 slave2.xincan.cn <none> <none>
mysql-2 1/2 Error 0 18s 192.168.34.90 slave2.xincan.cn <none> <none>
mysql-2 1/2 Running 1 19s 192.168.34.90 slave2.xincan.cn <none> <none>
mysql-2 2/2 Running 1 24s 192.168.34.90 slave2.xincan.cn <none> <none>
[root@master mysql-multiple]# kubectl delete -f ../mysql-multiple/
namespace "mysql" deleted
serviceaccount "mysql-nfs-client-provisioner" deleted
clusterrole.rbac.authorization.k8s.io "mysql-nfs-client-provisioner-runner" deleted
clusterrolebinding.rbac.authorization.k8s.io "run-mysql-nfs-client-provisioner" deleted
role.rbac.authorization.k8s.io "leader-locking-mysql-nfs-client-provisioner" deleted
rolebinding.rbac.authorization.k8s.io "leader-locking-mysql-nfs-client-provisioner" deleted
deployment.apps "mysql-nfs-client-provisioner" deleted
storageclass.storage.k8s.io "mysql-nfs-storage" deleted
secret "mysql-secret" deleted
configmap "mysql" deleted
service "mysql-master" deleted
service "mysql-read" deleted
statefulset.apps "mysql" deleted
[root@master mysql-multiple]#
- 所有服务启动后需要在mysql-master、mysql-read中指定主库匹配那个Service
- 根据StatefulSet无头服务中制定规则得知，Master 节点（序号为 0）也就是其中一个Pod名称为mysql-0为mysql主节点
- 将mysql-0这个Pod与Service mysql-master关联起来，代码如下
- 修改Service mysql-master资源，在selecter下增加statefulset.kubernetes.io/pod-name=mysql-0并保存
- 注意“=”号要换成“:”号
[root@master mysql-multiple]# kubectl -n mysql get svc,pod
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/mysql-master NodePort 10.1.67.39 <none> 3306:32060/TCP 70s
service/mysql-read NodePort 10.1.204.111 <none> 3306:32070/TCP 70s
NAME READY STATUS RESTARTS AGE
pod/mysql-0 2/2 Running 0 70s
pod/mysql-1 2/2 Running 1 56s
pod/mysql-2 2/2 Running 1 33s
pod/mysql-nfs-client-provisioner-6fb45b9598-dsplc 1/1 Running 0 70s
[root@master mysql-multiple]#
[root@master mysql-multiple]# kubectl -n mysql get pods --show-labels | grep mysql-0 | awk '{print $6}' | awk -F, '{print $3}'
statefulset.kubernetes.io/pod-name=mysql-0
[root@master mysql-multiple]#
[root@master mysql-multiple]# kubectl -n mysql edit svc mysql-master
kind: Service
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"mysql"},"name":"mysql-master","namespace":"mysql"},"spec":{"ports":[{"name":"mysql","nodePort":32060,"port":3306,"protocol":"TCP","targetPort":3306}],"selector":{"app":"mysql"},"type":"NodePort"}}
creationTimestamp: "2021-04-27T08:36:08Z"
labels:
app: mysql
name: mysql-master
namespace: mysql
resourceVersion: "945910"
selfLink: /api/v1/namespaces/mysql/services/mysql-master
uid: 5752f224-298e-454b-8f10-fee9942174c2
spec:
clusterIP: 10.1.67.39
clusterIPs:
- 10.1.67.39
externalTrafficPolicy: Cluster
ports:
- name: mysql
nodePort: 32060
port: 3306
protocol: TCP
targetPort: 3306
selector:
# 在此处增加
statefulset.kubernetes.io/pod-name: mysql-0
app: mysql
sessionAffinity: None
type: NodePort
status:
loadBalancer: {}
六、效果展示
- 通过
kubectl -n mysql get secret,pv,pvc,sc，查询mysql命名空间下对应的secret加密资源，pv（persistentvolume）数据挂载资源，pvc资源，sc资源 - 通过
kubectl -n mysql get all，查询mysql命名空间下对应的pod资源，service对外服务暴露资源 - 验证NFS服务端下mysql集群存储文件
[root@master mysql-multiple]# kubectl -n mysql get secret,pv,pvc,sc
NAME TYPE DATA AGE
secret/default-token-9pvqv kubernetes.io/service-account-token 3 31m
secret/mysql-nfs-client-provisioner-token-5mjhg kubernetes.io/service-account-token 3 31m
secret/mysql-secret Opaque 1 31m
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pvc-1ece6e9b-81c3-4fb5-a89a-3a6f224f1383 2Gi RWO Delete Bound mysql/data-mysql-1 mysql-nfs-storage 31m
persistentvolume/pvc-29d97ecc-1e05-4c3d-90f8-059417d063cc 2Gi RWO Delete Bound mysql/data-mysql-0 mysql-nfs-storage 31m
persistentvolume/pvc-802976c3-d19b-4c47-aa8f-1b512d44d79c 2Gi RWO Delete Bound mysql/data-mysql-2 mysql-nfs-storage 30m
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/data-mysql-0 Bound pvc-29d97ecc-1e05-4c3d-90f8-059417d063cc 2Gi RWO mysql-nfs-storage 31m
persistentvolumeclaim/data-mysql-1 Bound pvc-1ece6e9b-81c3-4fb5-a89a-3a6f224f1383 2Gi RWO mysql-nfs-storage 31m
persistentvolumeclaim/data-mysql-2 Bound pvc-802976c3-d19b-4c47-aa8f-1b512d44d79c 2Gi RWO mysql-nfs-storage 30m
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
storageclass.storage.k8s.io/mysql-nfs-storage fuseim.pri/ifs Delete Immediate false 31m
[root@master mysql-multiple]#
[root@master mysql-multiple]# kubectl -n mysql get all
NAME READY STATUS RESTARTS AGE
pod/mysql-0 2/2 Running 0 34m
pod/mysql-1 2/2 Running 1 34m
pod/mysql-2 2/2 Running 1 33m
pod/mysql-nfs-client-provisioner-6fb45b9598-dsplc 1/1 Running 0 34m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/mysql-master NodePort 10.1.67.39 <none> 3306:32060/TCP 34m
service/mysql-read NodePort 10.1.204.111 <none> 3306:32070/TCP 34m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/mysql-nfs-client-provisioner 1/1 1 1 34m
NAME DESIRED CURRENT READY AGE
replicaset.apps/mysql-nfs-client-provisioner-6fb45b9598 1 1 1 34m
NAME READY AGE
statefulset.apps/mysql 3/3 34m
[root@master mysql-multiple]#
# 验证NFS服务端下mysql集群存储文件
[root@base mysql-multiple]# pwd
/nfs/data/mysql-multiple
[root@base mysql-multiple]# ll
总用量 0
drwxrwxrwx 3 root root 19 4月 27 16:36 mysql-data-mysql-0-pvc-29d97ecc-1e05-4c3d-90f8-059417d063cc
drwxrwxrwx 3 root root 19 4月 27 16:36 mysql-data-mysql-1-pvc-1ece6e9b-81c3-4fb5-a89a-3a6f224f1383
drwxrwxrwx 3 root root 19 4月 27 16:36 mysql-data-mysql-2-pvc-802976c3-d19b-4c47-aa8f-1b512d44d79c
[root@base mysql-multiple]#
七、测试主从同步，外部navicat连接
- 命令方式测试主从同步：
- 进入主库：mysql-1这个pod，此pod为主节点，所以具有写的权限，在此执行
create database hatech，提示创建成功，并进行查询验证操作。
[root@master mysql-multiple]# kubectl -n mysql exec -it mysql-0 -- mysql -uroot -p123456
Defaulting container name to mysql.
Use 'kubectl describe pod/mysql-0 -n mysql' to see all of the containers in this pod.
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 513
Server version: 5.7.28-log MySQL Community Server (GPL)
Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+------------------------+
| Database |
+------------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
| xtrabackup_backupfiles |
+------------------------+
5 rows in set (0.00 sec)
mysql> create database hatech;
Query OK, 1 row affected (0.00 sec)
mysql> show databases;
+------------------------+
| Database |
+------------------------+
| information_schema |
| hatech |
| mysql |
| performance_schema |
| sys |
| xtrabackup_backupfiles |
+------------------------+
6 rows in set (0.00 sec)
mysql>
- 进入从库mysql-1这个pod，通过
show databases;进行验证，可以看到数据库列表中已经出现hatech这个数据库，接着删除此库，发现删除不了,提示--super-read-only，至此从库只读模式验证完毕。
[root@master mysql-multiple]# kubectl -n mysql exec -it mysql-1 -- mysql -uroot -p123456
Defaulting container name to mysql.
Use 'kubectl describe pod/mysql-1 -n mysql' to see all of the containers in this pod.
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 850
Server version: 5.7.28-log MySQL Community Server (GPL)
Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+------------------------+
| Database |
+------------------------+
| information_schema |
| hatech |
| mysql |
| performance_schema |
| sys |
| xtrabackup_backupfiles |
+------------------------+
6 rows in set (0.00 sec)
mysql> drop database hatech;
ERROR 1290 (HY000): The MySQL server is running with the --super-read-only option so it cannot execute this statement
mysql>
- Navicat连接测试
-
查询kuernetes集群下
service中，mysql涉及到的对外服务，可以看出有两个NodePort对外暴露的服务，友情提示：msql-read，此服务读取时包括所有的主库和从库，应为kubernetes中service本身具备服务注册功能
[root@master mysql-multiple]# kubectl -n mysql get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
mysql-master NodePort 10.1.118.85 <none> 3306:32060/TCP 48m
mysql-read NodePort 10.1.78.105 <none> 3306:32070/TCP 48m
[root@master mysql-localVolume]#
- Navicat体现
- 创建32060主库端口数据库连接成功
- 创建32070主库端口数据库连接成功
- 通过界面化操作，在主节点上创建hatech数据库后，刷新从库，则同步完成
- 通过界面化操作，在从节点上删除hatech数据库后，提示删除失败，从库提供只读功能，至此验证完毕
八：结束语
生产环境官方强烈建议利用nfs做mysql的动态存储，至此介绍完毕
