说明
本示例使用Heketi将GlusterFS集成到K8s中
Heketi介绍
- Heketi（github.com/heketi/heke…），是一个基于RESTful API的GlusterFS卷管理框架。
- Heketi可以方便地和云平台整合，提供RESTful API供Kubernetes调用，实现多GlusterFS集群的卷管理
- Heketi还有保证bricks和它对应的副本均匀分布在集群中的不同可用区的优点。
GlusterFS集群安装
下载Heketi
wget https://github.com/heketi/heketi/releases/download/v10.4.0/heketi-v10.4.0-release-10.linux.amd64.tar.gz
解压文件
root@node1:~# tar -xf heketi-v10.4.0-release-10.linux.amd64.tar.gz
root@node1:~# cd heketi/
root@node1:~/heketi# ls
heketi heketi-cli heketi.json
将解压的可执行程序复制到指定的环境变量路径中
cp heketi/{heketi,heketi-cli} /usr/local/bin
配置Heketi Server
为 Heketi 创建配置和数据路径
sudo mkdir -p /var/lib/heketi /etc/heketi /var/log/heketi
sudo cp heketi/heketi.json /etc/heketi
修改配置文件heketi.json
json不能有注释，创建配置文件时候需要将下面配置文件里面的中文去掉
{
"_port_comment": "Heketi Server Port Number",
"port": "18080", 修改为18080，防止与其它端口冲突
"_use_auth": "Enable JWT authorization. Please enable for deployment",
"use_auth": true, 开启用户认证
"_jwt": "Private keys for access",
"jwt": {
"_admin": "Admin has access to all APIs",
"admin": {
"key": "adminkey" 用户认证的key
},
"_user": "User only has access to /volumes endpoint",
"user": {
"key": "My Secret"
}
},
"_glusterfs_comment": "GlusterFS Configuration",
"glusterfs": {
"_executor_comment": [
"Execute plugin. Possible choices: mock, ssh",
"mock: This setting is used for testing and development.",
" It will not send commands to any node.",
"ssh: This setting will notify Heketi to ssh to the nodes.",
" It will need the values in sshexec to be configured.",
"kubernetes: Communicate with GlusterFS containers over",
" Kubernetes exec api."
],
"executor": "ssh", 访问glusterfs集群的方法
"_sshexec_comment": "SSH username and private key file information",
"sshexec": {
"keyfile": "/etc/heketi/heketi_key", 访问glusterfs集群使用的私钥，需要提前在k8s集群master节点生成并copy到glusterfs集群所有节点,需要从/root/.ssh/id_rsa复制到此处才可以使用。
"user": "root", 认证使用的用户
"port": "22", ssh连接使用的端口
"fstab": "/etc/fstab" 挂载的文件系统
},
"_kubeexec_comment": "Kubernetes configuration",
"kubeexec": {
"host" :"https://kubernetes.host:8443",
"cert" : "/path/to/crt.file",
"insecure": false,
"user": "kubernetes username",
"password": "password for kubernetes user",
"namespace": "OpenShift project or Kubernetes namespace",
"fstab": "Optional: Specify fstab file on node. Default is /etc/fstab"
},
"_db_comment": "Database file name",
"db": "/var/lib/heketi/heketi.db", 数据库位置
"_loglevel_comment": [
"Set log level. Choices are:",
" none, critical, error, warning, info, debug",
"Default is warning"
],
"loglevel" : "warning" 修改日志级别
}
}
说明: heketi有三种executor，分别为mock、ssh、kubernetes，建议在测试环境使用mock，生产环境使用ssh，当glusterfs以容器的方式部署在kubernetes上时，才使用kubernetes。我们这里将glusterfs和heketi独立部署，使用ssh的方式。
创建heketi用户
因为下面的Service服务文件是使用的heketi用户启动的Heketi服务
sudo groupadd --system heketi
sudo useradd -s /sbin/nologin --system -g heketi heketi
使用ed25519加密方法生成ssh密钥对
因为rsa加密方法在最新版openssh被禁用了
root@node1:~# ssh-keygen -t ed25519
复制master节点的公钥到GlusterFS的各个服务器上
ssh-copy-id root@192.168.202.201
ssh-copy-id root@192.168.202.202
ssh-copy-id root@192.168.202.203
master节点上复制私钥到/etc/heketi目录
root@node1:~# cp .ssh/id_ed25519 /etc/heketi/heketi_key
为 Heketi 创建服务文件
tee /etc/systemd/system/heketi.service <<EOF
[Unit]
Description=Heketi Server
[Service]
Type=simple
WorkingDirectory=/var/lib/heketi
EnvironmentFile=-/etc/heketi/heketi.env
User=heketi
ExecStart=/usr/local/bin/heketi --config=/etc/heketi/heketi.json
Restart=on-failure
StandardOutput=syslog
StandardError=syslog
[Install]
WantedBy=multi-user.target
EOF
修改文件权限
/etc/heketi及/var/lib/heketi目录所有者是root, 但是安装提供的service文件的user又是heketi. 导致不修改权限就是启动不起来,因此需要修改权限再启动服务。
chown heketi:heketi /etc/heketi/ -R
chown heketi:heketi /var/lib/heketi -R
创建数据库文件
root@node1:~# touch /var/lib/heketi/heketi.db
启动Heketi Server
root@node1:~# systemctl daemon-reload
root@node1:~# systemctl start heketi
root@node1:~# systemctl enable heketi
查看Heketi服务启动情况
root@node1:~# systemctl status heketi
● heketi.service - Heketi Server
Loaded: loaded (/etc/systemd/system/heketi.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2023-11-26 10:41:42 CST; 1s ago
Main PID: 60574 (heketi)
Tasks: 8 (limit: 4516)
Memory: 6.0M
CPU: 20ms
CGroup: /system.slice/heketi.service
└─60574 /usr/local/bin/heketi --config=/etc/heketi/heketi.json
Nov 26 10:41:42 node1 systemd[1]: Started Heketi Server.
Nov 26 10:41:42 node1 heketi[60574]: Heketi v10.4.0-release-10 (using go: go1.15.14)
Nov 26 10:41:42 node1 heketi[60574]: 2023/11/26 10:41:42 no SSH_KNOWN_HOSTS specified, skipping ssh host verification
Nov 26 10:41:42 node1 heketi[60574]: Listening on port 18080
Heketi使用验证
验证是否可以创建集群
heketi-cli --user admin --secret adminkey --server http://192.168.202.151:18080 --json cluster create
{"id":"0268aaa43dd0bd4051bcae3b93e7962c","nodes":[],"volumes":[],"block":true,"file":true,"blockvolumes":[]}
参数说明
--json表示输出时候以json字符串格式输出
删除已创建的集群
root@node1:~# heketi-cli --user admin --secret adminkey --server http://192.168.202.151:18080 --json cluster delete 747ec7ce501970bafd1e594bf86989f7
Cluster 747ec7ce501970bafd1e594bf86989f7 deleted
添加gluster节点
heketi-cli --user admin --secret adminkey --server http://192.168.202.151:18080 --json node add --cluster "0268aaa43dd0bd4051bcae3b93e7962c" --management-host-name 192.168.202.202 --storage-host-name 192.168.202.202 --zone 1
heketi-cli --user admin --secret adminkey --server http://192.168.202.151:18080 --json node add --cluster "0268aaa43dd0bd4051bcae3b93e7962c" --management-host-name 192.168.202.202 --storage-host-name 192.168.202.202 --zone 1
heketi-cli --user admin --secret adminkey --server http://192.168.202.151:18080 --json node add --cluster "0268aaa43dd0bd4051bcae3b93e7962c" --management-host-name 192.168.202.203 --storage-host-name 192.168.202.203 --zone 1
查看集群中node列表
root@node1:~# heketi-cli --user admin --secret adminkey --server http://192.168.202.151:18080 node list
Id:68488df5848b6bfbe212dde1d9e1073c Cluster:0268aaa43dd0bd4051bcae3b93e7962c
Id:8759be7fdd397707d860ec9f718056e1 Cluster:0268aaa43dd0bd4051bcae3b93e7962c
Id:f1d9d8bc34de87f1351d2efaa3196a36 Cluster:0268aaa43dd0bd4051bcae3b93e7962c
添加设备
添加GlusterFS集群节点中的设备到Heketi集群
heketi-cli --user admin --secret adminkey --server http://192.168.202.151:18080 device add --name "/dev/nvme0n3" --node 8759be7fdd397707d860ec9f718056e1
Device added successfully
heketi-cli --user admin --secret adminkey --server http://192.168.202.151:18080 device add --name "/dev/nvme0n3" --node 68488df5848b6bfbe212dde1d9e1073c
Device added successfully
heketi-cli --user admin --secret adminkey --server http://192.168.202.151:18080 device add --name "/dev/nvme0n3" --node f1d9d8bc34de87f1351d2efaa3196a36
Device added successfully
验证磁盘添加情况
root@node1:~# heketi-cli --user admin --secret adminkey --server http://192.168.202.151:18080 topology info
Cluster Id: 0268aaa43dd0bd4051bcae3b93e7962c
File: true
Block: true
Volumes:
Nodes:
Node Id: 68488df5848b6bfbe212dde1d9e1073c
State: online
Cluster Id: 0268aaa43dd0bd4051bcae3b93e7962c
Zone: 1
Management Hostnames: 192.168.202.202
Storage Hostnames: 192.168.202.202
Devices:
Id:c96a2115ec2fe84184cd3da7b586f88c State:online Size (GiB):19 Used (GiB):0 Free (GiB):19
Known Paths: /dev/disk/by-id/nvme-eui.5d61a68862036f32000c29619df89930 /dev/disk/by-diskseq/3 /dev/disk/by-path/pci-0000:0b:00.0-nvme-3 /dev/disk/by-id/nvme-VMware_Virtual_NVMe_Disk_VMware_NVME_0000 /dev/nvme0n3
Bricks:
Node Id: 8759be7fdd397707d860ec9f718056e1
State: online
Cluster Id: 0268aaa43dd0bd4051bcae3b93e7962c
Zone: 1
Management Hostnames: 192.168.202.201
Storage Hostnames: 192.168.202.201
Devices:
Id:ba1b06d03d16c07c21b817eb55e93ea8 State:online Size (GiB):19 Used (GiB):0 Free (GiB):19
Known Paths: /dev/disk/by-path/pci-0000:0b:00.0-nvme-3 /dev/disk/by-id/nvme-eui.8b9abff5ba7db3e7000c2965767ca009 /dev/disk/by-id/nvme-VMware_Virtual_NVMe_Disk_VMware_NVME_0000 /dev/disk/by-diskseq/3 /dev/nvme0n3
Bricks:
Node Id: f1d9d8bc34de87f1351d2efaa3196a36
State: online
Cluster Id: 0268aaa43dd0bd4051bcae3b93e7962c
Zone: 1
Management Hostnames: 192.168.202.203
Storage Hostnames: 192.168.202.203
Devices:
Id:e639ba3204d6ad137920e8eaf20c6536 State:online Size (GiB):19 Used (GiB):0 Free (GiB):19
Known Paths: /dev/disk/by-id/nvme-eui.304e0491fdf8142a000c29658e2f2350 /dev/disk/by-id/nvme-VMware_Virtual_NVMe_Disk_VMware_NVME_0000 /dev/disk/by-diskseq/3 /dev/disk/by-path/pci-0000:0b:00.0-nvme-3 /dev/nvme0n3
Bricks:
测试通过Heketi在GlusterFS集群中添加volume
创建一个复制卷，卷的大小为5G
root@node1:~# heketi-cli --user admin --secret adminkey --server http://192.168.202.151:18080 volume create --size=5 --replica=2
Name: vol_96caf17e6de1fdfc094083d0fe4dc0fb
Size: 5
Volume Id: 96caf17e6de1fdfc094083d0fe4dc0fb
Cluster Id: 0268aaa43dd0bd4051bcae3b93e7962c
Mount: 192.168.202.202:vol_96caf17e6de1fdfc094083d0fe4dc0fb
Mount Options: backup-volfile-servers=192.168.202.201,192.168.202.203
Block: false
Free Size: 0
Reserved Size: 0
Block Hosting Restriction: (none)
Block Volumes: []
Durability Type: replicate
Distribute Count: 1
Replica Count: 2
验证卷是否创建
root@node1:~# heketi-cli --user admin --secret adminkey --server http://192.168.202.151:18080 volume list
Id:96caf17e6de1fdfc094083d0fe4dc0fb Cluster:0268aaa43dd0bd4051bcae3b93e7962c Name:vol_96caf17e6de1fdfc094083d0fe4dc0fb
在GlusterFS集群中查看创建的卷
gluster volume list
vol_96caf17e6de1fdfc094083d0fe4dc0fb
