ceph的配置
在ceph集群中执行如下命令：
bash-4.4$ ceph -s
cluster:
id: 6779d759-5604-425d-bc98-c95a5f54e31d
health: HEALTH_OK
services:
mon: 3 daemons, quorum a,b,c (age 4h)
mgr: a(active, since 22h), standbys: b
mds: 1/1 daemons up, 1 hot standby
osd: 12 osds: 12 up (since 22h), 12 in (since 22h)
data:
volumes: 1/1 healthy
pools: 4 pools, 97 pgs
objects: 46 objects, 4.0 MiB
usage: 362 MiB used, 72 TiB / 72 TiB avail
pgs: 97 active+clean
io:
client: 1.2 KiB/s rd, 2 op/s rd, 0 op/s wr
bash-4.4$ ceph mon stat
e3: 3 mons at {a=[v2:192.168.5.163:3300/0,v1:192.168.5.163:6789/0],b=[v2:192.168.5.162:3300/0,v1:192.168.5.162:6789/0],c=[v2:192.168.5.165:3300/0,v1:192.168.5.165:6789/0]}, election epoch 44, leader 0 a, quorum 0,1,2 a,b,c
bash-4.4$
| 标题 | |
|---|---|
部署ceph-csi版本
涉及三方的版本：ceph（Octopus），kubernetes (v1.24.0)，ceph-sci版本
现阶段对应ceph csi与k8s版本对应如下：
| Ceph CSI Version | Container Orchestrator Name | Version Tested |
|---|---|---|
| v3.7.2 | Kubernetes | v1.22,v1.23,v1.24 |
| v3.7.1 | Kubernetes | v1.22,v1.23,v1.24 |
| v3.7.0 | Kubernetes | v1.22,v1.23,v1.24 |
| v3.6.1 | Kubernetes | v1.21,v1.22,v1.23 |
| v3.6.0 | Kubernetes | v1.21,v1.22,v1.23 |
目前使用的kubernetes版本是1.24，所以ceph-sci版本就使用最新版本v3.7.2
目前使用的Ceph的版本是rook-1.10.5(对应的ceph是qincy-17版本)版，ceph与Ceph CSI版本的对应关系，因为太多了，所以参照：ceph-sci插件官网
总上，部署ceph-csi v3.7.2 版本就可以了
rook搭建方法，见我的另一篇博客
需要特别指出的是，按照官网描述，对外提供暴露接口的rook需要将网络设置成共享主机网络，如下图：
本例中，需要先修改monitor的网络为主机网络host：
然后，需要在创建完成operator和cluster之后，在启动mon和osd之前，修改configmap:rook-config-override，如果mon已经启动，需要依次delete掉mon
[root@node1 examples]# kubectl -n rook-ceph edit cm/rook-config-override
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
config: |
[global]
public network = 192.168.5.0/24
cluster network = 192.168.5.0/24
public addr = ""
cluster addr = ""
kind: ConfigMap
metadata:
creationTimestamp: "2022-12-01T07:04:14Z"
name: rook-config-override
namespace: rook-ceph
ownerReferences:
- apiVersion: ceph.rook.io/v1
blockOwnerDeletion: true
controller: true
kind: CephCluster
name: rook-ceph
uid: abcd525d-8ccb-4cd8-a967-98e9c4c74d3c
resourceVersion: "1359217"
uid: 4dd0074b-c9ad-4321-9331-10af9ff8054b
然后，mon和osd的网络为主机网络：
[root@node1 examples]# kubectl -n rook-ceph get pods -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
csi-cephfsplugin-6t74r 2/2 Running 0 23h 192.168.5.164 node4 <none> <none>
csi-cephfsplugin-b4n7w 2/2 Running 0 23h 192.168.5.165 node5 <none> <none>
csi-cephfsplugin-provisioner-75875b5887-c8mfj 5/5 Running 0 23h 10.244.33.159 node5 <none> <none>
csi-cephfsplugin-provisioner-75875b5887-rtk9c 5/5 Running 0 23h 10.244.139.25 node6 <none> <none>
csi-cephfsplugin-sxftk 2/2 Running 0 23h 192.168.5.166 node6 <none> <none>
csi-rbdplugin-6kmzl 2/2 Running 0 23h 192.168.5.164 node4 <none> <none>
csi-rbdplugin-6sl5k 2/2 Running 0 23h 192.168.5.165 node5 <none> <none>
csi-rbdplugin-hlclm 2/2 Running 0 23h 192.168.5.166 node6 <none> <none>
csi-rbdplugin-provisioner-56d69f5d8-8t4q8 5/5 Running 0 23h 10.244.139.26 node6 <none> <none>
csi-rbdplugin-provisioner-56d69f5d8-cfksv 5/5 Running 0 23h 10.244.3.94 node4 <none> <none>
rook-ceph-crashcollector-node1-5bbd979996-hxmwx 1/1 Running 0 23h 192.168.5.161 node1 <none> <none>
rook-ceph-crashcollector-node2-859655cbbf-7qmpc 1/1 Running 0 23h 192.168.5.162 node2 <none> <none>
rook-ceph-crashcollector-node3-5c667cbf58-622w4 1/1 Running 0 23h 192.168.5.163 node3 <none> <none>
rook-ceph-crashcollector-node4-66857584b7-z67dv 1/1 Running 0 3h10m 192.168.5.164 node4 <none> <none>
rook-ceph-crashcollector-node5-fb97fc69d-qkchf 1/1 Running 0 3h8m 192.168.5.165 node5 <none> <none>
rook-ceph-crashcollector-node6-7549898d55-t842x 1/1 Running 0 3h8m 192.168.5.166 node6 <none> <none>
rook-ceph-mds-cephfs-a-6b7c594bf9-nj9t7 2/2 Running 0 3h8m 192.168.5.165 node5 <none> <none>
rook-ceph-mds-cephfs-b-6f58ff6fb-fzcqv 2/2 Running 0 3h8m 192.168.5.166 node6 <none> <none>
rook-ceph-mgr-a-5cf47df66-qwmfx 3/3 Running 0 23h 192.168.5.163 node3 <none> <none>
rook-ceph-mgr-b-86c4c55c47-pxvkh 3/3 Running 0 23h 192.168.5.164 node4 <none> <none>
rook-ceph-mon-a-5b5b97c6d6-9wz24 2/2 Running 0 23h 192.168.5.163 node3 <none> <none>
rook-ceph-mon-b-5c86569bfb-sd2zg 2/2 Running 0 23h 192.168.5.162 node2 <none> <none>
rook-ceph-mon-c-66954f5666-clqsl 2/2 Running 0 23h 192.168.5.165 node5 <none> <none>
rook-ceph-operator-64fb475fcb-nvmgq 1/1 Running 0 23h 10.244.3.92 node4 <none> <none>
rook-ceph-osd-0-54d6b497bc-wlqqr 2/2 Running 0 23h 192.168.5.163 node3 <none> <none>
rook-ceph-osd-1-654988fbd7-kr7s7 2/2 Running 0 23h 192.168.5.161 node1 <none> <none>
rook-ceph-osd-10-bcc5d45d4-2ltnb 2/2 Running 0 23h 192.168.5.165 node5 <none> <none>
rook-ceph-osd-11-68b66b7fff-d2nmn 2/2 Running 0 23h 192.168.5.166 node6 <none> <none>
rook-ceph-osd-2-7f9b869456-cz72h 2/2 Running 0 23h 192.168.5.162 node2 <none> <none>
rook-ceph-osd-3-f8b487f7b-2tnxq 2/2 Running 0 23h 192.168.5.164 node4 <none> <none>
rook-ceph-osd-4-58d5587d96-rts64 2/2 Running 0 23h 192.168.5.165 node5 <none> <none>
rook-ceph-osd-5-64f769c76b-4gwxp 2/2 Running 0 23h 192.168.5.166 node6 <none> <none>
rook-ceph-osd-6-d8fbc667c-cnzv2 2/2 Running 0 23h 192.168.5.163 node3 <none> <none>
rook-ceph-osd-7-7644d54bb6-zjw2x 2/2 Running 0 23h 192.168.5.161 node1 <none> <none>
rook-ceph-osd-8-75d4ffcc95-cvrx8 2/2 Running 0 23h 192.168.5.162 node2 <none> <none>
rook-ceph-osd-9-58969b5b8b-69kzp 2/2 Running 0 23h 192.168.5.164 node4 <none> <none>
rook-ceph-osd-prepare-node1-n92sv 0/1 Completed 0 4h46m 192.168.5.161 node1 <none> <none>
rook-ceph-osd-prepare-node2-cbd2p 0/1 Completed 0 4h46m 192.168.5.162 node2 <none> <none>
rook-ceph-osd-prepare-node3-pzlsx 0/1 Completed 0 4h46m 192.168.5.163 node3 <none> <none>
rook-ceph-osd-prepare-node4-fncgz 0/1 Completed 0 4h46m 192.168.5.164 node4 <none> <none>
rook-ceph-osd-prepare-node5-gqgr5 0/1 Completed 0 4h46m 192.168.5.165 node5 <none> <none>
rook-ceph-osd-prepare-node6-gq7jr 0/1 Completed 0 4h45m 192.168.5.166 node6 <none> <none>
rook-ceph-tools-5679b7d8f-4nv4l 1/1 Running 0 22h 10.244.33.161 node5 <none> <none>
rook-discover-254zb 1/1 Running 0 23h 10.244.3.93 node4 <none> <none>
rook-discover-6wxbs 1/1 Running 0 23h 10.244.139.23 node6 <none> <none>
rook-discover-vg9kz 1/1 Running 0 23h 10.244.33.157 node5 <none> <none>
下载ceph-csi
下载ceph-csi 3.6.1的源码：下载地址 解压缩后，进入到deploy目录下的rbd目录下的内容
[root@server14 deploy]# pwd
/root/ceph-csi-3.7.2/deploy
[root@server14 deploy]# tree -L 3
.
├── cephcsi
│ └── image
│ └── Dockerfile
├── cephfs
│ └── kubernetes
│ ├── csi-cephfsplugin-provisioner.yaml
│ ├── csi-cephfsplugin.yaml
│ ├── csi-config-map.yaml
│ ├── csidriver.yaml
│ ├── csi-nodeplugin-psp.yaml
│ ├── csi-nodeplugin-rbac.yaml
│ ├── csi-provisioner-psp.yaml
│ └── csi-provisioner-rbac.yaml
├── Makefile
├── rbd
│ └── kubernetes
│ ├── csi-config-map.yaml
│ ├── csidriver.yaml
│ ├── csi-nodeplugin-psp.yaml
│ ├── csi-nodeplugin-rbac.yaml
│ ├── csi-provisioner-psp.yaml
│ ├── csi-provisioner-rbac.yaml
│ ├── csi-rbdplugin-provisioner.yaml
│ └── csi-rbdplugin.yaml
└── scc.yaml
部署rbd
过程参照文档：github.com/ceph/ceph-c…
- 将ceph-csi/deploy/rbd/kubernetes/下的所有yaml文件拷贝到本地
- 创建csi-config-map.yaml
clusterID（集群ID）： ceph -s即可获得\- 如果ceph部署在host主机上面，则
monitors 在/var/lib/ceph/6779d759-5604-425d-bc98-c95a5f54e31d/mon.node1/config中 - 如果是rook，在toolbox中，通过/etc/ceph/ceph.conf 可以看到：
bash-4.4$ ceph mon dump epoch 3 fsid 6779d759-5604-425d-bc98-c95a5f54e31d last_changed 2022-12-01T07:04:57.529729+0000 created 2022-12-01T07:04:29.459509+0000 min_mon_release 17 (quincy) election_strategy: 1 0: [v2:192.168.5.163:3300/0,v1:192.168.5.163:6789/0] mon.a 1: [v2:192.168.5.162:3300/0,v1:192.168.5.162:6789/0] mon.b 2: [v2:192.168.5.165:3300/0,v1:192.168.5.165:6789/0] mon.c dumped monmap epoch 3
- 如果ceph部署在host主机上面，则
则编辑csi-config-map.yaml内容如下：
[root@server14 caoyong]# vim csi-config-map.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
name: "ceph-csi-config"
data:
config.json: |
[
{
"clusterID": "6779d759-5604-425d-bc98-c95a5f54e31d",
"monitors": [
"192.168.5.163:6789",
"192.168.5.162:6789",
"192.168.5.165:6789"
]
}
]
文档参照官网
该文件主要是给csi-ceph-plugin访问ceph提供访问地址和集群ID（多集群情况？）
3. 创建csi-kms-config-map.yaml,也可以不创建，但是需要将csi-rbdplugin-provisioner.yaml和csi-rbdplugin.yaml中kms有关内容注释掉
apiVersion: v1
kind: ConfigMap
data:
config.json: |-
{}
metadata:
name: ceph-csi-encryption-kms-config
- 创建ceph-config-map.yaml
host本地安装的话，ceph.conf就是复制ceph集群的配置文件，也就是/etc/ceph/ceph.conf文件中的对应内容# This is a sample configmap that helps define a Ceph configuration as required # by the CSI plugins. # Sample ceph.conf available at # https://github.com/ceph/ceph/blob/master/src/sample.ceph.conf Detailed # documentation is available at # https://docs.ceph.com/en/latest/rados/configuration/ceph-conf/ apiVersion: v1 kind: ConfigMap data: ceph.conf: | [global] auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx # keyring is a required key and its value should be empty keyring: | metadata: name: ceph-config - 创建k8s_rbd块儿存储池
- 在host上面搭建ceph：
[root@node1 ~]# ceph osd pool create rbdpool pool 'rbdpool' created [root@node1 ~]# rbd pool init rbdpool - 在rook上面搭建ceph
[root@xy-3-server26 examples]# vim pool.yaml apiVersion: ceph.rook.io/v1 kind: CephBlockPool metadata: name: rbdpool namespace: rook-ceph # namespace:cluster spec: failureDomain: host replicated: size: 3 # Disallow setting pool with replica 1, this could lead to data loss without recovery. # Make sure you're *ABSOLUTELY CERTAIN* that is what you want requireSafeReplicaSize: true # gives a hint (%) to Ceph in terms of expected consumption of the total cluster capacity of a given pool # for more info: https://docs.ceph.com/docs/master/rados/operations/placement-groups/#specifying-expected-pool-size #targetSizeRatio: .5 [root@xy-3-server26 examples]# kubectl apply -f pool.yamlbash-4.4$ ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=rbdpool' mgr 'profile rbd pool=rbdpool' [client.kubernetes] key = AQD1ZYhjeyhkGRAAw/aPtECWBnZE2qFudCbJWw==bash-4.4$ ceph auth get client.kubernetes [client.kubernetes] key = AQD1ZYhjeyhkGRAAw/aPtECWBnZE2qFudCbJWw== caps mgr = "profile rbd pool=rbdpool" caps mon = "profile rbd" caps osd = "profile rbd pool=rbdpool" exported keyring for client.kubernetes
-
为storage-class创建secret
编辑csi-rbd-secret.yamlapiVersion: v1 kind: Secret metadata: name: csi-rbd-secret namespace: default stringData: userID: kubernetes userKey: AQD1ZYhjeyhkGRAAw/aPtECWBnZE2qFudCbJWw== encryptionPassphrase: test_passphrase -
apply以上的所有文件\
- 创建 CSIDriver 对象：
[root@server14 kubernetes]# kubectl apply -f csidriver.yaml csidriver.storage.k8s.io/rbd.csi.ceph.com created- 为 sidecar 容器和节点插件部署 RBAC：
kubectl create -f csi-provisioner-rbac.yaml kubectl create -f csi-nodeplugin-rbac.yaml- 为 CSI 插件部署 ConfigMap：
kubectl create -f csi-config-map.yaml- 为 CSI pod 部署 Ceph 配置 ConfigMap：
kubectl create -f ceph-config-map.yaml- 部署 CSI sidecar 容器：
kubectl create -f csi-rbdplugin-provisioner.yaml部署配置部署，包括外部供应商、外部附加器、csi-snapshotter sidecar 容器和 CSI RBD 插件。
- 部署 RBD CSI 驱动程序：
kubectl create -f csi-rbdplugin.yaml部署包含两个容器的守护程序集：CSI node-driver-registrar 和 CSI RBD 驱动程序。 将看到下面的pod
[root@server14 caoyong]# kubectl get pods NAME READY STATUS RESTARTS AGE csi-rbdplugin-kv56n 3/3 Running 0 22h csi-rbdplugin-pj775 3/3 Running 0 22h csi-rbdplugin-provisioner-576966445d-h2jm4 7/7 Running 0 22h csi-rbdplugin-provisioner-576966445d-r8zz4 7/7 Running 0 22h csi-rbdplugin-provisioner-576966445d-z9b42 7/7 Running 0 22h -
创建StorageClass
编辑并执行storage.class.yaml
[root@server14 caoyong]# cat storage-class.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
#provisioner: ceph-csi-rbd.rbd.csi.ceph.com
#provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
clusterID: 6779d759-5604-425d-bc98-c95a5f54e31d
pool: rbdpool #之前创建pool的名称
imageFeatures: layering
csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
csi.storage.k8s.io/provisioner-secret-namespace: default
csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
csi.storage.k8s.io/controller-expand-secret-namespace: default
csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
csi.storage.k8s.io/node-stage-secret-namespace: default
csi.storage.k8s.io/fstype: ext4
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
- discard
[root@server14 caoyong]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
csi-rbd-sc rbd.csi.ceph.com Delete Immediate true 36m
[root@server14 caoyong]# kubectl apply -f storage-class.yaml
storageclass.storage.k8s.io/csi-rbd-sc unchanged
创建rbd-pvc.yaml
[root@server14 caoyong]# cat rbd-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: rbd-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: csi-rbd-sc
[root@server14 caoyong]# kubectl apply -f rbd-pvc.yaml
persistentvolumeclaim/rbd-pvc unchanged
[root@server14 caoyong]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
rbd-pvc Bound pvc-058e4fb9-a63b-44c5-baeb-e672453aacec 10Gi RWO csi-rbd-sc 38m
[root@server14 caoyong]# kubectl describe pvc/rbd-pvc
Name: rbd-pvc
Namespace: default
StorageClass: csi-rbd-sc
Status: Bound
Volume: pvc-058e4fb9-a63b-44c5-baeb-e672453aacec
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: rbd.csi.ceph.com
volume.kubernetes.io/storage-provisioner: rbd.csi.ceph.com
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 10Gi
Access Modes: RWO
VolumeMode: Filesystem
Used By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ProvisioningFailed 38m persistentvolume-controller storageclass.storage.k8s.io "csi-rbd-sc" not found
Normal ExternalProvisioning 38m (x2 over 38m) persistentvolume-controller waiting for a volume to be created, either by external provisioner "rbd.csi.ceph.com" or manually created by system administrator
Normal Provisioning 38m rbd.csi.ceph.com_csi-rbdplugin-provisioner-576966445d-z9b42_4a3cc722-53b1-4012-816b-37546f091491 External provisioner is provisioning volume for claim "default/rbd-pvc"
Normal ProvisioningSucceeded 38m rbd.csi.ceph.com_csi-rbdplugin-provisioner-576966445d-z9b42_4a3cc722-53b1-4012-816b-37546f091491 Successfully provisioned volume pvc-058e4fb9-a63b-44c5-baeb-e672453aacec
部署文件系统
- 在ceph集群上面创建文件系统
[root@node1 ~]# ceph osd pool create cephfs-metadata 32 32
pool 'cephfs_metadata' created
[root@node1 ~]# ceph osd pool create cephfs-data 32 32
pool 'cephfs_data' created
[root@node1 ~]# ceph fs new cephfs cephfs-metadata cephfs-data
new fs with metadata pool 3 and data pool 4
[root@node1 mon.node1]# cat config
# minimal ceph.conf for 365b02aa-db0c-11ec-b243-525400ce981f
[global]
fsid = 365b02aa-db0c-11ec-b243-525400ce981f
mon_host = [v2:172.70.10.181:3300/0,v1:172.70.10.181:6789/0] [v2:172.70.10.182:3300/0,v1:172.70.10.182:6789/0] [v2:172.70.10.183:3300/0,v1:172.70.10.183:6789/0]
# 下面的这一步不能少，cephfs一定要启动mds服务，才能正常对外提供服务，一般来说，这步操作在cephadm shell命令行下执行比较好
[root@node1 kubernetes]# cephadm shell
Inferring fsid 365b02aa-db0c-11ec-b243-525400ce981f
Inferring config /var/lib/ceph/365b02aa-db0c-11ec-b243-525400ce981f/mon.node1/config
Using recent ceph image quay.io/ceph/ceph@sha256:f2822b57d72d07e6352962dc830d2fa93dd8558b725e2468ec0d07af7b14c95d
[ceph: root@node1 /]# ceph orch apply mds cephfs --placement="3 node1 node2 node3"
Scheduled mds.cephfs update...
或者使用rook： 编辑filesystem.yaml 并执行
[root@node1 fs]# cat filesystem.yaml
#################################################################################################################
# Create a filesystem with settings with replication enabled for a production environment.
# A minimum of 3 OSDs on different nodes are required in this example.
# kubectl create -f filesystem.yaml
#################################################################################################################
apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
name: cephfs
namespace: rook-ceph # namespace:cluster
spec:
# The metadata pool spec. Must use replication.
metadataPool:
replicated:
size: 3
requireSafeReplicaSize: true
parameters:
# Inline compression mode for the data pool
# Further reference: https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#inline-compression
compression_mode:
none
# gives a hint (%) to Ceph in terms of expected consumption of the total cluster capacity of a given pool
# for more info: https://docs.ceph.com/docs/master/rados/operations/placement-groups/#specifying-expected-pool-size
#target_size_ratio: ".5"
# The list of data pool specs. Can use replication or erasure coding.
dataPools:
- name: data
failureDomain: host
replicated:
size: 3
# Disallow setting pool with replica 1, this could lead to data loss without recovery.
# Make sure you're *ABSOLUTELY CERTAIN* that is what you want
requireSafeReplicaSize: true
parameters:
# Inline compression mode for the data pool
# Further reference: https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#inline-compression
compression_mode:
none
# gives a hint (%) to Ceph in terms of expected consumption of the total cluster capacity of a given pool
# for more info: https://docs.ceph.com/docs/master/rados/operations/placement-groups/#specifying-expected-pool-size
#target_size_ratio: ".5"
# Whether to preserve filesystem after CephFilesystem CRD deletion
preserveFilesystemOnDelete: true
# The metadata service (mds) configuration
metadataServer:
# The number of active MDS instances
activeCount: 1
# Whether each active MDS instance will have an active standby with a warm metadata cache for faster failover.
# If false, standbys will be available, but will not have a warm cache.
activeStandby: true
# The affinity rules to apply to the mds deployment
placement:
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: role
# operator: In
# values:
# - mds-node
# topologySpreadConstraints:
# tolerations:
# - key: mds-node
# operator: Exists
# podAffinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- rook-ceph-mds
# topologyKey: kubernetes.io/hostname will place MDS across different hosts
topologyKey: kubernetes.io/hostname
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- rook-ceph-mds
# topologyKey: */zone can be used to spread MDS across different AZ
# Use <topologyKey: failure-domain.beta.kubernetes.io/zone> in k8s cluster if your cluster is v1.16 or lower
# Use <topologyKey: topology.kubernetes.io/zone> in k8s cluster is v1.17 or upper
topologyKey: topology.kubernetes.io/zone
# A key/value list of annotations
# annotations:
# key: value
# A key/value list of labels
# labels:
# key: value
# resources:
# The requests and limits set here, allow the filesystem MDS Pod(s) to use half of one CPU core and 1 gigabyte of memory
# limits:
# cpu: "500m"
# memory: "1024Mi"
# requests:
# cpu: "500m"
# memory: "1024Mi"
priorityClassName: system-cluster-critical
livenessProbe:
disabled: false
startupProbe:
disabled: false
# Filesystem mirroring settings
# mirroring:
# enabled: true
# list of Kubernetes Secrets containing the peer token
# for more details see: https://docs.ceph.com/en/latest/dev/cephfs-mirroring/#bootstrap-peers
# Add the secret name if it already exists else specify the empty list here.
# peers:
#secretNames:
#- secondary-cluster-peer
# specify the schedule(s) on which snapshots should be taken
# see the official syntax here https://docs.ceph.com/en/latest/cephfs/snap-schedule/#add-and-remove-schedules
# snapshotSchedules:
# - path: /
# interval: 24h # daily snapshots
# The startTime should be mentioned in the format YYYY-MM-DDTHH:MM:SS
# If startTime is not specified, then by default the start time is considered as midnight UTC.
# see usage here https://docs.ceph.com/en/latest/cephfs/snap-schedule/#usage
# startTime: 2022-07-15T11:55:00
# manage retention policies
# see syntax duration here https://docs.ceph.com/en/latest/cephfs/snap-schedule/#add-and-remove-retention-policies
# snapshotRetention:
# - path: /
# duration: "h 24"
经过上面步骤，可以看到ceph相关信息
bash-4.4$ ceph -s
cluster:
id: 6779d759-5604-425d-bc98-c95a5f54e31d
health: HEALTH_OK
services:
mon: 3 daemons, quorum a,b,c (age 9h)
mgr: a(active, since 27h), standbys: b
mds: 1/1 daemons up, 1 hot standby
osd: 12 osds: 12 up (since 27h), 12 in (since 27h)
data:
volumes: 1/1 healthy
pools: 4 pools, 97 pgs
objects: 46 objects, 4.0 MiB
usage: 408 MiB used, 72 TiB / 72 TiB avail
pgs: 97 active+clean
io:
client: 1.2 KiB/s rd, 2 op/s rd, 0 op/s wr
bash-4.4$ ceph auth get client.admin
[client.admin]
key = AQDvUYhjLUrIBhAAzcewDI9x6Qq5S8HMDPoGGA==
caps mds = "allow *"
caps mgr = "allow *"
caps mon = "allow *"
caps osd = "allow *"
exported keyring for client.admin
bash-4.4$ ceph mon stat
e3: 3 mons at {a=[v2:192.168.5.163:3300/0,v1:192.168.5.163:6789/0],b=[v2:192.168.5.162:3300/0,v1:192.168.5.162:6789/0],c=[v2:192.168.5.165:3300/0,v1:192.168.5.165:6789/0]}, election epoch 44, leader 0 a, quorum 0,1,2 a,b,c
- 创建ceph-csi-config configmap
下载下来之后进入 deploy/cephfs/kubernetes
使用rbd创建过的ceph-csi-config创建ConfigMap
[root@server14 caoyong]# cat csi-config-map.yaml
#
# /!\ DO NOT MODIFY THIS FILE
#
# This file has been automatically generated by Ceph-CSI yamlgen.
# The source for the contents can be found in the api/deploy directory, make
# your modifications there.
#
---
apiVersion: v1
kind: ConfigMap
metadata:
name: "ceph-csi-config"
data:
config.json: |
[
{
"clusterID": "6779d759-5604-425d-bc98-c95a5f54e31d",
"monitors": [
"192.168.5.163:6789",
"192.168.5.162:6789",
"192.168.5.165:6789"
]
}
]
- 安装csi-cephfs插件
kubectl apply -f deploy/rbd/kubernetes/csi-provisioner-rbac.yaml
kubectl apply -f deploy/rbd/kubernetes/csi-nodeplugin-rbac.yaml
kubectl apply -f deploy/rbd/kubernetes/csi-cephfsplugin-provisioner.yaml
kubectl apply -f deploy/rbd/kubernetes/csi-cephfsplugin.yaml
- 为storageClass创建secret
首先获取admin的keybash-4.4$ ceph auth get client.admin [client.admin] key = AQDvUYhjLUrIBhAAzcewDI9x6Qq5S8HMDPoGGA== caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *" exported keyring for client.admin[root@server14 caoyong]# cat secret.yaml --- apiVersion: v1 kind: Secret metadata: name: csi-cephfs-secret namespace: default stringData: # Required for statically provisioned volumes userID: admin userKey: AQDvUYhjLUrIBhAAzcewDI9x6Qq5S8HMDPoGGA== # Required for dynamically provisioned volumes adminID: admin adminKey: AQDvUYhjLUrIBhAAzcewDI9x6Qq5S8HMDPoGGA== [root@node1 fs]# kubectl apply -f secret.yaml secret/csi-cephfs-secret created [root@node1 fs]# k get secret NAME TYPE DATA AGE csi-cephfs-secret Opaque 4 11s csi-rbd-secret Opaque 2 28h
5.创建storageClass和pvc 编辑并执行
[root@server14 caoyong]# cat storage-class.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-cephfs-sc
provisioner: cephfs.csi.ceph.com
parameters:
clusterID: 6779d759-5604-425d-bc98-c95a5f54e31d
fsName: cephfs
pool: cephfs-data
# mounter: fuse 挂载方式
csi.storage.k8s.io/provisioner-secret-name: csi-cephfs-secret
csi.storage.k8s.io/provisioner-secret-namespace: default
csi.storage.k8s.io/controller-expand-secret-name: csi-cephfs-secret
csi.storage.k8s.io/controller-expand-secret-namespace: default
csi.storage.k8s.io/node-stage-secret-name: csi-cephfs-secret
csi.storage.k8s.io/node-stage-secret-namespace: default
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
- discard
编辑并执行
[root@server14 caoyong]# cat ceph-fs-pvc.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ceph-fs-pvc
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Gi
storageClassName: csi-cephfs-sc
可以看到相应的pvc信息
[root@server14 caoyong]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
csi-cephfs-sc cephfs.csi.ceph.com Delete Immediate true 5h31m
csi-rbd-sc rbd.csi.ceph.com Delete Immediate true 62m
[root@server14 caoyong]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
ceph-fs-pvc Bound pvc-637f3f94-f1de-44da-a73c-64b4a2847780 10Gi RWX csi-cephfs-sc 5h30m
rbd-pvc Bound pvc-058e4fb9-a63b-44c5-baeb-e672453aacec 10Gi RWO csi-rbd-sc 62m
[root@server14 caoyong]# kubectl apply -f ceph-fs-pvc.yaml
persistentvolumeclaim/ceph-fs-pvc created
[root@server14 caoyong]# kubectl describe pvc/ceph-fs-pvc
Name: ceph-fs-pvc
Namespace: default
StorageClass: csi-cephfs-sc
Status: Bound
Volume: pvc-7bfcc296-5e5e-4720-bef8-8e5e17a50176
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: cephfs.csi.ceph.com
volume.kubernetes.io/storage-provisioner: cephfs.csi.ceph.com
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 10Gi
Access Modes: RWX
VolumeMode: Filesystem
Used By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ExternalProvisioning 2s persistentvolume-controller waiting for a volume to be created, either by external provisioner "cephfs.csi.ceph.com" or manually created by system administrator
Normal Provisioning 2s cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-778c5d48c9-glkhv_4d18688f-9ac1-467c-a2ff-72be068f5727 External provisioner is provisioning volume for claim "default/ceph-fs-pvc"
Normal ProvisioningSucceeded 0s cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-778c5d48c9-glkhv_4d18688f-9ac1-467c-a2ff-72be068f5727 Successfully provisioned volume pvc-7bfcc296-5e5e-4720-bef8-8e5e17a50176
然后部署nginx挂载pvc 发现如下问题：
[root@server14 ~]# kubectl -n ceph-csi describe pod/my-nginx-test-fs-5846844f68-k5vc2
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 24m default-scheduler Successfully assigned ceph-csi/my-nginx-test-fs-5846844f68-k5vc2 to server19
Warning FailedMount 6m32s kubelet Unable to attach or mount volumes: unmounted volumes=[rbd], unattached volumes=[kube-api-access-2fdkx rbd]: timed out waiting for the condition
Warning FailedAttachVolume 2m51s (x7 over 20m) attachdetach-controller AttachVolume.Attach failed for volume "pvc-aab38778-4358-414a-b939-8ecce15cb3e0" : timed out waiting for external-attacher of cephfs.csi.ceph.com CSI driver to attach volume 0001-0024-6779d759-5604-425d-bc98-c95a5f54e31d-0000000000000003-e07d9cea-84eb-11ed-a5ed-3abcde10cfd9
Warning FailedMount 2m1s (x9 over 22m) kubelet Unable to attach or mount volumes: unmounted volumes=[rbd], unattached volumes=[rbd kube-api-access-2fdkx]: timed out waiting for the condition
发现只有rbd.csi.ceph.com一个驱动，没有cephfs的驱动cephfs.csi.ceph.com
[root@server14 caoyong]# kubectl get csidriver
NAME ATTACHREQUIRED PODINFOONMOUNT STORAGECAPACITY TOKENREQUESTS REQUIRESREPUBLISH MODES AGE
rbd.csi.ceph.com true false false <unset> false Persistent 23d
然后添加
[root@server14 kubernetes]# kubectl apply -f /root/ceph-csi-3.7.2/deploy/cephfs/kubernetes/csidriver.yaml
csidriver.storage.k8s.io/cephfs.csi.ceph.com created
[root@server14 kubernetes]# kubectl get csidriver
NAME ATTACHREQUIRED PODINFOONMOUNT STORAGECAPACITY TOKENREQUESTS REQUIRESREPUBLISH MODES AGE
cephfs.csi.ceph.com false false false <unset> false Persistent 5s
rbd.csi.ceph.com true false false <unset> false Persistent 23d
然后pod就可以正常挂载pvc了。。。 6. 遇到的问题
[root@node1 ~]# k get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
fs-pvc Pending csi-cephfs-sc 27m
rbd-pvc Bound pvc-80d393f0-8664-4d70-8e0d-d7a0550d4417 10Gi RWO csi-rbd-sc 7h22m
[root@node1 ~]# kd pvc/fs-pvc
Name: fs-pvc
Namespace: default
StorageClass: csi-cephfs-sc
Status: Pending
Volume:
Labels: <none>
Annotations: volume.beta.kubernetes.io/storage-provisioner: cephfs.csi.ceph.com
volume.kubernetes.io/storage-provisioner: cephfs.csi.ceph.com
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Provisioning 3m55s (x14 over 27m) cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-794b8d9f95-jwmw4_ac816be6-acdb-4447-a41a-e034c43d1b2b External provisioner is provisioning volume for claim "default/fs-pvc"
Warning ProvisioningFailed 3m55s (x4 over 24m) cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-794b8d9f95-jwmw4_ac816be6-acdb-4447-a41a-e034c43d1b2b failed to provision volume with StorageClass "csi-cephfs-sc": rpc error: code = DeadlineExceeded desc = context deadline exceeded
Warning ProvisioningFailed 3m55s (x10 over 24m) cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-794b8d9f95-jwmw4_ac816be6-acdb-4447-a41a-e034c43d1b2b failed to provision volume with StorageClass "csi-cephfs-sc": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-aaed8aa7-c202-44c2-8def-f3479ea27ffe already exists
Normal ExternalProvisioning 2m26s (x102 over 27m) persistentvolume-controller waiting for a volume to be created, either by external provisioner "cephfs.csi.ceph.com" or manually created by system administrator
# 到ceph集群中，查看集群健康状态
[root@node1 fs]# ceph health
HEALTH_ERR 1 filesystem is offline; 1 filesystem is online with fewer MDS than max_mds
# 该问题出现是因为cephfs没有启动mds，下面启动mds便可回复正常
[root@node1 kubernetes]# cephadm shell
Inferring fsid 365b02aa-db0c-11ec-b243-525400ce981f
Inferring config /var/lib/ceph/365b02aa-db0c-11ec-b243-525400ce981f/mon.node1/config
Using recent ceph image quay.io/ceph/ceph@sha256:f2822b57d72d07e6352962dc830d2fa93dd8558b725e2468ec0d07af7b14c95d
[ceph: root@node1 /]# ceph orch apply mds cephfs --placement="3 node1 node2 node3"
Scheduled mds.cephfs update...
#回到k8s环境中
[root@node1 ~]# k delete pvc/fs-pvc
persistentvolumeclaim "fs-pvc" deleted
[root@node1 fs]# k apply -f fs-pvc.yaml
persistentvolumeclaim/fs-pvc created
[root@node1 fs]# k get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
fs-pvc Bound pvc-01d43d98-3375-4642-9bd2-b4818ce59f77 11Gi RWX csi-cephfs-sc 6s
rbd-pvc Bound pvc-80d393f0-8664-4d70-8e0d-d7a0550d4417 10Gi RWO csi-rbd-sc 7h23m
至此，整合rbd和cephfs的过程结束。
对象存储
对于ceph对象存储，本身ceph提供的是基于七层协议的接口，直接通过对象存储s3协议访问即可，不需要通过csi进行集成。
