从RADOS移除OSD
Ceph集群中的一个OSD是一个node节点的服务进程且对应于一个物理磁盘设备，是一个专用的守护进程，在某个OSD设备出现故障，或管理员出于管理需要确实需要移除特定的osd设备时，需要先停止相关的守护进程，然后再进行移除操作，对于Luminous及其之后的版本，停止和移除的命令如下
# 具体操作如下
1， 停止设备：ceph osd out {osd-num}
2， 停止进程： systemctl stop ceph-osd@{osd-num}
3, 移除设备： ceph osd purge {id} --yes-i-really-mean-it
对于Luminous之前的版本来说，需要执行下面的操作
1,ceph osd crush remove {name}
2,移除osd的认证：ceph auth del osd.{osd.num}
3,最后移除osd设备：ceph osd rm {osd-num}
测试
查看osd信息
root@node1:~# ceph osd dump
epoch 104
fsid e7f2f395-d8f8-4345-bc48-78e48868fb0f
created 2023-10-22T18:35:26.710127+0800
modified 2023-10-22T19:36:55.395271+0800
flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit
crush_version 41
full_ratio 0.95
backfillfull_ratio 0.9
nearfull_ratio 0.85
require_min_compat_client luminous
min_compat_client jewel
require_osd_release reef
stretch_mode_enabled false
pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 103 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 20.00
max_osd 20
osd.0 up in weight 1 up_from 5 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:192.168.202.151:6800/110288,v1:192.168.202.151:6801/110288] [v2:192.168.146.101:6800/110288,v1:192.168.146.101:6801/110288] exists,up 1ba73f70-47f2-4850-a3a8-bbbed83eb899
osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:192.168.202.151:6804/112187,v1:192.168.202.151:6805/112187] [v2:192.168.146.101:6804/112187,v1:192.168.146.101:6805/112187] exists,up 29cc3fd2-786c-4947-a9d6-959076428184
osd.2 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:192.168.202.151:6808/114576,v1:192.168.202.151:6809/114576] [v2:192.168.146.101:6808/114576,v1:192.168.146.101:6809/114576] exists,up bcfc3a4e-e170-4237-86f2-73d53e14ebba
osd.3 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:192.168.202.151:6812/116428,v1:192.168.202.151:6813/116428] [v2:192.168.146.101:6812/116428,v1:192.168.146.101:6813/116428] exists,up d5ba90a9-050e-4221-b130-7dbe52de820d
osd.4 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:192.168.202.152:6800/123852,v1:192.168.202.152:6801/123852] [v2:192.168.146.102:6800/123852,v1:192.168.146.102:6801/123852] exists,up 049490db-3303-4a4c-91cb-db422276efd8
osd.5 up in weight 1 up_from 32 up_thru 32 down_at 0 last_clean_interval [0,0) [v2:192.168.202.152:6804/125632,v1:192.168.202.152:6805/125632] [v2:192.168.146.102:6804/125632,v1:192.168.146.102:6805/125632] exists,up bd216e0f-f9cf-4f4b-96b9-266411788306
osd.6 up in weight 1 up_from 37 up_thru 37 down_at 0 last_clean_interval [0,0) [v2:192.168.202.152:6808/127447,v1:192.168.202.152:6809/127447] [v2:192.168.146.102:6808/127447,v1:192.168.146.102:6809/127447] exists,up 2481f78e-02dd-4187-b89b-0ddc4de19c51
osd.7 up in weight 1 up_from 41 up_thru 41 down_at 0 last_clean_interval [0,0) [v2:192.168.202.152:6812/129255,v1:192.168.202.152:6813/129255] [v2:192.168.146.102:6812/129255,v1:192.168.146.102:6813/129255] exists,up 4cb56e8e-d054-4f8d-8558-6a72117fccfb
osd.8 up in weight 1 up_from 46 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:192.168.202.153:6800/112691,v1:192.168.202.153:6801/112691] [v2:192.168.146.103:6800/112691,v1:192.168.146.103:6801/112691] exists,up 7dd23d72-dc1f-41b0-87be-d99951423732
osd.9 up in weight 1 up_from 51 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:192.168.202.153:6804/114550,v1:192.168.202.153:6805/114550] [v2:192.168.146.103:6804/114550,v1:192.168.146.103:6805/114550] exists,up af18f0c6-c142-4c51-82c9-5f244390f8d7
osd.10 up in weight 1 up_from 56 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:192.168.202.153:6808/116364,v1:192.168.202.153:6809/116364] [v2:192.168.146.103:6808/116364,v1:192.168.146.103:6809/116364] exists,up ec424b18-710f-4859-8b05-546510b4d7a9
osd.11 up in weight 1 up_from 61 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:192.168.202.153:6812/118125,v1:192.168.202.153:6813/118125] [v2:192.168.146.103:6812/118125,v1:192.168.146.103:6813/118125] exists,up 178e8516-5d99-44ec-9ee0-5e32b6275709
osd.12 up in weight 1 up_from 66 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:192.168.202.154:6802/111661,v1:192.168.202.154:6803/111661] [v2:192.168.146.104:6800/111661,v1:192.168.146.104:6801/111661] exists,up 2d0d7e62-ed8a-4fd0-9835-b485842e519e
osd.13 up in weight 1 up_from 71 up_thru 103 down_at 0 last_clean_interval [0,0) [v2:192.168.202.154:6806/113441,v1:192.168.202.154:6807/113441] [v2:192.168.146.104:6804/113441,v1:192.168.146.104:6805/113441] exists,up c174fe12-6806-474f-89bd-8a8bfa6ecc9c
osd.14 up in weight 1 up_from 76 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:192.168.202.154:6810/115282,v1:192.168.202.154:6811/115282] [v2:192.168.146.104:6808/115282,v1:192.168.146.104:6809/115282] exists,up 3cfa6472-3262-42c8-bc8b-8a670835ae69
osd.15 up in weight 1 up_from 81 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:192.168.202.154:6814/117084,v1:192.168.202.154:6815/117084] [v2:192.168.146.104:6812/117084,v1:192.168.146.104:6813/117084] exists,up 62237977-d96a-4d5b-ad9a-6a497112dd9c
osd.16 up in weight 1 up_from 86 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:192.168.202.155:6800/115140,v1:192.168.202.155:6801/115140] [v2:192.168.146.105:6800/115140,v1:192.168.146.105:6801/115140] exists,up fd6f245e-09f9-4296-b21a-14aed0aedca5
osd.17 up in weight 1 up_from 91 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:192.168.202.155:6804/116921,v1:192.168.202.155:6805/116921] [v2:192.168.146.105:6804/116921,v1:192.168.146.105:6805/116921] exists,up a34cd1b4-4bbc-4bc9-bb30-fdd41f02bf51
osd.18 up in weight 1 up_from 96 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:192.168.202.155:6808/118802,v1:192.168.202.155:6809/118802] [v2:192.168.146.105:6808/118802,v1:192.168.146.105:6809/118802] exists,up bef46459-3a2e-49df-b127-ceae0140a10f
osd.19 up in weight 1 up_from 101 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:192.168.202.155:6812/120557,v1:192.168.202.155:6813/120557] [v2:192.168.146.105:6812/120557,v1:192.168.146.105:6813/120557] exists,up df6a1739-a59e-40b0-829e-fff89eaed7e2
root@node1:~#
移除osd.19
root@node1:~# ceph osd out 19
marked out osd.19.
root@node1:~#
停止服务
root@node5:~# systemctl status ceph-osd@19
● ceph-osd@19.service - Ceph object storage daemon osd.19
Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: enabled)
Active: active (running) since Sun 2023-10-22 19:26:44 CST; 44min ago
Process: 120553 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 19 (code=exited, status=0/SUCCESS)
Main PID: 120557 (ceph-osd)
Tasks: 62
Memory: 77.4M
CPU: 9.127s
CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@19.service
└─120557 /usr/bin/ceph-osd -f --cluster ceph --id 19 --setuser ceph --setgroup ceph
Oct 22 19:26:43 node5 systemd[1]: Starting Ceph object storage daemon osd.19...
Oct 22 19:26:44 node5 systemd[1]: Started Ceph object storage daemon osd.19.
Oct 22 19:26:46 node5 ceph-osd[120557]: 2023-10-22T19:26:46.671+0800 7fd9204e3640 -1 osd.19 0 log_to_monitors true
Oct 22 19:26:48 node5 ceph-osd[120557]: 2023-10-22T19:26:48.211+0800 7fd91c48c640 -1 osd.19 0 waiting for initial osdmap
Oct 22 19:26:48 node5 ceph-osd[120557]: 2023-10-22T19:26:48.235+0800 7fd915ac2640 -1 osd.19 100 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory
root@node5:~# systemctl stop ceph-osd@19
root@node5:~# ceph osd purge 19 --yes-i-really-mean-it
purged osd.19
查看当前osd信息
可以看到没有19了
root@node5:~# ceph osd dump
epoch 107
fsid e7f2f395-d8f8-4345-bc48-78e48868fb0f
created 2023-10-22T18:35:26.710127+0800
modified 2023-10-22T20:12:03.124259+0800
flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit
crush_version 42
full_ratio 0.95
backfillfull_ratio 0.9
nearfull_ratio 0.85
require_min_compat_client luminous
min_compat_client jewel
require_osd_release reef
stretch_mode_enabled false
pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 103 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 18.75
max_osd 20
osd.0 up in weight 1 up_from 5 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:192.168.202.151:6800/110288,v1:192.168.202.151:6801/110288] [v2:192.168.146.101:6800/110288,v1:192.168.146.101:6801/110288] exists,up 1ba73f70-47f2-4850-a3a8-bbbed83eb899
osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:192.168.202.151:6804/112187,v1:192.168.202.151:6805/112187] [v2:192.168.146.101:6804/112187,v1:192.168.146.101:6805/112187] exists,up 29cc3fd2-786c-4947-a9d6-959076428184
osd.2 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:192.168.202.151:6808/114576,v1:192.168.202.151:6809/114576] [v2:192.168.146.101:6808/114576,v1:192.168.146.101:6809/114576] exists,up bcfc3a4e-e170-4237-86f2-73d53e14ebba
osd.3 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:192.168.202.151:6812/116428,v1:192.168.202.151:6813/116428] [v2:192.168.146.101:6812/116428,v1:192.168.146.101:6813/116428] exists,up d5ba90a9-050e-4221-b130-7dbe52de820d
osd.4 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:192.168.202.152:6800/123852,v1:192.168.202.152:6801/123852] [v2:192.168.146.102:6800/123852,v1:192.168.146.102:6801/123852] exists,up 049490db-3303-4a4c-91cb-db422276efd8
osd.5 up in weight 1 up_from 32 up_thru 32 down_at 0 last_clean_interval [0,0) [v2:192.168.202.152:6804/125632,v1:192.168.202.152:6805/125632] [v2:192.168.146.102:6804/125632,v1:192.168.146.102:6805/125632] exists,up bd216e0f-f9cf-4f4b-96b9-266411788306
osd.6 up in weight 1 up_from 37 up_thru 37 down_at 0 last_clean_interval [0,0) [v2:192.168.202.152:6808/127447,v1:192.168.202.152:6809/127447] [v2:192.168.146.102:6808/127447,v1:192.168.146.102:6809/127447] exists,up 2481f78e-02dd-4187-b89b-0ddc4de19c51
osd.7 up in weight 1 up_from 41 up_thru 41 down_at 0 last_clean_interval [0,0) [v2:192.168.202.152:6812/129255,v1:192.168.202.152:6813/129255] [v2:192.168.146.102:6812/129255,v1:192.168.146.102:6813/129255] exists,up 4cb56e8e-d054-4f8d-8558-6a72117fccfb
osd.8 up in weight 1 up_from 46 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:192.168.202.153:6800/112691,v1:192.168.202.153:6801/112691] [v2:192.168.146.103:6800/112691,v1:192.168.146.103:6801/112691] exists,up 7dd23d72-dc1f-41b0-87be-d99951423732
osd.9 up in weight 1 up_from 51 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:192.168.202.153:6804/114550,v1:192.168.202.153:6805/114550] [v2:192.168.146.103:6804/114550,v1:192.168.146.103:6805/114550] exists,up af18f0c6-c142-4c51-82c9-5f244390f8d7
osd.10 up in weight 1 up_from 56 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:192.168.202.153:6808/116364,v1:192.168.202.153:6809/116364] [v2:192.168.146.103:6808/116364,v1:192.168.146.103:6809/116364] exists,up ec424b18-710f-4859-8b05-546510b4d7a9
osd.11 up in weight 1 up_from 61 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:192.168.202.153:6812/118125,v1:192.168.202.153:6813/118125] [v2:192.168.146.103:6812/118125,v1:192.168.146.103:6813/118125] exists,up 178e8516-5d99-44ec-9ee0-5e32b6275709
osd.12 up in weight 1 up_from 66 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:192.168.202.154:6802/111661,v1:192.168.202.154:6803/111661] [v2:192.168.146.104:6800/111661,v1:192.168.146.104:6801/111661] exists,up 2d0d7e62-ed8a-4fd0-9835-b485842e519e
osd.13 up in weight 1 up_from 71 up_thru 103 down_at 0 last_clean_interval [0,0) [v2:192.168.202.154:6806/113441,v1:192.168.202.154:6807/113441] [v2:192.168.146.104:6804/113441,v1:192.168.146.104:6805/113441] exists,up c174fe12-6806-474f-89bd-8a8bfa6ecc9c
osd.14 up in weight 1 up_from 76 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:192.168.202.154:6810/115282,v1:192.168.202.154:6811/115282] [v2:192.168.146.104:6808/115282,v1:192.168.146.104:6809/115282] exists,up 3cfa6472-3262-42c8-bc8b-8a670835ae69
osd.15 up in weight 1 up_from 81 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:192.168.202.154:6814/117084,v1:192.168.202.154:6815/117084] [v2:192.168.146.104:6812/117084,v1:192.168.146.104:6813/117084] exists,up 62237977-d96a-4d5b-ad9a-6a497112dd9c
osd.16 up in weight 1 up_from 86 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:192.168.202.155:6800/115140,v1:192.168.202.155:6801/115140] [v2:192.168.146.105:6800/115140,v1:192.168.146.105:6801/115140] exists,up fd6f245e-09f9-4296-b21a-14aed0aedca5
osd.17 up in weight 1 up_from 91 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:192.168.202.155:6804/116921,v1:192.168.202.155:6805/116921] [v2:192.168.146.105:6804/116921,v1:192.168.146.105:6805/116921] exists,up a34cd1b4-4bbc-4bc9-bb30-fdd41f02bf51
osd.18 up in weight 1 up_from 96 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:192.168.202.155:6808/118802,v1:192.168.202.155:6809/118802] [v2:192.168.146.105:6808/118802,v1:192.168.146.105:6809/118802] exists,up bef46459-3a2e-49df-b127-ceae0140a10f
root@node5:~# ceph status
cluster:
id: e7f2f395-d8f8-4345-bc48-78e48868fb0f
health: HEALTH_OK
services:
mon: 3 daemons, quorum node1,node2,node3 (age 97m)
mgr: node4(active, since 93m), standbys: node5
osd: 19 osds: 19 up (since 2m), 19 in (since 2m) # 显示了总共19个osd，之前是20个
data:
pools: 1 pools, 1 pgs
objects: 2 objects, 449 KiB
usage: 521 MiB used, 379 GiB / 380 GiB avail
pgs: 1 active+clean
