@TOC
1 前言
参考：ceph接入openstack配置 ==#本文解释从Netonline大佬转载过来的，用来解释接入集群的一些配置== Openstack环境中，数据存储可分为临时性存储与永久性存储。
临时性存储：主要由本地文件系统提供，并主要用于nova虚拟机的本地系统与临时数据盘，以及存储glance上传的系统镜像；
永久性存储：主要由cinder提供的块存储与swift提供的对象存储构成，以cinder提供的块存储应用最为广泛，块存储通常以云盘的形式挂载到虚拟机中使用。
Openstack中需要进行数据存储的三大项目主要是nova项目（虚拟机镜像文件），glance项目（共用模版镜像）与cinder项目（块存储）。
下图为cinder，glance与nova访问ceph集群的逻辑图：
ceph与openstack集成主要用到ceph的rbd服务，ceph底层为rados存储集群，ceph通过librados库实现对底层rados的访问；
openstack各项目客户端调用librbd，再由librbd调用librados访问底层rados； 实际使用中，nova需要使用libvirtdriver驱动以通过libvirt与qemu调用librbd；cinder与glance可直接调用librbd；
写入ceph集群的数据被条带切分成多个object，object通过hash函数映射到pg（构成pg容器池pool），然后pg通过几圈crush算法近似均匀地映射到物理存储设备osd（osd是基于文件系统的物理存储设备，如xfs，ext4等）。
2 openstack集群上的操作
#在openstack所有控制和计算节点安装ceph nautilus源码包，centos8有默认安装，但是版本一定要跟你连接的ceph版本一模一样！ #默认安装源码包的方法如下：
yum install centos-release-ceph-nautilus.noarch
#但是我这里使用的是nautilus14.2.10，因此
[Ceph]
name=Ceph packages for $basearch
baseurl=https://mirrors.aliyun.com/ceph/rpm-nautilus/el8/$basearch
enabled=1
gpgcheck=0
type=rpm-md
[Ceph-noarch]
name=Ceph noarch packages
baseurl=https://mirrors.aliyun.com/ceph/rpm-nautilus/el8/noarch
enabled=1
gpgcheck=0
type=rpm-md
[ceph-source]
name=Ceph source packages
baseurl=https://mirrors.aliyun.com/ceph/rpm-nautilus/el8/SRPMS
enabled=1
gpgcheck=0
type=rpm-md
#glance-api服务所在节点需要安装python3-rbd； #这里glance-api服务运行在3个控制节点，因此三台都必须安装
yum install python3-rbd -y
#cinder-volume与nova-compute服务所在节点需要安装ceph-common； #这里cinder-volume与nova-compute服务运行在2个计算（存储）节点
yum install ceph-common -y
3 ceph集群上的操作
#==ceph集群创建，在OpenStack Ussuri 集群部署教程 - centos8我有提供两个选择，自行选择== #在ceph集群和openstack集群添加hosts #vim /etc/hosts
172.16.1.131 ceph131
172.16.1.132 ceph132
172.16.1.133 ceph133
172.16.1.160 controller160
172.16.1.161 controller161
172.16.1.162 controller162
172.16.1.168 controller168
172.16.1.163 compute163
172.16.1.164 compute164
3.1 创建openstack集群将要使用的pool
#Ceph默认使用pool的形式存储数据，pool是对若干pg进行组织管理的逻辑划分，pg里的对象被映射到不同的osd，因此pool分布到整个集群里。 #可以将不同的数据存入1个pool，但如此操作不便于客户端数据区分管理，因此一般是为每个客户端分别创建pool。 #为cinder，nova，glance分别创建pool，命名为：volumes，vms，images ==#这里volumes池是永久性存储，vms是实例临时后端存储，images是镜像存储== ==#pg数是有算法的，可以使用官网计算器去计算！== ==#PG数量的预估 集群中单个池的PG数计算公式如下：PG 总数 = (OSD 数 * 100) / 最大副本数 / 池数 (结果必须舍入到最接近2的N次幂的值)== #在ceph集群里操作，创建pools
[root@ceph131 ~]# ceph osd pool create volumes 16 16 replicated
pool 'volumes' created
[root@ceph131 ~]# ceph osd pool create vms 16 16 replicated
pool 'vms' created
[root@ceph131 ~]# ceph osd pool create images 16 16 replicated
pool 'images' created
[root@ceph131 ~]# ceph osd lspools
1 cephfs_data
2 cephfs_metadata
3 rbd_storage
4 .rgw.root
5 default.rgw.control
6 default.rgw.meta
7 default.rgw.log
10 volumes
11 vms
12 images
3.2 ceph授权设置
3.2.1 创建用户
#ceph默认启用cephx authentication，需要为nova/cinder与glance客户端创建新的用户并授权； #可在管理节点上分别为运行cinder-volume与glance-api服务的节点创建client.glance与client.cinder用户并设置权限； #针对pool设置权限，pool名对应创建的pool
[root@ceph131 ~]# ceph auth get-or-create client.cinder mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rx pool=images'
[client.cinder]
key = AQDYA/9eFE4vIBAArdMpCCNxKxLUpSKaKc6nDg==
[root@ceph131 ~]# ceph auth get-or-create client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images'
[client.glance]
key = AQDkA/9eCaGUERAAxKTqRS5Vk7iuLugYnEP5BQ==
3.2.2 推送client.glance&client.cinder秘钥
#配置节点免密操作
[cephdeploy@ceph131 ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:CsvXYKm8mRzasMFwgWVLx5LvvfnPrRc5S1wSb6kPytM root@ceph131
The key's randomart image is:
+---[RSA 2048]----+
| +o. |
| =oo. . |
|. oo o . |
| .. . . = |
|. ....+ S . * |
| + o.=.+ O |
| + * oo.. + * |
| B *o .+.E . |
| o * ...++. |
+----[SHA256]-----+
#推送密钥至各openstack集群节点
ssh-copy-id root@controller160
ssh-copy-id root@controller161
ssh-copy-id root@controller162
ssh-copy-id root@compute163
ssh-copy-id root@compute164
#这里nova-compute服务与nova-volume服务运行在相同节点，不必重复操作。
#将创建client.glance用户生成的秘钥推送到运行glance-api服务的节点
[root@ceph131 ~]# ceph auth get-or-create client.glance | tee /etc/ceph/ceph.client.glance.keyring
/ceph/ceph.client.glance.keyring
[root@ceph131 ~]# ceph auth get-or-create client.glance | ssh root@controller160 tee /etc/ceph/ceph.client.glance.keyring
[client.glance]
key = AQDkA/9eCaGUERAAxKTqRS5Vk7iuLugYnEP5BQ==
[root@ceph131 ~]# ceph auth get-or-create client.glance | ssh root@controller161 tee /etc/ceph/ceph.client.glance.keyring
[client.glance]
key = AQDkA/9eCaGUERAAxKTqRS5Vk7iuLugYnEP5BQ==
[root@ceph131 ~]# ceph auth get-or-create client.glance | ssh root@controller162 tee /etc/ceph/ceph.client.glance.keyring
[client.glance]
key = AQDkA/9eCaGUERAAxKTqRS5Vk7iuLugYnEP5BQ==
#同时修改秘钥文件的属主与用户组
#chown glance:glance /etc/ceph/ceph.client.glance.keyring
ssh root@controller160 chown glance:glance /etc/ceph/ceph.client.glance.keyring
ssh root@controller161 chown glance:glance /etc/ceph/ceph.client.glance.keyring
ssh root@controller162 chown glance:glance /etc/ceph/ceph.client.glance.keyring
#将创建client.cinder用户生成的秘钥推送到运行cinder-volume服务的节点
[root@ceph131 ceph]# ceph auth get-or-create client.cinder | ssh root@compute163 tee /etc/ceph/ceph.client.cinder.keyring
[client.cinder]
key = AQDYA/9eFE4vIBAArdMpCCNxKxLUpSKaKc6nDg==
[root@ceph131 ceph]# ceph auth get-or-create client.cinder | ssh root@compute164 tee /etc/ceph/ceph.client.cinder.keyring
[client.cinder]
key = AQDYA/9eFE4vIBAArdMpCCNxKxLUpSKaKc6nDg==
#同时修改秘钥文件的属主与用户组
ssh root@compute163 chown cinder:cinder /etc/ceph/ceph.client.cinder.keyring
ssh root@compute164 chown cinder:cinder /etc/ceph/ceph.client.cinder.keyring
3.2.3 libvirt秘钥
#nova-compute所在节点需要将client.cinder用户的秘钥文件存储到libvirt中；当基于ceph后端的cinder卷被attach到虚拟机实例时，libvirt需要用到该秘钥以访问ceph集群；
#在管理节点向计算（存储）节点推送client.cinder秘钥文件，生成的文件是临时性的，将秘钥添加到libvirt后可删除
[root@ceph131 ceph]# ceph auth get-key client.cinder | ssh root@compute164 tee /etc/ceph/client.cinder.key
AQDYA/9eFE4vIBAArdMpCCNxKxLUpSKaKc6nDg==
[root@ceph131 ceph]# ceph auth get-key client.cinder | ssh root@compute163 tee /etc/ceph/client.cinder.key
AQDYA/9eFE4vIBAArdMpCCNxKxLUpSKaKc6nDg==
#在计算（存储）节点将秘钥加入libvirt，以compute163节点为例； #首先生成1个uuid，==全部计算（存储）节点可共用此uuid（其他节点不用操作此步）==； #uuid后续配置nova.conf文件时也会用到，请保持一致
[root@compute163 ~]# uuidgen
e9776771-b980-481d-9e99-3ddfdbf53d1e
[root@compute163 ~]# cd /etc/ceph/
[root@compute163 ceph]# touch secret.xml
[root@compute163 ceph]# vim secret.xml
<secret ephemeral='no' private='no'>
<uuid>cb26bb6c-2a84-45c2-8187-fa94b81dd53d</uuid>
<usage type='ceph'>
<name>client.cinder secret</name>
</usage>
</secret>
[root@compute163 ceph]# virsh secret-define --file secret.xml
Secret cb26bb6c-2a84-45c2-8187-fa94b81dd53d created
[root@compute163 ceph]# virsh secret-set-value --secret cb26bb6c-2a84-45c2-8187-fa94b81dd53d --base64 $(cat /etc/ceph/client.cinder.key)
Secret value set
#推送ceph.conf
[root@ceph131 ceph]# scp ceph.conf root@controller160:/etc/ceph/
ceph.conf 100% 514 407.7KB/s 00:00
[root@ceph131 ceph]# scp ceph.conf root@controller161:/etc/ceph/
ceph.conf 100% 514 631.5KB/s 00:00
[root@ceph131 ceph]# scp ceph.conf root@controller162:/etc/ceph/
ceph.conf 100% 514 218.3KB/s 00:00
[root@ceph131 ceph]# scp ceph.conf root@compute163:/etc/ceph/
ceph.conf 100% 514 2.3KB/s 00:00
[root@ceph131 ceph]# scp ceph.conf root@compute164:/etc/ceph/
ceph.conf 100% 514 3.6KB/s 00:00
4 Glance集成Ceph
4.1 配置glance-api.conf
#在运行glance-api服务的节点修改glance-api.conf文件，含3个控制节点，以controller160节点为例 #以下只列出涉及glance集成ceph的相关 #vim /etc/glance/glance-api.conf
[DEFAULT]
#打开copy-on-write功能
show_image_direct_url = True
[glance_store]
stores = rbd
default_store = rbd
rbd_store_chunk_size = 8
rbd_store_pool = images
rbd_store_user = glance
rbd_store_ceph_conf = /etc/ceph/ceph.conf
#stores = file,http
#default_store = file
#filesystem_store_datadir = /var/lib/glance/images/
#变更配置文件，重启服务
systemctl restart openstack-glance-api.service
#上传cirros镜像
[root@controller160 ~]# glance image-create --name "rbd_cirros-05" \
--file cirros-0.4.0-x86_64-disk.img \
--disk-format qcow2 --container-format bare \
--visibility=public
+------------------+----------------------------------------------------------------------------------+
| Property | Value |
+------------------+----------------------------------------------------------------------------------+
| checksum | 443b7623e27ecf03dc9e01ee93f67afe |
| container_format | bare |
| created_at | 2020-07-06T15:25:40Z |
| direct_url | rbd://76235629-6feb-4f0c-a106-4be33d485535/images/f6da37cd-449a-436c-b321-e0c1c0 |
| | 6761d8/snap |
| disk_format | qcow2 |
| id | f6da37cd-449a-436c-b321-e0c1c06761d8 |
| min_disk | 0 |
| min_ram | 0 |
| name | rbd_cirros-05 |
| os_hash_algo | sha512 |
| os_hash_value | 6513f21e44aa3da349f248188a44bc304a3653a04122d8fb4535423c8e1d14cd6a153f735bb0982e |
| | 2161b5b5186106570c17a9e58b64dd39390617cd5a350f78 |
| os_hidden | False |
| owner | d3dda47e8c354d86b17085f9e382948b |
| protected | False |
| size | 12716032 |
| status | active |
| tags | [] |
| updated_at | 2020-07-06T15:26:06Z |
| virtual_size | Not available |
| visibility | public |
+------------------+----------------------------------------------------------------------------------+
#远程查看images pool有没有此文件ID，要能够在controller节点使用rbd命令，需要安装ceph-common
[root@controller161 ~]# rbd -p images --id glance -k /etc/ceph/ceph.client.glance.keyring ls
f6da37cd-449a-436c-b321-e0c1c06761d8
[root@ceph131 ceph]# ceph df
RAW STORAGE:
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 96 GiB 93 GiB 169 MiB 3.2 GiB 3.30
TOTAL 96 GiB 93 GiB 169 MiB 3.2 GiB 3.30
POOLS:
POOL ID STORED OBJECTS USED %USED MAX AVAIL
cephfs_data 1 0 B 0 0 B 0 29 GiB
cephfs_metadata 2 8.9 KiB 22 1.5 MiB 0 29 GiB
rbd_storage 3 33 MiB 20 100 MiB 0.11 29 GiB
.rgw.root 4 1.2 KiB 4 768 KiB 0 29 GiB
default.rgw.control 5 0 B 8 0 B 0 29 GiB
default.rgw.meta 6 369 B 2 384 KiB 0 29 GiB
default.rgw.log 7 0 B 207 0 B 0 29 GiB
volumes 10 0 B 0 0 B 0 29 GiB
vms 11 0 B 0 0 B 0 29 GiB
images 12 12 MiB 8 37 MiB 0.04 29 GiB
[root@ceph131 ceph]# rbd ls images
f6da37cd-449a-436c-b321-e0c1c06761d8
#查看ceph集群，发现有个HEALTH_WARN，原因是刚刚创建的未定义pool池类型，可定义为'cephfs', 'rbd', 'rgw'等
[root@ceph131 ceph]# ceph -s
cluster:
id: 76235629-6feb-4f0c-a106-4be33d485535
health: HEALTH_WARN
application not enabled on 1 pool(s)
services:
mon: 3 daemons, quorum ceph131,ceph132,ceph133 (age 3d)
mgr: ceph131(active, since 4d), standbys: ceph132, ceph133
mds: cephfs_storage:1 {0=ceph132=up:active}
osd: 3 osds: 3 up (since 3d), 3 in (since 4d)
rgw: 1 daemon active (ceph131)
task status:
scrub status:
mds.ceph132: idle
data:
pools: 10 pools, 224 pgs
objects: 271 objects, 51 MiB
usage: 3.2 GiB used, 93 GiB / 96 GiB avail
pgs: 224 active+clean
io:
client: 4.1 KiB/s rd, 0 B/s wr, 4 op/s rd, 2 op/s wr
[root@ceph131 ceph]# ceph health detail
HEALTH_WARN application not enabled on 1 pool(s)
POOL_APP_NOT_ENABLED application not enabled on 1 pool(s)
application not enabled on pool 'images'
use 'ceph osd pool application enable <pool-name> <app-name>', where <app-name> is 'cephfs', 'rbd', 'rgw', or freeform for custom applications.
#解决方法，定义pool类型为rbd
[root@ceph131 ceph]# ceph osd pool application enable images rbd
enabled application 'rbd' on pool 'images'
[root@ceph131 ceph]# ceph osd pool application enable volumes rbd
enabled application 'rbd' on pool 'volumes'
[root@ceph131 ceph]# ceph osd pool application enable vms rbd
enabled application 'rbd' on pool 'vms'
#验证方法如下：
[root@ceph131 ceph]# ceph health detail
HEALTH_OK
[root@ceph131 ceph]# ceph osd pool application get images
{
"rbd": {}
}
5 Cinder集成Ceph
4.1 配置cinder.conf
#cinder利用插件式结构，支持同时使用多种后端存储，在cinder-volume所在节点设置cinder.conf中设置相应的ceph rbd驱动即可，以compute163为例 #vim /etc/cinder/cinder.conf
# 后端使用ceph存储
[DEFAULT]
#enabled_backends = lvm #注释掉本行
enabled_backends = ceph
[ceph]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
rbd_pool = volumes
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_flatten_volume_from_snapshot = false
rbd_max_clone_depth = 5
rbd_store_chunk_size = 4
rados_connect_timeout = -1
glance_api_version = 2
rbd_user = cinder
#注意替换uuid
rbd_secret_uuid = cb26bb6c-2a84-45c2-8187-fa94b81dd53d
volume_backend_name = ceph
#变更配置文件，重启服务
[root@compute163 ceph]# systemctl restart openstack-cinder-volume.service
#验证
[root@controller160 ~]# openstack volume service list
+------------------+-----------------+------+---------+-------+----------------------------+
| Binary | Host | Zone | Status | State | Updated At |
+------------------+-----------------+------+---------+-------+----------------------------+
| cinder-scheduler | controller160 | nova | enabled | up | 2020-07-06T16:10:36.000000 |
| cinder-scheduler | controller162 | nova | enabled | up | 2020-07-06T16:10:39.000000 |
| cinder-scheduler | controller161 | nova | enabled | up | 2020-07-06T16:10:33.000000 |
| cinder-volume | compute163@lvm | nova | enabled | down | 2020-07-06T16:07:09.000000 |
| cinder-volume | compute164@lvm | nova | enabled | down | 2020-07-06T16:07:04.000000 |
| cinder-volume | compute164@ceph | nova | enabled | up | 2020-07-06T16:10:38.000000 |
| cinder-volume | compute163@ceph | nova | enabled | up | 2020-07-06T16:10:39.000000 |
+------------------+-----------------+------+---------+-------+----------------------------+
4.2 创建一个volume
#设置卷类型，在控制节点为cinder的ceph后端存储创建对应的type，在配置多存储后端时可区分类型；可通过“cinder type-list”查看
[root@controller160 ~]# cinder type-create ceph
+--------------------------------------+------+-------------+-----------+
| ID | Name | Description | Is_Public |
+--------------------------------------+------+-------------+-----------+
| bc90d094-a76b-409f-affa-f8329d2b54d5 | ceph | - | True |
+--------------------------------------+------+-------------+-----------+
#为ceph type设置扩展规格，键值” volume_backend_name”，value值”ceph”
[root@controller160 ~]# cinder type-key ceph set volume_backend_name=ceph
[root@controller160 ~]# cinder extra-specs-list
+--------------------------------------+-------------+---------------------------------+
| ID | Name | extra_specs |
+--------------------------------------+-------------+---------------------------------+
| 0aacd847-535a-447e-914c-895289bf1a19 | __DEFAULT__ | {} |
| bc90d094-a76b-409f-affa-f8329d2b54d5 | ceph | {'volume_backend_name': 'ceph'} |
+--------------------------------------+-------------+---------------------------------+
#创建一个volume
[root@controller160 ~]# cinder create --volume-type ceph --name ceph-volume 1
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2020-07-06T16:14:36.000000 |
| description | None |
| encrypted | False |
| group_id | None |
| id | 63cb956c-2e4f-434e-a21a-9280530f737e |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | ceph-volume |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | d3dda47e8c354d86b17085f9e382948b |
| provider_id | None |
| replication_status | None |
| service_uuid | None |
| shared_targets | True |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | None |
| user_id | ec8c820dba1046f6a9d940201cf8cb06 |
| volume_type | ceph |
+--------------------------------+--------------------------------------+
#验证
[root@controller160 ~]# openstack volume list
+--------------------------------------+-------------+-----------+------+-------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+-------------+-----------+------+-------------+
| 63cb956c-2e4f-434e-a21a-9280530f737e | ceph-volume | available | 1 | |
| 9575c54a-d44e-46dd-9187-0c464c512c01 | test1 | available | 2 | |
+--------------------------------------+-------------+-----------+------+-------------+
[root@ceph131 ceph]# rbd ls volumes
volume-63cb956c-2e4f-434e-a21a-9280530f737e
6 Nova集成Ceph
6.1 配置ceph.conf
#如果需要从ceph rbd中启动虚拟机，必须将ceph配置为nova的临时后端； #推荐在计算节点的配置文件中启用rbd cache功能； #为了便于故障排查，配置admin socket参数，这样每个使用ceph rbd的虚拟机都有1个socket将有利于虚拟机性能分析与故障解决； #相关配置只涉及全部计算节点ceph.conf文件的[client]与[client.cinder]字段，以compute163节点为例
[root@compute163 ~]# vim /etc/ceph/ceph.conf
[client]
rbd cache = true
rbd cache writethrough until flush = true
admin socket = /var/run/ceph/guests/$cluster-$type.$id.$pid.$cctid.asok
log file = /var/log/qemu/qemu-guest-$pid.log
rbd concurrent management ops = 20
[client.cinder]
keyring = /etc/ceph/ceph.client.cinder.keyring
# 创建ceph.conf文件中指定的socker与log相关的目录，并更改属主
[root@compute163 ~]# mkdir -p /var/run/ceph/guests/ /var/log/qemu/
[root@compute163 ~]# chown qemu:libvirt /var/run/ceph/guests/ /var/log/qemu/
6.2 配置nova.conf
#在全部计算节点配置nova后端使用ceph集群的vms池，以compute163节点为例
[root@compute01 ~]# vim /etc/nova/nova.conf
[DEFAULT]
vif_plugging_is_fatal = False
vif_plugging_timeout = 0
[libvirt]
images_type = rbd
images_rbd_pool = vms
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = cinder
rbd_secret_uuid = cb26bb6c-2a84-45c2-8187-fa94b81dd53d #uuid前后一致
disk_cachemodes="network=writeback"
live_migration_flag="VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_PERSIST_DEST,VIR_MIGRATE_TUNNELLED"
# 禁用文件注入
inject_password = false
inject_key = false
inject_partition = -2
# 虚拟机临时root磁盘discard功能，”unmap”参数在scsi接口类型磁盘释放后可立即释放空间
hw_disk_discard = unmap
#变更配置文件，重启计算服务
[root@compute163 ~]# systemctl restart libvirtd.service openstack-nova-compute.service
[root@compute163 ~]# systemctl status libvirtd.service openstack-nova-compute.service
6.3 配置live-migration
6.3.1 修改/etc/libvirt/libvirtd.conf
#在全部计算节点操作，以compute163节点为例； #以下给出libvirtd.conf文件的修改处所在的行num
[root@compute163 ~]# egrep -vn "^$|^#" /etc/libvirt/libvirtd.conf
# 取消以下三行的注释
22:listen_tls = 0
33:listen_tcp = 1
45:tcp_port = "16509"
# 取消注释，并修改监听端口
55:listen_addr = "172.16.1.163"
# 取消注释，同时取消认证
158:auth_tcp = "none"
6.3.2 修改/etc/sysconfig/libvirtd
#在全部计算节点操作，以compute163节点为例； #以下给出libvirtd文件的修改处所在的行num
[root@compute163 ~]# egrep -vn "^$|^#" /etc/sysconfig/libvirtd
# 取消注释
9:LIBVIRTD_ARGS="--listen"
6.3.3 计算节点设置免密访问
#所有计算节点都必须设置免密，迁移必备！
[root@compute163 ~]# usermod -s /bin/bash nova
[root@compute163 ~]# passwd nova
Changing password for user nova.
New password:
BAD PASSWORD: The password contains the user name in some form
Retype new password:
passwd: all authentication tokens updated successfully.
[root@compute163 ~]# su - nova
[nova@compute163 ~]$ ssh-keygen -t rsa -N '' -f ~/.ssh/id_rsa
Generating public/private rsa key pair.
Your identification has been saved in /var/lib/nova/.ssh/id_rsa.
Your public key has been saved in /var/lib/nova/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:bnGCcG6eRvSG3Bb58eu+sXwEAnb72hUHmlNja2bQLBU nova@compute163
The key's randomart image is:
+---[RSA 3072]----+
| +E. |
| o.. o B |
| . o.oo.. B + |
| * = ooo= * .|
| B S oo.* o |
| + = + ..o |
| + o ooo |
| . . .o.o. |
| .*o |
+----[SHA256]-----+
[nova@compute163 ~]$ ssh-copy-id nova@compute164
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/var/lib/nova/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
nova@compute164's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'nova@compute164'"
and check to make sure that only the key(s) you wanted were added.
[nova@compute163 ~]$ ssh nova@172.16.1.164
Activate the web console with: systemctl enable --now cockpit.socket
Last failed login: Wed Jul 8 00:24:02 CST 2020 from 172.16.1.163 on ssh:notty
There were 5 failed login attempts since the last successful login.
[nova@compute164 ~]$
6.3.4 设置iptables - ==前面已经关闭了iptables，因此不用设置==
#live-migration时，源计算节点主动连接目的计算节点tcp16509端口，可以使用”virsh -c qemu+tcp://{node_ip or node_name}/system”连接目的计算节点测试； #迁移前后，在源目计算节点上的被迁移instance使用tcp49152~49161端口做临时通信； #因虚拟机已经启用iptables相关规则，此时切忌随意重启iptables服务，尽量使用插入的方式添加规则； #同时以修改配置文件的方式写入相关规则，切忌使用”iptables saved”命令； #在全部计算节点操作，以compute163节点为例
[root@compute163 ~]# iptables -I INPUT -p tcp -m state --state NEW -m tcp --dport 16509 -j ACCEPT
[root@compute163 ~]# iptables -I INPUT -p tcp -m state --state NEW -m tcp --dport 49152:49161 -j ACCEPT
6.3.5 重启服务
systemctl mask libvirtd.socket libvirtd-ro.socket \
libvirtd-admin.socket libvirtd-tls.socket libvirtd-tcp.socket
service libvirtd restart
systemctl restart openstack-nova-compute.service
#验证
[root@compute163 ~]# netstat -lantp|grep libvirtd
tcp 0 0 172.16.1.163:16509 0.0.0.0:* LISTEN 582582/libvirtd
6.4 验证是否集成
6.4.1创建基于ceph存储的bootable存储卷
#当nova从rbd启动instance时，镜像格式必须是raw格式，否则虚拟机在启动时glance-api与cinder均会报错； #首先进行格式转换，将*.img文件转换为*.raw文件 #cirros-0.4.0-x86_64-disk.img 这个文件网上自己下载
[root@controller160 ~]# qemu-img convert -f qcow2 -O raw ~/cirros-0.4.0-x86_64-disk.img ~/cirros-0.4.0-x86_64-disk.raw
# 生成raw格式镜像
[root@controller160 ~]# openstack image create "cirros-raw" \
> --file ~/cirros-0.4.0-x86_64-disk.raw \
> --disk-format raw --container-format bare \
> --public
+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| checksum | ba3cd24377dde5dfdd58728894004abb |
| container_format | bare |
| created_at | 2020-07-07T02:13:06Z |
| disk_format | raw |
| file | /v2/images/459f5ddd-c094-4b0f-86e5-f55baa33595c/file |
| id | 459f5ddd-c094-4b0f-86e5-f55baa33595c |
| min_disk | 0 |
| min_ram | 0 |
| name | cirros-raw |
| owner | d3dda47e8c354d86b17085f9e382948b |
| properties | direct_url='rbd://76235629-6feb-4f0c-a106-4be33d485535/images/459f5ddd-c094-4b0f-86e5-f55baa33595c/snap', os_hash_algo='sha512', os_hash_value='b795f047a1b10ba0b7c95b43b2a481a59289dc4cf2e49845e60b194a911819d3ada03767bbba4143b44c93fd7f66c96c5a621e28dff51d1196dae64974ce240e', os_hidden='False', owner_specified.openstack.md5='ba3cd24377dde5dfdd58728894004abb', owner_specified.openstack.object='images/cirros-raw', owner_specified.openstack.sha256='87ddf8eea6504b5eb849e418a568c4985d3cea59b5a5d069e1dc644de676b4ec', self='/v2/images/459f5ddd-c094-4b0f-86e5-f55baa33595c' |
| protected | False |
| schema | /v2/schemas/image |
| size | 46137344 |
| status | active |
| tags | |
| updated_at | 2020-07-07T02:13:42Z |
| visibility | public |
+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
#使用新镜像创建bootable卷
[root@controller160 ~]# cinder create --image-id 459f5ddd-c094-4b0f-86e5-f55baa33595c --volume-type ceph --name ceph-boot 1
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2020-07-07T02:17:42.000000 |
| description | None |
| encrypted | False |
| group_id | None |
| id | 46a45564-e148-4f85-911b-a4542bdbd4f0 |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | ceph-boot |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | d3dda47e8c354d86b17085f9e382948b |
| provider_id | None |
| replication_status | None |
| service_uuid | None |
| shared_targets | True |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | None |
| user_id | ec8c820dba1046f6a9d940201cf8cb06 |
| volume_type | ceph |
+--------------------------------+--------------------------------------+
#查看新创建的bootable卷，创建需要时间，过会查status才会是available
[root@controller160 ~]# cinder list
+--------------------------------------+-----------+-------------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-------------+------+-------------+----------+-------------+
| 62a21adb-0a22-4439-bf0b-121442790515 | available | ceph-boot | 1 | ceph | true | |
| 63cb956c-2e4f-434e-a21a-9280530f737e | available | ceph-volume | 1 | ceph | false | |
| 9575c54a-d44e-46dd-9187-0c464c512c01 | available | test1 | 2 | __DEFAULT__ | false | |
+--------------------------------------+-----------+-------------+------+-------------+----------+-------------+
#从基于ceph后端的volumes新建实例； #“--boot-volume”指定具有”bootable”属性的卷，启动后，虚拟机运行在volumes卷
#创建云主机类型
[root@controller160 ~]# openstack flavor create --id 1 --vcpus 1 --ram 256 --disk 1 m1.nano
+----------------------------+---------+
| Field | Value |
+----------------------------+---------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| disk | 1 |
| id | 1 |
| name | m1.nano |
| os-flavor-access:is_public | True |
| properties | |
| ram | 256 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 1 |
+----------------------------+---------+
#安全规则
[root@controller160 ~]# openstack security group rule create --proto icmp default
[root@controller160 ~]# openstack security group rule create --proto tcp --dst-port 22 'default'
#创建虚拟网络
openstack network create --share --external \
--provider-physical-network provider \
--provider-network-type flat provider-eth1
#创建子网，根据实际进行修改
openstack subnet create --network provider-eth1 \
--allocation-pool start=172.16.2.220,end=172.16.2.229 \
--dns-nameserver 114.114.114.114 --gateway 172.16.2.254 --subnet-range 172.16.2.0/24 \
172.16.2.0/24
#云主机可用类型
openstack flavor list
#可用镜像
openstack image list
#可用的安全组
openstack security group list
#可用的网络
openstack network list
#创建实例，也可以通过web创建
[root@controller160 ~]# nova boot --flavor m1.nano \
--boot-volume d3770a82-068c-49ad-a9b7-ef863bb61a5b \
--nic net-id=53b98327-3a47-4316-be56-cba37e8f20f2 \
--security-group default \
ceph-boot02
[root@controller160 ~]# nova show c592ca1a-dbce-443a-9222-c7e47e245725
+--------------------------------------+----------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | compute163 |
| OS-EXT-SRV-ATTR:hostname | ceph-boot02 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute163 |
| OS-EXT-SRV-ATTR:instance_name | instance-00000025 |
| OS-EXT-SRV-ATTR:kernel_id | |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | |
| OS-EXT-SRV-ATTR:reservation_id | r-ty5v9w8n |
| OS-EXT-SRV-ATTR:root_device_name | /dev/vda |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2020-07-07T10:55:39.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2020-07-07T10:53:30Z |
| description | - |
| flavor:disk | 1 |
| flavor:ephemeral | 0 |
| flavor:extra_specs | {} |
| flavor:original_name | m1.nano |
| flavor:ram | 256 |
| flavor:swap | 0 |
| flavor:vcpus | 1 |
| hostId | 308132ea4792b277acfae8d3c5d88439d3d5d6ba43d8b06395581d77 |
| host_status | UP |
| id | c592ca1a-dbce-443a-9222-c7e47e245725 |
| image | Attempt to boot from volume - no image supplied |
| key_name | - |
| locked | False |
| locked_reason | - |
| metadata | {} |
| name | ceph-boot02 |
| os-extended-volumes:volumes_attached | [{"id": "d3770a82-068c-49ad-a9b7-ef863bb61a5b", "delete_on_termination": false}] |
| progress | 0 |
| provider-eth1 network | 172.16.2.228 |
| security_groups | default |
| server_groups | [] |
| status | ACTIVE |
| tags | [] |
| tenant_id | d3dda47e8c354d86b17085f9e382948b |
| trusted_image_certificates | - |
| updated | 2020-07-07T10:55:40Z |
| user_id | ec8c820dba1046f6a9d940201cf8cb06 |
+--------------------------------------+----------------------------------------------------------------------------------+
6.4.2 从ceph rbd启动虚拟机
#--nic：net-id指网络id，非subnet-id；
#最后“cirros-cephrbd-instance1”为instance名称
[root@controller160 ~]# openstack server create --flavor m1.nano --image cirros-raw --nic net-id=53b98327-3a47-4316-be56-cba37e8f20f2 --security-group default cirros-cephrbd-instance1
+-------------------------------------+---------------------------------------------------+
| Field | Value |
+-------------------------------------+---------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | None |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None |
| OS-EXT-SRV-ATTR:instance_name | |
| OS-EXT-STS:power_state | NOSTATE |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | None |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | |
| adminPass | CRiNuZoK6ftt |
| config_drive | |
| created | 2020-07-07T15:13:08Z |
| flavor | m1.nano (1) |
| hostId | |
| id | 6ea79ec0-1ec6-47ff-b185-233c565b1fab |
| image | cirros-raw (459f5ddd-c094-4b0f-86e5-f55baa33595c) |
| key_name | None |
| name | cirros-cephrbd-instance1 |
| progress | 0 |
| project_id | d3dda47e8c354d86b17085f9e382948b |
| properties | |
| security_groups | name='eea8a6b4-2b6d-4f11-bfe8-12b56bafe36c' |
| status | BUILD |
| updated | 2020-07-07T15:13:10Z |
| user_id | ec8c820dba1046f6a9d940201cf8cb06 |
| volumes_attached | |
+-------------------------------------+---------------------------------------------------+
#查询生成的instance
[root@controller160 ~]# nova list
+--------------------------------------+--------------------------+--------+------------+-------------+----------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------------------------+--------+------------+-------------+----------------------------+
| c592ca1a-dbce-443a-9222-c7e47e245725 | ceph-boot02 | ACTIVE | - | Running | provider-eth1=172.16.2.228 |
| 6ea79ec0-1ec6-47ff-b185-233c565b1fab | cirros-cephrbd-instance1 | ACTIVE | - | Running | provider-eth1=172.16.2.226 |
+--------------------------------------+--------------------------+--------+------------+-------------+----------------------------+
6.4.3 对rbd启动的虚拟机进行live-migration
#使用“nova show 6ea79ec0-1ec6-47ff-b185-233c565b1fab”得知从rbd启动的instance在迁移前位于compute163节点；
#或使用”nova hypervisor-servers compute163”进行验证；
[root@controller01 ~]# nova live-migration cirros-cephrbd-instance1 compute164
# 迁移过程中可查看状态
[root@controller160 ~]# nova list
+--------------------------------------+--------------------------+-----------+------------+-------------+----------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------------------------+-----------+------------+-------------+----------------------------+
| c592ca1a-dbce-443a-9222-c7e47e245725 | ceph-boot02 | ACTIVE | - | Running | provider-eth1=172.16.2.228 |
| 6ea79ec0-1ec6-47ff-b185-233c565b1fab | cirros-cephrbd-instance1 | MIGRATING | migrating | Running | provider-eth1=172.16.2.226 |
+--------------------------------------+--------------------------+-----------+------------+-------------+----------------------------+
# 迁移完成后，查看instacn所在节点；
# 或使用“nova show 6ea79ec0-1ec6-47ff-b185-233c565b1fab”命令查看”hypervisor_hostname”
[root@controller01 ~]# nova hypervisor-servers compute163
[root@controller01 ~]# nova hypervisor-servers compute164
X.过程中遇到的问题
eg1.2020-07-04 00:39:56.394 671959 ERROR glance.common.wsgi rados.ObjectNotFound: [errno 2] error calling conf_read_file
原因是：找不到ceph.conf配置文件
解决方案：从ceph集群复制ceph.conf配置至各节点/etc/ceph/里面
eg2.2020-07-04 01:01:27.736 1882718 ERROR glance_store._drivers.rbd [req-fd768a6d-e7e2-476b-b1d3-d405d7a560f2 ec8c820dba1046f6a9d940201cf8cb06 d3dda47e8c354d86b17085f9e382948b - default default] Error con
necting to ceph cluster.: rados.ObjectNotFound: [errno 2] error connecting to the cluster
eg3.libvirtd[580770]: --listen parameter not permitted with systemd activation sockets, see 'man libvirtd' for further guidance
原因是：默认使用了systemd模式，要恢复到传统模式，所有的systemd必须被屏蔽
解决方案：
systemctl mask libvirtd.socket libvirtd-ro.socket \
libvirtd-admin.socket libvirtd-tls.socket libvirtd-tcp.socket
然后使用以下命令重启即可：
service libvirtd restart
eg4.AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/guests/ceph-client.cinder.596406.94105140863224.asok': (13) Permission denied
原因是：这个是因为/var/run/ceph目录权限有问题，qemu起的这些虚拟机示例，其属主属组都是qemu，但是/var/run/ceph目录的属主属组是ceph:ceph，权限是770
解决方案：直接将/var/run/ceph目录的权限改为777，另外，/var/log/qemu/也最好设置一下权限，设置为777
eg5.Error on AMQP connection <0.9284.0> (172.16.1.162:55008 -> 172.16.1.162:5672, state: starting):
AMQPLAIN login refused: user 'guest' can only connect via localhost
原因是：从3版本开始不支持guest远程登陆
解决方案：vim /etc/rabbitmq/rabbitmq.config添加以下字段，有点！！然后重启rabbitmq服务
[{rabbit, [{loopback_users, []}]}].
eg6.Failed to allocate network(s): nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed
在计算节点的/etc/nova/nova.conf中添加下面两句，然后重启
vif_plugging_is_fatal = False
vif_plugging_timeout = 0
eg7.Resize error: not able to execute ssh command: Unexpected error while running command.
原因是：openstack迁移是以ssh为基础，因此需要计算节点间都配置免密
解决方案：见6.3.3
