Kubernetes备份恢复工具velero
Velero 是一款云原生时代的灾难恢复和迁移工具，采用 Go 语言编写，并在 github 上进行了开源，开源地址为：github.com/vmware-tanz… 源于西班牙语，意思为帆船，非常符合 Kubernetes 社区的命名风格。
利用 velero 用户可以安全的备份、恢复和迁移 Kubernetes 集群资源和持久卷。它的基本原理就是将集群的数据，例如集群资源和持久化数据卷备份到对象存储中，在恢复的时候将数据从对象存储中拉取下来。除了灾备之外它还能做资源移转，支持把容器应用从一个集群迁移到另一个集群，这也是 velero 一个非常成功的使用场景。
Velero 主要包括两个核心组件，分别为服务端和客户端。服务端运行在具体的 Kubernetes 集群中，客户端是运行在本地的命令行工具，只要配置好 kubectl 及 kubeconfig 即可使用，非常简单。
Velero 基于其实现的 kubernetes 资源备份能力，可以轻松实现 Kubernetes 集群的数据备份和恢复、复制 kubernetes 集群资源到其他 kubernetes 集群或者快速复制生产环境到测试环境等功能。
在资源备份方面，velero 支持将数据备份到众多的云存储中，例如AWS S3或S3兼容的存储系统、Azure Blob、Google Cloud存储、Aliyun OSS等。与备份整个 kubernetes 的数据存储引擎 etcd 相比，velero 的控制更加细化，可以对 Kubernetes 集群内对象级别进行备份，还可以通过对 Type、Namespace、Label 等对象进行分类备份或者恢复。
Velero与etcd快照备份的区别
- etcd 快照是全局备份，在即使一个资源对象需要恢复，也需要做全局恢复到备份的状态，即会影响其它namespace中pod运行服务。
- Velero可以有针对性的备份，比如按照namespace单独备份、只备份单独的资源对象等，在恢复的时候只恢复单独的namespace或资源对象，而不影响其它namespace中pod运行服务。
- velero支持ceph、oss等对象存储，etcd 快照是一个为本地文件。
- velero支持任务计划实现周期备份，但etcd 快照也可以基于cronjob实现。
Velero工作流程
流程图：
以核心的数据备份为例，当执行velero backup create my-backup时：
-
Velero 客户端首先调用 Kubernetes API 服务器以创建 Backup 对象
-
BackupController 将收到通知有新的 Backup 对象被创建并执行验证
-
BackupController 开始备份过程，它通过查询 API 服务器以获取资源来收集数据以进行备份
-
BackupController 将调用对象存储服务，例如，AWS S3 -上传备份文件。默认情况下，velero backup create支持任何持久卷的磁盘快照，可以通过指定其他标志来调整快照，运行velero backup create --help可以查看可用的标志，也可以使用--snapshot-volumes=false选项禁用快照
Velero恢复流程
- Velero客户端调用Kubernetes API服务器创建一个Restore对象
- RestoreController注意到新的Restore对象并执行验证
- RestoreController从对象存储服务(兼容openstack swift接口)获取备份信息。然后，它在备份的资源上运行一些预处理，以确保这些资源可以在新的集群上工作。例如，使用备份的API版本来验证恢复资源是否可以在目标集群上工作
- RestoreController启动恢复过程，每次恢复一个符合条件的资源
Velero特性
Velero 目前包含以下特性：
- 支持 Kubernetes 集群数据备份和恢复
- 支持复制当前 Kubernetes 集群的资源到其它 Kubernetes 集群
- 支持复制生产环境到开发以及测试环境
Velero组件
Velero 组件一共分两部分，分别是服务端和客户端。
- 服务端：运行在你 Kubernetes 的集群中
- 客户端：是一些运行在本地的命令行的工具，需要已配置好 kubectl 及集群 kubeconfig 的机器上
minio安装
添加minio helm 仓库
helm repo add minio https://helm.min.io/ (官方的)
helm repo add bitnami https://charts.bitnami.com/bitnami
拉取minio 的chart 包, 解压并配置
helm pull bitnami/minio (个人喜欢用bitnami版本的minio)
tar -xvf minio-12.1.7.tgz
[root@k8s-cka-master minio]# ls
Chart.lock charts Chart.yaml ci README.md templates values.yaml
[root@k8s-cka-master minio]# kubectl get sc (查看自己的存储类)
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
managed-nfs-storage fuseim.pri/ifs Delete Immediate false 30d
[root@k8s-cka-master minio]# vim values.yaml
## @section Global parameters
## Global Docker image parameters
## Please, note that this will override the image parameters, including dependencies, configured to use the global value
## Current available global Docker image parameters: imageRegistry, imagePullSecrets and storageClass
## @param global.imageRegistry Global Docker image registry
## @param global.imagePullSecrets Global Docker registry secret names as an array
## @param global.storageClass Global StorageClass for Persistent Volume(s)
##
global:
imageRegistry: ""
## e.g.
## imagePullSecrets:
## - myRegistryKeySecretName
##
imagePullSecrets: []
storageClass: "managed-nfs-storage" # 添加存储类
mode: standalone
## MinIO® authentication parameters
##
auth:
## @param auth.rootUser MinIO® root username
##
rootUser: admin
## @param auth.rootPassword Password for MinIO® root user
##
rootPassword: "abc123456" # 添加登录密码 这里有个坑密码长度必须大于8 然后就是报错 minio pod会一直重启
## @param auth.existingSecret Use existing secret for credentials details (`auth.rootUser` and `auth.rootPassword` will be ignored and picked up from this secret). The secret has to contain the keys `root-user` and `root-password`)
##
existingSecret: ""
安装 helm 安装 minio
[root@k8s-cka-master minio]# helm install minio .
NAME: minio
LAST DEPLOYED: Fri Mar 24 10:26:04 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: minio
CHART VERSION: 12.1.7
APP VERSION: 2023.2.17
** Please be patient while the chart is being deployed **
MinIO® can be accessed via port on the following DNS name from within your cluster:
minio.default.svc.cluster.local
To get your credentials run:
export ROOT_USER=$(kubectl get secret --namespace default minio -o jsonpath="{.data.root-user}" | base64 -d)
export ROOT_PASSWORD=$(kubectl get secret --namespace default minio -o jsonpath="{.data.root-password}" | base64 -d)
To connect to your MinIO® server using a client:
- Run a MinIO® Client pod and append the desired command (e.g. 'admin info'):
kubectl run --namespace default minio-client \
--rm --tty -i --restart='Never' \
--env MINIO_SERVER_ROOT_USER=$ROOT_USER \
--env MINIO_SERVER_ROOT_PASSWORD=$ROOT_PASSWORD \
--env MINIO_SERVER_HOST=minio \
--image docker.io/bitnami/minio-client:2023.1.28-debian-11-r6 -- admin info minio
To access the MinIO® web UI:
- Get the MinIO® URL:
echo "MinIO® web URL: http://127.0.0.1:9001/minio"
kubectl port-forward --namespace default svc/minio 9001:9001
更改 serivce 访问 minio 9001对应的端口
[root@k8s-cka-master minio]# kubectl get pod
NAME READY STATUS RESTARTS AGE
minio-5847b9b778-kjbr2 1/1 Running 0 68s
nfs-client-provisioner-6f5b9ddf9f-c8h8w 1/1 Running 8 30d
[root@k8s-cka-master minio]# kubectl edit svc minio
service/minio edited
[root@k8s-cka-master minio]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 32d
kustomize-guestbook-ui ClusterIP 10.100.208.233 <none> 80/TCP 31d
minio NodePort 10.105.90.13 <none> 9000:31371/TCP,9001:32766/TCP 98s
nginx ClusterIP 10.105.137.255 <none> 80/TCP 24d
velero安装
添加helm源
helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts
helm pull vmware-tanzu/velero
修改相关配置
# 开启初始化容器 awk相关插件
initContainers:
# - name: velero-plugin-for-csi
# image: velero/velero-plugin-for-csi:v0.4.2
# imagePullPolicy: IfNotPresent
# volumeMounts:
# - mountPath: /target
# name: plugins
- name: velero-plugin-for-aws
image: velero/velero-plugin-for-aws:v1.6.1
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: /target
name: plugins
# backupStorageLocation 配置
configuration:
# Cloud provider being used (e.g. aws, azure, gcp).
provider: aws
# Parameters for the `default` BackupStorageLocation. See
# https://velero.io/docs/v1.6/api-types/backupstoragelocation/
backupStorageLocation:
# name is the name of the backup storage location where backups should be stored. If a name is not provided,
# a backup storage location will be created with the name "default". Optional.
name: default
# provider is the name for the backup storage location provider. If omitted
# `configuration.provider` will be used instead.
provider: aws
# bucket is the name of the bucket to store backups in. Required.
bucket: test-date
# caCert defines a base64 encoded CA bundle to use when verifying TLS connections to the provider. Optional.
caCert:
# prefix is the directory under which all Velero data should be stored within the bucket. Optional.
prefix:
# default indicates this location is the default backup storage location. Optional.
default:
# accessMode determines if velero can write to this backup storage location. Optional.
# default to ReadWrite, ReadOnly is used during migrations and restores.
accessMode: ReadWrite
# Additional provider-specific configuration. See link above
# for details of required/optional fields for your provider.
config:
region: minio
s3ForcePathStyle: true
s3Url: http://ip:9000
volumeSnapshotLocation:
# name is the name of the volume snapshot location where snapshots are being taken. Required.
name: default
# provider is the name for the volume snapshot provider. If omitted
# `configuration.provider` will be used instead.
provider: aws
# Additional provider-specific configuration. See link above
# for details of required/optional fields for your provider.
config:
region: minio
# velero连接 minio 相关认证信息
credentials:
# Whether a secret should be used. Set to false if, for examples:
# - using kube2iam or kiam to provide AWS IAM credentials instead of providing the key file. (AWS only)
# - using workload identity instead of providing the key file. (GCP only)
useSecret: true
# Name of the secret to create if `useSecret` is true and `existingSecret` is empty
name:
# Name of a pre-existing secret (if any) in the Velero namespace
# that should be used to get IAM account credentials. Optional.
existingSecret:
# Data to be stored in the Velero secret, if `useSecret` is true and `existingSecret` is empty.
# As of the current Velero release, Velero only uses one secret key/value at a time.
# The key must be named `cloud`, and the value corresponds to the entire content of your IAM credentials file.
# Note that the format will be different for different providers, please check their documentation.
# Here is a list of documentation for plugins maintained by the Velero team:
# [AWS] https://github.com/vmware-tanzu/velero-plugin-for-aws/blob/main/README.md
# [GCP] https://github.com/vmware-tanzu/velero-plugin-for-gcp/blob/main/README.md
# [Azure] https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/blob/main/README.md
secretContents:
cloud: |
[default]
aws_access_key_id=admin
aws_secret_access_key=admin123
# Velero ConfigMaps.
# Eg:
configMaps:
fs-restore-action-config:
labels:
velero.io/plugin-config: ""
velero.io/pod-volume-restore: RestoreItemAction
data:
image: velero/velero-restore-helper:v1.10.2
安装velero 二进制工具
[root@k8s-master velero]# wget https://github.com/vmware-tanzu/velero/releases/download/v1.10.2/velero-v1.10.2-linux-amd64.tar.gz
velero 常用命令
# 基于名称空间级别备份
[root@k8s-master ~]# velero backup create backup-ns-kube-system --include-namespaces kube-system --namespace default
Backup request "backup-ns-kube-system" submitted successfully.
Run `velero backup describe backup-ns-kube-system` or `velero backup logs backup-ns-kube-system` for more details.
# 基于资源级别备份 例如：备份 default 名称空间下的deployment/nginx
[root@k8s-master ~]# velero backup create backup-deployment-nginx --include-cluster-resources=true --ordered-resources 'deployments=default/nginx' --namespace default --snapshot-volumes=false
Backup request "backup-deployment-nginx" submitted successfully.
Run `velero backup describe backup-deployment-nginx` or `velero backup logs backup-deployment-nginx` for more details.
# 查看备份情况
[root@k8s-master ~]# velero backup get -n default
