这是我参与更文挑战的第7天，活动详情查看： 更文挑战
1、背景
当Kubernetes容器编排管理工具已成为企业首选时，各个公有云厂商的云Kubernetes凭借在使用成本、运维成本、方便性、长期稳定性上大大超过了用户自建自维护Kubernets集群，有不少用户纷纷想把之前自己维护Kubernetes负载迁移到公有云Kubernetes服务上。
在迁移过程中，当镜像个数较少时，可以通过docker pull或docker push命令完成镜像迁移，如果涉及到成千上百个镜像，甚至几T的镜像仓库数据时，迁移过程就变得非常漫长，并且可能丢失数据。此时，用户在各种容器镜像仓库之间迁移时，期望有镜像同步复制的能力。
2、image-syncer
2.1 简介
一个简单、易用的批量镜像迁移和镜像同步复制工具，支持几乎所有目前主流的基于Docker Registry V2搭建的镜像存储服务，例如ACR、Docker、Hub、Quay、自建Harbor等，目前已经初步经过了TB级别的生产环境镜像迁移验证
2.2 安装
wget https://github.com/AliyunContainerService/image-syncer/releases/download/v1.0.3/image-syncer-v1.0.3-linux-amd64.tar.gz
tar zxf image-syncer-v1.0.3-linux-amd64.tar.gz && mv image-syncer /usr/local/bin/
3、使用
3.1 Harbor To Aliyun ACR
配置文件模板
{
"auth": { // 认证字段，其中每个对象为一个registry的一个账号和
// 密码；通常，同步源需要具有pull以及访问tags权限，
// 同步目标需要拥有push以及创建仓库权限，如果没有提供，则默认匿名访问
"quay.io": { // registry的url，需要和下面images中对应registry的url相同
"username": "xxx", // 用户名，可选
"password": "xxxxxxxxx", // 密码，可选
"insecure": true // registry是否是http服务，如果是，insecure字段需要为true
},
"registry.cn-beijing.aliyuncs.com": {
"username": "xxx",
"password": "xxxxxxxxx"
},
"registry.hub.docker.com": {
"username": "xxx",
"password": "xxxxxxxxxx"
}
},
images: {
"quay.io/coreos/kube-rbac-proxy": "quay.io/ruohe/kube-rbac-proxy",
"xxxx":"xxxxx",
"xxx/xxx/xx:tag1,tag2,tag3":"xxx/xxx/xx"
}
}
1、同步镜像规则字段，其中一条规则包括一个源仓库（键）和一个目标仓库（值）
2、同步的最大单位是仓库（repo），不支持通过一条规则同步整个namespace以及registry
3、源仓库和目标仓库的格式与docker pull/push命令使用的镜像url类似（registry/namespace/repository:tag）
4、源仓库和目标仓库（如果目标仓库不为空字符串）都至少包含registry/namespace/repository
5、源仓库字段不能为空，如果需要将一个源仓库同步到多个目标仓库需要配置多条规则
6、目标仓库名可以和源仓库名不同（tag也可以不同），此时同步功能类似于：docker pull + docker tag + docker push
7、当源仓库字段中不包含tag时，表示将该仓库所有tag同步到目标仓库，此时目标仓库不能包含tag 8、当源仓库字段中包含tag时，表示只同步源仓库中的一个tag到目标仓库，如果目标仓库中不包含tag，则默认使用源tag 9、源仓库字段中的tag可以同时包含多个（比如"a/b/c:1,2,3"），tag之间通过","隔开，此时目标仓库不能包含tag，并且默认使用原来的tag
10、当目标仓库为空字符串时，会将源镜像同步到默认registry的默认namespace下，并且repo以及tag与源仓库相同，默认registry和默认namespace可以通过命令行参数以及环境变量配置，参考下面的描述
创建配置文件
{
"auth": {
"172.20.5.10": {
"username": "admin",
"password": "Xxzx@789",
"insecure": true
},
"registry.cn-beijing.aliyuncs.com": {
"username": "352@qq.com",
"password": "docker"
}
},
"images": {
"172.20.5.10/library/kube-apiserver": ""
}
}
开始迁移
# 并发数为10，重试次数为10
# 日志输出到./log文件下，不存在会自动创建，不指定的话默认会将日志打印到Stderr
# 指定配置文件为harbor-to-acr.json，内容如上所述
image-syncer --proc=10 --config=./harbor_to_acr.json --registry=registry.cn-beijing.aliyuncs.com --namespace=cloud-devops --retries=10 --log=./log
Start to generate sync tasks, please wait ...
Start to handle sync tasks, please wait ...
Finished, 0 sync tasks failed, 0 tasks generate failed
日志
{"level":"info","msg":"Get tags of 172.20.5.10/library/kube-apiserver successfully: [v1.20.1]","time":"2021-01-08 10:54:23"}
{"level":"info","msg":"Generate a task for 172.20.5.10/library/kube-apiserver:v1.20.1 to registry.cn-beijing.aliyuncs.com/cloud-devops/kube-apiserver:v1.20.1","time":"2021-01-08 10:54:23"}
{"level":"info","msg":"Get manifest from 172.20.5.10/library/kube-apiserver:v1.20.1","time":"2021-01-08 10:54:23"}
{"level":"info","msg":"Get a blob sha256:f398b465657ed53ee83af22197ef61be9daec6af791c559ee5220dee5f3d94fe(647423) from 172.20.5.10/library/kube-apiserver:v1.20.1 success","time":"2021-01-08 10:54:23"}
{"level":"info","msg":"Put blob sha256:f398b465657ed53ee83af22197ef61be9daec6af791c559ee5220dee5f3d94fe(647423) to registry.cn-beijing.aliyuncs.com/cloud-devops/kube-apiserver:v1.20.1 success","time":"2021-01-08 10:54:24"}
{"level":"info","msg":"Get a blob sha256:cbcdf8ef32b41cd954f25c9d85dee61b05acc3b20ffa8620596ed66ee6f1ae1d(723022) from 172.20.5.10/library/kube-apiserver:v1.20.1 success","time":"2021-01-08 10:54:24"}
{"level":"info","msg":"Put blob sha256:cbcdf8ef32b41cd954f25c9d85dee61b05acc3b20ffa8620596ed66ee6f1ae1d(723022) to registry.cn-beijing.aliyuncs.com/cloud-devops/kube-apiserver:v1.20.1 success","time":"2021-01-08 10:54:24"}
{"level":"info","msg":"Get a blob sha256:d7d21f5bdd8303a60bac834f99867a58e6f3e1abcb6d486158a1ccb67dbf85bf(29033259) from 172.20.5.10/library/kube-apiserver:v1.20.1 success","time":"2021-01-08 10:54:24"}
{"level":"info","msg":"Put blob sha256:d7d21f5bdd8303a60bac834f99867a58e6f3e1abcb6d486158a1ccb67dbf85bf(29033259) to registry.cn-beijing.aliyuncs.com/cloud-devops/kube-apiserver:v1.20.1 success","time":"2021-01-08 10:54:24"}
{"level":"info","msg":"Get a blob sha256:75c7f711208082c548b935ab31e681ea30acccdce6b7abeecabae5bbfd326627(2630) from 172.20.5.10/library/kube-apiserver:v1.20.1 success","time":"2021-01-08 10:54:24"}
{"level":"info","msg":"Put blob sha256:75c7f711208082c548b935ab31e681ea30acccdce6b7abeecabae5bbfd326627(2630) to registry.cn-beijing.aliyuncs.com/cloud-devops/kube-apiserver:v1.20.1 success","time":"2021-01-08 10:54:25"}
{"level":"info","msg":"Put manifest to registry.cn-beijing.aliyuncs.com/cloud-devops/kube-apiserver:v1.20.1","time":"2021-01-08 10:54:25"}
{"level":"info","msg":"Synchronization successfully from 172.20.5.10/library/kube-apiserver:v1.20.1 to registry.cn-beijing.aliyuncs.com/cloud-devops/kube-apiserver:v1.20.1","time":"2021-01-08 10:54:25"}
{"level":"info","msg":"Finished, 0 sync tasks failed, 0 tasks generate failed","time":"2021-01-08 10:54:25"}
3.2 Harbor To Huawei SWR
创建配置文件
{
"auth": {
"172.20.5.10": {
"username": "admin",
"password": "Xxzx@789",
"insecure": true
},
"swr.cn-east-3.myhuaweicloud.com": {
"username": "cn-east-3@8B5B8GC",
"password": "8b0f1e69f2"
}
},
"images": {
"172.20.5.10/library/kube-apiserver": "",
"172.20.5.10/library/controller": ""
}
}
启动任务
image-syncer --proc=10 --config=./harbor_to_swr.json --registry=swr.cn-east-3.myhuaweicloud.com --namespace=cloud-devops --retries=10 --log=./log
Start to generate sync tasks, please wait ...
Start to handle sync tasks, please wait ...
Finished, 0 sync tasks failed, 0 tasks generate failed
3.3 Huawei SWR To Aliyun ACR
创建配置文件
{
"auth": {
"swr.cn-east-3.myhuaweicloud.com": {
"username": "cn-east-3@8B5B8GC",
"password": "8b0f1e6a46af8dff0ae519a09f2"
},
"registry.cn-beijing.aliyuncs.com": {
"username": "352@qq.com",
"password": "docker"
}
},
"images": {
"swr.cn-east-3.myhuaweicloud.com/cloud-devops/kube-apiserver": "",
"swr.cn-east-3.myhuaweicloud.com/cloud-devops/controller": ""
}
}
开始迁移
image-syncer --proc=10 --config=./harbor_to_swr.json --registry=registry.cn-beijing.aliyuncs.com --namespace=cloud-devops --retries=10 --log=./log
Start to generate sync tasks, please wait ...
Start to handle sync tasks, please wait ...
Finished, 0 sync tasks failed, 0 tasks generate failed
