一、harbor 2.3.3安装[master]
- harbor，价值之一在于比如在master提供私有源服务，各node统一从master拉取服务。避免后续很多被墙组件需要重复搞定镜像pull的各种报错
- 所以，本文在master节点安装harbor服务，并提交各种如gcr.io镜像。各node节点 拉取镜像
1.1、docker-compose 2.0.1安装
#[root@VM-16-14-centos ~]# curl -L https://github.com/docker/compose/releases/download/2.0.1/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
[root@VM-16-14-centos ~]# mkdir -p /data/
[root@VM-16-14-centos ~]# cd /data/
[root@VM-16-14-centos data]# rz
[root@VM-16-14-centos data]# cp docker-compose-linux-x86_64 /usr/local/bin/docker-compose
[root@VM-16-14-centos data]# chmod +x /usr/local/bin/docker-compose
[root@VM-16-14-centos data]# docker-compose --version
Docker Compose version v2.0.1
1.2、harbor 2.3.3安装
- github.com/goharbor/ha…
- github.com/goharbor/ha…
- #rz/ssh 速度无异，但ssh 可显示速度和进展。rz不行。
[root@VM-16-14-centos harbor]# mkdir -p /var/log/harbor/
#[root@VM-16-14-centos data]# wget https://github.com/goharbor/harbor/archive/v1.8.1.tar.gz
[root@VM-16-14-centos data]# rz
[root@VM-16-14-centos data]# tar zxvf harbor-offline-installer-v2.3.3.tgz
[root@VM-16-14-centos data]# mv harbor /usr/local/harbor
[root@VM-16-14-centos data]# cd /usr/local/harbor/
[root@VM-16-14-centos harbor]# cp harbor.yml.tmpl harbor.yml
[root@VM-16-14-centos harbor]# vim harbor.yml
[root@VM-16-14-centos harbor]# vim /usr/local/harbor/harbor.yml
#设置hostname
hostname = 1.*.*.220
harbor_admin_password: Harbor12345
port: 81（默认80）
注释 #https:
#port: 443
#certificate: /your/certificate/path
# private_key: /your/private/key/path
[root@VM-16-14-centos harbor]# /usr/local/harbor/prepare
prepare base dir is set to /usr/local/harbor WARNING:root:WARNING: HTTP protocol is insecure. Harbor will deprecate http protocol in the future. Please make sure to upgrade to https
[root@VM-16-14-centos harbor]# /usr/local/harbor/install.sh
[root@VM-16-14-centos harbor]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
920da4112ed1 goharbor/harbor-jobservice:v2.3.3 "/harbor/entrypoint.…" About a minute ago Up About a minute (healthy) harbor-jobservice
d54576943b93 goharbor/nginx-photon:v2.3.3 "nginx -g 'daemon of…" About a minute ago Up About a minute (healthy) 0.0.0.0:80->8080/tcp, :::80->8080/tcp nginx
d7a7dbfe0c6d goharbor/harbor-core:v2.3.3 "/harbor/entrypoint.…" About a minute ago Up About a minute (healthy) harbor-core
875db9b1d4b1 goharbor/harbor-portal:v2.3.3 "nginx -g 'daemon of…" About a minute ago Up About a minute (healthy) harbor-portal
2942147e4342 goharbor/harbor-registryctl:v2.3.3 "/home/harbor/start.…" About a minute ago Up About a minute (healthy) registryctl
d4128df6575c goharbor/redis-photon:v2.3.3 "redis-server /etc/r…" About a minute ago Up About a minute (healthy) redis
7bde15f063ff goharbor/harbor-db:v2.3.3 "/docker-entrypoint.…" About a minute ago Up About a minute (healthy) harbor-db
333429ccf2b5 goharbor/registry-photon:v2.3.3 "/home/harbor/entryp…" About a minute ago Up About a minute (healthy) registry
160ede620fee goharbor/harbor-log:v2.3.3 "/bin/sh -c /usr/loc…" About a minute ago Up About a minute (healthy) 127.0.0.1:1514->10514/tcp harbor-log
[root@VM-16-14-centos harbor]# docker-compose ps
NAME COMMAND SERVICE STATUS PORTS
harbor-core "/harbor/entrypoint.…" core running (healthy)
harbor-db "/docker-entrypoint.…" postgresql running (healthy)
harbor-jobservice "/harbor/entrypoint.…" jobservice running (healthy)
harbor-log "/bin/sh -c /usr/loc…" log running (healthy) 127.0.0.1:1514->10514/tcp
harbor-portal "nginx -g 'daemon of…" portal running (healthy)
nginx "nginx -g 'daemon of…" proxy running (healthy) 0.0.0.0:80->8080/tcp, :::80->8080/tcp
redis "redis-server /etc/r…" redis running (healthy)
registry "/home/harbor/entryp…" registry running (healthy)
registryctl "/home/harbor/start.…" registryctl running (healthy)
[root@VM-16-14-centos harbor]# cd /usr/local/harbor/
[root@VM-16-14-centos harbor]# docker-compose start
[root@VM-16-14-centos harbor]# docker-compose restart
- 这里注意，如果后期配置变化，可以重新修改配置，重新执行 prepare、install.sh
Note: stopping existing Harbor instance ...
[+] Running 10/10
⠿ Container harbor-jobservice Removed 10.4s
⠿ Container registryctl Removed 10.3s
⠿ Container nginx Removed 0.3s
⠿ Container harbor-portal Removed 0.2s
⠿ Container harbor-core Removed 10.2s
⠿ Container harbor-db Removed 0.4s
⠿ Container registry Removed 10.3s
⠿ Container redis Removed 0.3s
⠿ Container harbor-log Removed 10.2s
⠿ Network harbor_harbor Removed 0.1s
[Step 5]: starting Harbor ...
[+] Running 10/10
⠿ Network harbor_harbor Created 0.1s
⠿ Container harbor-log Started 0.7s
⠿ Container harbor-db Started 2.1s
⠿ Container harbor-portal Started 2.5s
⠿ Container registry Started 2.3s
⠿ Container redis Started 2.4s
⠿ Container registryctl Started 2.4s
⠿ Container harbor-core Started 3.2s
⠿ Container nginx Started 4.5s
⠿ Container harbor-jobservice Started 4.3s
✔ ----Harbor has been installed and started successfully.----
1.3、访问harbor
http://1.*.*.220:*/harbor/projects
admin/***，密码 见 harbor_admin_password
二、harbor使用
2.1、查看状态
[root@VM-16-14-centos ~]# docker images
[root@VM-16-14-centos ~]# docker ps -a
2.2、注册各类镜像源账号与获取token
2.3、新建项目
- 这里在v2之后具备镜像代理能力，这里不选是因为本机无访问被墙镜像源能力
- 镜像代理能力是指，客户端是拉取特定源时，若首次拉取会缓存镜像。下次直接从私有源缓存获取。
2.4、push镜像至私有源
- 登录
[root@VM-16-14-centos ~]# docker login -u admin -p Harbor12345 http://127.0.0.1:81
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
- 拉取镜像
[root@VM-16-14-centos ~]# docker pull httpd
Using default tag: latest
latest: Pulling from library/httpd
7d63c13d9b9b: Pull complete
ca52f3eeea66: Pull complete
448256567156: Pull complete
21d69ac90caf: Pull complete
462e88bc3074: Pull complete
Digest: sha256:f70876d78442771406d7245b8d3425e8b0a86891c79811af94fb2e12af0fadeb
Status: Downloaded newer image for httpd:latest
docker.io/library/httpd:latest
- 查看镜像
[root@VM-16-14-centos ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
httpd latest 1132a4fc88fa 8 days ago 143MB
- 打tag
[root@VM-16-14-centos ~]# docker tag httpd:latest 127.0.0.1:81/mylibary/httpd:latest
[root@VM-16-14-centos ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
127.0.0.1:81/mylibary/httpd latest 1132a4fc88fa 8 days ago 143MB
httpd latest 1132a4fc88fa 8 days ago 143MB
- 提交push镜像
[root@VM-16-14-centos ~]# docker push 127.0.0.1:81/mylibary/httpd:latest
The push refers to repository [127.0.0.1:81/mylibary/httpd]
4dcdec0b7a0e: Pushed
c86537ee54f9: Pushed
ecd2b49ef243: Pushed
7511c367f47a: Pushed
e8b689711f21: Pushed
latest: digest: sha256:73c9b78280a693050838e9e3519e7f5723d742ada3e42c45f10744b4d88f486e size: 1365
2.5、从master私有源pull镜像
[root@VM-16-6-centos ~]# docker login -u admin -p Harbor12345 http://10.206.16.14:81
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get "https://10.206.16.14:81/v2/": http: server gave HTTP response to HTTPS client
E2.5.1、增加insecure-registries配置，导致docker启动失败
- docker配置文件，不要使用json，后续在配置改动中会导致格式异常，进而导致docker启动失败。修改格式为conf。
[root@VM-16-6-centos ~]# vim /etc/docker/daemon.json
增加
"insecure-registries" : ["10.206.16.14:81",
[root@VM-16-6-centos ~]# sudo systemctl start docker
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.
[root@VM-16-6-centos ~]# journalctl -xe
-- Support: https://access.redhat.com/support
--
-- The unit docker.service has entered the 'failed' state with result 'exit-code'.
10月 30 07:46:58 VM-16-6-centos systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: docker.service 单元已失败
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- docker.service 单元已失败。
--
-- 结果为“failed”。
10月 30 07:46:58 VM-16-6-centos systemd[1]: docker.socket: Failed with result 'service-start-limit-hit'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- The unit docker.socket has entered the 'failed' state with result 'service-start-limit-hit'.
10月 30 07:46:58 VM-16-6-centos sudo[2372925]: pam_unix(sudo:session): session closed for user root
10月 30 07:46:58 VM-16-6-centos kubelet[1200]: E1030 07:46:58.604435 1200 remote_runtime.go:207] "ListPodSandbox with filter from runtime service failed" err="rpc error: code = Unknown >
10月 30 07:46:58 VM-16-6-centos kubelet[1200]: E1030 07:46:58.604466 1200 kuberuntime_sandbox.go:281] "Failed to list pod sandboxes" err="rpc error: code = Unknown desc = Cannot connect>
10月 30 07:46:58 VM-16-6-centos kubelet[1200]: E1030 07:46:58.604487 1200 generic.go:205] "GenericPLEG: Unable to retrieve pods" err="rpc error: code = Unknown desc = Cannot connect to >
10月 30 07:46:59 VM-16-6-centos kubelet[1200]: E1030 07:46:59.378918 1200 kubelet.go:1991] "Skipping pod synchronization" err="[container runtime is down, container runtime not ready: R>
10月 30 07:46:59 VM-16-6-centos kubelet[1200]: E1030 07:46:59.606161 1200 remote_runtime.go:207] "ListPodSandbox with filter from runtime service failed" err="rpc error: code = Unknown >
10月 30 07:46:59 VM-16-6-centos kubelet[1200]: E1030 07:46:59.606212 1200 kuberuntime_sandbox.go:281] "Failed to list pod sandboxes" err="rpc error: code = Unknown desc = Cannot connect>
10月 30 07:46:59 VM-16-6-centos kubelet[1200]: E1030 07:46:59.606232 1200 generic.go:205] "GenericPLEG: Unable to retrieve pods" err="rpc error: code = Unknown desc = Cannot connect to >
10月 30 07:47:00 VM-16-6-centos kubelet[1200]: E1030 07:47:00.607705 1200 remote_runtime.go:207] "ListPodSandbox with filter from runtime service failed" err="rpc error: code = Unknown >
10月 30 07:47:00 VM-16-6-centos kubelet[1200]: E1030 07:47:00.607741 1200 kuberuntime_sandbox.go:281] "Failed to list pod sandboxes" err="rpc error: code = Unknown desc = Cannot connect>
10月 30 07:47:00 VM-16-6-centos kubelet[1200]: E1030 07:47:00.607764 1200 generic.go:205] "GenericPLEG: Unable to retrieve pods" err="rpc error: code = Unknown desc = Cannot connect to >
10月 30 07:47:01 VM-16-6-centos CROND[2372946]: (root) CMD (flock -xn /tmp/stargate.lock -c '/usr/local/qcloud/stargate/admin/start.sh > /dev/null 2>&1 &')
10月 30 07:47:01 VM-16-6-centos kubelet[1200]: E1030 07:47:01.609355 1200 remote_runtime.go:207] "ListPodSandbox with filter from runtime service failed" err="rpc error: code = Unknown >
10月 30 07:47:01 VM-16-6-centos kubelet[1200]: E1030 07:47:01.609428 1200 kuberuntime_sandbox.go:281] "Failed to list pod sandboxes" err="rpc error: code = Unknown desc = Cannot connect>
10月 30 07:47:01 VM-16-6-centos kubelet[1200]: E1030 07:47:01.609448 1200 generic.go:205] "GenericPLEG: Unable to retrieve pods" err="rpc error: code = Unknown desc = Cannot connect to >
10月 30 07:47:02 VM-16-6-centos kubelet[1200]: E1030 07:47:02.558832 1200 kubelet.go:2342] "Container runtime not ready" runtimeReady="RuntimeReady=false reason:DockerDaemonNotReady mes>
10月 30 07:47:02 VM-16-6-centos kubelet[1200]: E1030 07:47:02.561016 1200 remote_image.go:71] "ListImages with filter from image service failed" err="rpc error: code = Unknown desc = Ca>
10月 30 07:47:02 VM-16-6-centos kubelet[1200]: E1030 07:47:02.561042 1200 kuberuntime_image.go:136] "Failed to list images" err="rpc error: code = Unknown desc = Cannot connect to the D>
10月 30 07:47:02 VM-16-6-centos kubelet[1200]: E1030 07:47:02.561066 1200 eviction_manager.go:255] "Eviction manager: failed to get summary stats" err="failed to get imageFs stats: fail>
10月 30 07:47:02 VM-16-6-centos kubelet[1200]: E1030 07:47:02.597993 1200 remote_runtime.go:86] "Version from runtime service failed" err="rpc error: code = Unknown desc = failed to get>
10月 30 07:47:02 VM-16-6-centos kubelet[1200]: E1030 07:47:02.610425 1200 remote_runtime.go:207] "ListPodSandbox with filter from runtime service failed" err="rpc error: code = Unknown >
10月 30 07:47:02 VM-16-6-centos kubelet[1200]: E1030 07:47:02.610455 1200 kuberuntime_sandbox.go:281] "Failed to list pod sandboxes" err="rpc error: code = Unknown desc = Cannot connect>
10月 30 07:47:02 VM-16-6-centos kubelet[1200]: E1030 07:47:02.610476 1200 generic.go:205] "GenericPLEG: Unable to retrieve pods" err="rpc error: code = Unknown desc = Cannot connect to >
- 排障
[root@VM-16-6-centos ~]# systemctl start docker
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.
[root@VM-16-6-centos ~]# systemctl status docker.service
[root@VM-16-6-centos ~]# journalctl -xe
[root@VM-16-6-centos ~]# mv /etc/docker/daemon.json /etc/docker/daemon.conf
[root@VM-16-6-centos ~]# sudo systemctl daemon-reload
[root@VM-16-6-centos ~]# sudo systemctl restart docker.service
- 重新login，依旧报错
- 必须修改 /usr/lib/systemd/system/docker.service
- 之后无论http/https登录均可
[root@VM-16-6-centos ~]# docker login -u admin -p Harbor12345 https://10.206.16.14:81
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get "https://10.206.16.14:81/v2/": http: server gave HTTP response to HTTPS client
[root@VM-16-6-centos ~]# vi /usr/lib/systemd/system/docker.service
修改
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd
为
ExecStart=/usr/bin/dockerd -H fd:// --insecure-registry 10.206.16.14:81 --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd
即增加 --insecure-registry 10.206.16.14:81
[root@VM-16-6-centos ~]# docker login -u admin -p Harbor12345 https://10.206.16.14:81
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
[root@VM-16-6-centos ~]# docker login -u admin -p Harbor12345 http://10.206.16.14:81
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
2.6、node从master私有源pull镜像
[root@VM-16-6-centos ~]# docker pull 10.206.16.14:81/mylibary/httpd
Using default tag: latest
latest: Pulling from mylibary/httpd
7d63c13d9b9b: Pull complete
ca52f3eeea66: Pull complete
448256567156: Pull complete
21d69ac90caf: Pull complete
462e88bc3074: Pull complete
Digest: sha256:73c9b78280a693050838e9e3519e7f5723d742ada3e42c45f10744b4d88f486e
Status: Downloaded newer image for 10.206.16.14:81/mylibary/httpd:latest
10.206.16.14:81/mylibary/httpd:latest
[root@VM-16-6-centos ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
10.206.16.14:81/mylibary/httpd latest 1132a4fc88fa 8 days ago 143MB
2.7、node补齐master上的k8s.gcr.io、quay.io等镜像
三、后期维护
- 若出现docker异常，或重启服务器后，harbor进程会异常或缺失
- 这里就明显异常，缺少nginx等pod
- 进入/usr/local/harbor/，重新执行docker-compose start即可
[root@VM-16-14-centos ~]# cd /usr/local/harbor/
[root@VM-16-14-centos harbor]# docker-compose start
[+] Running 5/5
⠿ Container registry Started 0.9s
⠿ Container harbor-portal Started 1.0s
⠿ Container registryctl Started 0.7s
⠿ Container harbor-jobservice Started 0.6s
⠿ Container nginx Started
- 正常重启后，pod清单
