1.28.11-配置Harbor私有镜像仓库一、环境准备喷一嘴: 网上大多配置方案在小弟这里都跑不通, 翻阅大量资料

1.28.11-配置Harbor私有镜像仓库

一、环境准备

喷一嘴: 网上大多配置方案在小弟这里都跑不通, 翻阅大量资料, 逐步整理, 请君参考, 望没有把大家带跑偏, 欢迎交流

环境信息
- Kubernetes版本及信息

[root@master01 troila]# kubectl get nodes -o wide 
NAME                  STATUS   ROLES           AGE    VERSION    INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION              CONTAINER-RUNTIME
master01.troila.com   Ready    control-plane   119m   v1.28.11   172.27.109.1   <none>        CentOS Linux 7 (Core)   6.1.0-1.el7.elrepo.x86_64   containerd://1.7.19
slave02.troila.com    Ready    <none>          119m   v1.28.11   172.27.109.2   <none>        CentOS Linux 7 (Core)   6.1.0-1.el7.elrepo.x86_64   containerd://1.7.19
slave03.troila.com    Ready    <none>          119m   v1.28.11   172.27.109.3   <none>        CentOS Linux 7 (Core)   6.1.0-1.el7.elrepo.x86_64   containerd://1.7.19
slave04.troila.com    Ready    <none>          119m   v1.28.11   172.27.109.4   <none>        CentOS Linux 7 (Core)   6.1.0-1.el7.elrepo.x86_64   containerd://1.7.19
slave05.troila.com    Ready    <none>          119m   v1.28.11   172.27.109.5   <none>        CentOS Linux 7 (Core)   6.1.0-1.el7.elrepo.x86_64   containerd://1.7.19
[root@master01 troila]#

Nerdctl 版本信息


[root@master01 troila]# nerdctl version
WARN[0000] unable to determine buildctl version: exec: "buildctl": executable file not found in $PATH 
Client:
 Version:       v1.7.6
 OS/Arch:       linux/amd64
 Git commit:    845e989f69d25b420ae325fedc8e70186243fd93
 buildctl:
  Version:

Server:
 containerd:
  Version:      v1.7.19
  GitCommit:    2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
 runc:
  Version:      1.1.13
  GitCommit:    v1.1.13-0-g58aa9203-dirty
[root@master01 troila]# 

[root@master01 troila]# nerdctl info
Client:
 Namespace:     default
 Debug Mode:    false

Server:
 Server Version: v1.7.19
 Storage Driver: overlayfs
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Log: fluentd journald json-file syslog
  Storage: native overlayfs
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 6.1.0-1.el7.elrepo.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.36GiB
 Name: master01.troila.com
 ID: e2933e96-95fa-444a-a51b-ff3734d26643

Crictl 版本及信息

[root@master01 troila]# crictl version
Version:  0.1.0
RuntimeName:  containerd
RuntimeVersion:  v1.7.19
RuntimeApiVersion:  v1
[root@master01 troila]#

Harbor配置信息
- 地址: base.troila.com:9000
- 假域名😭, 没钱,!!!!!
- 使用Http请求

注意
- toml文件执行有顺序约束, 请严格按照一下文档配置
- toml文件执行有顺序约束, 请严格按照一下文档配置
- toml文件执行有顺序约束, 请严格按照一下文档配置

二、开始配置

在/etc/containerd/certs.d/下创建base.troila.com:9000文件夹, 并在此文件夹下创建hosts.toml

因配置私有仓库扫描文件地址为: config_path = "/etc/containerd/certs.d"
/etc/containerd/certs.d/base.troila.com:9000/hosts.toml请君自行创建,如下

[root@master01 base.troila.com:9000]# pwd
/etc/containerd/certs.d/base.troila.com:9000
[root@master01 base.troila.com:9000]# ls
hosts.toml
[root@master01 base.troila.com:9000]#

在hosts.toml文件中做如下配置, 此处不解释, 懂的都懂:

[root@master01 base.troila.com:9000]# cat hosts.toml 

server = "http://base.troila.com:9000"

[host."http://base.troila.com:9000"]
  capabilities = ["pull", "resolve", "push"]
  skip_verify = true
  
[root@master01 base.troila.com:9000]#

修改配置文件, 地址为: /etc/containerd/config.toml
- 在[plugins."io.containerd.grpc.v1.cri".registry.configs]下配置如下代码
- insecure_skip_verify = true 跳过安全校验

# 跳过Https请求
[plugins."io.containerd.grpc.v1.cri".registry.configs."base.troila.com:9000".tls]
  insecure_skip_verify = true

# 配置私有仓库Harbor登录信息
[plugins."io.containerd.grpc.v1.cri".registry.configs."base.troila.com:9000".auth]
  username = "admin"
  password = "Troila@123"

修改配置文件, 地址为: /etc/containerd/config.toml
- 在[plugins."io.containerd.grpc.v1.cri".registry.mirrors]下配置如下代码
- 放到[plugins."io.containerd.grpc.v1.cri".registry.mirrors]层级下第一个子集
```
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."base.troila.com:9000"]
  endpoint = ["http://base.troila.com:9000"]
```
参考配置层级

[plugins."io.containerd.grpc.v1.cri".registry]

[plugins."io.containerd.grpc.v1.cri".registry.mirrors]

[plugins."io.containerd.grpc.v1.cri".registry.configs]

注意以上config.toml配置中, 私有和公有mirros一定要放到config_path = "/etc/containerd/certs.d"前面
注意以上config.toml配置中, 私有和公有mirros一定要放到config_path = "/etc/containerd/certs.d"前面
注意以上config.toml配置中, 私有和公有mirros一定要放到config_path = "/etc/containerd/certs.d"前面
整体配置

[root@master01 troila]# cat /etc/containerd/config.toml 
version = 2
root = "/var/lib/containerd"
state = "/run/containerd"
oom_score = 0

[grpc]
  address = "/run/containerd/containerd.sock"
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[debug]
  address = "/run/containerd/containerd-debug.sock"
  uid = 0
  gid = 0
  level = "warn"

[timeouts]
  "io.containerd.timeout.shim.cleanup" = "5s"
  "io.containerd.timeout.shim.load" = "5s"
  "io.containerd.timeout.shim.shutdown" = "3s"
  "io.containerd.timeout.task.state" = "2s"

[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    sandbox_image = "sealos.hub:5000/pause:3.9"
    max_container_log_line_size = 16384
    max_concurrent_downloads = 20
    disable_apparmor = true
    [plugins."io.containerd.grpc.v1.cri".containerd]
      snapshotter = "overlayfs"
      default_runtime_name = "runc"
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          runtime_type = "io.containerd.runc.v2"
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            SystemdCgroup = true
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.crun]
          runtime_type = "io.containerd.runc.v2"
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.crun.options]
            BinaryName = "/usr/bin/crun"
            SystemdCgroup = true
  
  
    [plugins."io.containerd.grpc.v1.cri".registry]
  
      # mirrors公共资源镜像配置一定要放到config_path之前
  		# mirrors公共资源镜像配置一定要放到config_path之前
      # mirrors公共资源镜像配置一定要放到config_path之前
  		# mirrors公共资源镜像配置一定要放到config_path之前,Toml文件是有序的, config_path的一般指定的是私有仓库地址, 会覆盖掉registry, mirrors配置
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
  
        # 配置私有mirrors
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."base.troila.com:9000"]
          endpoint = ["http://base.troila.com:9000"]

        # 配置公有mirrors docker.io
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
          endpoint = ["https://k8s.m.daocloud.io","https://45hrqeao.mirror.aliyuncs.com","https://registry.docker-cn.com"]
  
        # 配置公有mirrors registry.k8s.io
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."registry.k8s.io"]
          endpoint = ["https://45hrqeao.mirror.aliyuncs.com", "https://k8s.m.daocloud.io", "https://docker.mirrors.ustc.edu.cn", "https://hub-mirror.c.163.com", "registry.cn-hangzhou.aliyuncs.com","https://registry.docker-cn.com"]

      # 配置私有仓库地址
      config_path = "/etc/containerd/certs.d"

      [plugins."io.containerd.grpc.v1.cri".registry.configs]
  
        [plugins."io.containerd.grpc.v1.cri".registry.configs."sealos.hub:5000".auth]
          username = "admin"
          password = "passw0rd"

        # 配置私有仓库镜像拉取上传跳过认证
        [plugins."io.containerd.grpc.v1.cri".registry.configs."base.troila.com:9000".tls]
          insecure_skip_verify = true

        # 配置私有Harbor仓库
        [plugins."io.containerd.grpc.v1.cri".registry.configs."base.troila.com:9000".auth]
          username = "admin"
          password = "Troila@123"

三、重启 containerd

systemctl restart containerd

四、验证

验证CRI

[root@master01 troila]# crictl info

......................................... 

    "registry": {
      "configPath": "",
      "mirrors": {
        "base.troila.com:9000": {
          "endpoint": [
            "http://base.troila.com:9000"
          ]
        },
        "docker.io": {
          "endpoint": [
            "https://k8s.m.daocloud.io",
            "https://45hrqeao.mirror.aliyuncs.com",
            "https://registry.docker-cn.com"
          ]
        },
        "registry.k8s.io": {
          "endpoint": [
            "https://45hrqeao.mirror.aliyuncs.com",
            "https://k8s.m.daocloud.io",
            "https://docker.mirrors.ustc.edu.cn",
            "https://hub-mirror.c.163.com",
            "registry.cn-hangzhou.aliyuncs.com",
            "https://registry.docker-cn.com"
          ]
        }
      },
      "configs": {
        "base.troila.com:9000": {
          "auth": {
            "username": "admin",
            "password": "Troila@123",
            "auth": "",
            "identitytoken": ""
          },
          "tls": {
            "insecure_skip_verify": true,
            "caFile": "",
            "certFile": "",
            "keyFile": ""
          }
        },
        "sealos.hub:5000": {
          "auth": {
            "username": "admin",
            "password": "passw0rd",
            "auth": "",
            "identitytoken": ""
          },
          "tls": null
        }
      },
      "auths": null,
      "headers": null
    },
.........................................    
}

登录私有镜像仓库
- Nerdctil 登录

[root@slave03 ~]# nerdctl login -u admin base.troila.com:9000
Enter Password: 
INFO[0005] host will try HTTPS first since it is configured for HTTP with a TLS configuration, consider changing host to HTTPS or removing unused TLS configuration  host="base.troila.com:9000"
WARNING: Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
[root@slave03 ~]#

CRI下载镜像

[root@slave03 ~]# crictl pull base.troila.com:9000/library/busybox:1.36.1
Image is up to date for sha256:65ad0d468eb1c558bf7f4e64e790f586e9eda649ee9f130cd0e835b292bbc5ac

[root@slave03 ~]# crictl images
IMAGE                                                          TAG                 IMAGE ID            SIZE
base.troila.com:9000/library/busybox                           1.36.1              65ad0d468eb1c       2.22MB
base.troila.com:9000/library/nfs-subdir-external-provisioner   v4.0.0              3beaacba3ff48       17.9MB
sealos.hub:5000/calico/cni                                     v3.28.0             107014d9f4c89       94.5MB
sealos.hub:5000/calico/csi                                     v3.28.0             1a094aeaf1521       9.09MB
sealos.hub:5000/calico/node-driver-registrar                   v3.28.0             0f80feca743f4       11.6MB
sealos.hub:5000/calico/node                                    v3.28.0             4e42b6f329bc1       115MB
sealos.hub:5000/calico/pod2daemon-flexvol                      v3.28.0             587b28ecfc62e       6.59MB
sealos.hub:5000/calico/typha                                   v3.28.0             a9372c0f51b54       30.9MB
sealos.hub:5000/kube-proxy                                     v1.28.11            a3eea76ce409e       28.1MB
sealos.hub:5000/labring/lvscare                                v4.3.7              7c3f2d8e222cb       13.6MB
sealos.hub:5000/pause                                          3.9                 e6f1816883972       319kB
[root@slave03 ~]#

Nerdctl 下载镜像

[root@slave03 ~]# crictl rmi 65
Deleted: base.troila.com:9000/library/busybox:1.36.1

[root@slave03 ~]# nerdctl pull base.troila.com:9000/library/busybox:1.36.1
INFO[0000] host will try HTTPS first since it is configured for HTTP with a TLS configuration, consider changing host to HTTPS or removing unused TLS configuration  host="base.troila.com:9000"
INFO[0000] host will try HTTPS first since it is configured for HTTP with a TLS configuration, consider changing host to HTTPS or removing unused TLS configuration  host="base.troila.com:9000"
base.troila.com:9000/library/busybox:1.36.1:                                      resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:28e01ab32c9dbcbaae96cf0d5b472f22e231d9e603811857b295e61197e40a9b: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:65ad0d468eb1c558bf7f4e64e790f586e9eda649ee9f130cd0e835b292bbc5ac:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:d0f42ecf7e6cd9d1b2a6eb22f45ada31d70a854a985ff6a5c24149147bbb6287:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 0.7 s                                                                    total:  2.1 Mi (3.0 MiB/s)  

[root@slave03 ~]# nerdctl images
REPOSITORY                              TAG       IMAGE ID        CREATED          PLATFORM       SIZE       BLOB SIZE
base.troila.com:9000/library/busybox    1.36.1    28e01ab32c9d    8 seconds ago    linux/amd64    4.1 MiB    2.1 MiB
[root@slave03 ~]#

ok, 完工, 一天天的瞎折腾

1.28.11-配置Harbor私有镜像仓库

一、 环境准备

二、开始配置

三、重启 containerd

四、验证

一、环境准备