一、镜像打包工具的选择
- docker官方buildkit github.com/moby/buildk…
4.4K star Latest:v0.9.2
传统docker build，需要宿主机root权限
- Kaniko, Google 主导 github.com/GoogleConta…
9.2K star Latest:v1.7.0
kaniko，可以运行在k8s集群（即无docker守护进程的环境中，默认打包在docker环境中挂载宿主机socket然后docker build。或者–privileged 用特权执行打包）
- Img, Jessie Frazelle 发起
- Buildah, Red Hat 主导
- Jib也是Google开源的一款Java容器镜像构建工具，通过使用Jib，Java 开发人员可以使用他们熟悉的Java构建工具来构建容器镜像。 Jib负责处理将应用程序打包到容器镜像中所需的所有步骤，它不需要我们编写Dockerfile或安装docker daemon，而是直接把镜像构建功能集成到了java构建工具gradle和maven中(通过将插件添加到构建中)，即用java的构建工具直接完成容器镜像的构建。
二、buildkit 0.9.2打包镜像
2.1、启动buildkitd服务端
[root@VM-16-14-centos data]# mkdir -p buildkit-v0.9.2
[root@VM-16-14-centos data]# mv buildkit-v0.9.2.linux-amd64.tar.gz buildkit-v0.9.2/
[root@VM-16-14-centos data]# cd buildkit-v0.9.2
[root@VM-16-14-centos buildkit-v0.9.2]# tar zxvf buildkit-v0.9.2.linux-amd64.tar.gz
[root@VM-16-14-centos buildkit-v0.9.2]# ./bin/buildkitd
INFO[2021-11-13T00:14:56+08:00] auto snapshotter: using overlayfs
WARN[2021-11-13T00:14:56+08:00] using host network as the default
INFO[2021-11-13T00:14:56+08:00] found worker "ldsfqoyorohnyskxntuc4govs", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:VM-16-14-centos org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/386]
WARN[2021-11-13T00:14:56+08:00] using host network as the default
INFO[2021-11-13T00:14:56+08:00] found worker "iu2l2hh9iceyztc0z31lx4mn0", labels=map[org.mobyproject.buildkit.worker.containerd.namespace:buildkit org.mobyproject.buildkit.worker.containerd.uuid:c2e6caec-3bcb-4fc5-90e5-f18b206415e4 org.mobyproject.buildkit.worker.executor:containerd org.mobyproject.buildkit.worker.hostname:VM-16-14-centos org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/386]
INFO[2021-11-13T00:14:56+08:00] found 2 workers, default="ldsfqoyorohnyskxntuc4govs"
WARN[2021-11-13T00:14:56+08:00] currently, only the default worker can be used.
INFO[2021-11-13T00:14:56+08:00] running server on /run/buildkit/buildkitd.sock
2.2、buildctl打包镜像[golang]
2.2.1、范例1：buildctl打包镜像[golang]
- Dockerfile
[root@VM-16-14-centos buildkit-v0.9.2]# vim Dockerfile
FROM golang:alpine AS builder
RUN mkdir /app
ADD . /app/
WORKDIR /app
ENV GO111MODULE=off
RUN go build -o hello .
FROM alpine
RUN mkdir /app
WORKDIR /app
COPY --from=builder /app/hello .
CMD ["./hello"]
[root@VM-16-14-centos buildkit-v0.9.2]# vim main.go
package main
import (
"fmt"
"runtime"
)
func main() {
fmt.Printf("Hello, %s!\n", runtime.GOARCH)
}
-
先登录harbor或使用证书
-
首次打包，下载基础镜像 耗时超长
-
pull镜像
[root@VM-16-14-centos buildkit-v0.9.2]# ./bin/buildctl build --output type=image,name=127.0.0.1:81/mylibary/go_hello:v1,push=true
--export-cache type=inline
--frontend=dockerfile.v0
--local context=.
--local dockerfile=.
[+] Building 2.5s (18/18) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 242B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/alpine:latest 1.9s
=> [internal] load metadata for docker.io/library/golang:alpine 1.9s
=> [builder 1/5] FROM docker.io/library/golang:alpine@sha256:d1b1456acc7317f562ba81698ae4f0971a0a2e84ddc4e746a8e3671bf88df1bb 0.0s
=> => resolve docker.io/library/golang:alpine@sha256:d1b1456acc7317f562ba81698ae4f0971a0a2e84ddc4e746a8e3671bf88df1bb 0.0s
=> [stage-1 1/4] FROM docker.io/library/alpine@sha256:635f0aa53d99017b38d1a0aa5b2082f7812b03e3cdb299103fe77b5c8a07f1d2 0.0s
=> => resolve docker.io/library/alpine@sha256:635f0aa53d99017b38d1a0aa5b2082f7812b03e3cdb299103fe77b5c8a07f1d2 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 612B 0.0s
=> CACHED [stage-1 2/4] RUN mkdir /app 0.0s
=> CACHED [stage-1 3/4] WORKDIR /app 0.0s
=> CACHED [builder 2/5] RUN mkdir /app 0.0s
=> CACHED [builder 3/5] ADD . /app/ 0.0s
=> CACHED [builder 4/5] WORKDIR /app 0.0s
=> CACHED [builder 5/5] RUN go build -o hello . 0.0s
=> CACHED [stage-1 4/4] COPY --from=builder /app/hello . 0.0s
=> exporting to image 0.4s
=> => exporting layers 0.0s
=> => exporting manifest sha256:82d56b77d57d500d45ea2a25ae4173f793b2361ff6a303d93631baa48a33f8b1 0.0s
=> => exporting config sha256:87ded8c63742ed5d4fdca9ef8a5ddfb06bc86f260c23787d61ad32c7d8ac953d 0.0s
=> => pushing layers 0.3s
=> => pushing manifest for 127.0.0.1:81/mylibary/go_hello:v1@sha256:82d56b77d57d500d45ea2a25ae4173f793b2361ff6a303d93631baa48a33f8b1 0.1s
=> [auth] mylibary/go_hello:pull,push token for 127.0.0.1:81 0.0s
=> [auth] mylibary/go_hello:pull,push token for 127.0.0.1:81 0.0s
=> exporting cache 0.1s
=> => preparing build cache for export
2.2.2、范例2：buildctl打包镜像[golang web]
- Dockerfile
[root@VM-16-14-centos go8080]# pwd
/data/buildkit-v0.9.2/go8080
[root@VM-16-14-centos go8080]# cat Dockerfile
from golang:alpine
workdir /app
copy . /app/
# run go build main.go
# expose 8080
# entrypoint ["./main"]
expose 8080
entrypoint go run main.go
[root@VM-16-14-centos go8080]# cat main.go
package main
import(
"net/http"
"fmt"
)
func main(){
fmt.Println("开启后端")
http.HandleFunc("/Hello",PrintHello)
err:=http.ListenAndServe(":8080",nil)
if err==nil{
fmt.Println("开启8080端口")
}
}
func PrintHello(w http.ResponseWriter, r *http.Request){
if r.Method!="GET"{
return
}
fmt.Fprint(w,"hello world")
}
- pull镜像
[root@VM-16-14-centos go8080]# ../bin/buildctl build --output type=image,name=127.0.0.1:81/mylibary/go_hello_8080:v1,push=true --export-cache type=inline --frontend=dockerfile.v0 --local context=. --local dockerfile=.
[+] Building 16.1s (12/12) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 185B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/golang:alpine 2.4s
=> CACHED [1/3] FROM docker.io/library/golang:alpine@sha256:d1b1456acc7317f562ba81698ae4f0971a0a2e84ddc4e746a8e3671bf88df1bb 0.0s
=> => resolve docker.io/library/golang:alpine@sha256:d1b1456acc7317f562ba81698ae4f0971a0a2e84ddc4e746a8e3671bf88df1bb 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 561B 0.0s
=> [2/3] WORKDIR /app 0.1s
=> [3/3] COPY . /app/ 0.1s
=> exporting to image 3.2s
=> => exporting layers 0.0s
=> => exporting manifest sha256:561c258aad3f32839aba1802dac850df1e91b8242343c358ceedb10d871db43e 0.0s
=> => exporting config sha256:66b8cfebd2022dd56628042ddda891d3ea12f1b5723ddfa39c589bb9eff18072 0.0s
=> => pushing layers 1.7s
=> => pushing manifest for 127.0.0.1:81/mylibary/go_hello_8080:v1@sha256:561c258aad3f32839aba1802dac850df1e91b8242343c358ceedb10d871db43e 1.4s
=> [auth] mylibary/go_hello_8080:pull,push token for 127.0.0.1:81 0.0s
=> [auth] mylibary/go_hello_8080:pull,push token for 127.0.0.1:81 0.0s
=> [auth] mylibary/go_hello_8080:pull,push token for 127.0.0.1:81 0.0s
=> exporting cache 0.0s
=> => preparing build cache for export
三、k8s拉取harbor私有源镜像、deploy部署
3.1、手动验证harbor镜像拉取
验证方法：
harbor默认https验证，若取消https验证，则必须确保insecure-registries配置
[root@VM-16-14-centos go_hello]# docker pull *.*.*.*:81/mylibary/go_hello:v1
v1: Pulling from mylibary/go_hello
97518928ae5f: Pull complete
532ffc939e35: Pull complete
abc7ba6fd3d7: Pull complete
Digest: sha256:82d56b77d57d500d45ea2a25ae4173f793b2361ff6a303d93631baa48a33f8b1
Status: Downloaded newer image for
*.*.*.*:81/mylibary/go_hello:v1
*.*.*.*:81/mylibary/go_hello:v1
[root@VM-16-14-centos go_hello]# docker pull 127.0.0.1:81/mylibary/go_hello:v1
v1: Pulling from mylibary/go_hello
Digest: sha256:82d56b77d57d500d45ea2a25ae4173f793b2361ff6a303d93631baa48a33f8b1
Status: Downloaded newer image for 127.0.0.1:81/mylibary/go_hello:v1
127.0.0.1:81/mylibary/go_hello:v1
3.2、k8s deploy部署配置
[root@VM-16-14-centos go_hello]# pwd
/data/buildkit-v0.9.2/go_hello
[root@VM-16-14-centos go_hello]# cat go.dep.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: go-hello-deployment
spec:
replicas: 1
selector:
matchLabels:
app: go-hello-pod
template:
metadata:
labels:
app: go-hello-pod
spec:
containers:
- name: go-hello-container
image: *.*.*.*:81/mylibary/go_hello:latest #镜像名称+版本
imagePullPolicy: Never #表示镜像来源，IfNotPresent本地没有就从hub仓库拉取，Never表示只从本地
ports:
- containerPort: 8081
[root@VM-16-14-centos go_hello]# kubectl apply -f go.dep.yml
deployment.apps/go-hello-deployment created
3.3、镜像pull失败解决
3.3.1 方案1：基于现有Docker凭据创建secret
-
镜像pull错误
-
基于现有Docker凭据创建secret
[root@VM-16-14-centos go_hello]# cat ~/.docker/config.json
{
"auths": {
"*.*.*.*:81": {
"auth": "YWRtaW46SGFyYm9yMTIzNDU="
},
"127.0.0.1:81": {
"auth": "YWRtaW46SGFyYm9yMTIzNDU="
}
}
}
[root@VM-16-14-centos go_hello]# k get secrets
NAME TYPE DATA AGE
default-token-b8xch kubernetes.io/service-account-token 3 19d
ingress-nginx-admission-token-2sjj7 kubernetes.io/service-account-token 3 18d
sh.helm.release.v1.traefik.v1 helm.sh/release.v1 1 12d
sh.helm.release.v1.traefik.v2 helm.sh/release.v1 1 12d
sh.helm.release.v1.traefik.v3 helm.sh/release.v1 1 12d
sh.helm.release.v1.traefik.v4 helm.sh/release.v1 1 12d
sh.helm.release.v1.traefik.v5 helm.sh/release.v1 1 12d
sh.helm.release.v1.traefik.v6 helm.sh/release.v1 1 12d
traefik-token-h79wc kubernetes.io/service-account-token 3 12d
[root@VM-16-14-centos go_hello]# kubectl create secret generic harborsecret --from-file=.dockerconfigjson=/root/.docker/config.json --type=kubernetes.io/dockerconfigjson secret/harborsecret created
harborsecret 表示key名
/root/.docker/config.json 表示docker认证文件
[root@VM-16-14-centos go_hello]# k get secrets
NAME TYPE DATA AGE
default-token-b8xch kubernetes.io/service-account-token 3 19d
harborsecret kubernetes.io/dockerconfigjson 1 85s
ingress-nginx-admission-token-2sjj7 kubernetes.io/service-account-token 3 18d
sh.helm.release.v1.traefik.v1 helm.sh/release.v1 1 12d
sh.helm.release.v1.traefik.v2 helm.sh/release.v1 1 12d
sh.helm.release.v1.traefik.v3 helm.sh/release.v1 1 12d
sh.helm.release.v1.traefik.v4 helm.sh/release.v1 1 12d
sh.helm.release.v1.traefik.v5 helm.sh/release.v1 1 12d
sh.helm.release.v1.traefik.v6 helm.sh/release.v1 1 12d
traefik-token-h79wc kubernetes.io/service-account-token 3 12d
[root@VM-16-14-centos go_hello]# kubectl get secrets harborsecret --output="jsonpath={.data.\.dockerconfigjson}" | base64 -d
{
"auths": {
"*.*.*.*:81": {
"auth": "YWRtaW46SGFyYm9yMTIzNDU="
},
"127.0.0.1:81": {
"auth": "YWRtaW46SGFyYm9yMTIzNDU="
}
}
}
3.3.2 方案2：使用命令给harbor帐户创建secret[建议使用]
- harbor，新建帐户，创建secret
- 重新修改deploy文件，引用imagePullSecrets
# 举例：机器人账号
[root@VM-16-14-centos go_hello]# kubectl create secret docker-registry harbor-registry-key --docker-server=*.*.*.*:81 --docker-username=robot$mylibary+mylibary --docker-password=6GX3YKCJ8EDheTbAFAyLTA4CjfbtlSWz
secret/harbor-registry-key created
# 实际：新建账号
[root@VM-16-14-centos go_hello]# kubectl create secret docker-registry harbor-registry-key --docker-server=127.0.0.1:81 --docker-username=mylibary --docker-password=Mylibary_A1
[root@VM-16-14-centos go_hello]# kubectl create secret docker-registry harbor-registry-key --docker-server=*.*.*.*:81 --docker-username=mylibary --docker-password=Mylibary_A1
[root@VM-16-14-centos go_hello]# k get se
secrets serviceaccounts services
serverstransports.traefik.containo.us servicemonitors.monitoring.coreos.com
[root@VM-16-14-centos go_hello]# k get secrets
NAME TYPE DATA AGE
default-token-b8xch kubernetes.io/service-account-token 3 19d
harbor-registry-key kubernetes.io/dockerconfigjson 1 11s
harborsecret kubernetes.io/dockerconfigjson 1 21m
ingress-nginx-admission-token-2sjj7 kubernetes.io/service-account-token 3 18d
sh.helm.release.v1.traefik.v1 helm.sh/release.v1 1 12d
sh.helm.release.v1.traefik.v2 helm.sh/release.v1 1 12d
sh.helm.release.v1.traefik.v3 helm.sh/release.v1 1 12d
sh.helm.release.v1.traefik.v4 helm.sh/release.v1 1 12d
sh.helm.release.v1.traefik.v5 helm.sh/release.v1 1 12d
sh.helm.release.v1.traefik.v6 helm.sh/release.v1 1 12d
traefik-token-h79wc kubernetes.io/service-account-token 3 12d
3.4、修改deploy配置引入imagePullSecrets
- 重新修改deploy文件
[root@VM-16-14-centos go_hello]# cat go.dep.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: go-hello-deployment
spec:
replicas: 1
selector:
matchLabels:
app: go-hello-pod
template:
metadata:
labels:
app: go-hello-pod
spec:
containers:
- name: go-hello-container
#image: 127.0.0.1:81/mylibary/go_hello:v1 #镜像名称+版本
image: 1.*.*.*.:81/mylibary/go_hello:v1 #镜像名称+版本
imagePullPolicy: Always #表示镜像来源，IfNotPresent本地没有就从hub仓库拉取，Never表示只从本地
ports:
- containerPort: 8081
imagePullSecrets:
#- name: harborsecret
- name: harbor-registry-key
[root@VM-16-14-centos harbor]# k describe pod go-hello-deployment-6544f88c8c-58kt9
- deploy部署成功，成功pull镜像
四、pod状态异常排查
4.1、pod Back-off restarting failed container解决
- 原因：镜像有问题，更新镜像Dockerfile
[root@VM-16-14-centos buildkit-v0.9.2]# cd go_hello_8080/
[root@VM-16-14-centos go_hello_8080]# ll
总用量 8
-rw-r--r-- 1 root root 146 11月 13 10:14 Dockerfile
-rw-r--r-- 1 root root 338 11月 13 10:14 main.go
[root@VM-16-14-centos go_hello_8080]# vim Dockerfile
[root@VM-16-14-centos go_hello_8080]# vi main.go
[root@VM-16-14-centos go_hello_8080]# cp ../go_hello/
Dockerfile go.dep.yml main.go
[root@VM-16-14-centos go_hello_8080]# cp ../go_hello/go.dep.yml .
[root@VM-16-14-centos go_hello_8080]# vim go.dep.yml
[root@VM-16-14-centos go_hello_8080]# kubectl apply -f go.dep.yml
deployment.apps/go-hello8080-deployment created
4.2、验证镜像以及服务状态
[root@VM-16-14-centos go_hello_8080]# k describe pod go-hello8080-deployment-5988c6db6c-b4xqz
Status: Running
IP: 172.30.2.194
IPs:
IP: 172.30.2.194
Containers:
go-hello8080-container:
Port: 8080/TCP
Host Port: 0/TCP
State: Running
Started: Sat, 13 Nov 2021 20:19:39 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qxhwt (ro)
[root@VM-16-14-centos go_hello_8080]# curl 172.30.2.194:8080/Hello
- 镜像打包的go服务，启动8080端口，GET请求/Hello返回hello_world
