前端可以了解的buildkit知识本篇的主要目的是站在前端的角度来看下docker buildkit怎么使用，及在实际使

本篇的主要目的是站在前端的角度来看下docker buildkit怎么使用，及在实际使用的过程中可能会碰到什么问题，并通过什么样的方式去解决

背景

在CI场景使用buildkit构建前端项目镜像，可能会碰到一些问题，具体的问题如下所示

常见错误

1: yarn install错误`ENOTEMPTY: directory not empty, rmdir`等，如下图所示

2: webpack build错误 `Module not found: Error:`等,如下图所示

3: 执行build命令的时候提示`/bin/sh cross-env: not found`,如下图所示

解决方法1

如果未使用公司的nodejs镜像，需要替换成公司的nodejs镜像，然后重新接测原因是：公司的nodejs镜像内处理了一些 install 通用错误，会进行错误缓存删除与重新install 如果已经使用了公司nodejs镜像，还是无法正常install or build成功，有两种可能

全局缓存目录内的npm包有问题
/app/node_modules下的包有问题

# 添加一行新的RUN命令,用于清除错误缓存
RUN --mount=type=cache,target=/app/node_modules,id=trade-znsk-back-test,sharing=locked --mount=type=cache,target=/usr/local/share/.cache/yarn/v6 --mount=type=cache,target=/usr/local/share/.cache/yarn/v4 rm -rf /usr/local/share/.cache/yarn/v6/npm-lodash-4.* && rm -rf /app/node_modules

注意一点要先通过--mount=type=cache,target挂载了缓存目录之后才可以删除

解决方法2

在CI构建时添加配置--no-cache参数，如果是急着上线的，可以考虑使用这种方式，但是等接测成功，或者分支上了之后，还是需要将--no-cache参数去掉，避免无法复用缓存，导致CI场景构建速度变慢，如果去掉之后还是出现类似问题，推荐通过解决方法1解决

Options: [
  BuildCtlArgs: "--no-cache"
]

上面列举了碰到的问题，及提供的解决方法，但是为什么会出现这些问题，下面是一些关于buildkit的具体实践与验证，有兴趣可以往下看

buildkit简单使用

启用buildkit

如果Docker Desktop and Docker Engine >= v23.0，那么默认是开启了buildkit的如果是Docker Desktop and Docker Engine < v23.0，那么需要通过DOCKER_BUILDKIT参数开启

DOCKER_BUILDKIT=1 docker build .

buildkit参数

以前端的npm全局缓存与项目内的node_modules为例

# 工作目录
WORKDIR /app

ADD ./package.json /app/package.json
ADD ./yarn.lock /app/yarn.lock
RUN --mount=type=cache,target=/app/node_modules,id=trade-znsk-back-test,sharing=locked --mount=type=cache,target=/usr/local/share/.cache/yarn/v6 --mount=type=cache,target=/usr/local/share/.cache/yarn/v4 npm config set registry https://registry.npmmirror.com && yarn --frozen-lockfile --check-files
ARG  IMAGE_TAG
ENV IMAGE_TAG=$IMAGE_TAG
ADD . /app
RUN --mount=type=cache,target=/app/node_modules,id=trade-znsk-back-test,sharing=locked --mount=type=cache,target=/usr/local/share/.cache/yarn/v6 --mount=type=cache,target=/usr/local/share/.cache/yarn/v4 yarn build

挂载项目下/app/node_modules缓存 --mount=type=cache,target=/app/node_modules,id=trade-znsk-back-test,sharing=locked有id，并且sharing=locked

挂载yarn全局缓存 --mount=type=cache,target=/usr/local/share/.cache/yarn/v6没有id，sharing=shared

/app/node_modules是按项目分支来进行复用，sharing=locked所以并行构建时按照顺序使用缓存目录，这样做是比较合理的，即可以最大程度的复用缓存，又可以避免频繁出问题 /usr/local/share/.cache/yarn/v6 所有项目共用，sharing=shared所以并行构建时同时使用缓存目录，所以会存在竞争写的问题，这样设置是合理的，因为包是不具有项目相关的属性的，所以可以给所有项目共用，又不能设置成locked，会因为排队导致构建总时间提升，所以设置为shared更合理

buildkit缓存作用范围与生命周期

下面以该Dockerfile配置为例，来构建镜像，确认buildkit缓存作用范围与生命周期

FROM yunke-registry.cn-hangzhou.cr.aliyuncs.com/yued/node-image-base:v12.22.11-alpine-rc

WORKDIR /app

ADD ./package.json /app/package.json
ADD ./yarn.lock /app/yarn.lock

# 安装依赖
RUN --mount=type=cache,target=/app/node_modules,id=first-1,sharing=locked --mount=type=cache,target=/usr/local/share/.cache/yarn/v6,id=v6-1 ls /app/node_modules && echo 'xxxxxx install' && ls /usr/local/share/.cache/yarn/v6 && yarn --frozen-lockfile --check-files --verbose

# 将docker构建参数转换到环境变量
ARG IMAGE_TAG
ENV IMAGE_TAG=$IMAGE_TAG

ADD . /app
# 构建代码
RUN --mount=type=cache,target=/app/node_modules,id=first-1,sharing=locked --mount=type=cache,target=/usr/local/share/.cache/yarn/v6,id=v6-1 ls /app/node_modules && echo 'xxxxxx build' && ls /usr/local/share/.cache/yarn/v6 && yarn build

第一次构建镜像

PROGRESS_NO_TRUNC=1 docker build --progress plain -t cache-test .

输出日志

#10 [stage-0 5/7] RUN --mount=type=cache,target=/app/node_modules,id=first-1,sharing=locked --mount=type=cache,target=/usr/local/share/.cache/yarn/v6,id=v6-1 ls /app/node_modules && echo 'xxxxxx install' && ls /usr/local/share/.cache/yarn/v6 && yarn --frozen-lockfile --check-files --verbose
[1/4] Resolving packages...
[2/4] Fetching packages...
verbose 0.969130601 Performing "GET" request to "https://registry.npmmirror.com/md5/-/md5-2.3.0.tgz".
verbose 1.033409254 Performing "GET" request to "https://registry.npmmirror.com/is-buffer/-/is-buffer-1.1.6.tgz".
[3/4] Linking dependencies...
verbose 1.350317255 Creating directory "/app/node_modules/crypt".
verbose 1.35079673 Creating directory "/app/node_modules/charenc".
[4/4] Building fresh packages...
Done in 0.86s.
#10 2.849 安装结束: id:bukit_2023_04_24_10_23_49.358_Dof2MwDSj1682303029361 number:0 code:0
#10 DONE 2.9s

#11 [stage-0 6/7] ADD . /app
#11 sha256:3739680a59588f8ec252f84cde74662094159630c3d9b14cf140eec23bb29c4f
#11 DONE 0.0s

#12 [stage-0 7/7] RUN --mount=type=cache,target=/app/node_modules,id=first-1,sharing=locked --mount=type=cache,target=/usr/local/share/.cache/yarn/v6,id=v6-1 ls /app/node_modules && echo 'xxxxxx build' && ls /usr/local/share/.cache/yarn/v6 && yarn build
#12 sha256:98fddd45964f8d6c204bd9edf7f8c18265d1d0d80317d9de943779de14abb792
#12 0.322 charenc
#12 0.322 crypt
#12 0.322 is-buffer
#12 0.322 md5
#12 0.323 xxxxxx build
#12 0.325 npm-charenc-0.0.2-c0a1d2f3a7092e03774bfa83f14c0fc5790a8667-integrity
#12 0.325 npm-crypt-0.0.2-88d7ff7ec0dfb86f713dc87bbb42d044d3e6c41b-integrity
#12 0.325 npm-is-buffer-1.1.6-efaa2ea9daa0d7ab2ea13a97b2b8ad51fefbe8be-integrity
#12 0.325 npm-md5-2.3.0-c3da9a6aae3a30b46b7b0c349b87b110dc3bda4f-integrity
#12 0.430 userHome:  /usr/local/share
#12 0.460 当前yinstall version: 0.0.1-beta.8
#12 0.460 实际执行命令(yarn): yarnpkg build
#12 0.770 yarn run v1.22.17
#12 0.878 $ echo 'build1'
#12 0.920 build1
#12 0.920 Done in 0.16s.
#12 DONE 1.0s

从构建日志可以看出第一个RUN执行时，挂载的target=/app/node_modules为空，挂载的/usr/local/share/.cache/yarn/v6也为空第二个RUN执行时，挂载的target=/app/node_modules就有对应的缓存，挂载的/usr/local/share/.cache/yarn/v6也有对应缓存

通过 docker system df -v --format '{{ .BuildCache | json }}'命令查看docker创建的缓存

可以看到创建了两条新的缓存类型，分别是/app/node_modules与/usr/local/share/.cache/yarn/v6，但是注意到这里的ID并非是RUN --mount=type=cache,target=/app/node_modules,id=first-1命令中的这个id

第二次构建镜像

修改package.json，重新构建镜像，构建日志如下所示

#6 [stage-0 3/7] ADD ./package.json /app/package.json
#6 sha256:51328cf253e9807b3545c36fb20fa2f4398d2d0711fe681bec9d4d9459ed7ffa
#6 DONE 0.1s

#7 [stage-0 4/7] ADD ./yarn.lock /app/yarn.lock
#7 sha256:bc39fb6837acc411f5d9855f3cf1a3489566d11cb6a33cbbb6db5605265f3f93
#7 DONE 0.0s

#8 [stage-0 5/7] RUN --mount=type=cache,target=/app/node_modules,id=first-1,sharing=locked --mount=type=cache,target=/usr/local/share/.cache/yarn/v6,id=v6-1 ls /app/node_modules && echo 'xxxxxx install' && ls /usr/local/share/.cache/yarn/v6 && yarn --frozen-lockfile --check-files --verbose
#8 sha256:65f1568dbee5606c6031b3bc0159fc72d0b5a10585ea6396aee4aec8652ed2e9
#8 0.354 charenc
#8 0.354 crypt
#8 0.354 is-buffer
#8 0.354 md5
#8 0.355 xxxxxx install
#8 0.356 npm-charenc-0.0.2-c0a1d2f3a7092e03774bfa83f14c0fc5790a8667-integrity
#8 0.356 npm-crypt-0.0.2-88d7ff7ec0dfb86f713dc87bbb42d044d3e6c41b-integrity
#8 0.356 npm-is-buffer-1.1.6-efaa2ea9daa0d7ab2ea13a97b2b8ad51fefbe8be-integrity
#8 0.356 npm-md5-2.3.0-c3da9a6aae3a30b46b7b0c349b87b110dc3bda4f-integrity
#8 0.486 userHome:  /usr/local/share
#8 0.514 当前yinstall version: 0.0.1-beta.8
#8 0.516 实际执行命令(yinstall): yarnpkg --frozen-lockfile --check-files --verbose
yarn install v1.22.17
verbose 0.382990853 current time: 2023-04-24T02:34:44.316Z
[1/4] Resolving packages...
success Already up-to-date.
Done in 0.28s.
#8 1.107 安装结束: id:bukit_2023_04_24_10_34_43.920_nhh5j3Aa_1682303683921 number:0 code:0
#8 DONE 1.2s

#9 [stage-0 6/7] ADD . /app
#9 sha256:f5eeabfe203bfe2fcdec8f94dc68260a3666a5073aafa72295b780a89e5ebc2d
#9 DONE 0.1s

#10 [stage-0 7/7] RUN --mount=type=cache,target=/app/node_modules,id=first-1,sharing=locked --mount=type=cache,target=/usr/local/share/.cache/yarn/v6,id=v6-1 ls /app/node_modules && echo 'xxxxxx build' && ls /usr/local/share/.cache/yarn/v6 && yarn build
#10 sha256:8f18e1171530483bbb3dc8282c843f3bfccbb1ce6d149064a4b17ae6a8699ada
#10 0.299 charenc
#10 0.299 crypt
#10 0.299 is-buffer
#10 0.299 md5
#10 0.299 xxxxxx build
#10 0.301 npm-charenc-0.0.2-c0a1d2f3a7092e03774bfa83f14c0fc5790a8667-integrity
#10 0.301 npm-crypt-0.0.2-88d7ff7ec0dfb86f713dc87bbb42d044d3e6c41b-integrity
#10 0.301 npm-is-buffer-1.1.6-efaa2ea9daa0d7ab2ea13a97b2b8ad51fefbe8be-integrity
#10 0.301 npm-md5-2.3.0-c3da9a6aae3a30b46b7b0c349b87b110dc3bda4f-integrity
#10 0.410 userHome:  /usr/local/share
#10 0.429 当前yinstall version: 0.0.1-beta.8
#10 0.429 实际执行命令(yarn): yarnpkg build
#10 0.797 yarn run v1.22.17
#10 0.919 $ echo 'build1'
#10 0.955 build1
#10 0.958 Done in 0.19s.
#10 DONE 1.0s

从日志可以看到第一个RUN执行时，挂载的target=/app/node_modules有上一次的缓存内容，挂载的/usr/local/share/.cache/yarn/v6也有对应的缓存第二个RUN执行时，挂载的target=/app/node_modules也有对应的缓存，挂载的/usr/local/share/.cache/yarn/v6也有对应的缓存

虽然层缓存没有了，但是--mount挂载的缓存还是有的

再次通过docker system df -v --format '{{ .BuildCache | json }}'查看数据，ID是没有变的，变的是使用次数及时间相关的字段

小结

buildkit缓存，不受层缓存影响
buildkit根据id来进行复用

no-cache 作用

当我们不想复用缓存的时候可以通过--no-cache参数来禁用缓存

第三次构建镜像

执行build命令，第三次构建镜像，添加--no-cache参数

PROGRESS_NO_TRUNC=1 docker build --no-cache --progress plain -t cache-test .

具体日志如下所示

#6 [stage-0 3/7] ADD ./package.json /app/package.json
#6 sha256:a4f4f42f21e8aac43b1b34bf2907e1aca42f41a9e23e3655813dda92c612dcca
#6 DONE 0.0s

#7 [stage-0 4/7] ADD ./yarn.lock /app/yarn.lock
#7 sha256:09f715ce302d285d108d365de1c6841298a61d501bf3f8ce1e87c98a80207a0c
#7 DONE 0.0s

#8 [stage-0 5/7] RUN --mount=type=cache,target=/app/node_modules,id=first-1,sharing=locked --mount=type=cache,target=/usr/local/share/.cache/yarn/v6,id=v6-1 ls /app/node_modules && echo 'xxxxxx install' && ls /usr/local/share/.cache/yarn/v6 && yarn --frozen-lockfile --check-files --verbose
#8 sha256:28beb65dcd4aca56201454aeb1e675abce5e190eee3e77379730ad39c7e946aa
#8 0.333 xxxxxx install
#8 0.442 userHome:  /usr/local/share
#8 0.467 当前yinstall version: 0.0.1-beta.8
#8 0.469 实际执行命令(yinstall): yarnpkg --frozen-lockfile --check-files --verbose
yarn install v1.22.17
verbose 0.424501455 current time: 2023-04-24T02:45:22.493Z
[1/4] Resolving packages...
[2/4] Fetching packages...
verbose 0.755493378 Performing "GET" request to "https://registry.npmmirror.com/md5/-/md5-2.3.0.tgz".
[3/4] Linking dependencies...
verbose 1.25078219 Creating directory "/app/node_modules/charenc".
verbose 1.251225593 Creating directory "/app/node_modules/crypt".
verbose 1.263046235 Copying "/usr/local/share/.cache/yarn/v6/npm-charenc-0.0.2-c0a1d2f3a7092e03774bfa83f14c0fc5790a8667-integrity/node_modules/charenc/LICENSE.mkd" to "/app/node_modules/charenc/LICENSE.mkd".
verbose 1.264316963 Copying "/usr/local/share/.cache/yarn/v6/npm-charenc-0.0.2-c0a1d2f3a7092e03774bfa83f14c0fc5790a8667-integrity/node_modules/charenc/README.js" to "/app/node_modules/charenc/README.js".
[4/4] Building fresh packages...
Done in 1.07s.
#8 1.895 安装结束: id:bukit_2023_04_24_10_45_22.052_wWX8Dfztj1682304322053 number:0 code:0
#8 DONE 2.0s

#9 [stage-0 6/7] ADD . /app
#9 sha256:27e579ffe721c474bfe6f32507a2ecb03e5c104cd67c03b67704a01aa9104d47
#9 DONE 0.1s

#10 [stage-0 7/7] RUN --mount=type=cache,target=/app/node_modules,id=first-1,sharing=locked --mount=type=cache,target=/usr/local/share/.cache/yarn/v6,id=v6-1 ls /app/node_modules && echo 'xxxxxx build' && ls /usr/local/share/.cache/yarn/v6 && yarn build
#10 sha256:83ef8426c52ec657b2cc31b7019c2c02f5f38d938997562bd12002bed74161b9
#10 0.319 charenc
#10 0.319 crypt
#10 0.319 is-buffer
#10 0.319 md5
#10 0.319 xxxxxx build
#10 0.321 npm-charenc-0.0.2-c0a1d2f3a7092e03774bfa83f14c0fc5790a8667-integrity
#10 0.322 npm-crypt-0.0.2-88d7ff7ec0dfb86f713dc87bbb42d044d3e6c41b-integrity
#10 0.322 npm-is-buffer-1.1.6-efaa2ea9daa0d7ab2ea13a97b2b8ad51fefbe8be-integrity
#10 0.322 npm-md5-2.3.0-c3da9a6aae3a30b46b7b0c349b87b110dc3bda4f-integrity
#10 0.450 userHome:  /usr/local/share
#10 0.485 当前yinstall version: 0.0.1-beta.8
#10 0.486 实际执行命令(yarn): yarnpkg build
#10 0.812 yarn run v1.22.17
#10 0.920 $ echo 'build1'
#10 0.963 build1
#10 0.965 Done in 0.17s.
#10 DONE 1.0s

使用--no-cache之后第一个RUN命令内，挂载的target=/app/node_modules为空，挂载的/usr/local/share/.cache/yarn/v6也为空第二个RUN执行时，挂载的target=/app/node_modules就有对应的缓存，挂载的/usr/local/share/.cache/yarn/v6也有对应的问题了

使用docker system df -v --format '{{ .BuildCache | json }}'查看缓存数据从上面的数据可以看到，之前旧的cache被取代，生成了一个新的ID，及重新计数

第四次构建镜像

跨项目复用

PROGRESS_NO_TRUNC=1 docker build --progress plain -t cache-test-project2 .

日志如下

#8 [stage-0 3/7] ADD ./package.json /app/package.json
#8 sha256:00993009cc2393484705e1eab8ed592a275290677d729cdebc35e26a22a479f3
#8 DONE 0.1s

#9 [stage-0 4/7] ADD ./yarn.lock /app/yarn.lock
#9 sha256:2a7cea640800abb9d8e43069b3989df36d75d2179bd9cef236eb6cf63acfa09d
#9 DONE 0.0s

#10 [stage-0 5/7] RUN --mount=type=cache,target=/app/node_modules,id=first-1,sharing=locked --mount=type=cache,target=/usr/local/share/.cache/yarn/v6,id=v6-1 ls /app/node_modules && echo 'xxxxxx install' && ls /usr/local/share/.cache/yarn/v6 && yarn --frozen-lockfile --check-files --verbose
#10 sha256:2e6b55b763984d4ed3167e5ba8dbf060bc35d16f59635d1a026a9caa76f74c71
#10 0.374 charenc
#10 0.374 crypt
#10 0.374 is-buffer
#10 0.374 md5
#10 0.374 xxxxxx install
#10 0.375 npm-charenc-0.0.2-c0a1d2f3a7092e03774bfa83f14c0fc5790a8667-integrity
#10 0.375 npm-crypt-0.0.2-88d7ff7ec0dfb86f713dc87bbb42d044d3e6c41b-integrity
#10 0.375 npm-is-buffer-1.1.6-efaa2ea9daa0d7ab2ea13a97b2b8ad51fefbe8be-integrity
#10 0.375 npm-md5-2.3.0-c3da9a6aae3a30b46b7b0c349b87b110dc3bda4f-integrity
#10 0.493 userHome:  /usr/local/share
#10 0.527 当前yinstall version: 0.0.1-beta.8
#10 0.529 实际执行命令(yinstall): yarnpkg --frozen-lockfile --check-files --verbose
yarn install v1.22.17
verbose 0.471145432 current time: 2023-04-24T03:11:06.656Z
[1/4] Resolving packages...
success Already up-to-date.
Done in 0.23s.
#10 1.177 安装结束: id:bukit_project2_2023_04_24_11_11_06.169_nRetDcLFF1682305866170 number:0 code:0
#10 DONE 1.3s

#11 [stage-0 6/7] ADD . /app
#11 sha256:a3fb93ecc4236a6e5e5423b8aa9cdac06371dff2badcda3eeaea42e1df504e85
#11 DONE 0.1s

#12 [stage-0 7/7] RUN --mount=type=cache,target=/app/node_modules,id=first-1,sharing=locked --mount=type=cache,target=/usr/local/share/.cache/yarn/v6,id=v6-1 ls /app/node_modules && echo 'xxxxxx build' && ls /usr/local/share/.cache/yarn/v6 && yarn build
#12 sha256:41ccc41ec1bdba60fa3e226e65a41b4d19141c6984d4531fe5d0d6b83b7b073f
#12 0.326 charenc
#12 0.326 crypt
#12 0.326 is-buffer
#12 0.326 md5
#12 0.327 xxxxxx build
#12 0.328 npm-charenc-0.0.2-c0a1d2f3a7092e03774bfa83f14c0fc5790a8667-integrity
#12 0.328 npm-crypt-0.0.2-88d7ff7ec0dfb86f713dc87bbb42d044d3e6c41b-integrity
#12 0.328 npm-is-buffer-1.1.6-efaa2ea9daa0d7ab2ea13a97b2b8ad51fefbe8be-integrity
#12 0.328 npm-md5-2.3.0-c3da9a6aae3a30b46b7b0c349b87b110dc3bda4f-integrity
#12 0.522 userHome:  /usr/local/share
#12 0.542 当前yinstall version: 0.0.1-beta.8
#12 0.542 实际执行命令(yarn): yarnpkg build
#12 0.845 yarn run v1.22.17
#12 0.949 $ echo 'build1'
#12 0.982 build1
#12 0.984 Done in 0.15s.
#12 DONE 1.0s

从日志可以看到，第一次与第二次RUN命令内都直接复用了mount cache缓存

从数据可以看到ID没变，使用数增加，也就是没有生成新的cache

小结

--no-cache不会复用层缓存
--no-cache会让--mount=type=cache 同一个cache id 失效，但是后面有相同的--mount=type=cache还是会继续复用
buildkit缓存会跨镜像复用

结论

buildkit缓存可以通过id区分复用范围，通过sharing区分挂载时机，如果不主动清除，缓存会一直存在
- CI场景下/app/node_modules是以项目分支维度来进行隔离，所以缓存是项目分支级别，并通过sharing=locked，按顺序控制缓存目录挂载
- CI场景下/usr/local/share/.cache/yarn/v6是以路径维度进行隔离，所以缓存是全局级别，同一个docker构建的镜像项目都可以复用，并通过sharing=shared，并行控制缓存目录挂载，所以会存在yarn install竞争写的问题，可以通过yinstall解决(公司nodejs基础镜像内置)
--no-cache参数，既可以禁用docker 层缓存，也可以禁用首次包含挂载目录命令内的buildkit缓存

前端可以了解的buildkit知识

背景