在工作中，我遇到了一个需要从系统网页下载 Docker 镜像包的需求，目标是直接将结果下载为 tar 包。我们使用的镜像服务是 Harbor，它提供了丰富的系统管理 API，但遗憾的是，似乎没有找到用于下载或导出镜像的接口。这让我感到非常困惑。
经过反复思考，我决定从 Docker pull 协议入手。值得注意的是，Docker pull 是基于权限的 HTTP 协议，支持镜像各个层的下载，但它并不直接包含 manifest.json 文件。这意味着我需要自己编写代码来组装这些信息。在这个过程中，我还需要深入分析 tar 包的结构，以及 manifest.json 与各个层（layers）之间的关系。同时，我还必须理解加密验证的逻辑，以确保下载的镜像包是完整且有效的。
这个过程并不简单。首先，我需要了解 Docker 镜像的层级结构。每个 Docker 镜像由多个层组成，这些层通过 layers 字段在 manifest.json 中被描述。每一层都是只读的，可以被多个镜像共享，这也使得镜像的存储更加高效。其次，manifest.json 文件包含了关于镜像的元数据，比如标签、架构和层的哈希值等信息，这对下载和验证镜像的完整性至关重要。
此外，还要考虑到安全性和权限管理。Harbor 通常会对访问镜像的请求进行身份验证，确保只有授权用户可以下载镜像。因此，我需要处理相应的身份验证流程，获取所需的访问令牌，以便能够顺利执行下载操作。
目前已经通过这一系列的步骤，成功下载到所需的 Docker 镜像包，并确保它的完整性和可用性。这不仅是一个技术挑战，也让我对 Docker 镜像的内部机制有了更深刻的理解。
以下就是具体代码实现的过程，逻辑因为实验性质，所以并没有做任何抽象，只在Main类里直接过程形式书写，注释较少还请见谅。
java 依赖
dependencies {
implementation 'org.json:json:20231013'
implementation 'org.apache.commons:commons-compress:1.26.0'
}
java 代码
package org.example;
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import com.google.gson.JsonArray;
import com.google.gson.JsonObject;
import org.apache.commons.codec.binary.Base64;
import org.apache.commons.compress.archivers.tar.TarArchiveEntry;
import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream;
import java.io.*;
import java.net.HttpURLConnection;
import java.net.URL;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.*;
import java.util.stream.Collectors;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.nio.charset.StandardCharsets;
public class Main {
static String projectName = "project-name/";
public static void main(String[] args) throws Exception {
String registryUrl = "http://127.0.0.1:8008";
String baseImageName = "alpine";
String tag = "v1.0.0";
String username = "admin";
String password = "*********";
try {
String imageDir = downloadImage(projectName+baseImageName, registryUrl, tag, username, password);
createTarArchive(imageDir, imageDir + ".tar");
}catch (Exception e){
e.printStackTrace();
}
}
private static void createTarArchive(String sourceDir, String tarFile) throws IOException {
try (TarArchiveOutputStream tarOs = new TarArchiveOutputStream(new FileOutputStream(tarFile))) {
tarOs.setLongFileMode(TarArchiveOutputStream.LONGFILE_POSIX);
Path sourcePath = Paths.get(sourceDir);
Files.walk(sourcePath)
.filter(path -> !Files.isDirectory(path))
.forEach(path -> {
try {
String entryName = sourcePath.relativize(path).toString();
TarArchiveEntry entry = new TarArchiveEntry(path.toFile(), entryName);
tarOs.putArchiveEntry(entry);
Files.copy(path, tarOs);
tarOs.closeArchiveEntry();
} catch (IOException e) {
throw new RuntimeException("Failed to add file to tar: " + path, e);
}
});
tarOs.finish();
}
}
private static String downloadImage(String imageName, String registryUrl, String tag, String username, String password) throws Exception {
// 创建目标目录结构
String imageDir = imageName.replace("/", "_");
Files.createDirectories(Paths.get(imageDir, "blobs", "sha256"));
// 下载 manifest
String manifestJson = downloadManifest(registryUrl, imageName, tag, username, password);
Map manifest = new Gson().fromJson(manifestJson, Map.class);
List<Map> layers = (List<Map>) manifest.get("layers");
for (Map layer : layers) {
String digest = (String) layer.get("digest");
String sha256 = digest.replace("sha256:", "");
downloadBlob(registryUrl, imageName, sha256, username, password,
Paths.get(imageDir, "blobs", "sha256", sha256));
}
String dockerManifest = createDockerManifest(imageName, tag, manifest);
Files.write(Paths.get(imageDir, "manifest.json"), dockerManifest.getBytes());
Map config = (Map) manifest.get("config");
String configDigest = ((String) config.get("digest")).replace("sha256:", "");
downloadBlob(registryUrl, imageName, configDigest, username, password,
Paths.get(imageDir, "blobs", "sha256", configDigest));
String ociLayout = "{"imageLayoutVersion": "1.0.0"}";
Files.write(Paths.get(imageDir, "oci-layout"), ociLayout.getBytes());
String indexJson = createIndexJson(manifest, imageName, tag);
Files.write(Paths.get(imageDir, "index.json"), indexJson.getBytes());
String repoJson = createRepositoriesJson(imageName, tag, manifest);
Files.write(Paths.get(imageDir, "repositories"), repoJson.getBytes());
return imageDir;
}
private static String downloadManifest(String registryUrl, String imageName, String tag,
String username, String password) throws Exception {
String auth = Base64.encodeBase64String((username + ":" + password).getBytes());
String manifestUrl = registryUrl + "/v2/" + imageName + "/manifests/" + tag;
URL url = new URL(manifestUrl);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.setRequestProperty("Authorization", "Basic " + auth);
conn.setRequestProperty("Accept", "application/vnd.docker.distribution.manifest.v2+json");
try (BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getInputStream()))) {
return reader.lines().collect(Collectors.joining("\n"));
}
}
private static void downloadBlob(String registryUrl, String imageName, String digest,
String username, String password, Path destPath) throws Exception {
String auth = Base64.encodeBase64String((username + ":" + password).getBytes());
String blobUrl = registryUrl + "/v2/" + imageName + "/blobs/sha256:" + digest;
URL url = new URL(blobUrl);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.setRequestProperty("Authorization", "Basic " + auth);
try (InputStream in = conn.getInputStream();
OutputStream out = Files.newOutputStream(destPath)) {
byte[] buffer = new byte[8192];
int bytesRead;
while ((bytesRead = in.read(buffer)) != -1) {
out.write(buffer, 0, bytesRead);
}
}
}
private static String createDockerManifest(String imageName, String tag, Map<String, Object> manifest) {
JsonArray manifestArray = new JsonArray();
JsonObject manifestEntry = new JsonObject();
// 设置 Config
Map<String, Object> config = (Map<String, Object>) manifest.get("config");
String configDigest = (String) config.get("digest");
manifestEntry.addProperty("Config", "blobs/sha256/" + configDigest.replace("sha256:", ""));
// 设置 RepoTags
JsonArray repoTags = new JsonArray();
repoTags.add(imageName.replace(Main.projectName,"") + ":" + tag);
manifestEntry.add("RepoTags", repoTags);
// 设置 Layers
JsonArray layerPaths = new JsonArray();
List<Map<String, Object>> layers = (List<Map<String, Object>>) manifest.get("layers");
for (Map<String, Object> layer : layers) {
String digest = (String) layer.get("digest");
layerPaths.add("blobs/sha256/" + digest.replace("sha256:", ""));
}
manifestEntry.add("Layers", layerPaths);
// 设置 LayerSources
JsonObject layerSources = new JsonObject();
for (Map<String, Object> layer : layers) {
String digest = (String) layer.get("digest");
JsonObject layerInfo = new JsonObject();
layerInfo.addProperty("mediaType", (String) layer.get("mediaType"));
layerInfo.addProperty("size", ((Number) layer.get("size")).longValue());
layerInfo.addProperty("digest", digest);
layerSources.add(digest, layerInfo);
}
manifestEntry.add("LayerSources", layerSources);
// 添加到数组中
manifestArray.add(manifestEntry);
return new GsonBuilder().setPrettyPrinting().create().toJson(manifestArray);
}
// 添加计算 SHA256 的辅助方法
private static String calculateSha256(String input) {
try {
MessageDigest digest = MessageDigest.getInstance("SHA-256");
byte[] hash = digest.digest(input.getBytes(StandardCharsets.UTF_8));
StringBuilder hexString = new StringBuilder();
for (byte b : hash) {
String hex = Integer.toHexString(0xff & b);
if (hex.length() == 1) hexString.append('0');
hexString.append(hex);
}
return hexString.toString();
} catch (NoSuchAlgorithmException e) {
throw new RuntimeException("Failed to calculate SHA-256", e);
}
}
private static String createIndexJson(Map<String, Object> manifest, String imageName, String tag) {
JsonObject index = new JsonObject();
// 设置基本属性
index.addProperty("schemaVersion", 2);
index.addProperty("mediaType", "application/vnd.oci.image.index.v1+json");
// 创建 manifests 数组
JsonArray manifests = new JsonArray();
JsonObject manifestEntry = new JsonObject();
// 设置 manifest 条目的属性
manifestEntry.addProperty("mediaType", "application/vnd.oci.image.manifest.v1+json");
// 计算 manifest 的 digest
String manifestJson = new Gson().toJson(manifest);
String manifestDigest = calculateSha256(manifestJson);
manifestEntry.addProperty("digest", "sha256:" + manifestDigest);
// 设置 size
manifestEntry.addProperty("size", manifestJson.length());
// 添加 annotations
JsonObject annotations = new JsonObject();
annotations.addProperty("io.containerd.image.name", imageName.replace(Main.projectName,"") + ":" + tag);
annotations.addProperty("org.opencontainers.image.ref.name", tag);
manifestEntry.add("annotations", annotations);
manifests.add(manifestEntry);
index.add("manifests", manifests);
return new GsonBuilder().setPrettyPrinting().create().toJson(index);
}
private static String createRepositoriesJson(String imageName, String tag, Map manifest) {
Map<String, Object> repositories = new HashMap<>();
Map<String, Object> tags = new HashMap<>();
Map config = (Map) manifest.get("config");
String configDigest = (String) config.get("digest");
tags.put(tag, configDigest);
repositories.put(imageName, tags);
return new GsonBuilder().setPrettyPrinting().create().toJson(repositories);
}
}
运行结果
09:04:00: 正在执行 ':org.example.Main.main()'…
> Task :compileJava UP-TO-DATE
> Task :processResources NO-SOURCE
> Task :classes UP-TO-DATE
> Task :org.example.Main.main()
BUILD SUCCESSFUL in 612ms
2 actionable tasks: 1 executed, 1 up-to-date
09:04:01: 执行完成 ':org.example.Main.main()'。
拉取组装后的目录结构
与初始创建的目录结构有一些区别，但可以正常导出到docker里
导入本地 docker hub测试
PS：本文的内容是本人实验性的测试，毫无保留的分享给大家，但这不是标准答案，请慎用，我们共同研究
