protobuf-maven-plugin 插件详解

982 阅读3分钟

插件源码地址

github.com/xolstice/pr…

插件属性 - configuration 解析

protocArtifact

指定具体的protobuf编译器

com.google.protobuf:protoc:${protobuf.version}:exe:${os.detected.classifier}

这里 protobuf.version 是maven里用户自定义的属性值,这里选的是3.21.7,其他的版本可以上maven repo里搜索。 os.detected.classifier 变量的值是通过os-maven-plugin执行器获取,具体可以参考底部章节

<extensions>
    <extension>
        <groupId>kr.motd.maven</groupId>
        <artifactId>os-maven-plugin</artifactId>
        <version>1.7.0</version>
    </extension>
</extensions>

这里主要目的是兼容开发和测试等各个环境基础平台的差异。

pluginId

文档里很清楚的说明了,这是一个唯一id,对于protoc插件,且不能使用下面的内置值: java, javanano, js, csharp, cpp, python, descriptor-set

pluginArtifact

编译成java代码的插件,推荐用官方提供的生成器插件

io.grpc:protoc-gen-grpc-java:${grpc.version}:exe:${os.detected.classifier}

protoSourceRoot

.proto 文件的目录,默认为 ${basedir}/src/main/proto ,指定了编译器从哪里文件目录读取最终需要编译文件成源码的文件。

outputDirectory

输出目录,指将编译的结果java文件放到哪个文件目录下

clearOutputDirectory

是否清理输出的目的目录,默认为true。这个属性在每一次执行时都会用到,对于有多个exectution和 goal的,都建议设置为false。

additionalProtoPathElements

额外的proto文件目录,可以指定多个目录,语法格式:

<additionalProtoPathElements>
    <additionalProtoPathElement>${basedir}/src/main/proto</additionalProtoPathElement>
    <additionalProtoPathElement>${project.build.directory}/proto-shared</additionalProtoPathElement>
</additionalProtoPathElements>

protoc 命令解析

基本语法


protoc [选项] 输入文件.proto

常用选项

选项格式说明
--<lang>_out=目录指定生成代码的目标语言和输出目录(如 --java_out=src/main/java
--proto_path=目录 / -I 目录指定 .proto 文件的搜索路径(类似 C 语言的 -I 选项),默认是当前目录
--plugin=protoc-gen-xxx=路径指定第三方插件(如 gRPC 代码生成器)的路径
--version查看 protoc 版本

具体示例与解析

protoc --proto_path=/Users/zhangsan/IdeaProjects/fan-grpc-server/src/main/proto  \ 
 --proto_path=/Users/zhangsan/IdeaProjects/fotile-grpc-server/target/protoc-dependencies/6b63959f70068927950ba8f7060f3dde \ 
 --proto_path=/Users/zhangsan/IdeaProjects/fotile-grpc-server/target/protoc-dependencies/8bc9516eb52724c330bdee3a4872d438  \
 --proto_path=/Users/zhangsan/IdeaProjects/fotile-grpc-server/target/proto-shared  \
 --java_out=/Users/zhangsan/IdeaProjects/fotile-grpc-server/target/generated-sources/protobuf/java \ 
 /Users/zhangsan/IdeaProjects/fotile-grpc-server/src/main/proto/myUser.proto

我们来看下, --prot_path 可以多次指定目录,表示需要依赖或者引用proto文件来自哪里,通常用在被编译的.proto文件有依赖其他的.proto内容,需要去哪个地方找对应的.proto文件 --java_out 则指定了要将编译后的源码输出到哪个目录 最后的protowen文件,就是需要被编译的原始proto文件

os-maven-plugin 插件介绍

其作用主要用于在构建过程中识别当前操作系统的信息(如操作系统名称、架构等),并将这些信息作为 Maven 属性暴露出来,方便在 pom.xml 中根据不同操作系统进行条件化配置。

配置后可以使用的主要属性包括:

  • ${os.name}:操作系统名称(如 Windows、Linux、Mac OS X)
  • ${os.arch}:系统架构(如 x86、amd64、aarch64)
  • ${os.detected.name}:标准化的操作系统名称(如 windows、linux、osx)
  • ${os.detected.arch}:标准化的系统架构(如 x86_64、x86、arm64)
  • ${os.detected.classifier}:组合的分类器(如 windows-x86_64、linux-arm64)

插件源码解读 基于tag=0.6.1

源文件

org.xolstice.maven.plugin.protobuf.AbstractProtocMojo#execute() throws MojoExecutionException, MojoFailureException {

        if (skipMojo()) {
            return;
        }

        try {
            checkParameters();
        } catch (final MojoConfigurationException e) {
            throw new MojoExecutionException("Configuration error: " + e.getMessage(), e);
        } catch (final MojoInitializationException e) {
            throw new MojoExecutionException(e.getMessage(), e);
        }

        getLog().info("START TO DOODODODODODOOD");
        final File protoSourceRoot = getProtoSourceRoot();
        if (protoSourceRoot.exists()) {
            try {
                final List<File> protoFiles = findProtoFilesInDirectory(protoSourceRoot);
                final File outputDirectory = getOutputDirectory();
                final List<File> outputFiles = findGeneratedFilesInDirectory(getOutputDirectory());

                if (protoFiles.isEmpty()) {
                    getLog().info("No proto files to compile.");
                } else if (!hasDelta(protoFiles)) {
                    getLog().info("Skipping compilation because build context has no changes.");
                    doAttachFiles();
                } else if (checkStaleness && checkFilesUpToDate(protoFiles, outputFiles)) {
                    getLog().info("Skipping compilation because target directory newer than sources.");
                    doAttachFiles();
                } else {
                    final List<File> derivedProtoPathElements =
                            makeProtoPathFromJars(temporaryProtoFileDirectory, getDependencyArtifactFiles());
                    FileUtils.mkdir(outputDirectory.getAbsolutePath());

                    if (clearOutputDirectory) {
                        try {
                            cleanDirectory(outputDirectory);
                        } catch (final IOException e) {
                            throw new MojoInitializationException("Unable to clean output directory", e);
                        }
                    }

                    if (writeDescriptorSet) {
                        final File descriptorSetOutputDirectory = getDescriptorSetOutputDirectory();
                        FileUtils.mkdir(descriptorSetOutputDirectory.getAbsolutePath());
                        if (clearOutputDirectory) {
                            try {
                                cleanDirectory(descriptorSetOutputDirectory);
                            } catch (final IOException e) {
                                throw new MojoInitializationException(
                                        "Unable to clean descriptor set output directory", e);
                            }
                        }
                    }

                    createProtocPlugins();

                    //get toolchain from context
                    final Toolchain tc = toolchainManager.getToolchainFromBuildContext("protobuf", session); //NOI18N
                    if (tc != null) {
                        getLog().info("Toolchain in protobuf-maven-plugin: " + tc);
                        //when the executable to use is explicitly set by user in mojo's parameter, ignore toolchains.
                        if (protocExecutable != null) {
                            getLog().warn(
                                    "Toolchains are ignored, 'protocExecutable' parameter is set to " + protocExecutable);
                        } else {
                            //assign the path to executable from toolchains
                            protocExecutable = tc.findTool("protoc"); //NOI18N
                        }
                    }
                    if (protocExecutable == null && protocArtifact != null) {
                        final Artifact artifact = createDependencyArtifact(protocArtifact);
                        final File file = resolveBinaryArtifact(artifact);
                        protocExecutable = file.getAbsolutePath();
                    }
                    if (protocExecutable == null) {
                        // Try to fall back to 'protoc' in $PATH
                        getLog().warn("No 'protocExecutable' parameter is configured, using the default: 'protoc'");
                        protocExecutable = "protoc";
                    }

                    final Protoc.Builder protocBuilder =
                            new Protoc.Builder(protocExecutable)
                                    .addProtoPathElement(protoSourceRoot)
                                    .addProtoPathElements(derivedProtoPathElements)
                                    .addProtoPathElements(asList(additionalProtoPathElements))
                                    .addProtoFiles(protoFiles);
                    addProtocBuilderParameters(protocBuilder);
                    final Protoc protoc = protocBuilder.build();

                    if (getLog().isDebugEnabled()) {
                        getLog().debug("Proto source root:");
                        getLog().debug(" " + protoSourceRoot);

                        if (derivedProtoPathElements != null && !derivedProtoPathElements.isEmpty()) {
                            getLog().debug("Derived proto paths:");
                            for (final File path : derivedProtoPathElements) {
                                getLog().debug(" " + path);
                            }
                        }

                        if (additionalProtoPathElements != null && additionalProtoPathElements.length > 0) {
                            getLog().debug("Additional proto paths:");
                            for (final File path : additionalProtoPathElements) {
                                getLog().debug(" " + path);
                            }
                        }
                    }
                    protoc.logExecutionParameters(getLog());

                    getLog().info(format("Compiling %d proto file(s) to %s", protoFiles.size(), outputDirectory));

                    final int exitStatus = protoc.execute(getLog());
                    if (StringUtils.isNotBlank(protoc.getOutput())) {
                        getLog().info("PROTOC: " + protoc.getOutput());
                    }
                    if (exitStatus != 0) {
                        getLog().error("PROTOC FAILED: " + protoc.getError());
                        for (File pf : protoFiles) {
                            buildContext.removeMessages(pf);
                            buildContext.addMessage(pf, 0, 0, protoc.getError(), BuildContext.SEVERITY_ERROR, null);
                        }
                        throw new MojoFailureException(
                                "protoc did not exit cleanly. Review output for more information.");
                    } else if (StringUtils.isNotBlank(protoc.getError())) {
                        getLog().warn("PROTOC: " + protoc.getError());
                    }
                    doAttachFiles();
                }
            } catch (final MojoConfigurationException e) {
                throw new MojoExecutionException("Configuration error: " + e.getMessage(), e);
            } catch (final MojoInitializationException e) {
                throw new MojoExecutionException(e.getMessage(), e);
            } catch (final CommandLineException e) {
                throw new MojoExecutionException("An error occurred while invoking protoc: " + e.getMessage(), e);
            } catch (final InterruptedException e) {
                getLog().info("Process interrupted");
            }
        } else {
            getLog().info(format("%s does not exist. Review the configuration or consider disabling the plugin.",
                    protoSourceRoot));
        }
}

这里是读取最终需要编译的proto文件列表 image.png 注意上面,如果这个protoSourceRoot参数重定义值配置错误时,会导致protoc直接不执行,可以查看maven插件执行时的日志:

does not exist. Review the configuration or consider disabling the plugin.

如果出现这个说明你的配置有问题。

image.png 这里 derivedProtoPathElements 的变量值,看下来自哪里:

image.png 其中 temporaryProtoFileDirectory 默认值为: ${project.build.directory}/protoc-dependencies 即 target/protoc-dependencies目录, 参考我本地的情况:

image.png

image.png

关于target/protoc-dependencies目录下的文件是怎么产生的?

image.png

image.png 在这个for循环里,其实就是对依赖的包里的文件进行查找

for (final File classpathElementFile : classpathElementFiles) {
    // for some reason under IAM, we receive poms as dependent files
    // I am excluding .xml rather than including .jar as there may be other extensions in use (sar, har, zip)
    if (classpathElementFile.isFile() && classpathElementFile.canRead() &&
            !classpathElementFile.getName().endsWith(".xml")) {

        // create the jar file. the constructor validates.
        final JarFile classpathJar;
        try {
            classpathJar = new JarFile(classpathElementFile);
        } catch (final IOException e) {
            throw new MojoInitializationException(
                    "Not a readable JAR artifact: " + classpathElementFile.getAbsolutePath(), e);
        }
        final Enumeration<JarEntry> jarEntries = classpathJar.entries();
        while (jarEntries.hasMoreElements()) {
            final JarEntry jarEntry = jarEntries.nextElement();
            final String jarEntryName = jarEntry.getName();
            // TODO try using org.codehaus.plexus.util.SelectorUtils.matchPath() with DEFAULT_INCLUDES
            // 这里就是在jar包里查找 .proto的文件
            if (jarEntryName.endsWith(PROTO_FILE_SUFFIX)) {
                final File jarDirectory;
                try {
                    jarDirectory = new File(temporaryProtoFileDirectory, truncatePath(classpathJar.getName()));
                    // Check for Zip Slip vulnerability
                    // https://snyk.io/research/zip-slip-vulnerability
                    final String canonicalJarDirectoryPath = jarDirectory.getCanonicalPath();
                    final File uncompressedCopy = new File(jarDirectory, jarEntryName);
                    final String canonicalUncompressedCopyPath = uncompressedCopy.getCanonicalPath();
                    if (!canonicalUncompressedCopyPath.startsWith(canonicalJarDirectoryPath + File.separator)) {
                        throw new MojoInitializationException(
                                "ZIP SLIP: Entry " + jarEntry.getName() +
                                        " in " + classpathJar.getName() + " is outside of the target dir");
                    }
                    FileUtils.mkdir(uncompressedCopy.getParentFile().getAbsolutePath());
                    copyStreamToFile(
                            new RawInputStreamFacade(classpathJar.getInputStream(jarEntry)),
                            uncompressedCopy);
                } catch (final IOException e) {
                    throw new MojoInitializationException("Unable to unpack proto files", e);
                }
                protoDirectories.add(jarDirectory);
            }
        }
    } else if (classpathElementFile.isDirectory()) {
        final List<File> protoFiles;
        try {
            protoFiles = getFiles(classpathElementFile, DEFAULT_INCLUDES, null);
        } catch (final IOException e) {
            throw new MojoInitializationException(
                    "Unable to scan for proto files in: " + classpathElementFile.getAbsolutePath(), e);
        }
        if (!protoFiles.isEmpty()) {
            protoDirectories.add(classpathElementFile);
        }
    }
}

image.png 看下这里,根据配置 和jarPath生成初始化的目录 jarDirectory, 然后,把这个文件拷贝到目标目录下。

image.png 这里对.proto后缀的文件进行处理

image.png 这里是真正的执行protoc的过程。

org.xolstice.maven.plugin.protobuf#execute(final Log log) throws CommandLineException, InterruptedException

image.png

参考这里,组建命令行参数

image.png 这里就是写明了--proto_path值的来源 从图片中就看出来了 additionalProtoPathElements 这个配置,针对的是增加--proto_path 的值,而不是 需要编译的 proto 文件

image.png 这里就是指明了需要编译的 .proto文件

总结

  1. 只有protoSourceRoot属性里的文件目录才是最终能被编译的文件
  2. 被编译的.proto文件如果有依赖外部的.proto文件,一般通过dependency引入即可,如果有些依赖的.proto文件在工程其他地方,建议通过 additionalProtoPathElements 这个配置。
  3. 一般不建议改动protoSourceRoot属性值,遵循 约定大于配置的原则。