Java基准测试神器JMH

编程学习指南
180 阅读3分钟

我们在很多场景需要对代码进行性能的测试：

  • 在进行性能优化的时候已经把瓶颈定位在了某个局部的代码，现在需要逐步进行优化，以及验证
  • 对多种代码实现方案的性能比对，挑选出性能优的方案

这个时候我们就需要一款能够对Java代码/方法进行性能测试的工具。JMH（Java Microbenchmark Harness）是一款小型的Java基准测试工具。何为Microbenchmark？能够进行method级别的benchmark测试，精确到微妙级。

更多分布式系统设计资料：
github.com/xiajunhust/…

从一个示例开始

我们看如下一段测试代码，用来比较2个字符串拼接方法的性能差异：

maven依赖：

<dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-core</artifactId>
    <version>1.23</version>
</dependency>
<dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-generator-annprocess</artifactId>
    <version>1.23</version>
</dependency>

\

package com.alipay.worktest.jmh;

import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.infra.Blackhole;
import org.openjdk.jmh.results.format.ResultFormatType;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

import java.util.concurrent.TimeUnit;

@BenchmarkMode(Mode.AverageTime)
@Warmup(iterations = 3, time = 1)
@Measurement(iterations = 5, time = 5)
@Threads(4)
@Fork(1)
@State(value = Scope.Benchmark)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
public class StringConnectTest {
    
    @Param(value = {"10", "50", "100"})
    private int length;
    
    @Benchmark
    public void testStringAdd(Blackhole blackhole) {
        String a = "";
        for (int i = 0; i < length; i++) {
            a += i;
        }
        blackhole.consume(a);
    }
    
    @Benchmark
    public void testStringBuilderAdd(Blackhole blackhole) {
        StringBuilder sb = new StringBuilder();
        for (int i = 0; i < length; i++) {
            sb.append(i);
        }
        blackhole.consume(sb.toString());
    }
    
    public static void main(String[] args) throws RunnerException {
        Options opt = new OptionsBuilder()
            .include(StringConnectTest.class.getSimpleName())
            .result("result.json")
            .resultFormat(ResultFormatType.JSON).build();
        new Runner(opt).run();
    }
}

\

跑出来的结果是一个JSON格式数据：

[
  {
    "jmhVersion" : "1.23",
    "benchmark" : "com.alipay.worktest.jmh.StringConnectTest.testStringAdd",
    "mode" : "avgt",
    "threads" : 4,
    "forks" : 1,
    "jvm" : "/Library/Java/JavaVirtualMachines/jdk1.8.0_321.jdk/Contents/Home/jre/bin/java",
    "jvmArgs" : [
      "-javaagent:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=53328:/Applications/IntelliJ IDEA.app/Contents/bin",
      "-Dfile.encoding=UTF-8"
    ],
    "jdkVersion" : "1.8.0_321",
    "vmName" : "Java HotSpot(TM) 64-Bit Server VM",
    "vmVersion" : "25.321-b07",
    "warmupIterations" : 3,
    "warmupTime" : "1 s",
    "warmupBatchSize" : 1,
    "measurementIterations" : 5,
    "measurementTime" : "5 s",
    "measurementBatchSize" : 1,
    "params" : {
      "length" : "10"
    },
    "primaryMetric" : {
      "score" : 104.37353532768762,
      "scoreError" : 54.40382524005513,
      "scoreConfidence" : [
        49.96971008763249,
        158.77736056774273
      ],
      "scorePercentiles" : {
        "0.0" : 96.11150190679969,
        "50.0" : 96.59461051460224,
        "90.0" : 129.00380282412652,
        "95.0" : 129.00380282412652,
        "99.0" : 129.00380282412652,
        "99.9" : 129.00380282412652,
        "99.99" : 129.00380282412652,
        "99.999" : 129.00380282412652,
        "99.9999" : 129.00380282412652,
        "100.0" : 129.00380282412652
      },
      "scoreUnit" : "ns/op",
      "rawData" : [
        [
          129.00380282412652,
          103.69461443995105,
          96.59461051460224,
          96.11150190679969,
          96.46314695295848
        ]
      ]
    },
    "secondaryMetrics" : {
    }
  },
  {
    "jmhVersion" : "1.23",
    "benchmark" : "com.alipay.worktest.jmh.StringConnectTest.testStringAdd",
    "mode" : "avgt",
    "threads" : 4,
    "forks" : 1,
    "jvm" : "/Library/Java/JavaVirtualMachines/jdk1.8.0_321.jdk/Contents/Home/jre/bin/java",
    "jvmArgs" : [
      "-javaagent:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=53328:/Applications/IntelliJ IDEA.app/Contents/bin",
      "-Dfile.encoding=UTF-8"
    ],
    "jdkVersion" : "1.8.0_321",
    "vmName" : "Java HotSpot(TM) 64-Bit Server VM",
    "vmVersion" : "25.321-b07",
    "warmupIterations" : 3,
    "warmupTime" : "1 s",
    "warmupBatchSize" : 1,
    "measurementIterations" : 5,
    "measurementTime" : "5 s",
    "measurementBatchSize" : 1,
    "params" : {
      "length" : "50"
    },
    "primaryMetric" : {
      "score" : 839.7015408571585,
      "scoreError" : 1012.0904808153387,
      "scoreConfidence" : [
        -172.38893995818012,
        1851.7920216724972
      ],
      "scorePercentiles" : {
        "0.0" : 695.2649561315446,
        "50.0" : 720.1534550275057,
        "90.0" : 1305.5735933226465,
        "95.0" : 1305.5735933226465,
        "99.0" : 1305.5735933226465,
        "99.9" : 1305.5735933226465,
        "99.99" : 1305.5735933226465,
        "99.999" : 1305.5735933226465,
        "99.9999" : 1305.5735933226465,
        "100.0" : 1305.5735933226465
      },
      "scoreUnit" : "ns/op",
      "rawData" : [
        [
          720.1534550275057,
          695.2649561315446,
          695.3827943248662,
          782.13290547923,
          1305.5735933226465
        ]
      ]
    },
    "secondaryMetrics" : {
    }
  },
  {
    "jmhVersion" : "1.23",
    "benchmark" : "com.alipay.worktest.jmh.StringConnectTest.testStringAdd",
    "mode" : "avgt",
    "threads" : 4,
    "forks" : 1,
    "jvm" : "/Library/Java/JavaVirtualMachines/jdk1.8.0_321.jdk/Contents/Home/jre/bin/java",
    "jvmArgs" : [
      "-javaagent:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=53328:/Applications/IntelliJ IDEA.app/Contents/bin",
      "-Dfile.encoding=UTF-8"
    ],
    "jdkVersion" : "1.8.0_321",
    "vmName" : "Java HotSpot(TM) 64-Bit Server VM",
    "vmVersion" : "25.321-b07",
    "warmupIterations" : 3,
    "warmupTime" : "1 s",
    "warmupBatchSize" : 1,
    "measurementIterations" : 5,
    "measurementTime" : "5 s",
    "measurementBatchSize" : 1,
    "params" : {
      "length" : "100"
    },
    "primaryMetric" : {
      "score" : 32956.27123766383,
      "scoreError" : 267012.8315788902,
      "scoreConfidence" : [
        -234056.56034122637,
        299969.102816554
      ],
      "scorePercentiles" : {
        "0.0" : 1841.5207789439423,
        "50.0" : 1875.3263286044403,
        "90.0" : 156999.3648593761,
        "95.0" : 156999.3648593761,
        "99.0" : 156999.3648593761,
        "99.9" : 156999.3648593761,
        "99.99" : 156999.3648593761,
        "99.999" : 156999.3648593761,
        "99.9999" : 156999.3648593761,
        "100.0" : 156999.3648593761
      },
      "scoreUnit" : "ns/op",
      "rawData" : [
        [
          156999.3648593761,
          2223.449094824512,
          1841.5207789439423,
          1875.3263286044403,
          1841.6951265701437
        ]
      ]
    },
    "secondaryMetrics" : {
    }
  },
  {
    "jmhVersion" : "1.23",
    "benchmark" : "com.alipay.worktest.jmh.StringConnectTest.testStringBuilderAdd",
    "mode" : "avgt",
    "threads" : 4,
    "forks" : 1,
    "jvm" : "/Library/Java/JavaVirtualMachines/jdk1.8.0_321.jdk/Contents/Home/jre/bin/java",
    "jvmArgs" : [
      "-javaagent:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=53328:/Applications/IntelliJ IDEA.app/Contents/bin",
      "-Dfile.encoding=UTF-8"
    ],
    "jdkVersion" : "1.8.0_321",
    "vmName" : "Java HotSpot(TM) 64-Bit Server VM",
    "vmVersion" : "25.321-b07",
    "warmupIterations" : 3,
    "warmupTime" : "1 s",
    "warmupBatchSize" : 1,
    "measurementIterations" : 5,
    "measurementTime" : "5 s",
    "measurementBatchSize" : 1,
    "params" : {
      "length" : "10"
    },
    "primaryMetric" : {
      "score" : 73.26168608974095,
      "scoreError" : 6.80320933870852,
      "scoreConfidence" : [
        66.45847675103244,
        80.06489542844946
      ],
      "scorePercentiles" : {
        "0.0" : 71.66782608452925,
        "50.0" : 72.71042944925414,
        "90.0" : 76.2384487740947,
        "95.0" : 76.2384487740947,
        "99.0" : 76.2384487740947,
        "99.9" : 76.2384487740947,
        "99.99" : 76.2384487740947,
        "99.999" : 76.2384487740947,
        "99.9999" : 76.2384487740947,
        "100.0" : 76.2384487740947
      },
      "scoreUnit" : "ns/op",
      "rawData" : [
        [
          71.66782608452925,
          72.71042944925414,
          73.31208404057011,
          72.3796421002565,
          76.2384487740947
        ]
      ]
    },
    "secondaryMetrics" : {
    }
  },
  {
    "jmhVersion" : "1.23",
    "benchmark" : "com.alipay.worktest.jmh.StringConnectTest.testStringBuilderAdd",
    "mode" : "avgt",
    "threads" : 4,
    "forks" : 1,
    "jvm" : "/Library/Java/JavaVirtualMachines/jdk1.8.0_321.jdk/Contents/Home/jre/bin/java",
    "jvmArgs" : [
      "-javaagent:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=53328:/Applications/IntelliJ IDEA.app/Contents/bin",
      "-Dfile.encoding=UTF-8"
    ],
    "jdkVersion" : "1.8.0_321",
    "vmName" : "Java HotSpot(TM) 64-Bit Server VM",
    "vmVersion" : "25.321-b07",
    "warmupIterations" : 3,
    "warmupTime" : "1 s",
    "warmupBatchSize" : 1,
    "measurementIterations" : 5,
    "measurementTime" : "5 s",
    "measurementBatchSize" : 1,
    "params" : {
      "length" : "50"
    },
    "primaryMetric" : {
      "score" : 382.5654880443052,
      "scoreError" : 3.175081894080194,
      "scoreConfidence" : [
        379.390406150225,
        385.7405699383854
      ],
      "scorePercentiles" : {
        "0.0" : 381.1825172868479,
        "50.0" : 382.8092182310429,
        "90.0" : 383.30706169689176,
        "95.0" : 383.30706169689176,
        "99.0" : 383.30706169689176,
        "99.9" : 383.30706169689176,
        "99.99" : 383.30706169689176,
        "99.999" : 383.30706169689176,
        "99.9999" : 383.30706169689176,
        "100.0" : 383.30706169689176
      },
      "scoreUnit" : "ns/op",
      "rawData" : [
        [
          382.8092182310429,
          382.52083165866816,
          383.30706169689176,
          383.0078113480757,
          381.1825172868479
        ]
      ]
    },
    "secondaryMetrics" : {
    }
  },
  {
    "jmhVersion" : "1.23",
    "benchmark" : "com.alipay.worktest.jmh.StringConnectTest.testStringBuilderAdd",
    "mode" : "avgt",
    "threads" : 4,
    "forks" : 1,
    "jvm" : "/Library/Java/JavaVirtualMachines/jdk1.8.0_321.jdk/Contents/Home/jre/bin/java",
    "jvmArgs" : [
      "-javaagent:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=53328:/Applications/IntelliJ IDEA.app/Contents/bin",
      "-Dfile.encoding=UTF-8"
    ],
    "jdkVersion" : "1.8.0_321",
    "vmName" : "Java HotSpot(TM) 64-Bit Server VM",
    "vmVersion" : "25.321-b07",
    "warmupIterations" : 3,
    "warmupTime" : "1 s",
    "warmupBatchSize" : 1,
    "measurementIterations" : 5,
    "measurementTime" : "5 s",
    "measurementBatchSize" : 1,
    "params" : {
      "length" : "100"
    },
    "primaryMetric" : {
      "score" : 778.4593415225656,
      "scoreError" : 62.30734983271735,
      "scoreConfidence" : [
        716.1519916898483,
        840.766691355283
      ],
      "scorePercentiles" : {
        "0.0" : 762.777875969865,
        "50.0" : 772.686315788206,
        "90.0" : 803.2454160577608,
        "95.0" : 803.2454160577608,
        "99.0" : 803.2454160577608,
        "99.9" : 803.2454160577608,
        "99.99" : 803.2454160577608,
        "99.999" : 803.2454160577608,
        "99.9999" : 803.2454160577608,
        "100.0" : 803.2454160577608
      },
      "scoreUnit" : "ns/op",
      "rawData" : [
        [
          762.777875969865,
          803.2454160577608,
          768.1863916764114,
          772.686315788206,
          785.400708120585
        ]
      ]
    },
    "secondaryMetrics" : {
    }
  }
]

更多的示例可参考：

github.com/openjdk/jmh…

JMH基础知识介绍

@BenchmarkMode

JMH能够以不同的模式来执行基准测试，JMH支持如下几种模式：

  • Throughput：吞吐率，即每秒被测试方法被执行的次数
  • Average Time：方法平均执行耗时
  • Sample Time：随机采样，展示其分布
  • Single Shot Time：只执行一次，获取其耗时，一般用于测试冷启动的性能。

@State

用于定义作用范围。

  • Scope.Benchmark：所有测试线程共享一个实例，测试有状态实例在多线程共享下的性能
  • Scope.Group：同一个线程在同一个 group 里共享实例
  • Scope.Thread：默认的 State，每个测试线程分配一个实例

@Warmup

预热配置参数。

  • iterations：预热的次数
  • time：每次预热的时间
  • timeUnit：时间的单位，默认秒
  • batchSize：批处理大小，每次操作调用几次方法

@Threads

每个进程的测试线程数。

@Fork

进行 fork 的次数，可用于类或者方法上。如果 fork 数是 2 的话，则 JMH 会 fork 出两个进程来进行测试。

JMH使用注意事项

预热-JVM JIT机制

JIT是指Java中的即时编译器（Just-in-time）跟静态编译相比，即时编译器选择性的编译热点代码，省去编译时间。因此我们在进行基准测试的时候，需要进行预热，以获得真实情况下的测试结果。

JIT编译优化-死码消除

JVM可能会把从未使用的变量相关的方法的整个代码全部移除，这样会导致测试结果不准确。解决办法是一种是增加return变量来显式引用，另一种是通过 Blackhole 的 consume 来避免。

public void testMethod() {
    int a = 1;
    int b = 2;
    int sum = a + b;
  }

解决死码示例：

public void testMethod(Blackhole blackhole) {
    int a = 1;
    int b = 2;
    int sum = a + b;
    blackhole.consume(sum);
}

避免循环

JVM会对循环进行优化，这样会导致获取的测试结果不准确。

JMH结果可视化展示

JMH测试结果是一个json串，我们也可以通过一些工具进行可视化直观展示。

avatar
文章
阅读
粉丝