Flink 在 IDE 开发环境下使用 savepoint 恢复状态数据

应用场景：开发环境下的任务运行时只可以配置checkpoint，不会自动保存savepoint，但是有些场景如恢复广播状态数据等需要用到 savepoint。

解决方案

版本： flink:2.2.0

大体步骤

• 准备一个用于开发环境测试的 savepoint
• 配置 savepoint 恢复路径，即可。

准备 savepoint

• IDE 启动 flink 任务，在控制台打印的日志中要到 Job ID，格式如：f3e53fabb5d6eaf209fd57a759fdde06
• 使用如下命令即可保存一份 savepoint：

  curl -X POST "http://localhost:8081/jobs/{Job ID}/savepoints" \
  -H "Content-Type: application/json" \
  -d '{
  "target-directory": "file:///d:/flink-savepoints-stream/dev",
  "cancel-job": true,
  "format": "canonical"
  }'
  

  • 需要先开启 flink 开发环境的 Web UI，然后将 Job ID 替换掉，并配置保存路径

IDE 配置恢复路径并重新启动

整体配置的代码如下：

        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
//        // ==================== 临时检查点配置 ====================
        env.setParallelism(2);
        env.enableCheckpointing(10000); // 每10秒一次checkpoint
        env.getCheckpointConfig().setCheckpointingConsistencyMode(CheckpointingMode.EXACTLY_ONCE);
        env.getCheckpointConfig().setMinPauseBetweenCheckpoints(8000); // 最小间隔8秒
        env.getCheckpointConfig().setCheckpointTimeout(20000); // 超时20秒
        env.getCheckpointConfig().setMaxConcurrentCheckpoints(1); // 最大并发数
        env.getCheckpointConfig().setTolerableCheckpointFailureNumber(2); // 允许失败次数
        env.getCheckpointConfig().setExternalizedCheckpointRetention(ExternalizedCheckpointRetention.RETAIN_ON_CANCELLATION);
        Configuration flinkConfig = new Configuration();
        flinkConfig.set(StateBackendOptions.STATE_BACKEND, "hashmap");
        flinkConfig.set(CheckpointingOptions.CHECKPOINT_STORAGE, "filesystem");
        flinkConfig.set(CheckpointingOptions.CHECKPOINTS_DIRECTORY, "file:///d:/flink-checkpoints-stream/dev");
        flinkConfig.set(CheckpointingOptions.SAVEPOINT_DIRECTORY, "file:///d:/flink-savepoints-stream/dev");
        flinkConfig.setString("execution.savepoint.path", "file:///d:/flink-savepoints-stream/dev/savepoint-f3e53f-ea5a1be94d9e");
        env.configure(flinkConfig, ControlApplication.class.getClassLoader());

其中的有关 savepoint 部分仅:

        flinkConfig.set(CheckpointingOptions.SAVEPOINT_DIRECTORY, "file:///d:/flink-savepoints-stream/dev");
        flinkConfig.setString("execution.savepoint.path", "file:///d:/flink-savepoints-stream/dev/savepoint-f3e53f-ea5a1be94d9e");

如果有命令报错找不到主类就在配置 flinkConfig 的时候加上主类配置，如：env.configure(flinkConfig, 主类名.class.getClassLoader());