flink 源码阅读(1) yarn-per-job 提交流程

sunlovefly
695 阅读4分钟

0 前言 最近在整理之前源码阅读的内容,将spark和flink同时记录一下 作为对比和学习 本文主要探索的工作最常用的yarn-per-job的方式来

1 flink启动和配置脚本

1.0 启动脚本

我们主要是用 flink-bin\bin\flink.sh脚本来启动flink 其中关键在于

exec $JAVA_RUN $JVM_ARGS $FLINK_ENV_JAVA_OPTS "${log_setting[@]}" -classpath manglePathList "$CC_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS""org.apache.flink.client.cli.CliFrontend "$@"

所以flink 程序的入口在于org.apache.flink.client.cli.CliFrontend这个类


#!/usr/bin/env bash
################################################################################
#  Licensed to the Apache Software Foundation (ASF) under one
#  or more contributor license agreements.  See the NOTICE file
#  distributed with this work for additional information
#  regarding copyright ownership.  The ASF licenses this file
#  to you under the Apache License, Version 2.0 (the
#  "License"); you may not use this file except in compliance
#  with the License.  You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#  See the License for the specific language governing permissions and
# limitations under the License.
################################################################################

target="$0"
# For the case, the executable has been directly symlinked, figure out
# the correct bin path by following its symlink up to an upper bound.
# Note: we can't use the readlink utility here if we want to be POSIX
# compatible.
iteration=0
while [ -L "$target" ]; do
    if [ "$iteration" -gt 100 ]; then
        echo "Cannot resolve path: You have a cyclic symlink in $target."
        break
    fi
    ls=`ls -ld -- "$target"`
    target=`expr "$ls" : '.* -> (.*)$'`
    iteration=$((iteration + 1))
done

# Convert relative path to absolute path
bin=`dirname "$target"`

# get flink config
. "$bin"/config.sh

if [ "$FLINK_IDENT_STRING" = "" ]; then
        FLINK_IDENT_STRING="$USER"
fi

CC_CLASSPATH=`constructFlinkClassPath`

log=$FLINK_LOG_DIR/flink-$FLINK_IDENT_STRING-client-$HOSTNAME.log
log_setting=(-Dlog.file="$log" -Dlog4j.configuration=file:"$FLINK_CONF_DIR"/log4j-cli.properties -Dlog4j.configurationFile=file:"$FLINK_CONF_DIR"/log4j-cli.properties -Dlogback.configurationFile=file:"$FLINK_CONF_DIR"/logback.xml)

# Add Client-specific JVM options
FLINK_ENV_JAVA_OPTS="${FLINK_ENV_JAVA_OPTS} ${FLINK_ENV_JAVA_OPTS_CLI}"

# Add HADOOP_CLASSPATH to allow the usage of Hadoop file systems
exec $JAVA_RUN $JVM_ARGS $FLINK_ENV_JAVA_OPTS "${log_setting[@]}" -classpath "`manglePathList "$CC_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`" org.apache.flink.client.cli.CliFrontend "$@"

1.1 配置脚本

flink/bin/config.sh(相关环境配置都在这里)

2 org.apache.flink.client.cli.CliFrontend 初探

2.1 关键看一下main方法中的

int retCode = SecurityUtils.getInstalledContext() .runSecured(() -> cli.parseParameters(args));

public static void main(final String[] args) {
   //打印一下配置的参数
   EnvironmentInformation.logEnvironmentInfo(LOG, "Command Line Client", args);

   // 1. find the configuration directory 加载一下配置文件
   final String configurationDirectory = getConfigurationDirectoryFromEnv();

   // 2. load the global configuration 加载一下全局的配置
   final Configuration configuration = GlobalConfiguration.loadConfiguration(configurationDirectory);

   // 3. load the custom command lines 加载自定义命令行
   final List<CustomCommandLine> customCommandLines = loadCustomCommandLines(
      configuration,
      configurationDirectory);

   try {
      final CliFrontend cli = new CliFrontend(
         configuration,
         customCommandLines);
      // 安全配置
      SecurityUtils.install(new SecurityConfiguration(cli.configuration));
      int retCode = SecurityUtils.getInstalledContext()
            .runSecured(() -> cli.parseParameters(args));
      System.exit(retCode);
   }
   catch (Throwable t) {
      final Throwable strippedThrowable = ExceptionUtils.stripException(t, UndeclaredThrowableException.class);
      LOG.error("Fatal error while running command line interface.", strippedThrowable);
      strippedThrowable.printStackTrace();
      System.exit(31);
   }
}

进入到parseParameters 这个方法中 根据action我们选择case ACTION_RUN: run(params);中的run() 方法中去看

public int parseParameters(String[] args) {

   // check for action
   if (args.length < 1) {
      CliFrontendParser.printHelp(customCommandLines);
      System.out.println("Please specify an action.");
      return 1;
   }

   // get action
   String action = args[0];

   // remove action from parameters
   final String[] params = Arrays.copyOfRange(args, 1, args.length);

   try {
      // do action
      switch (action) {
         case ACTION_RUN:
            run(params);
            return 0;
         case ACTION_RUN_APPLICATION:
            runApplication(params);
            return 0;
         case ACTION_LIST:
            list(params);
            return 0;
         case ACTION_INFO:
            info(params);
            return 0;
         case ACTION_CANCEL:
            cancel(params);
            return 0;
         case ACTION_STOP:
            stop(params);
            return 0;
         case ACTION_SAVEPOINT:
            savepoint(params);
            return 0;
         case "-h":
         case "--help":
            CliFrontendParser.printHelp(customCommandLines);
            return 0;
         case "-v":
         case "--version":
            String version = EnvironmentInformation.getVersion();
            String commitID = EnvironmentInformation.getRevisionInformation().commitId;
            System.out.print("Version: " + version);
            System.out.println(commitID.equals(EnvironmentInformation.UNKNOWN) ? "" : ", Commit ID: " + commitID);
            return 0;
         default:
            System.out.printf(""%s" is not a valid action.\n", action);
            System.out.println();
            System.out.println("Valid actions are "run", "list", "info", "savepoint", "stop", or "cancel".");
            System.out.println();
            System.out.println("Specify the version option (-v or --version) to print Flink version.");
            System.out.println();
            System.out.println("Specify the help option (-h or --help) to get help on the command.");
            return 1;
      }
   } catch (CliArgsException ce) {
      return handleArgException(ce);
   } catch (ProgramParametrizationException ppe) {
      return handleParametrizationException(ppe);
   } catch (ProgramMissingJobException pmje) {
      return handleMissingJobException();
   } catch (Exception e) {
      return handleError(e);
   }
}

run() 方法中 其中关键有 validateAndGetActiveCommandLine,getEffectiveConfiguration和 executeProgram这三个方法我们依次往里面看一下

protected void run(String[] args) throws Exception {
   LOG.info("Running 'run' command.");
   //获取command选项
   final Options commandOptions = CliFrontendParser.getRunCommandOptions();
   final CommandLine commandLine = getCommandLine(commandOptions, args, true);

   // evaluate help flag
   if (commandLine.hasOption(HELP_OPTION.getOpt())) {
      CliFrontendParser.printHelpForRun(customCommandLines);
      return;
   }
   // 校验commandLine 
   final CustomCommandLine activeCommandLine =
         validateAndGetActiveCommandLine(checkNotNull(commandLine));
   // 程序参数选项
   final ProgramOptions programOptions = ProgramOptions.create(commandLine);

   final PackagedProgram program =
         getPackagedProgram(programOptions);
   
   // 得到jar包的地址
   final List<URL> jobJars = program.getJobJarAndDependencies();
   final Configuration effectiveConfiguration = getEffectiveConfiguration(
         activeCommandLine, commandLine, programOptions, jobJars);

   LOG.debug("Effective executor configuration: {}", effectiveConfiguration);
    // 执行程序       
   try {
      executeProgram(effectiveConfiguration, program);
   } finally {
      program.deleteExtractedLibraries();
   }
}

我们此处肯定有疑问之前我们阅读Spark 源码的时候是根据 反射来创建不同任务的提交方法的，那flink 以何种方式提交的呢？ 那还是得先回到上面org.apache.flink.client.cli.CliFrontend这个类中关键的loadCustomCommandLines方法里面，可以看到有三个add()，这个地方依次添加了 Generic、Yarn 和 Default 三种命令行客户端（后面根据 isActive()按顺序 选择） 看三个add() 方法

public static List<CustomCommandLine> loadCustomCommandLines(Configuration configuration, String configurationDirectory) {
   List<CustomCommandLine> customCommandLines = new ArrayList<>();
   customCommandLines.add(new GenericCLI(configuration, configurationDirectory));

   // Command line interface of the YARN session, with a special initialization here
   // to prefix all options with y/yarn.
   final String flinkYarnSessionCLI = "org.apache.flink.yarn.cli.FlinkYarnSessionCli";
   try {
      customCommandLines.add(
         loadCustomCommandLine(flinkYarnSessionCLI,
            configuration,
            configurationDirectory,
            "y",
            "yarn"));
   } catch (NoClassDefFoundError | Exception e) {
      final String errorYarnSessionCLI = "org.apache.flink.yarn.cli.FallbackYarnSessionCli";
      try {
         LOG.info("Loading FallbackYarnSessionCli");
         customCommandLines.add(
               loadCustomCommandLine(errorYarnSessionCLI, configuration));
      } catch (Exception exception) {
         LOG.warn("Could not load CLI class {}.", flinkYarnSessionCLI, e);
      }
   }

   // Tips: DefaultCLI must be added at last, because getActiveCustomCommandLine(..) will get the
   //       active CustomCommandLine in order and DefaultCLI isActive always return true.
   customCommandLines.add(new DefaultCLI(configuration));

   return customCommandLines;
}

在我们点到 validateAndGetActiveCommandLine 这个方法里面可以点到这个接口方法isActive 的实现类 FlinkYarnSessionCli方法中去 可以发现这段逻辑是 FlinkYarnSessionCli 为 active 时优先返回 FlinkYarnSessionCli。 或者对于 DefaultCli，它的 isActive 方法总是返回 true。

/**
 * Gets the custom command-line for the arguments.
 * @param commandLine The input to the command-line.
 * @return custom command-line which is active (may only be one at a time)
 */
public CustomCommandLine validateAndGetActiveCommandLine(CommandLine commandLine) {
   LOG.debug("Custom commandlines: {}", customCommandLines);
   for (CustomCommandLine cli : customCommandLines) {
      LOG.debug("Checking custom commandline {}, isActive: {}", cli, cli.isActive(commandLine));
      if (cli.isActive(commandLine)) {
         return cli;
      }
   }
   throw new IllegalStateException("No valid command-line found.");
}

关键在这一段 YarnJobClusterExecutor.NAME.equalsIgnoreCase(configuration.get(DeploymentOptions.TARGET))

Name

public static final String NAME = YarnDeploymentTarget.PER_JOB.getName();

PER_JOB("yarn-per-job")

public boolean isActive(CommandLine commandLine) {
   final String jobManagerOption = commandLine.getOptionValue(addressOption.getOpt(), null);
   final boolean yarnJobManager = ID.equals(jobManagerOption);
   final boolean hasYarnAppId = commandLine.hasOption(applicationId.getOpt())
         || configuration.getOptional(YarnConfigOptions.APPLICATION_ID).isPresent();
   final boolean hasYarnExecutor = YarnSessionClusterExecutor.NAME.equalsIgnoreCase(configuration.get(DeploymentOptions.TARGET))
         || YarnJobClusterExecutor.NAME.equalsIgnoreCase(configuration.get(DeploymentOptions.TARGET));
   return hasYarnExecutor || yarnJobManager || hasYarnAppId || (isYarnPropertiesFileMode(commandLine) && yarnApplicationIdFromYarnProperties != null);
}

我们再进入到getEffectiveConfiguration中去其中关键在于 applyCommandLineOptionsToConfiguration这个接口方法我们看一下它的实现类

private <T> Configuration getEffectiveConfiguration(
      final CustomCommandLine activeCustomCommandLine,
      final CommandLine commandLine,
      final ProgramOptions programOptions,
      final List<T> jobJars) throws FlinkException {

   final ExecutionConfigAccessor executionParameters = ExecutionConfigAccessor.fromProgramOptions(
         checkNotNull(programOptions),
         checkNotNull(jobJars));

   final Configuration executorConfig = checkNotNull(activeCustomCommandLine)
         .applyCommandLineOptionsToConfiguration(commandLine);

   final Configuration effectiveConfiguration = new Configuration(executorConfig);

   executionParameters.applyToConfiguration(effectiveConfiguration);
   LOG.debug("Effective executor configuration: {}", effectiveConfiguration);
   return effectiveConfiguration;
}

@Override
public Configuration applyCommandLineOptionsToConfiguration(CommandLine commandLine) throws FlinkException {
   // we ignore the addressOption because it can only contain "yarn-cluster"
   final Configuration effectiveConfiguration = new Configuration(configuration);

   applyDescriptorOptionToConfig(commandLine, effectiveConfiguration);

   final ApplicationId applicationId = getApplicationId(commandLine);
   if (applicationId != null) {
      final String zooKeeperNamespace;
      if (commandLine.hasOption(zookeeperNamespace.getOpt())){
         zooKeeperNamespace = commandLine.getOptionValue(zookeeperNamespace.getOpt());
      } else {
         zooKeeperNamespace = effectiveConfiguration.getString(HA_CLUSTER_ID, applicationId.toString());
      }

      effectiveConfiguration.setString(HA_CLUSTER_ID, zooKeeperNamespace);
      effectiveConfiguration.setString(YarnConfigOptions.APPLICATION_ID, ConverterUtils.toString(applicationId));
      effectiveConfiguration.setString(DeploymentOptions.TARGET, YarnSessionClusterExecutor.NAME);
   } else {
      effectiveConfiguration.setString(DeploymentOptions.TARGET, YarnJobClusterExecutor.NAME);
   }

   if (commandLine.hasOption(jmMemory.getOpt())) {
      String jmMemoryVal = commandLine.getOptionValue(jmMemory.getOpt());
      if (!MemorySize.MemoryUnit.hasUnit(jmMemoryVal)) {
         jmMemoryVal += "m";
      }
      effectiveConfiguration.set(JobManagerOptions.TOTAL_PROCESS_MEMORY, MemorySize.parse(jmMemoryVal));
   }

   if (commandLine.hasOption(tmMemory.getOpt())) {
      String tmMemoryVal = commandLine.getOptionValue(tmMemory.getOpt());
      if (!MemorySize.MemoryUnit.hasUnit(tmMemoryVal)) {
         tmMemoryVal += "m";
      }
      effectiveConfiguration.set(TaskManagerOptions.TOTAL_PROCESS_MEMORY, MemorySize.parse(tmMemoryVal));
   }

   if (commandLine.hasOption(slots.getOpt())) {
      effectiveConfiguration.setInteger(TaskManagerOptions.NUM_TASK_SLOTS, Integer.parseInt(commandLine.getOptionValue(slots.getOpt())));
   }

   dynamicPropertiesEncoded = encodeDynamicProperties(commandLine);
   if (!dynamicPropertiesEncoded.isEmpty()) {
      Map<String, String> dynProperties = getDynamicProperties(dynamicPropertiesEncoded);
      for (Map.Entry<String, String> dynProperty : dynProperties.entrySet()) {
         effectiveConfiguration.setString(dynProperty.getKey(), dynProperty.getValue());
      }
   }

   if (isYarnPropertiesFileMode(commandLine)) {
      return applyYarnProperties(effectiveConfiguration);
   } else {
      return effectiveConfiguration;
   }
}

关键在于这一段 最终会选择的提交方式是yarn-session和yarn-per-job

if (applicationId != null) {
   final String zooKeeperNamespace;
   if (commandLine.hasOption(zookeeperNamespace.getOpt())){
      zooKeeperNamespace = commandLine.getOptionValue(zookeeperNamespace.getOpt());
   } else {
      zooKeeperNamespace = effectiveConfiguration.getString(HA_CLUSTER_ID, applicationId.toString());
   }

   effectiveConfiguration.setString(HA_CLUSTER_ID, zooKeeperNamespace);
   effectiveConfiguration.setString(YarnConfigOptions.APPLICATION_ID, ConverterUtils.toString(applicationId));
   effectiveConfiguration.setString(DeploymentOptions.TARGET, YarnSessionClusterExecutor.NAME);
} else {
   effectiveConfiguration.setString(DeploymentOptions.TARGET, YarnJobClusterExecutor.NAME);
}

executeProgram进入到这个方法里面 program.invokeInteractiveModeForExecution() 一路点进去可以发现和Spark 异曲同工调用了反射的方式的来初始化我们程序的main方法

public static void executeProgram(
      PipelineExecutorServiceLoader executorServiceLoader,
      Configuration configuration,
      PackagedProgram program,
      boolean enforceSingleJobExecution,
      boolean suppressSysout) throws ProgramInvocationException {
   checkNotNull(executorServiceLoader);
   final ClassLoader userCodeClassLoader = program.getUserCodeClassLoader();
   final ClassLoader contextClassLoader = Thread.currentThread().getContextClassLoader();
   try {
      Thread.currentThread().setContextClassLoader(userCodeClassLoader);

      LOG.info("Starting program (detached: {})", !configuration.getBoolean(DeploymentOptions.ATTACHED));

      ContextEnvironment.setAsContext(
         executorServiceLoader,
         configuration,
         userCodeClassLoader,
         enforceSingleJobExecution,
         suppressSysout);

      StreamContextEnvironment.setAsContext(
         executorServiceLoader,
         configuration,
         userCodeClassLoader,
         enforceSingleJobExecution,
         suppressSysout);

      try {
         program.invokeInteractiveModeForExecution();
      } finally {
         ContextEnvironment.unsetAsContext();
         StreamContextEnvironment.unsetAsContext();
      }
   } finally {
      Thread.currentThread().setContextClassLoader(contextClassLoader);
   }
}

public void invokeInteractiveModeForExecution()throwsProgramInvocationException {
    callMainMethod(mainClass, args); 
} 
private static void callMainMethod(Class entryClass, String[] args) throws ProgramInvocationException { 
mainMethod = entryClass.getMethod("main", String[].class);  // 反射调用 main 函数 mainMethod.invoke(null, (Object) args);  }

3 execute 执行流程

3.1 org.apache.flink.streaming.api.environment.StreamExecutionEnvironment 初探 yarn 提交

首先我们一般运行flink调用的是execute 所以我们重点来看这个方法 其中 getStreamGraph这个方法方法我会再研究后面慢慢往里面看

public JobExecutionResult execute(String jobName) throws Exception { ... ... return execute(getStreamGraph(jobName)); 
}

我们往里面点进去可以看到 executeAsync这个同步方法 逻辑是提交模式选择匹配的 factory 根据提交模式选择匹配的 factory

public JobClient executeAsync(StreamGraph streamGraph) throws Exception {
   checkNotNull(streamGraph, "StreamGraph cannot be null.");
   checkNotNull(configuration.get(DeploymentOptions.TARGET), "No execution.target specified in your configuration file.");
/根据提交模式选择匹配的 factory
   final PipelineExecutorFactory executorFactory =
      executorServiceLoader.getExecutorFactory(configuration);

   checkNotNull(
      executorFactory,
      "Cannot find compatible factory for specified execution.target (=%s)",
      configuration.get(DeploymentOptions.TARGET));

   CompletableFuture<JobClient> jobClientFuture = executorFactory
      .getExecutor(configuration)
      .execute(streamGraph, configuration);

   try {
      JobClient jobClient = jobClientFuture.get();
      jobListeners.forEach(jobListener -> jobListener.onJobSubmitted(jobClient, null));
      return jobClient;
   } catch (ExecutionException executionException) {
      final Throwable strippedException = ExceptionUtils.stripExecutionException(executionException);
      jobListeners.forEach(jobListener -> jobListener.onJobSubmitted(null, strippedException));

      throw new FlinkException(
         String.format("Failed to execute job '%s'.", streamGraph.getJobName()),
         strippedException);
   }
}

flink里面使用了工厂模式来创建一个执行器， executeAsync这个方法里面关键在于这个接口方法execute(streamGraph, configuration); 我找到这个实现类 AbstractJobClusterExecutor 里面有一个很关键的地方 此处将流图转成了作业图我们后期再慢慢研究

final JobGraph jobGraph = PipelineExecutorUtils.getJobGraph(pipeline, configuration);

@Internal
public JobClient executeAsync(StreamGraph streamGraph) throws Exception {
   checkNotNull(streamGraph, "StreamGraph cannot be null.");
   checkNotNull(configuration.get(DeploymentOptions.TARGET), "No execution.target specified in your configuration file.");

   final PipelineExecutorFactory executorFactory =
      executorServiceLoader.getExecutorFactory(configuration);

   checkNotNull(
      executorFactory,
      "Cannot find compatible factory for specified execution.target (=%s)",
      configuration.get(DeploymentOptions.TARGET));

   CompletableFuture<JobClient> jobClientFuture = executorFactory
      .getExecutor(configuration)
      .execute(streamGraph, configuration);

   try {
      JobClient jobClient = jobClientFuture.get();
      jobListeners.forEach(jobListener -> jobListener.onJobSubmitted(jobClient, null));
      return jobClient;
   } catch (ExecutionException executionException) {
      final Throwable strippedException = ExceptionUtils.stripExecutionException(executionException);
      jobListeners.forEach(jobListener -> jobListener.onJobSubmitted(null, strippedException));

      throw new FlinkException(
         String.format("Failed to execute job '%s'.", streamGraph.getJobName()),
         strippedException);
   }
}

我们点到关键的方法 execute(streamGraph, configuration)中实现类AbstractJobClusterExecutor,此处有三个比较重要的方法 createClusterDescriptor, getClusterSpecification ,deployJobCluster 这个三个方法显而易见 就是创建集群,指定，发布 job

@Override
public CompletableFuture<JobClient> execute(@Nonnull final Pipeline pipeline, @Nonnull final Configuration configuration) throws Exception {
   // 在这个地方流图转出作业图
   final JobGraph jobGraph = PipelineExecutorUtils.getJobGraph(pipeline, configuration);

   try (final ClusterDescriptor<ClusterID> clusterDescriptor = clusterClientFactory.createClusterDescriptor(configuration)) {
      final ExecutionConfigAccessor configAccessor = ExecutionConfigAccessor.fromConfiguration(configuration);

      final ClusterSpecification clusterSpecification = clusterClientFactory.getClusterSpecification(configuration);

      final ClusterClientProvider<ClusterID> clusterClientProvider = clusterDescriptor
            .deployJobCluster(clusterSpecification, jobGraph, configAccessor.getDetachedMode());
      LOG.info("Job has been submitted with JobID " + jobGraph.getJobID());

      return CompletableFuture.completedFuture(
            new ClusterClientJobClientAdapter<>(clusterClientProvider, jobGraph.getJobID()));
   }
}

我们先看 createClusterDescriptor 点到 getClusterDescriptor 这个方法里面去

@Override
public YarnClusterDescriptor createClusterDescriptor(Configuration configuration) {
   checkNotNull(configuration);

   final String configurationDirectory =
         configuration.get(DeploymentOptionsInternal.CONF_DIR);
   YarnLogConfigUtil.setLogConfigFileInConfig(configuration, configurationDirectory);

   return getClusterDescriptor(configuration);
}

在这个地方我们可以很清晰的看到 创建了 Yarn客户端 并且启动yarnCliet

private YarnClusterDescriptor getClusterDescriptor(Configuration configuration) {
   final YarnClient yarnClient = YarnClient.createYarnClient();
   final YarnConfiguration yarnConfiguration = new YarnConfiguration();

   yarnClient.init(yarnConfiguration);
   yarnClient.start();

   return new YarnClusterDescriptor(
         configuration,
         yarnConfiguration,
         yarnClient,
         YarnClientYarnClusterInformationRetriever.create(yarnClient),
         false);
}

我们再来看一下getClusterSpecification 这个方法此处是个接口方法 我看一下他的实现类 AbstractContainerizedClusterClientFactory 此处可以看到是分配jobManagerMemoryMB 和 taskManagerMemoryMB 还有 slotsPerTaskManager

@Override
public ClusterSpecification getClusterSpecification(Configuration configuration) {
   checkNotNull(configuration);

   final int jobManagerMemoryMB = JobManagerProcessUtils.processSpecFromConfigWithNewOptionToInterpretLegacyHeap(
         configuration,
         JobManagerOptions.TOTAL_PROCESS_MEMORY)
      .getTotalProcessMemorySize()
      .getMebiBytes();

   final int taskManagerMemoryMB = TaskExecutorProcessUtils
      .processSpecFromConfig(TaskExecutorProcessUtils.getConfigurationMapLegacyTaskManagerHeapSizeToConfigOption(
         configuration, TaskManagerOptions.TOTAL_PROCESS_MEMORY))
      .getTotalProcessMemorySize()
      .getMebiBytes();

   int slotsPerTaskManager = configuration.getInteger(TaskManagerOptions.NUM_TASK_SLOTS);

   return new ClusterSpecification.ClusterSpecificationBuilder()
      .setMasterMemoryMB(jobManagerMemoryMB)
      .setTaskManagerMemoryMB(taskManagerMemoryMB)
      .setSlotsPerTaskManager(slotsPerTaskManager)
      .createClusterSpecification();
}

我最后来看 deployJobCluster这个接口方法看一下它的实现类 YarnClusterDescriptor 的方法

@Override
public ClusterClientProvider<ApplicationId> deployJobCluster(
   ClusterSpecification clusterSpecification,
   JobGraph jobGraph,
   boolean detached) throws ClusterDeploymentException {
   try {
      return deployInternal(
         clusterSpecification,
         "Flink per-job cluster",
         getYarnJobClusterEntrypoint(),
         jobGraph,
         detached);
   } catch (Exception e) {
      throw new ClusterDeploymentException("Could not deploy Yarn job cluster.", e);
   }
}

我们点到 return deployInternal(clusterSpecification,"Flink per-job cluster" getYarnJobClusterEntrypoint(),jobGraph,detached);这个方法里面去。这个方法很长 有 检查yarnClient 和创建yarn 应用 还有就是 上传 jar 包和配置文件到 HDFS 我们一点点往里面看 里面有一些关键逻辑

// 集群描述是准备好
isReadyForDeployment(clusterSpecification);

// ------------------ Check if the specified queue exists --------------------
// 检查yarnClient 是否存在
checkYarnQueues(yarnClient);

// ------------------ Check if the YARN ClusterClient has the requested resources --------------

// Create application via yarnClient
// 创建应用的
final YarnClientApplication yarnApplication = yarnClient.createApplication();

关键我们 startAppMaster看这个方法

ApplicationReport report = startAppMaster(
      flinkConfiguration,
      applicationName,
      yarnClusterEntrypoint,
      jobGraph,
      yarnClient,
      yarnApplication,
      validClusterSpecification);

点进去看我们首先能看到 初始化hdfs 文件系统

// ------------------ Initialize the file systems -------------------------
      
org.apache.flink.core.fs.FileSystem.initialize(
      configuration,
      PluginUtils.createPluginManagerFromRootFolder(configuration));

final FileSystem fs = FileSystem.get(yarnConfiguration);

// hard coded check for the GoogleHDFS client because its not overriding the getScheme() method.
if (!fs.getClass().getSimpleName().equals("GoogleHadoopFileSystem") &&
      fs.getScheme().startsWith("file")) {
   LOG.warn("The file system scheme is '" + fs.getScheme() + "'. This indicates that the "
         + "specified Hadoop configuration path is wrong and the system is using the default Hadoop configuration values."
         + "The Flink YARN client needs to store its files in a distributed file system");
}

ApplicationSubmissionContext appContext = yarnApplication.getApplicationSubmissionContext();

final List<Path> providedLibDirs = getRemoteSharedPaths(configuration);

final YarnApplicationFileUploader fileUploader = YarnApplicationFileUploader.from(
   fs,
   fs.getHomeDirectory(),
   providedLibDirs,
   appContext.getApplicationId(),
   getFileReplication());

// The files need to be shipped and added to classpath.
Set<File> systemShipFiles = new HashSet<>(shipFiles.size());
for (File file : shipFiles) {
   systemShipFiles.add(file.getAbsoluteFile());
}

final String logConfigFilePath = configuration.getString(YarnConfigOptionsInternal.APPLICATION_LOG_CONFIG_FILE);
if (logConfigFilePath != null) {
   systemShipFiles.add(new File(logConfigFilePath));
}

上传和配置 ApplicationMaster 的 jar 包：flink-dist*.jar 

final YarnLocalResourceDescriptor localResourceDescFlinkJar = fileUploader.uploadFlinkDist(flinkJarPath);

上传 flink 配置文件

String flinkConfigKey = "flink-conf.yaml"; 
Path remotePathConf = setupSingleLocalResource( flinkConfigKey, fs, appId, new Path(tmpConfigurationFile.getAbsolutePath()), localResources, homeDir, "");

// 将 JobGraph 写入 tmp 文件并添加到本地资源，并上传到 HDFS

fileUploader.registerSingleLocalResource( jobGraphFilename, new Path(tmpJobGraphFile.toURI()), "", true, false);

final JobManagerProcessSpec processSpec = JobManagerProcessUtils.processSpecFromConfigWithNewOptionToInterpretLegacyHeap( flinkConfiguration, JobManagerOptions.TOTAL_PROCESS_MEMORY); 
//封装启动 AM container 的 Java 命令 
final ContainerLaunchContext amContainer = setupApplicationMasterContainer( yarnClusterEntrypoint, hasKrb5, processSpec)

应用上下文设置amContainer

appContext.setApplicationName(customApplicationName);
appContext.setApplicationType(applicationType != null ? applicationType : "Apache Flink");
appContext.setAMContainerSpec(amContainer);
appContext.setResource(capability);

yarn 提交

yarnClient.submitApplication(appContext);

我们先重点看一下 setupApplicationMasterContainer 里面有个关键的JAVA_HOME/bin/java的一般这种都是在am 里面启动一个进程来启动

ContainerLaunchContext setupApplicationMasterContainer(
      String yarnClusterEntrypoint,
      boolean hasKrb5,
      JobManagerProcessSpec processSpec) {
   // ------------------ Prepare Application Master Container  ------------------------------

   // respect custom JVM options in the YAML file
   String javaOpts = flinkConfiguration.getString(CoreOptions.FLINK_JVM_OPTIONS);
   if (flinkConfiguration.getString(CoreOptions.FLINK_JM_JVM_OPTIONS).length() > 0) {
      javaOpts += " " + flinkConfiguration.getString(CoreOptions.FLINK_JM_JVM_OPTIONS);
   }
   //applicable only for YarnMiniCluster secure test run
   //krb5.conf file will be available as local resource in JM/TM container
   if (hasKrb5) {
      javaOpts += " -Djava.security.krb5.conf=krb5.conf";
   }

   // Set up the container launch context for the application master
   ContainerLaunchContext amContainer = Records.newRecord(ContainerLaunchContext.class);

   final  Map<String, String> startCommandValues = new HashMap<>();
   startCommandValues.put("java", "$JAVA_HOME/bin/java");

   String jvmHeapMem = JobManagerProcessUtils.generateJvmParametersStr(processSpec, flinkConfiguration);
   startCommandValues.put("jvmmem", jvmHeapMem);

   startCommandValues.put("jvmopts", javaOpts);
   startCommandValues.put("logging", YarnLogConfigUtil.getLoggingYarnCommand(flinkConfiguration));

   startCommandValues.put("class", yarnClusterEntrypoint);
   startCommandValues.put("redirects",
      "1> " + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/jobmanager.out " +
      "2> " + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/jobmanager.err");
   startCommandValues.put("args", "");

   final String commandTemplate = flinkConfiguration
         .getString(ConfigConstants.YARN_CONTAINER_START_COMMAND_TEMPLATE,
               ConfigConstants.DEFAULT_YARN_CONTAINER_START_COMMAND_TEMPLATE);
   final String amCommand =
      BootstrapTools.getStartCommand(commandTemplate, startCommandValues);

   amContainer.setCommands(Collections.singletonList(amCommand));

   LOG.debug("Application Master start command: " + amCommand);

   return amContainer;
}

// 封装 AM 的 classpath 和环境参数
final Map<String, String> appMasterEnv = new HashMap<>();
// set user specified app master environment variables
appMasterEnv.putAll(
   ConfigurationUtils.getPrefixedKeyValuePairs(ResourceManagerOptions.CONTAINERIZED_MASTER_ENV_PREFIX, configuration));
// set Flink app class path
appMasterEnv.put(YarnConfigKeys.ENV_FLINK_CLASSPATH, classPathBuilder.toString());

// set Flink on YARN internal configuration values
appMasterEnv.put(YarnConfigKeys.FLINK_DIST_JAR, localResourceDescFlinkJar.toString());
appMasterEnv.put(YarnConfigKeys.ENV_APP_ID, appId.toString());
appMasterEnv.put(YarnConfigKeys.ENV_CLIENT_HOME_DIR, fileUploader.getHomeDir().toString());
appMasterEnv.put(YarnConfigKeys.ENV_CLIENT_SHIP_FILES, encodeYarnLocalResourceDescriptorListToString(fileUploader.getEnvShipResourceList()));
appMasterEnv.put(YarnConfigKeys.ENV_ZOOKEEPER_NAMESPACE, getZookeeperNamespace());
appMasterEnv.put(YarnConfigKeys.FLINK_YARN_FILES, fileUploader.getApplicationDir().toUri().toString());

// https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnApplicationSecurity.md#identity-on-an-insecure-cluster-hadoop_user_name
appMasterEnv.put(YarnConfigKeys.ENV_HADOOP_USER_NAME, UserGroupInformation.getCurrentUser().getUserName());

if (localizedKeytabPath != null) {
   appMasterEnv.put(YarnConfigKeys.LOCAL_KEYTAB_PATH, localizedKeytabPath);
   String principal = configuration.getString(SecurityOptions.KERBEROS_LOGIN_PRINCIPAL);
   appMasterEnv.put(YarnConfigKeys.KEYTAB_PRINCIPAL, principal);
   if (remotePathKeytab != null) {
      appMasterEnv.put(YarnConfigKeys.REMOTE_KEYTAB_PATH, remotePathKeytab.toString());
   }
}

//To support Yarn Secure Integration Test Scenario
if (remoteYarnSiteXmlPath != null) {
   appMasterEnv.put(YarnConfigKeys.ENV_YARN_SITE_XML_PATH, remoteYarnSiteXmlPath.toString());
}
if (remoteKrb5Path != null) {
   appMasterEnv.put(YarnConfigKeys.ENV_KRB5_PATH, remoteKrb5Path.toString());
}

// set classpath from YARN configuration
Utils.setupYarnClassPath(yarnConfiguration, appMasterEnv);

amContainer.setEnvironment(appMasterEnv);

我们到 submitApplication 这个接口函数的实现类YarnClientImpl 里面 看一下具体的方法 至此 基本都是yarn里面的内容，后面会专门研究yarn的源码

public ApplicationId submitApplication(ApplicationSubmissionContext appContext) throws YarnException, IOException {
    ApplicationId applicationId = appContext.getApplicationId();
    if (applicationId == null) {
        throw new ApplicationIdNotProvidedException("ApplicationId is not provided in ApplicationSubmissionContext");
    } else {
        SubmitApplicationRequest request = (SubmitApplicationRequest)Records.newRecord(SubmitApplicationRequest.class);
        request.setApplicationSubmissionContext(appContext);
        this.rmClient.submitApplication(request);
        int pollCount = 0;
        long startTime = System.currentTimeMillis();

        while(true) {
            while(true) {
                try {
                    YarnApplicationState state = this.getApplicationReport(applicationId).getYarnApplicationState();
                    if (!state.equals(YarnApplicationState.NEW) && !state.equals(YarnApplicationState.NEW_SAVING)) {
                        LOG.info("Submitted application " + applicationId);
                        return applicationId;
                    }

                    long elapsedMillis = System.currentTimeMillis() - startTime;
                    if (this.enforceAsyncAPITimeout() && elapsedMillis >= this.asyncApiPollTimeoutMillis) {
                        throw new YarnException("Timed out while waiting for application " + applicationId + " to be submitted successfully");
                    }

                    ++pollCount;
                    if (pollCount % 10 == 0) {
                        LOG.info("Application submission is not finished, submitted application " + applicationId + " is still in " + state);
                    }

                    try {
                        Thread.sleep(this.submitPollIntervalMillis);
                    } catch (InterruptedException var11) {
                        LOG.error("Interrupted while waiting for application " + applicationId + " to be successfully submitted.");
                    }
                } catch (ApplicationNotFoundException var12) {
                    LOG.info("Re-submit application " + applicationId + "with the " + "same ApplicationSubmissionContext");
                    this.rmClient.submitApplication(request);
                }
            }
        }
    }
}

4 Dispatcher ResourceManager 创建

4.1 org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint 初探

我们在前面提到 YarnClusterDescriptor 中有可以看到

/**
 * 用于启动应用程序主控程序的类
 */
protected String getYarnJobClusterEntrypoint() {
   return YarnJobClusterEntrypoint.class.getName();
}

我们点到org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint 这个类看一下主方法

Configuration configuration = YarnEntrypointUtils.loadConfiguration(workingDirectory, env); 
YarnJobClusterEntrypoint yarnJobClusterEntrypoint = new 
YarnJobClusterEntrypoint(configuration); ClusterEntrypoint.runClusterEntrypoint(yarnJobClusterEntrypoint)

关键在于这个方法 runClusterEntrypoint 里面的 startCluster

public static void runClusterEntrypoint(ClusterEntrypoint clusterEntrypoint) {

   final String clusterEntrypointName = clusterEntrypoint.getClass().getSimpleName();
   try {
      clusterEntrypoint.startCluster();
   } catch (ClusterEntrypointException e) {
      LOG.error(String.format("Could not start cluster entrypoint %s.", clusterEntrypointName), e);
      System.exit(STARTUP_FAILURE_RETURN_CODE);
   }

   clusterEntrypoint.getTerminationFuture().whenComplete((applicationStatus, throwable) -> {
      final int returnCode;

      if (throwable != null) {
         returnCode = RUNTIME_FAILURE_RETURN_CODE;
      } else {
         returnCode = applicationStatus.processExitCode();
      }

      LOG.info("Terminating cluster entrypoint process {} with exit code {}.", clusterEntrypointName, returnCode, throwable);
      System.exit(returnCode);
   });
}

可以看到 使用了 一个实现 Callable接口的 runCluster方法

public void startCluster() throws ClusterEntrypointException {
   LOG.info("Starting {}.", getClass().getSimpleName());

   try {
      PluginManager pluginManager = PluginUtils.createPluginManagerFromRootFolder(configuration);
      configureFileSystems(configuration, pluginManager);

      SecurityContext securityContext = installSecurityContext(configuration);

      securityContext.runSecured((Callable<Void>) () -> {
         runCluster(configuration, pluginManager);

         return null;
      });
   } catch (Throwable t) {
      final Throwable strippedThrowable = ExceptionUtils.stripException(t, UndeclaredThrowableException.class);

      try {
         // clean up any partial state
         shutDownAsync(
            ApplicationStatus.FAILED,
            ExceptionUtils.stringifyException(strippedThrowable),
            false).get(INITIALIZATION_SHUTDOWN_TIMEOUT.toMilliseconds(), TimeUnit.MILLISECONDS);
      } catch (InterruptedException | ExecutionException | TimeoutException e) {
         strippedThrowable.addSuppressed(e);
      }

      throw new ClusterEntrypointException(
         String.format("Failed to initialize the cluster entrypoint %s.", getClass().getSimpleName()),
         strippedThrowable);
   }
}

初始化

private void runCluster(Configuration configuration, PluginManager pluginManager) throws Exception {
   synchronized (lock) {
      // 首先初始化服务
      initializeServices(configuration, pluginManager);

     
      configuration.setString(JobManagerOptions.ADDRESS, commonRpcService.getAddress());
      configuration.setInteger(JobManagerOptions.PORT, commonRpcService.getPort());
      // 1、创建 dispatcher、ResourceManager 对象的工厂类
      final DispatcherResourceManagerComponentFactory dispatcherResourceManagerComponentFactory = createDispatcherResourceManagerComponentFactory(configuration);
      clusterComponent = dispatcherResourceManagerComponentFactory.create(
         configuration,
         ioExecutor,
         commonRpcService,
         haServices,
         blobServer,
         heartbeatServices,
         metricRegistry,
         archivedExecutionGraphStore,
         new RpcMetricQueryServiceRetriever(metricRegistry.getMetricQueryServiceRpcService()),
         this);
      //
      clusterComponent.getShutDownFuture().whenComplete(
         (ApplicationStatus applicationStatus, Throwable throwable) -> {
            if (throwable != null) {
               shutDownAsync(
                  ApplicationStatus.UNKNOWN,
                  ExceptionUtils.stringifyException(throwable),
                  false);
            } else {
               // This is the general shutdown path. If a separate more specific shutdown was
               // already triggered, this will do nothing
               shutDownAsync(
                  applicationStatus,
                  null,
                  true);
            }
         });
   }
}

flink 的启动流程比Spark的跳转多的多 有些看累了 继续往里面 create 方法里面继续看 可以看到 1 有创建 dispatcherRunner 对象并启 2 启动 ResourceManager 我们重点来看 createResourceManager 创建ResourceManager这个方法 点击进去

@Override
public DispatcherResourceManagerComponent create(
     Configuration configuration,
     Executor ioExecutor,
     RpcService rpcService,
     HighAvailabilityServices highAvailabilityServices,
     BlobServer blobServer,
     HeartbeatServices heartbeatServices,
     MetricRegistry metricRegistry,
     ArchivedExecutionGraphStore archivedExecutionGraphStore,
     MetricQueryServiceRetriever metricQueryServiceRetriever,
     FatalErrorHandler fatalErrorHandler) throws Exception {

  LeaderRetrievalService dispatcherLeaderRetrievalService = null;
  LeaderRetrievalService resourceManagerRetrievalService = null;
  WebMonitorEndpoint<?> webMonitorEndpoint = null;
  ResourceManager<?> resourceManager = null;
  DispatcherRunner dispatcherRunner = null;

  try {
     dispatcherLeaderRetrievalService = highAvailabilityServices.getDispatcherLeaderRetriever();

     resourceManagerRetrievalService = highAvailabilityServices.getResourceManagerLeaderRetriever();

     final LeaderGatewayRetriever<DispatcherGateway> dispatcherGatewayRetriever = new RpcGatewayRetriever<>(
        rpcService,
        DispatcherGateway.class,
        DispatcherId::fromUuid,
        10,
        Time.milliseconds(50L));

     final LeaderGatewayRetriever<ResourceManagerGateway> resourceManagerGatewayRetriever = new RpcGatewayRetriever<>(
        rpcService,
        ResourceManagerGateway.class,
        ResourceManagerId::fromUuid,
        10,
        Time.milliseconds(50L));

     final ScheduledExecutorService executor = WebMonitorEndpoint.createExecutorService(
        configuration.getInteger(RestOptions.SERVER_NUM_THREADS),
        configuration.getInteger(RestOptions.SERVER_THREAD_PRIORITY),
        "DispatcherRestEndpoint");

     final long updateInterval = configuration.getLong(MetricOptions.METRIC_FETCHER_UPDATE_INTERVAL);
     final MetricFetcher metricFetcher = updateInterval == 0
        ? VoidMetricFetcher.INSTANCE
        : MetricFetcherImpl.fromConfiguration(
           configuration,
           metricQueryServiceRetriever,
           dispatcherGatewayRetriever,
           executor);

     webMonitorEndpoint = restEndpointFactory.createRestEndpoint(
        configuration,
        dispatcherGatewayRetriever,
        resourceManagerGatewayRetriever,
        blobServer,
        executor,
        metricFetcher,
        highAvailabilityServices.getClusterRestEndpointLeaderElectionService(),
        fatalErrorHandler);

     log.debug("Starting Dispatcher REST endpoint.");
     webMonitorEndpoint.start();

     final String hostname = RpcUtils.getHostname(rpcService);

     resourceManager = resourceManagerFactory.createResourceManager(
        configuration,
        ResourceID.generate(),
        rpcService,
        highAvailabilityServices,
        heartbeatServices,
        fatalErrorHandler,
        new ClusterInformation(hostname, blobServer.getPort()),
        webMonitorEndpoint.getRestBaseUrl(),
        metricRegistry,
        hostname);

     final HistoryServerArchivist historyServerArchivist = HistoryServerArchivist.createHistoryServerArchivist(configuration, webMonitorEndpoint, ioExecutor);

     final PartialDispatcherServices partialDispatcherServices = new PartialDispatcherServices(
        configuration,
        highAvailabilityServices,
        resourceManagerGatewayRetriever,
        blobServer,
        heartbeatServices,
        () -> MetricUtils.instantiateJobManagerMetricGroup(metricRegistry, hostname),
        archivedExecutionGraphStore,
        fatalErrorHandler,
        historyServerArchivist,
        metricRegistry.getMetricQueryServiceGatewayRpcAddress());

     log.debug("Starting Dispatcher.");
     dispatcherRunner = dispatcherRunnerFactory.createDispatcherRunner(
        highAvailabilityServices.getDispatcherLeaderElectionService(),
        fatalErrorHandler,
        new HaServicesJobGraphStoreFactory(highAvailabilityServices),
        ioExecutor,
        rpcService,
        partialDispatcherServices);

     log.debug("Starting ResourceManager.");
     resourceManager.start();

     resourceManagerRetrievalService.start(resourceManagerGatewayRetriever);
     dispatcherLeaderRetrievalService.start(dispatcherGatewayRetriever);

     return new DispatcherResourceManagerComponent(
        dispatcherRunner,
        resourceManager,
        dispatcherLeaderRetrievalService,
        resourceManagerRetrievalService,
        webMonitorEndpoint);

  } catch (Exception exception) {
     // clean up all started components
     if (dispatcherLeaderRetrievalService != null) {
        try {
           dispatcherLeaderRetrievalService.stop();
        } catch (Exception e) {
           exception = ExceptionUtils.firstOrSuppressed(e, exception);
        }
     }

     if (resourceManagerRetrievalService != null) {
        try {
           resourceManagerRetrievalService.stop();
        } catch (Exception e) {
           exception = ExceptionUtils.firstOrSuppressed(e, exception);
        }
     }

     final Collection<CompletableFuture<Void>> terminationFutures = new ArrayList<>(3);

     if (webMonitorEndpoint != null) {
        terminationFutures.add(webMonitorEndpoint.closeAsync());
     }

     if (resourceManager != null) {
        terminationFutures.add(resourceManager.closeAsync());
     }

     if (dispatcherRunner != null) {
        terminationFutures.add(dispatcherRunner.closeAsync());
     }

     final FutureUtils.ConjunctFuture<Void> terminationFuture = FutureUtils.completeAll(terminationFutures);

     try {
        terminationFuture.get();
     } catch (Exception e) {
        exception = ExceptionUtils.firstOrSuppressed(e, exception);
     }

     throw new FlinkException("Could not create the DispatcherResourceManagerComponent.", exception);
  }
}

点进去看我们可以发现里面关键的方法在于 createResourceManager

public ResourceManager<T> createResourceManager(
      Configuration configuration,
      ResourceID resourceId,
      RpcService rpcService,
      HighAvailabilityServices highAvailabilityServices,
      HeartbeatServices heartbeatServices,
      FatalErrorHandler fatalErrorHandler,
      ClusterInformation clusterInformation,
      @Nullable String webInterfaceUrl,
      MetricRegistry metricRegistry,
      String hostname) throws Exception {

   final ResourceManagerMetricGroup resourceManagerMetricGroup = ResourceManagerMetricGroup.create(metricRegistry, hostname);
   final SlotManagerMetricGroup slotManagerMetricGroup = SlotManagerMetricGroup.create(metricRegistry, hostname);

   final ResourceManagerRuntimeServices resourceManagerRuntimeServices = createResourceManagerRuntimeServices(
      configuration, rpcService, highAvailabilityServices, slotManagerMetricGroup);

   return createResourceManager(
      configuration,
      resourceId,
      rpcService,
      highAvailabilityServices,
      heartbeatServices,
      fatalErrorHandler,
      clusterInformation,
      webInterfaceUrl,
      resourceManagerMetricGroup,
      resourceManagerRuntimeServices);
}

我们点击看他的实现类 是 YarnResourceManagerFactory 这个 方法的关键是

@Override
public ResourceManager<YarnWorkerNode> createResourceManager(
      Configuration configuration,
      ResourceID resourceId,
      RpcService rpcService,
      HighAvailabilityServices highAvailabilityServices,
      HeartbeatServices heartbeatServices,
      FatalErrorHandler fatalErrorHandler,
      ClusterInformation clusterInformation,
      @Nullable String webInterfaceUrl,
      ResourceManagerMetricGroup resourceManagerMetricGroup,
      ResourceManagerRuntimeServices resourceManagerRuntimeServices) {

   return new YarnResourceManager(
      rpcService,
      resourceId,
      configuration,
      System.getenv(),
      highAvailabilityServices,
      heartbeatServices,
      resourceManagerRuntimeServices.getSlotManager(),
      ResourceManagerPartitionTrackerImpl::new,
      resourceManagerRuntimeServices.getJobLeaderIdService(),
      clusterInformation,
      fatalErrorHandler,
      webInterfaceUrl,
      resourceManagerMetricGroup);
}

创建 YarnResourceManager 时，创建了 SlotManager 我们点到 ResourceManagerFactory 这个里面去

private ResourceManagerRuntimeServices createResourceManagerRuntimeServices(
      Configuration configuration,
      RpcService rpcService,
      HighAvailabilityServices highAvailabilityServices,
      SlotManagerMetricGroup slotManagerMetricGroup) throws ConfigurationException {

   return ResourceManagerRuntimeServices.fromConfiguration(
      createResourceManagerRuntimeServicesConfiguration(configuration),
      highAvailabilityServices,
      rpcService.getScheduledExecutor(),
      slotManagerMetricGroup);
}

我们点到 fromConfiguration 里面去

public static ResourceManagerRuntimeServices fromConfiguration(
      ResourceManagerRuntimeServicesConfiguration configuration,
      HighAvailabilityServices highAvailabilityServices,
      ScheduledExecutor scheduledExecutor,
      SlotManagerMetricGroup slotManagerMetricGroup) {

   final SlotManager slotManager = new SlotManagerImpl(
      scheduledExecutor,
      configuration.getSlotManagerConfiguration(),
      slotManagerMetricGroup);

   final JobLeaderIdService jobLeaderIdService = new JobLeaderIdService(
      highAvailabilityServices,
      scheduledExecutor,
      configuration.getJobTimeout());

   return new ResourceManagerRuntimeServices(slotManager, jobLeaderIdService);
}

public static ResourceManagerRuntimeServices fromConfiguration(
      ResourceManagerRuntimeServicesConfiguration configuration,
      HighAvailabilityServices highAvailabilityServices,
      ScheduledExecutor scheduledExecutor,
      SlotManagerMetricGroup slotManagerMetricGroup) {

   final SlotManager slotManager = new SlotManagerImpl(
      scheduledExecutor,
      configuration.getSlotManagerConfiguration(),
      slotManagerMetricGroup);

   final JobLeaderIdService jobLeaderIdService = new JobLeaderIdService(
      highAvailabilityServices,
      scheduledExecutor,
      configuration.getJobTimeout());

   return new ResourceManagerRuntimeServices(slotManager, jobLeaderIdService);
}

创建并启动 Dispatcher 关键在上文提到的代码关键在于次

log.debug("Starting Dispatcher.");
dispatcherRunner = dispatcherRunnerFactory.createDispatcherRunner(
   highAvailabilityServices.getDispatcherLeaderElectionService(),
   fatalErrorHandler,
   new HaServicesJobGraphStoreFactory(highAvailabilityServices),
   ioExecutor,
   rpcService,
   partialDispatcherServices);

我们点到这个接口方法中去看它的实现类 DefaultDispatcherRunnerFactory 可以看到它的实现方法

@Override
public DispatcherRunner createDispatcherRunner(
      LeaderElectionService leaderElectionService,
      FatalErrorHandler fatalErrorHandler,
      JobGraphStoreFactory jobGraphStoreFactory,
      Executor ioExecutor,
      RpcService rpcService,
      PartialDispatcherServices partialDispatcherServices) throws Exception {

   final DispatcherLeaderProcessFactory dispatcherLeaderProcessFactory = dispatcherLeaderProcessFactoryFactory.createFactory(
      jobGraphStoreFactory,
      ioExecutor,
      rpcService,
      partialDispatcherServices,
      fatalErrorHandler);

   return DefaultDispatcherRunner.create(
      leaderElectionService,
      fatalErrorHandler,
      dispatcherLeaderProcessFactory);
}

启动 ResourceManager 我们回到 DefaultDispatcherResourceManagerComponentFactory这个类里面去 可以看到 启动 resourceManager

log.debug("Starting ResourceManager.");
resourceManager.start();

我们点到 ResourceManager 这个类中去可以看到 onStart 的方法

@Override
public void onStart() throws Exception {
   try {
      startResourceManagerServices();
   } catch (Exception e) {
      final ResourceManagerException exception = new ResourceManagerException(String.format("Could not start the ResourceManager %s", getAddress()), e);
      onFatalError(exception);
      throw exception;
   }
}

点到 我们可以发现

private void startResourceManagerServices() throws Exception {
   try {
      leaderElectionService = highAvailabilityServices.getResourceManagerLeaderElectionService();

      initialize();

      leaderElectionService.start(this);
      jobLeaderIdService.start(new JobLeaderIdActionsImpl());

      registerTaskExecutorMetrics();
   } catch (Exception e) {
      handleStartResourceManagerServicesException(e);
   }
}

先创建 ResourceManagerClient 客户端

protected void initialize() throws ResourceManagerException {
   try {
      resourceManagerClient = createAndStartResourceManagerClient(
         yarnConfig,
         yarnHeartbeatIntervalMillis,
         webInterfaceUrl);
   } catch (Exception e) {
      throw new ResourceManagerException("Could not start resource manager client.", e);
   }

   nodeManagerClient = createAndStartNodeManagerClient(yarnConfig);
}

Dispatcher 启动 JobManager

未完待续。。。。

6 这段时间有点忙 看flink 源码看的有些懵逼 在阿里开发者官网找了一下看到一个不错的总结 developer.aliyun.com/article/719… 可以好好看一下

avatar
文章
阅读
粉丝