Flink sql源码执行流程

参考：
FlinkSQL源码解析（一）转换流程
一文搞懂Flink SQL执行过程
Flink源码阅读之Flinksql执行流程
深入理解flinksql执行流程
FlinkSQL生成StreamGraph
Flink SQL 整体执行框架

流程总览

SQL语句经过Calcite解析生成抽象语法树SQLNode，基于生成的SQLNode并结合flink Catalog完成校验生成一颗Operation树，接下来blink planner将Operation树，接下来blink planner将Opearation树转为RelNode然后进行优化，最后生成Transformation变成流计算任务。

SteamTableEnvironment#executeSql->TableEnvironment#executeSql 
    ->ParserImpl#parse()  解析SQL
        ->CalciteParser#parse()   SQL 解析阶段，生成AST（抽象语法树），作用是SQL–>SqlNode
        ->SqlToOperationConverter#convert()     执行sql校验，并将SqlNode转换为Operation的过程
            ->FlinkPlannerImpl#validate(sqlNode)   sql校验
            ->SqlToOperationConverter#convertSqlQuery   将SqlNode转换为Operation,这里是select语句时调用
                ->SqlToOperationConverter#toQueryOperation 
                    ->FlinkPlannerImpl.rel()      使用 Calcite的SqlToRelConverter 将 SqlNode 转换成 RelNode
                    ->#注1 new PlannerQueryOperation(relational.project())   将 RelNode 封装成 PlannerQueryOperation   
    ->TableEnvironmentImpl#executeOperation()   优化并执行SQL
        ->TableEnvironmentImpl#executeInternal()    DML语句会调用这个（CRUD）
            ->TableEnvironmentImpl#translate()
                ->PlannerBase#translate()
                    ->PlannerBase#translateToRel()   对select字段进行校验，将ModifyOperation转换为Calcite的relational expression.也就是LogicalLegacySink
                    ->PlannerBase#optimize()         对生成的relational expression进行优化，默认使用StreamCommonSubGraphBasedOptimizer
                        ->CommonSubGraphBasedOptimizer#optimize()
                            ->StreamCommonSubGraphBasedOptimizer#doOptimize()
                                ->StreamCommonSubGraphBasedOptimizer#optimizeTree()
                                    ->#注2 FlinkChainedProgram#optimize()
                                        -> FlinkOptimizeProgram#optimize() FlinkOptimizeProgram具体子类的optimize()方法，比如FlinkHepProgram或者FlinkVolcanoProgram
                    ->PlannerBase#translateToExecNodePlan()    优化后的RelNode被转换FlinkPhysicalRel,生成执行节点图
                    ->StreamPlanner#translateToPlan()    将执行节点图转换为transformation
                        ->ExecNode#translateToPlan()   将每一个ExecNode转换为对应transformation
                            ->StreamExecSink#translateToPlanInternal()
                                ->CommonPhysicalSink#createSinkTransformation()
        

注1：我们知道，Flink 借助于 Calcite 完成 SQl 的解析和优化，而后续的优化部分其实都是直接基于 RelNode 来完成的，那么这里为什么又多出了一个 QueryOperation 的概念呢？这主要是因为，Flink SQL 是支持 SQL 语句和 Table Api 接口混合使用的，在 Table Api 接口中，主要的操作都是基于 Operation 接口来完成的。

注2：optimizeTree中默认会使用FlinkStreamProgram中提供的一组Program，Program通过FlinkChainedProgram进行连接。每一个Program又包含多个FlinkOptimizeProgram分别对应了一组优化rule，最终调用FlinkHepProgram或者FlinkVolcanoProgram的优化规则。

optimize优化详细解析

StreamCommonSubGraphBasedOptimizer.scala
private def optimizeTree(
relNode: RelNode,
updateBeforeRequired: Boolean,
miniBatchInterval: MiniBatchInterval,
isSinkBlock: Boolean): RelNode = {
....
// 使用FlinkStreamProgram提供的一组Program
val programs = calciteConfig.getStreamProgram
.getOrElse(FlinkStreamProgram.buildProgram(config.getConfiguration))
//调用FlinkChainedProgram.optimize
programs.optimize(relNode, new StreamOptimizeContext() {
...
})
}

optimizeTree中默认会使用FlinkStreamProgram中提供的一组Program，Program通过FlinkChainedProgram进行连接。每一个Program又包含多个FlinkOptimizeProgram分别对应了一组优化rule，最终调用FlinkHepProgram或者FlinkVolcanoProgram的优化规则。

我们来看一下FlinkStreamProgram的结构：

// 针对流表进行优化的相关优化
org.apache.flink.table.planner.plan.optimize.program.FlinkStreamProgram:

val SUBQUERY_REWRITE = "subquery_rewrite"
val TEMPORAL_JOIN_REWRITE = "temporal_join_rewrite"
val DECORRELATE = "decorrelate"
val TIME_INDICATOR = "time_indicator"
val DEFAULT_REWRITE = "default_rewrite"
val PREDICATE_PUSHDOWN = "predicate_pushdown"
val JOIN_REORDER = "join_reorder"
val LOGICAL = "logical"
val LOGICAL_REWRITE = "logical_rewrite"
val PHYSICAL = "physical"
val PHYSICAL_REWRITE = "physical_rewrite"

以谓词下推predicate_pushdown的优化规则举例，查看Flink如何扩展Calcite的优化规则。在FlinkHepRuleSetProgram包含了创建FlinkHepProgram相关配置，在optimize通过创建FlinkHepProgram进行Relnode的优化。

 chainedProgram.addLast(
      PREDICATE_PUSHDOWN,  
      FlinkGroupProgramBuilder.newBuilder[StreamOptimizeContext]
        .addProgram(
          FlinkHepRuleSetProgramBuilder.newBuilder
            // 规则类型为一组集合
            .setHepRulesExecutionType(HEP_RULES_EXECUTION_TYPE.RULE_COLLECTION)
            //  子节点 自下而上
            .setHepMatchOrder(HepMatchOrder.BOTTOM_UP)
            // 定义一组优化规则，使用Calcite内部的规则居多
            .add(FlinkStreamRuleSets.FILTER_PREPARE_RULES)
            .build(), "filter rules")
        .addProgram(
          FlinkHepRuleSetProgramBuilder.newBuilder
            .setHepRulesExecutionType(HEP_RULES_EXECUTION_TYPE.RULE_SEQUENCE)
            .setHepMatchOrder(HepMatchOrder.BOTTOM_UP)
            .add(FlinkStreamRuleSets.FILTER_TABLESCAN_PUSHDOWN_RULES)
            .build(), "push predicate into table scan")
        .addProgram(
          FlinkHepRuleSetProgramBuilder.newBuilder
            .setHepRulesExecutionType(HEP_RULES_EXECUTION_TYPE.RULE_SEQUENCE)
            .setHepMatchOrder(HepMatchOrder.BOTTOM_UP)
            .add(FlinkStreamRuleSets.PRUNE_EMPTY_RULES)
            .build(), "prune empty after predicate push down")
        .build())

predicate_pushdownProgram包含了"filter rules"，"push predicate into table scan"，"prune empty after predicate push down"三个子Program，这三个Program使用FlinkHepRuleSetProgram构建FlinkHepProgram，FlinkHepProgram最终对应Calcite的HepPlanner。

"push predicate into table scan"绑定了一组优化集合，我们查看其中一个PushFilterIntoTableSourceScanRule规则的扩展。每个规则会对应一个具体的RelOptRule类，其中会有matches和onMatch方法。这两个方法会在HepPlanner#findBestExp里调用，调用链如下：

-> FlinkChainedProgram#optimize()
    -> FlinkOptimizeProgram#optimize()
        -> FlinkHepRuleSetProgram#optimize()
            ->FlinkHepProgram#optimize()
                ->HepPlanner#findBestExp()

• matches()：判断是否能够使用下推规则
• onMatch()：规则能够匹配时，执行具体的下推逻辑。

 /**
    * RuleSet to do push predicate/partition into table scan
    */
  val FILTER_TABLESCAN_PUSHDOWN_RULES: RuleSet = RuleSets.ofList(
    // push a filter down into the table scan
    PushFilterIntoTableSourceScanRule.INSTANCE,
    PushFilterIntoLegacyTableSourceScanRule.INSTANCE,
    // push partition into the table scan
    PushPartitionIntoLegacyTableSourceScanRule.INSTANCE,
    // push partition into the dynamic table scan
    PushPartitionIntoTableSourceScanRule.INSTANCE
  )

// 下推过滤条件到TableSourceScan
public class PushFilterIntoTableSourceScanRule extends RelOptRule {
    //指定匹配规则：匹配Filter operand且第一个operand LogicalTableScan
    public PushFilterIntoTableSourceScanRule() {
        super(operand(Filter.class,
            operand(LogicalTableScan.class, none())),
            "PushFilterIntoTableSourceScanRule");
    }
    //判断是否能够使用下推规则
    @Override
    public boolean matches(RelOptRuleCall call) {
        
        Filter filter = call.rel(0);
        if (filter.getCondition() == null) {
            return false;
        }
        LogicalTableScan scan = call.rel(1);
        TableSourceTable tableSourceTable = scan.getTable().unwrap(TableSourceTable.class);
        // we can not push filter twice
        return tableSourceTable != null
            && tableSourceTable.tableSource() instanceof SupportsFilterPushDown
            && Arrays.stream(tableSourceTable.extraDigests()).noneMatch(str -> str.startsWith("filter=["));
    }
    // 规则能够匹配时，执行具体的下推逻辑。
    @Override
    public void onMatch(RelOptRuleCall call) {
        Filter filter = call.rel(0);
        LogicalTableScan scan = call.rel(1);
        TableSourceTable table = scan.getTable().unwrap(TableSourceTable.class);
        //  执行具体下推动作
        pushFilterIntoScan(call, filter, scan, table);
        .........
    }
}