Spark源码系列03-任务提交02

131 阅读1分钟

spark源码系列03-任务提交01:blog.csdn.net/qq_19968255…

 

12. DAGScheduler    getShuffleMapStage

创建父Stage

val stage =newOrUsedStage(……

 

13. DAGScheduler    newOrUsedStage

递归

val stage = newStage(rdd, numTasks, Some(shuffleDep), jobId, callSite)

 

14.DAGScheduler    handleJobSubmitted

……

if (shouldRunLocally) {

        // Compute very short actions like first() or take() with no parent stages locally.

        //TODO 本地运行

        listenerBus.post(

          SparkListenerJobStart(job.jobId, jobSubmissionTime, Seq.empty, properties))

        runLocally(job)

      } else {

        //TODO 集群模式