工程师晋升：DAG有向无环图实现高并发的业务代码编排

前言

很多同学每天都在写代码，写了三年四年的业务代码，感觉自己在技术上没有机会提升。

作为一个有追求的程序员，肯定不只希望自己每天都在写业务代码没有任何提升。其实在用合理的架构来写业务代码也是能让你的代码水平更上一层楼的

业务代码从软件系统的演进规律来看其实是也是分等级的。

函数 -> 类 -> 组件 -> 脚本 -> 服务 -> 系统 -> 分栈/层 -> 配置化/标准化 -> 自动化 -> 平台化 -> 产品化 -> 规划化

软件工程的本质就是应对规模化所带来的的复杂性。如何将复杂的东西变简单，使其可以应对更大规模的发展，这就是软件工程带来的价值，是高技术力的表现。

常见的业务问题

核心的API业务服务，调用下游多个RPC，成百上千的RPC请求互相依赖形成复杂的调用关系。

如何耗费时间最少完成调用链路？并且在框架上预防新人因为不熟悉代码添加了本来可以并行的RPC调用而导致的整个调用链路耗时增加？

这其实不仅仅是一个性能优化的问题，更是一个业务编排的问题

即：如何抽象业务逻辑,使其延迟等于所有串行RPC的耗时总和。

分层并发调用模型

我们在写代码的时候脑海中对于是否能够并发的服务都存在一个分层图。

可是这个分层图真的能达到最大的并发效果吗？其实真正完全并发的效果最终应该是形成一个有向无环图DAG。

DAG 有向无环图

每个RPC步骤是否真的需要等待整个层的执行成功才能向下执行？

最短的执行时间应该是最长依赖路径上所有任务的执行时间总和

用java代码实现DAG

领域设计

• DAG(流程): 有向无环图，描述有向无环图节点、边缘、依赖关系，以及负责合法性校验等；
• Node(节点): 流程节点，描述有向无环图中单个节点信息，在可执行的 DAG 中描述的就是任务；
• Task(任务): 执行任务，描述一类可执行的任务，负责具体任务执行的实现；
• Engine(引擎): DAG 加载器和执行器，负责配置化解析、流程引擎执行。构建任务并发执行的线程池。

DAG：有向无环图

public interface DAG<T> {  
    // 添加依赖关系  
    void addEdge(Node<T> from, Node<T> to);  
  
    // 获取DAG图中的所有节点  
    Set<Node<T>> getAllNodes();  
  
    // 验证DAG是否有效，是否存在循环  
    boolean validate();  
  
    // 获取DAG图的唯一ID  
    String getId();  
}

public class DefaultDAG<T> implements DAG<T> {  
    private static final Logger LOGGER = LogManager.getLogger(DefaultDAG.class);  
  
    // DAG图上所有的节点  
    private final Set<Node<T>> nodes = new HashSet<>();  
    // DAG图的唯一ID  
    private final String id;  
  
    // 生产DAG图的唯一ID  
    public DefaultDAG() {  
        this.id = UUID.randomUUID().toString();  
    }  
  
    /**  
    * 1.往DAG图中添加nodes  
    * 2.设置Node的父节点和子节点  
    * @param from  
    * @param to  
    */  
    @Override  
    public void addEdge(Node<T> from, Node<T> to) {  
    if (from.equals(to)) {  
        return;  
    }  
    from.addChildren(to);  
    to.addParent(from);  
    nodes.add(from);  
    nodes.add(to);  
}  
  
    @Override  
    public Set<Node<T>> getAllNodes() {  
        return nodes;  
    }  
  
    /**
     * 判断DAG图的有效性，只要不存在环即可
     * @return
     */
    @Override
    public boolean validate() {
        List<Node<T>> roots = new ArrayList<>();
        
        // 发现所有根节点（即没有父节点的节点）
        for (Node<T> node : nodes) {
            if (node.getParents().isEmpty()) {
                roots.add(node);
            }
        }
        
        if (CollectionUtils.isEmpty(roots)) {
            // 如果没有根节点，则说明图中有环或没有任何节点
            return false;
        }
        
        Map<String, Node<T>> visited;
        
        for (Node<T> root : roots) {
            visited = new HashMap<>();
            if (!dfs(root, visited)) {
                // 如果任何一个子树中有环，直接返回 false
                return false;
            }
        }
        // 如果所有子树都没有环，则返回 true
        return true;
    }
    
    
    private boolean dfs(Node<T> root, Map<String, Node<T>> visited) {
        if (root == null) {
            return true;
        }
        
        visited.put(root.getId(), root);
        for (Node<T> child : root.getChildren()) {
            if (visited.containsKey(child.getId())) {
                LOGGER.warn("Node: {} is circled", child.getId());
                return false;
            }
            
            visited.put(child.getId(), child);
            if (!dfs(child, visited)) {
                return false;
            }
        }
        visited.remove(root.getId());
        
        return true;
    }
      
    @Override  
    public String getId() {  
        return id;  
        }  
    }

Node：流程节点

public interface Node<T> {
    // 获取子节点
    Set<Node<T>> getChildren();
    // 获取父节点
    Set<Node<T>> getParents();
    // 添加子节点
    void addChildren(Node<T> child);
    // 添加父节点
    void addParent(Node<T> parent);

    String getId();

    void setId(String id);
    // 设置Node下的Task
    void setData(T data);

    T getData();
}

public class DefaultNode<T> implements Node<T> {
    private final Set<Node<T>> parents = new HashSet<>();
    private final Set<Node<T>> children = new HashSet<>();

    private String id;
    private T data;

    public DefaultNode(String id, T data) {
        this.id = id;
        this.data = data;
    }

    @Override
    public Set<Node<T>> getChildren() {
        return children;
    }

    @Override
    public Set<Node<T>> getParents() {
        return parents;
    }

    @Override
    public void addChildren(Node<T> child) {
        children.add(child);
    }

    @Override
    public void addParent(Node<T> parent) {
        parents.add(parent);
    }

    @Override
    public String getId() {
        return id;
    }

    @Override
    public void setId(String id) {
        this.id = id;
    }

    @Override
    public void setData(T data) {
        this.data = data;
    }

    @Override
    public T getData() {
        return data;
    }
}

Task：执行任务

public interface Task {
    // 获取任务名字
    public String getTaskName();

    // 开始执行任务
    TaskOutput run(TaskInput input);
}

public class PrintTask implements Task {
    private static final Logger LOGGER = LogManager.getLogger(PrintTask.class);
    // 自定义任务的名字
    private static final String TASK_NAME = "PrintTask";

    public PrintTask() {
    }

    public String getTaskName() {
        return TASK_NAME;
    }
    // 执行任务逻辑 
    // todo 这里还可以从上下文中获取这个任务是否需要执行的标志位。
    @Override
    public TaskOutput run(TaskInput input) {
        // LOGGER.info("Task: {}-{} Input: {} Output: {}", taskName, taskId, input.getParameters(), true);
        String taskId = input.getTaskId();
        System.out.printf("Thread: %s Task: %s-%s Output: %s\n", Thread.currentThread().getId(), getTaskName(), taskId, true);
        Map<String, Object> output = new LinkedHashMap<>();
        output.put(taskId + "_Result", Boolean.toString(true));

        return new TaskOutput()
                .setTaskId(taskId)
                .setSucceed(true)
                .setOutput(output);
    }
}

任务的入参 返回结果

public class TaskInput {
    private String taskId;
    // 任务的入参,从上下文context中获取
    private Map<String, Object> parameters;

    public String getTaskId() {
        return taskId;
    }

    public TaskInput setTaskId(String taskId) {
        this.taskId = taskId;
        return this;
    }

    public Map<String, Object> getParameters() {
        return parameters;
    }

    public TaskInput setParameters(Map<String, Object> parameters) {
        this.parameters = parameters;
        return this;
    }
}

public class TaskOutput {
    private String taskId;
    private boolean succeed;
    private String message;
    private Exception exception;
    // 任务执行完后的输出 这里的内容重新放回context上下文中
    private Map<String, Object> output;

    public String getTaskId() {
        return taskId;
    }

    public TaskOutput setTaskId(String taskId) {
        this.taskId = taskId;
        return this;
    }

    public boolean isSucceed() {
        return succeed;
    }

    public TaskOutput setSucceed(boolean succeed) {
        this.succeed = succeed;
        return this;
    }

    public String getMessage() {
        return message;
    }

    public TaskOutput setMessage(String message) {
        this.message = message;
        return this;
    }

    public Exception getException() {
        return exception;
    }

    public TaskOutput setException(Exception exception) {
        this.exception = exception;
        return this;
    }

    public Map<String, Object> getOutput() {
        return output;
    }

    public TaskOutput setOutput(Map<String, Object> output) {
        this.output = output;
        return this;
    }
}

Engine：引擎

@Data
public class Context {
    // 存放当前执行的DAG图
    private final DAG<Task> graph;
    // 存放已经执行过的Node
    private final Set<Node<Task>> processed = new ConcurrentHashSet<>();
    // 存放DAG图的上下文包括Task的输出结果
    private final Map<String, Object> parameters = new ConcurrentHashMap<>();
    // 存放Node节点的Map
    private final Map<String, Node<Task>> taskIdNodeMap = new ConcurrentHashMap<>();
    
    public Context(DAG<Task> graph) {
        this.graph = graph;
    }
    
    public Context(DAG<Task> graph, Map<String, Object> parameters) {
        this.graph = graph;
        this.parameters.putAll(parameters);
        Set<Node<Task>> allNodes = graph.getAllNodes();
        for (Node<Task> node : allNodes) {
            taskIdNodeMap.put(node.getId(), node);
        }
    }
    
    public DAG<Task> getGraph() {
        return graph;
    }
    
    public Set<Node<Task>> getProcessed() {
        return processed;
    }
    
    public Map<String, Object> getParameters() {
        return parameters;
    }
}

class InnerResult {
    private boolean succeed;
    private String message;
    private Exception exception;

    public boolean isSucceed() {
        return succeed;
    }

    public InnerResult setSucceed(boolean succeed) {
        this.succeed = succeed;
        return this;
    }

    public String getMessage() {
        return message;
    }

    public InnerResult setMessage(String message) {
        this.message = message;
        return this;
    }

    public Exception getException() {
        return exception;
    }

    public InnerResult setException(Exception exception) {
        this.exception = exception;
        return this;
    }
}

public class Result {
    private boolean succeed;
    private String message;
    private Exception exception;
    // 返回结果
    private Map<String, Object> output;

    public boolean isSucceed() {
        return succeed;
    }

    public Result setSucceed(boolean succeed) {
        this.succeed = succeed;
        return this;
    }

    public String getMessage() {
        return message;
    }

    public Result setMessage(String message) {
        this.message = message;
        return this;
    }

    public Exception getException() {
        return exception;
    }

    public Result setException(Exception exception) {
        this.exception = exception;
        return this;
    }

    public Map<String, Object> getOutput() {
        return output;
    }

    public Result setOutput(Map<String, Object> output) {
        this.output = output;
        return this;
    }
}

这种Engine不能达到DAG最大并发，最大并发请参考EngineV3

public class Engine {
    private static final Logger LOGGER = LogManager.getLogger(Engine.class);

    private static final int DEFAULT_EXEC_CONCURRENT = 1;
    private final ExecutorService executor;
    private final Map<String, Task> taskMapping = new HashMap<>();

    public Engine() {
        this.executor = Executors.newFixedThreadPool(DEFAULT_EXEC_CONCURRENT);
    }

    public Engine(int concurrent) {
        this.executor = Executors.newFixedThreadPool(concurrent);
    }

    public Engine(ExecutorService executor) {
        this.executor = executor;
    }

    public <T extends Task> void register(T task) {
        taskMapping.put(task.getTaskName(), task);
    }

    // 反序列化dag的json字符串为dag开始执行任务
    public DAG<Task> load(String dag) {
        DAGVO dagDO = JsonUtil.parse(dag, new TypeReference<DAGVO>(){});
        if (dagDO.getEdges() == null || dagDO.getEdges().isEmpty()) {
            LOGGER.warn("Config Attr:edges Is Empty");
            throw new RuntimeException("Empty Edges");
        }
        if (dagDO.getNodes() == null || dagDO.getNodes().isEmpty()) {
            LOGGER.warn("Config Attr:nodes Is Empty");
            throw new RuntimeException("Empty Nodes");
        }

        // parse node
        Map<String, Node<Task>> nodeMapping = new HashMap<>();
        for (NodeVO nodeVO : dagDO.getNodes()) {
            Task task = taskMapping.get(nodeVO.getTaskName());
            if (task == null) {
                throw new RuntimeException(String.format("Task %s Not Found", nodeVO.getTaskName()));
            }

            Node<Task> node = new DefaultNode<>(nodeVO.getId(), task);
            nodeMapping.put(node.getId(), node);
        }

        // build edges
        DAG<Task> graph = new DefaultDAG<>();
        for (EdgeVO edgeVO : dagDO.getEdges()) {
            Node<Task> from = nodeMapping.get(edgeVO.getFrom());
            Node<Task> to = nodeMapping.get(edgeVO.getTo());
            if (from == null || to == null) {
                throw new RuntimeException(String.format("Node %s or %s Not Found", edgeVO.getFrom(), edgeVO.getTo()));
            }
            graph.addEdge(from, to);
        }

        // validate dag
        if (!graph.validate()) {
            throw new RuntimeException("Dag Is Invalid");
        }

        return graph;
    }

    // 正常的dag图任务执行流程
    public Result execute(DAG<Task> graph, Map<String, Object> parameters) {
        // 1.校验图是否存在环
        if (!graph.validate()) {
            return new Result().setSucceed(false)
                    .setMessage("Graph Is Circled");
        }

        // 构造上下文 执行dag图的任务
        Context context = new Context(graph, parameters);
        InnerResult result = doExecute(graph.getAllNodes(), context);

        return new Result()
                .setSucceed(result.isSucceed())
                .setException(result.getException())
                .setMessage(result.getMessage())
                .setOutput(context.getParameters());
    }

    private InnerResult doExecute(Set<Node<Task>> nodes, Context context) {
        Set<Node<Task>> processed = context.getProcessed();
        Map<String, Object> parameters = context.getParameters();

        // pick up can execute node
        List<Node<Task>> canExecuteNodeList = new ArrayList<>();
        Map<String, Node<Task>> nodeMap = new HashMap<>();
        for (Node<Task> node : nodes) {
            nodeMap.put(node.getId(), node);
            if (!processed.contains(node) && processed.containsAll(node.getParents())) {
                canExecuteNodeList.add(node);
            }
        }

        // submit can execute node
        List<Future<TaskOutput>> futureList = new ArrayList<>();
        for (Node<Task> node : canExecuteNodeList) {
            Future<TaskOutput> future = executor.submit(() -> {
                return doExecute(node, context);
            });
            futureList.add(future);
        }

        // process node execute output
        for (Future<TaskOutput> future : futureList) {
            TaskOutput output;
            try {
                output = future.get();
            } catch (Throwable t) {
                throw new RuntimeException(t);
            }

            Node<Task> node = nodeMap.get(output.getTaskId());
            processed.add(node);

            // process output
            parameters.putAll(output.getOutput());
            if (!output.isSucceed()) {
                return new InnerResult().setSucceed(false).setMessage(output.getMessage());
            }

            // process children
            InnerResult result = doExecute(node.getChildren(), context);
            if (!result.isSucceed()) {
                return result;
            }
        }

        return new InnerResult().setSucceed(true);
    }

    private TaskOutput doExecute(Node<Task> node, Context context) {
        Task task = node.getData();

        TaskOutput output = new TaskOutput();
        output.setSucceed(true);
        output.setTaskId(node.getId());

        try {
            output = task.run(new TaskInput().setTaskId(node.getId()).setParameters(context.getParameters()));
        } catch (Exception e) {
            output.setSucceed(false);
            output.setException(e);
        }

        return output;
    }
}

/**
 *
 * 为什么这段代码能达到DAG的最大并发
 * 动态任务调度：代码使用了CompletionService来实时监控任务的完成情况，并根据任务的完成状态动态地提交其子任务进行执行。这样，任务一旦完成，其依赖的子任务就能立刻开始执行，不需要等待其他无关任务的完成。
 *
 * 最大化线程池利用：由于每个任务的执行都是异步的，且新任务可以在前一个任务完成后立即被提交，线程池中的线程得到了充分的利用。这种方式确保了任务能够尽可能并发地执行，避免了不必要的等待。
 *
 * 任务依赖处理：只有在所有依赖的父任务完成后，子任务才会被提交。这种精细的依赖管理保证了DAG图的正确执行顺序，同时也避免了过早或过晚的任务执行，从而实现了最大并发。
 *
 *  结论：这段代码通过合理使用线程池和并发工具，最大化了DAG任务的并发执行能力，确保任务一旦准备好就立即执行，实现了高效的DAG调度和执行。
 *
 * @version 1.0.0
 * @since 1.0.0
 */

@Service
public class EngineV3 {
    private static final Logger LOGGER = LogManager.getLogger(EngineV3.class);
    
    @Autowired
    @Qualifier("serviceExecutorService")
    private ExecutorService executor;
    
    public Result execute(DAG<Task> graph, Map<String, Object> parameters){
        // 1.校验图是否存在环
        if (!graph.validate()) {
            LOGGER.warn("Graph Is Circled");
            return new Result().setSucceed(false).setMessage("Graph Is Circled");
        }
        // 使用线程安全的ConcurrentHashMap来管理上下文参数
        Context context = new Context(graph, new ConcurrentHashMap<>(parameters));
        CompletionService<TaskOutput> completionService = new ExecutorCompletionService<>(executor);
        
        // 获取DAG图中所有的Node
        Set<Node<Task>> nodes = graph.getAllNodes();
        // 存放所有已执行完成的Node
        Set<Node<Task>> processed = context.getProcessed();
        
        int remainingTasks = nodes.size();
        
        // 先执行root节点
        for (Node<Task> node : nodes) {
            if (node.getParents().isEmpty()) {
                submitTask(node, context, completionService);
            }
        }
        
        while(remainingTasks > 0){
            try {
                // 从线程池中取出最早已经执行完成的那个任务
                TaskOutput output = completionService.take().get();
                // 找到已经执行完成的Node放入执行完成的集合processed中
                Node<Task> completedNode = context.getTaskIdNodeMap().get(output.getTaskId());
                processed.add(completedNode);
                // 剩余需要执行的任务数-1
                remainingTasks--;
                // 执行任务失败 直接抛出异常中断
                if (!output.isSucceed()) {
                    LOGGER.warn("任务:{} 执行失败.任务中断 message:{}", output.getTaskId(), output.getMessage());
                    return new Result().setSucceed(false).setMessage(output.getMessage());
                }
                // 合并输出到上下文参数中
                context.getParameters().putAll(output.getOutput());
                
                // 查看子节点是否有可以执行的，进行执行
                for(Node<Task> child : completedNode.getChildren()){
                    if (processed.containsAll(child.getParents())) {
                        submitTask(child, context, completionService);
                    }
                }
                
            }catch (Exception e){
                LOGGER.error("发生异常 执行失败", e);
                return new Result().setSucceed(false).setMessage(e.getMessage()).setException(e);
            }
        }
        
        return new Result().setSucceed(true).setOutput(context.getParameters());
    }
    
    private void submitTask(Node<Task> node, Context context, CompletionService<TaskOutput> completionService) {
        completionService.submit(() -> {
            Task task = node.getData();
            TaskOutput output = task.run(new TaskInput().setTaskId(node.getId()).setParameters(context.getParameters()));
            if (output == null) {
                output = new TaskOutput();
                output.setSucceed(false);
                output.setMessage("Task returned null output");
            }
            output.setTaskId(node.getId());
            return output;
        });
    }
}

初始化Graph

@Configuration
public class GraphEngineManager {

    
    @Autowired
    private ATask atask;
    
    @Autowired
    private BTask btask;
    
    @Bean("myGraph")
    public DAG<Task> init() {
        
        // 1.创建一个图
        DAG<Task> graph = new DefaultDAG<>();
        
        // 2.创建图上的Node并把每个Node需要执行Task创建好
        Node<Task> ataskNode = new DefaultNode<>("ATaskNode", atask);
        Node<Task> btaskNode = new DefaultNode<>("BTaskNode", atask);
        
        // 3.构建图的依赖边关系
        graph.addEdge(ataskNode, btaskNode);
        
        // 4.检查构建的图是否存在环
        if (!graph.validate()){
            throw new IllegalStateException("The DAG graph contains a cycle and is therefore invalid.");
        }
        return graph;
    }
}

初始化线程池

@Configuration
@Setter
public class ThreadPoolManager {
    
    /**
     * 获取当前机器核心数量
     */
    private final int CURRENT_MACHINE_CORE_NUM = Math.max(Runtime.getRuntime().availableProcessors(), 4);
    
    private BlockingQueue<Runnable> resource_thread_queue = new LinkedBlockingQueue<>(5000);
    

    public static int getThreadCountFactor() {
        return ConfigService.getAppConfig().getIntProperty("thread.count.factor", 12);
    }

    @Bean("serviceExecutorService")
    public ExecutorService getExecutorService() {
        return new ThreadPoolExecutor(CURRENT_MACHINE_CORE_NUM * getThreadCountFactor(),
                CURRENT_MACHINE_CORE_NUM * getThreadCountFactor(),
            60,
            TimeUnit.SECONDS,
            resource_thread_queue,
            new CustomizableThreadFactory("ServiceExecutorService-"));
    }
    
    
}

使用方式

使用方式支持代码和配置化两种方式进行初始化和运行。

方法一：代码初加载运行

public class EngineTest {
    private static final Logger LOGGER = LogManager.getLogger(EngineTest.class);

    @Test
    public void testExecute() {
        // 1.创建一个图
        DAG<Task> graph = new DefaultDAG<>();
        // 2.创建图上的Node并把每个Node需要执行Task创建好
        PrintTask printTask = new PrintTask();
        Node<Task> nodeA = new DefaultNode<>("A", printTask);
        Node<Task> nodeB = new DefaultNode<>("B", printTask);
        Node<Task> nodeC = new DefaultNode<>("C", printTask);
        Node<Task> nodeD = new DefaultNode<>("D", printTask);
        Node<Task> nodeE = new DefaultNode<>("E", printTask);
        Node<Task> nodeF = new DefaultNode<>("F", printTask);
        // 3.构建图的依赖边关系
        graph.addEdge(nodeA, nodeB);
        graph.addEdge(nodeB, nodeC);
        graph.addEdge(nodeC, nodeE);
        graph.addEdge(nodeA, nodeD);
        graph.addEdge(nodeD, nodeE);
        graph.addEdge(nodeE, nodeF);
        // 4.创建上下文
        Map<String, Object> parameters = new HashMap<>();
        parameters.put("FlowID", graph.getId());
        // 5.创建引擎
        Engine engine = new Engine();
        // 5.1 把要执行的任务放入引擎中，有多个任务则放入多个（json配置化执行方式需要，此处不需要）
        engine.register(printTask);
        // 6.执行DAG图的任务
        Result result = engine.execute(graph, parameters);
        LOGGER.info("Exec Flow:{} Succeed:{} Output:{}", graph.getId(), result.isSucceed(), result.getOutput());
    }
}

方法二：配置化加载运行

public class LoadTest {

    @Test
    public void testLoad() {
        String dag = "{\n" +
                "    "nodes":[\n" +
                "        {\n" +
                "            "id":"1",\n" +
                "            "task_name":"PrintTask"\n" +
                "        },\n" +
                "        {\n" +
                "            "id":"2",\n" +
                "            "task_name":"PrintTask"\n" +
                "        },\n" +
                "        {\n" +
                "            "id":"3",\n" +
                "            "task_name":"PrintTask"\n" +
                "        },\n" +
                "        {\n" +
                "            "id":"4",\n" +
                "            "task_name":"PrintTask"\n" +
                "        }\n" +
                "    ],\n" +
                "    "edges":[\n" +
                "        {\n" +
                "            "from":"1",\n" +
                "            "to":"2"\n" +
                "        },\n" +
                "        {\n" +
                "            "from":"1",\n" +
                "            "to":"3"\n" +
                "        },\n" +
                "        {\n" +
                "            "from":"2",\n" +
                "            "to":"4"\n" +
                "        },\n" +
                "        {\n" +
                "            "from":"3",\n" +
                "            "to":"4"\n" +
                "        }\n" +
                "    ],\n" +
                "    "parameters":{\n" +
                "\n" +
                "    }\n" +
                "}";

        System.out.println(dag);

        Engine engine = new Engine();
        PrintTask printTask = new PrintTask();
        engine.register(printTask);

        DAG<Task> graph = engine.load(dag);

        Map<String, Object> parameters = new HashMap<>();
        parameters.put("FlowID", graph.getId());

        Result result = engine.execute(graph, parameters);
        System.out.printf("Exec Flow:%s Succeed:%s Output:%s\n", graph.getId(), result.isSucceed(), result.getMessage());
    }
}

springboot使用方式

public class Service {
        
    @Autowired
    private Engine engine;
    
    @Autowired
    private EngineV3 engineV3;
    
    @Autowired
    @Qualifier("myGraph")
    private DAG<Task> myGraph;
    


    public Map<String, Object> execute(Req Req) {
        try{
            // 1.创建上下文
            Map<String, Object> parameters = new ConcurrentHashMap<>();
            parameters.put("Your parameters", Req);
            
            // 2.执行DAG图的任务
            Result result = engineV3.execute(myGraph, parameters);
            LOGGER.info("Exec Succeed:{} Output:{}", result.isSucceed(), result.getOutput());
            Map<String, Object> output = result.getOutput();
        
            return output;
        }catch (Exception e){
            LOGGER.error("发生异常. Req:{}", Req, e);
            return null;
        }
    }

}

代码仓库

详细参考 Github：github.com/AaronSheng/…

优化的点

在原有的基础上，优化了

• Task的返回结果定义为Map<String, Object> 会更方便我们传输Task返回的不同类型的结果
• 项目启动的时候，把DAG图构建好
• 项目启动的时候，初始化好线程池
• 检查图是否有环，把所有的root节点都检查了
• 改写了DAG的执行逻辑，用completionService实现了真正意义上的DAG最大并发，即父节点执行完，子节点就可以立刻开始执行。

待优化的点

• 父节点的角色：区分强弱依赖，弱依赖失败还可以继续执行，强依赖失败则该条链路可以不执行了。

参考资料

基于 DAG 任务编排实现

DAG计算框架-实现业务编排