第一次接触的商品推荐业务

一、场景

公司需要一个模块，根据给定用户的信息，根据预设的规则进行匹配，查询出来符合他的商品（一个商品存在多个规则，并且条件），比如用户男的A商品，用户女B商品。然后单个条件可以组成一个规则表达式。 比如：

条件A : user.sex = '男' 
条件B : user.age > 18 
最终 条件A && 条件B := user.sex = '男'  && user.age > 18 

二、第一次解决方案

使用ScriptEngineManager来构造一个js 脚本，然后执行命令js命令完成逻辑判断

规则结构

public class ProductHandlingRule extends BaseEntity{

    /**
    * 办理条件
    */
    private String expression;

    /**
    * 逻辑条件
    */
    @TableField(typeHandler = JacksonTypeHandler.class)
    private JSONArray logicalCondition;
 }

比如 表示 条件一成立 或者 条件二和三都成立

{
    "expression": "1||2&&3",
    "logicalCondition": [
        {
            "no": 1,
            "value": "100",
            "operate": "EQUAL",
            "matchFieldId": 1
        },
        {
            "no": 2,
            "value": "100",
            "operate": "GREAT_EQUAL",
            "matchFieldId": 2
        },
        {
            "no": 3,
            "value": "s",
            "operate": "CONTAIN",
            "matchFieldId": 3
        }
    ]
}

public static void main(String[] args) throws Exception {
    ScriptEngineManager manager = new ScriptEngineManager();
    ScriptEngine engine = manager.getEngineByName("js");
   
    System.out.println((boolean)engine.eval("function run(a) {return " + user.getSex()+"== 1;}"));
    System.out.println((boolean)engine.eval("function run(a) {return " + user.getSex()+"== '男';}"));
}

输出：
false
true

二、第一次解决方案

使用ScriptEngineManager来构造一个js 脚本，然后执行命令js命令完成逻辑判断, 每个商品的能同时办理多个规则，所以每次把商品全部拿出来循环一下，查出商品的全部规则，再一条条的执行规则，如果一个为false，就删除当前的商品，不返回。

public List<Product> recommendProduct() {
    //获取全部商品
    List<Product> productList = getAll();

    Iterator<Product> productIter = productList.iterator();
    while (productIter.hasNext()) {
        Product product = productIter.next();
        boolean flag = true;
        for (Rule rule : product.getRuleList()) {
            StringBuilder funcBodySb = new StringBuilder();
            funcBodySb.append("(");

            Map<Integer, String> expressionMap = new HashMap<>();
            List<KeyValueDTO<String, String>> argList = new ArrayList<>();

            String uniqueTime = String.valueOf(System.currentTimeMillis());
            for (Condition condition : rule.getLogicalCondition()) {
                String code = "从指定来源取出的值，比如男，18岁，身份证3123";

                String value = condition.getValue();
                if (value == null) {
                    value = "";
                }

                OperateTypeEnum operateTypeEnum = OperateTypeEnum.valueOf(condition.getOperate());
                switch (operateTypeEnum) {
                    case EQUAL -> expressionMap.put(condition.getNo(), "(" + code + " == " + value + ")");
                    case NOT_EQUAL -> expressionMap.put(condition.getNo(), "(" + code + " != " + value + ")");
                    case GREAT -> expressionMap.put(condition.getNo(), "(" + code + " > " + value + ")");
                    case GREAT_EQUAL -> expressionMap.put(condition.getNo(), "(" + code + " >= " + value + ")");
                    case LITTLE -> expressionMap.put(condition.getNo(), "(" + code + " < " + value + ")");
                    case LITTLE_EQUAL -> expressionMap.put(condition.getNo(), "(" + code + " <= " + value + ")");
                    case CONTAIN ->
                            expressionMap.put(condition.getNo(), "(" + code + ".indexOf(" + value + ") != -1)");
                    case NOT_CONTAIN ->
                            expressionMap.put(condition.getNo(), "(!" + code + ".indexOf(" + value + ") == -1)");
                    default -> expressionMap.put(condition.getNo(), "false");
                }

            }

            //遍历表达式，去除表达式的 数字，替换为表达式，比如条件1 替换为 arg1.indexOf(100) != -1
            int no = 0;
            for (int i = 0; i < rule.getExpression().length(); i++) {
                if (rule.getExpression().charAt(i) >= '0' && rule.getExpression().charAt(i) <= '9') {
                    no = no * 10 + rule.getExpression().charAt(i) - '0';
                } else {
                    if (no != 0) {
                        if (!expressionMap.containsKey(no)) {
                            throw new BizException("表达式条件不存在");
                        }
                        funcBodySb.append(expressionMap.get(no));
                        no = 0;
                    }
                    funcBodySb.append(rule.getExpression().charAt(i));
                }
            }
            if (no > 0) {
                if (!expressionMap.containsKey(no)) {
                    throw new BizException("表达式条件不存在");
                }
                funcBodySb.append(expressionMap.get(no));
            }

            funcBodySb.append(")");

            ScriptEngineManager manager = new ScriptEngineManager();
            ScriptEngine engine = manager.getEngineByName("js");
            flag = (boolean) engine.eval(funcBodySb.toString());
            if (!flag) {
                break;
            }
        }
        if (!flag) {
            productIter.remove();
        }
    }
    return productList;
}

这样方案可以实现，但是有一个问题，代码执行时间太长，每次生成表达式，有时候cpu使用率会达到100%，执行表达式，都会大量使用cpu，不符合要求。

二、第二次解决方案

使用ScriptEngineManager来构造一个js 脚本，在里面定义一个方法，然后传参调用

public static void main(String[] args) throws Exception {
    ScriptEngineManager manager = new ScriptEngineManager();
    ScriptEngine engine = manager.getEngineByName("js");
    engine.eval("function run(a) {return a == 1;}");

    Invocable invocable = (Invocable) engine;
    System.out.println(invocable.invokeFunction("run", "1"));
    System.out.println(invocable.invokeFunction("run", "2"));
}

输出：
true
false

在将生成的表达式本地缓存起来，使用的时候直接取就行。之前考虑放到redis，因为是分布式的，考虑到序列化和反序列化的时间，还是放本地缓存，热门的就提前预热一下本地缓存就行。

private static final Map<Long,Invocable> RULE_INVOCABLE_MAP = new HashMap<>(64);

public List<Product> recommendProduct() {
    //获取全部商品
    List<Product> productList = getAll();

    Iterator<Product> productIter = productList.iterator();
    while (productIter.hasNext()) {
        Product product = productIter.next();
        boolean flag = true;
        for (Rule rule : product.getRuleList()) {
            Invocable invocable = RULE_INVOCABLE_MAP.get(rule.getId());
            if (invocable == null){
                StringBuilder funcDefSb = new StringBuilder("function run{");
                StringBuilder funcBodySb = new StringBuilder();
                funcBodySb.append("(");

                Map<Integer, String> expressionMap = new HashMap<>();
                List<KeyValueDTO<String, String>> argList = new ArrayList<>();

                String uniqueTime = String.valueOf(System.currentTimeMillis());
                for (Condition condition : rule.getLogicalCondition()) {
                    String code = "arg"+rule.getId()+condition.getNo();
                    
                    funcDefSb.append(code).append(",");

                    String value = condition.getValue();
                    if (value == null) {
                        value = "";
                    }

                    OperateTypeEnum operateTypeEnum = OperateTypeEnum.valueOf(condition.getOperate());
                    switch (operateTypeEnum) {
                        case EQUAL -> expressionMap.put(condition.getNo(), "(" + code + " == " + value + ")");
                        case NOT_EQUAL -> expressionMap.put(condition.getNo(), "(" + code + " != " + value + ")");
                        case GREAT -> expressionMap.put(condition.getNo(), "(" + code + " > " + value + ")");
                        case GREAT_EQUAL -> expressionMap.put(condition.getNo(), "(" + code + " >= " + value + ")");
                        case LITTLE -> expressionMap.put(condition.getNo(), "(" + code + " < " + value + ")");
                        case LITTLE_EQUAL -> expressionMap.put(condition.getNo(), "(" + code + " <= " + value + ")");
                        case CONTAIN ->
                                expressionMap.put(condition.getNo(), "(" + code + ".indexOf(" + value + ") != -1)");
                        case NOT_CONTAIN ->
                                expressionMap.put(condition.getNo(), "(!" + code + ".indexOf(" + value + ") == -1)");
                        default -> expressionMap.put(condition.getNo(), "false");
                    }

                }

                //遍历表达式，去除表达式的 数字，替换为表达式，比如条件1 替换为 arg1.indexOf(100) != -1
                int no = 0;
                for (int i = 0; i < rule.getExpression().length(); i++) {
                    if (rule.getExpression().charAt(i) >= '0' && rule.getExpression().charAt(i) <= '9') {
                        no = no * 10 + rule.getExpression().charAt(i) - '0';
                    } else {
                        if (no != 0) {
                            if (!expressionMap.containsKey(no)) {
                                throw new BizException("表达式条件不存在");
                            }
                            funcBodySb.append(expressionMap.get(no));
                            no = 0;
                        }
                        funcBodySb.append(rule.getExpression().charAt(i));
                    }
                }
                if (no > 0) {
                    if (!expressionMap.containsKey(no)) {
                        throw new BizException("表达式条件不存在");
                    }
                    funcBodySb.append(expressionMap.get(no));
                }
                funcDefSb.setLength(funcDefSb.length() - 1);
                funcBodySb.append(")");

                ScriptEngineManager manager = new ScriptEngineManager();
                ScriptEngine engine = manager.getEngineByName("js");
                engine.eval(funcDefSb.append("){").append(funcBodySb).append(";}").toString());

                invocable = (Invocable) engine;
                RULE_INVOCABLE_MAP.put(rule.getId(),invocable);
            }
            flag = (boolean)invocable.invokeFunction("run","伱的参数列表");
            if (!flag){
                break;
            }
        }
        if (!flag) {
            productIter.remove();
        }
    }
    return productList;
}

速度是比第一次快了一点点，但是还是不够快

三、第三次解决方案

因为每个规则每次还是要查询，遍历，传参执行多个规则，就想有没有办法根据一个商品的维度来组合，可以将多个规则组合成一个js function，调用一次就行。

如果在每次添加商品和修改商品的的规则的时候，重新构造商品的规则，组合在一起就行。规则也可以提前缓存好，比如规则a，是参数arg1和arg2，规则b是arg3和arg4，到时候组合成function run(arg1,arg2,arg3,arg4),条件体也是同理，使用&&联合起来

private static final Map<String, RecommendInvocableDTO> PRODUCT_INVOCABLE_MAP = new HashMap<>(256);
private static final Map<Long, RuleDefineDTO> RULE_DEFINE_MAP = new HashMap<>(256);

public static void putRuleDefine(Long key, ProductHandlingRule productHandlingRule) {
    if (key == null || productHandlingRule == null) {
        return;
    }
    if (productHandlingRule.getLogicalCondition() == null) {
        productHandlingRule.setLogicalCondition(new JSONArray());
    }
    //获取办理规则逻辑条件 比如 user.name = 'aaa'
    List<ProductHandlingConditionDTO> handlingConditionDTOList = productHandlingRule.getLogicalCondition().toJavaList(ProductHandlingConditionDTO.class);
    handlingConditionDTOList.sort(Comparator.comparingInt(ProductHandlingConditionDTO::getNo));

    StringBuilder funcDefSb = new StringBuilder();
    StringBuilder funcBodySb = new StringBuilder();
    funcBodySb.append("(");

    Map<Integer, String> expressionMap = new HashMap<>(handlingConditionDTOList.size());

    String uniqueTime = String.valueOf(System.currentTimeMillis());
    for (ProductHandlingConditionDTO productHandlingConditionDTO : handlingConditionDTOList) {
        String code = "arg" + key + uniqueTime + productHandlingConditionDTO.getNo();
        funcDefSb.append(code).append(",");

        String value = productHandlingConditionDTO.getValue();
        if (value == null) {
            value = "";
        }
        // 构造表达式 比如 arg1 != 6
        OperateTypeEnum operateTypeEnum = OperateTypeEnum.valueOf(productHandlingConditionDTO.getOperate());
        switch (operateTypeEnum) {
            case EQUAL ->
                    expressionMap.put(productHandlingConditionDTO.getNo(), "(" + code + " == " + value + ")");
            case NOT_EQUAL ->
                    expressionMap.put(productHandlingConditionDTO.getNo(), "(" + code + " != " + value + ")");
            case GREAT ->
                    expressionMap.put(productHandlingConditionDTO.getNo(), "(" + code + " > " + value + ")");
            case GREAT_EQUAL ->
                    expressionMap.put(productHandlingConditionDTO.getNo(), "(" + code + " >= " + value + ")");
            case LITTLE ->
                    expressionMap.put(productHandlingConditionDTO.getNo(), "(" + code + " < " + value + ")");
            case LITTLE_EQUAL ->
                    expressionMap.put(productHandlingConditionDTO.getNo(), "(" + code + " <= " + value + ")");
            case CONTAIN ->
                    expressionMap.put(productHandlingConditionDTO.getNo(), "(" + code + ".indexOf(" + value + ") != -1)");
            case NOT_CONTAIN ->
                    expressionMap.put(productHandlingConditionDTO.getNo(), "(!" + code + ".indexOf(" + value + ") == -1)");
            default -> expressionMap.put(productHandlingConditionDTO.getNo(), "false");
        }

    }

    //将表达式 1&&2 替换成 (user.name == 'a') && (user.age = 2)
    int no = 0;
    for (int i = 0; i < productHandlingRule.getExpression().length(); i++) {
        if (productHandlingRule.getExpression().charAt(i) >= '0' && productHandlingRule.getExpression().charAt(i) <= '9') {
            no = no * 10 + productHandlingRule.getExpression().charAt(i) - '0';
        } else {
            if (no != 0) {
                if (!expressionMap.containsKey(no)) {
                    throw new BizException("表达式条件不存在");
                }
                funcBodySb.append(expressionMap.get(no));
                no = 0;
            }
            funcBodySb.append(productHandlingRule.getExpression().charAt(i));
        }
    }
    if (no > 0) {
        if (!expressionMap.containsKey(no)) {
            throw new BizException("表达式条件不存在");
        }
        funcBodySb.append(expressionMap.get(no));
    }

    if (!funcDefSb.isEmpty()){
        funcDefSb.setLength(funcDefSb.length() - 1);
    }
    funcBodySb.append(")");
    
    RuleDefineDTO ruleDefineDTO = new RuleDefineDTO();
    ruleDefineDTO.setArgString(funcDefSb.toString());
    ruleDefineDTO.setBodyString(funcBodySb.toString());

    RULE_DEFINE_MAP.put(key, ruleDefineDTO);
}

public static void removeRuleDefine(Long key) {
    RULE_DEFINE_MAP.remove(key);
}

public static RuleDefineDTO getRuleDefine(Long key) {
    return RULE_DEFINE_MAP.get(key);
}

public static void putProductInvocable(Long key,ProductDTO productDTO) {

    StringBuilder funcDefSb = new StringBuilder();
    StringBuilder funcBodySb = new StringBuilder();
    funcDefSb.append("function run(");
    funcBodySb.append("{return (");

    // 获取规则 然后凭借起来，比如 规则A arg1，arg2 规则B arg3 结果是  arg1,arg2,arg3 请求体用 （）&&（）拼接
    for (Rule rule : productDTO.getRuleList()) {
        RuleDefineDTO handlingRuleDefineDTO = getRuleDefine(rule.getId());
        if (handlingRuleDefineDTO == null){
            putRuleDefine(rule.getId(),rule);
            handlingRuleDefineDTO = getRuleDefine(rule.getId());
            if (handlingRuleDefineDTO == null) {
                continue;
            }
        }
        argList.addAll(handlingRuleDefineDTO.getArgList());
        funcDefSb.append(handlingRuleDefineDTO.getArgString()).append(",");
        funcBodySb.append(handlingRuleDefineDTO.getBodyString()).append("&&");
    }

    funcDefSb.setLength(funcDefSb.length() - 1);
    funcDefSb.append(")");
    if (funcBodySb.length() == 8){
        funcBodySb.append("true");
    }else {
        funcBodySb.setLength(funcBodySb.length() - 2);
    }
    funcBodySb.append(");}");
    ScriptEngineManager manager = new ScriptEngineManager();
    ScriptEngine engine = manager.getEngineByName("js");

    RecommendInvocableDTO recommendInvocableDTO = new RecommendInvocableDTO();

    try {
        String function = funcDefSb.append(funcBodySb).toString();
        engine.eval(function);

        recommendInvocableDTO.setFunctionString(function);
    } catch (Exception e) {
        throw new BizException("生成表达式失败，请检查");
    }
    log.info("generate product expression result id:{},head:{},body:{}", key, funcDefSb, funcBodySb);

    recommendInvocableDTO.setInvocable((Invocable) engine);
    recommendInvocableDTO.setArgList(argList);

    PRODUCT_INVOCABLE_MAP.put(key + ruleType, recommendInvocableDTO);
}

public static void removeProductInvocable(String key) {
    PRODUCT_INVOCABLE_MAP.remove(key);
}

public static RecommendInvocableDTO getProductInvocable(String key) {
    return PRODUCT_INVOCABLE_MAP.get(key);
}

其实还缺少一个过期的功能，这样是一个兜底方案。然后每次修改规则都要同步修改产品最后的表达式或者删除产品表达式，这样可以让产品函数保存正确。具体的看业务需求。

推荐产品的代码就比较简单了，弄一个线程池，多线程推荐，单线程速度还是比较慢，多个产品都是独立存在的不存在并发冲突问题

List<ProductDTO> productList = getAll();

//50个产品一组
List<List<ProductDTO>> productSplitList = CollUtil.split(productList, 50);
List<Future<Void>> futureList = new ArrayList<>(productSplitList.size());
for (List<ProductDTO> splitList : productSplitList) {
    CompletableFuture<Void> future = CompletableFuture.runAsync(() -> {
        Iterator<ProductDTO> iterator = splitList.iterator();
        while (iterator.hasNext()) {
            ProductDTO product = iterator.next();
            if (!isNeedRecommend(product)) {
                iterator.remove();
                continue;
            }
        }
    }, productRecommendPoolExecutor);
    futureList.add(future);
}
//等待所有线程完成计算
for (Future<Void> voidFuture : futureList) {
    try {
        voidFuture.get();
    } catch (Exception e) {
        throw new BizException(e.getMessage());
    }
}
//聚合结果
return BeanUtil.copyToList(productSplitList.stream()
        .flatMap(List::stream)
        .collect(Collectors.toList()), ProductDTO.class);

/**
 * 是否需要推荐
 *
 * @param product
 */
private static Boolean isNeedRecommend(ProductDTO product) {
    RecommendInvocableDTO productInvocable = ProductCacheUtils.getProductInvocable(product.getId() + ruleType);
    if (productInvocable == null) {
        ProductCacheUtils.putProductInvocable(product.getId(), ruleType, product);
        productInvocable = ProductCacheUtils.getProductInvocable(product.getId() + ruleType);
    }
   //这里来组装自己的参数列表
    Object[] argArr = new Object[argList.size()];
    try {
        Boolean checkResult = (Boolean) productInvocable.getInvocable().invokeFunction("run", argArr);
        return checkResult;
    } catch (Exception e) {
        throw new BizException("推荐商品错误");
    }
}

四、总结

总的来说就是 多线程和缓存提升速度，定时任务更新缓存，和启动更新缓存，广播更新缓存 来保证数据的分布式的一致性缓存。

第一次碰到这样的需求，给大伙分享分享哈蛤