前言

本章学习一下ShardingProxy的配置与启动流程。基于4.1.0版本。

一、配置

1、ShardingConfiguration

ShardingConfiguration是ShardingProxy的全局配置，配置可以分为两部分，一部分是服务端配置server.yaml，一部分是规则配置（如分片规则）。

@RequiredArgsConstructor
@Getter
public final class ShardingConfiguration {
    // sharding-proxy服务端配置
    private final YamlProxyServerConfiguration serverConfiguration;
    // schema - 规则配置
    private final Map<String, YamlProxyRuleConfiguration> ruleConfigurationMap;
}

2、YamlProxyServerConfiguration

YamlProxyServerConfiguration是ShardingProxy的服务端配置，一般是/conf/server.yaml。

orchestration:
  orchestration_ds:
    orchestrationType: registry_center,config_center,distributed_lock_manager
    instanceType: zookeeper
    serverLists: localhost:2181
    namespace: orchestration
    props:
      overwrite: false
      retryIntervalMilliseconds: 500
      timeToLiveSeconds: 60
      maxRetries: 3
      operationTimeoutMilliseconds: 500
authentication:
  users:
    root:
      password: root
    sharding:
      password: sharding
      authorizedSchemas: sharding_db
props:
  max.connections.size.per.query: 1
  acceptor.size: 16  # The default value is available processors count * 2.
  executor.size: 16  # Infinite by default.
  proxy.frontend.flush.threshold: 128  # The default value is 128.
  proxy.transaction.type: LOCAL
  proxy.opentracing.enabled: false
  proxy.hint.enabled: false
  query.with.cipher.column: true
  sql.show: true
  allow.range.query.with.inline.sharding: false

YamlProxyServerConfiguration配置又分为三部分：编排、授权、普通kv配置。

public final class YamlProxyServerConfiguration implements YamlConfiguration {
    // 授权配置 用户 - 密码&权限
    private YamlAuthenticationConfiguration authentication;
    // 编排名称 - 编排配置
    private Map<String, YamlCenterRepositoryConfiguration> orchestration;
    // 普通kv配置 包括netty、openTracing、sharding-jdbc相关配置
    private Properties props = new Properties();
}

3、YamlProxyRuleConfiguration

YamlProxyRuleConfiguration是sharding-jdbc的规则配置，一般是/conf/config-*.yaml。

schemaName: sharding_db

dataSources:
  ds_0:
    url: jdbc:mysql://127.0.0.1:3306/demo_ds_0?serverTimezone=UTC&useSSL=false
    username: root
    password:
    connectionTimeoutMilliseconds: 30000
    idleTimeoutMilliseconds: 60000
    maxLifetimeMilliseconds: 1800000
    maxPoolSize: 50
  ds_1:
    url: jdbc:mysql://127.0.0.1:3306/demo_ds_1?serverTimezone=UTC&useSSL=false
    username: root
    password:
    connectionTimeoutMilliseconds: 30000
    idleTimeoutMilliseconds: 60000
    maxLifetimeMilliseconds: 1800000
    maxPoolSize: 50

shardingRule:
  tables:
    t_order:
      actualDataNodes: ds_${0..1}.t_order_${0..1}
      tableStrategy:
        inline:
          shardingColumn: order_id
          algorithmExpression: t_order_${order_id % 2}
      keyGenerator:
        type: SNOWFLAKE
        column: order_id
    t_order_item:
      actualDataNodes: ds_${0..1}.t_order_item_${0..1}
      tableStrategy:
        inline:
          shardingColumn: order_id
          algorithmExpression: t_order_item_${order_id % 2}
      keyGenerator:
        type: SNOWFLAKE
        column: order_item_id
  bindingTables:
    - t_order,t_order_item
  broadcastTables:
    - t_address
  defaultDatabaseStrategy:
    inline:
      shardingColumn: user_id
      algorithmExpression: ds_${user_id % 2}
  defaultTableStrategy:
    none:

与之前sharding-jdbc的主要区别是，sharding-proxy多了schemaName，这个schemaName是客户端配置的jdbc.url，例如:jdbc:mysql://127.0.0.1:3307/sharding_db?serverTimezone=UTC&useSSL=false。

YamlProxyRuleConfiguration包含了schema、数据源、规则配置。

public final class YamlProxyRuleConfiguration implements YamlConfiguration {
    // 客户端jdbc.url指定的schema
    private String schemaName;
    // dataSource
    private Map<String, YamlDataSourceParameter> dataSources = new HashMap<>();
    // 默认dataSource
    private YamlDataSourceParameter dataSource;
    // 分片规则 类比 ShardingRuleConfiguration
    private YamlShardingRuleConfiguration shardingRule;
    // 主从规则 类比 MasterSlaveRuleConfiguration
    private YamlMasterSlaveRuleConfiguration masterSlaveRule;
    // 加密规则 类比 EncryptRuleConfiguration
    private YamlEncryptRuleConfiguration encryptRule;
    // 影子规则 类比 ShadowRuleConfiguration
    private YamlShadowRuleConfiguration shadowRule;
}

二、启动

sharding-proxy-bootstrap模块下只有一个引导程序，即BootStrap作为main方法入口。

public final class Bootstrap {
  public static void main(final String[] args) throws IOException, SQLException {
      // SPI加载XXXDecorator
      registerDecorator();
      // 获取Netty启动端口 默认3307
      int port = getPort(args);
      // 加载Yaml配置
      ShardingConfiguration shardingConfig = new ShardingConfigurationLoader().load(getConfigPath(args));
      // 启动
      if (null == shardingConfig.getServerConfiguration().getOrchestration()) {
          // 无编排治理启动 SideCar
          startWithoutRegistryCenter(shardingConfig.getRuleConfigurationMap(), shardingConfig.getServerConfiguration().getAuthentication(), shardingConfig.getServerConfiguration().getProps(), port);
      } else {
          // 有编排治理启动 docs/document/content/features/orchestration/*.cn.md
          // 1 配置集中管理（配置中心）zk nacos etcd apollo
          // 2 支持proxy实例熔断，支持数据源禁用（注册中心） zk etcd
          startWithRegistryCenter(shardingConfig.getServerConfiguration(), shardingConfig.getRuleConfigurationMap().keySet(), shardingConfig.getRuleConfigurationMap(), port);
      }
  }
  
   private static void startWithoutRegistryCenter(final Map<String, YamlProxyRuleConfiguration> ruleConfigs,
                                                   final YamlAuthenticationConfiguration authentication, final Properties prop, final int port) throws SQLException {
        // Yaml配置转Authentication授权信息
        Authentication authenticationConfiguration = getAuthentication(authentication);
        // 初始化单例ShardingProxy上下文
        ShardingProxyContext.getInstance().init(authenticationConfiguration, prop);
        // 加载LogicSchema 类比sharding-jdbc的ShardingRuntimeContext
        LogicSchemas.getInstance().init(getDataSourceParameterMap(ruleConfigs), getRuleConfiguration(ruleConfigs));
        // 启动Netty服务端
        ShardingProxy.getInstance().start(port);
    }
}

组件注册->加载配置->初始化ShardingProxyContext上下文->初始化LogicSchemas->Netty启动。

1、组件注册

ShardingProxy基本复用了ShardingJDBC的组件，registerDecorator方法负责注册这些组件。

private static void registerDecorator() {
      // 1 解析 SQLParserEngine
      // 2 路由
      NewInstanceServiceLoader.register(RouteDecorator.class);
      // 3 重写
      NewInstanceServiceLoader.register(SQLRewriteContextDecorator.class);
      // 4 执行 ExecutorEngine
      // 5 归并
      NewInstanceServiceLoader.register(ResultProcessEngine.class);
  }

2、配置加载

ShardingConfigurationLoader加载/conf路径下的ShardingConfiguration全局配置。对于规则配置，一个schema只能有一个配置文件。

public final class ShardingConfigurationLoader {
	// path 默认为/conf
    public ShardingConfiguration load(final String path) throws IOException {
        Collection<String> schemaNames = new HashSet<>();
        // 加载/conf/server.yaml
        YamlProxyServerConfiguration serverConfig = loadServerConfiguration(new File(ShardingConfigurationLoader.class.getResource(path + "/" + SERVER_CONFIG_FILE).getFile()));
        File configPath = new File(ShardingConfigurationLoader.class.getResource(path).getFile());
        Collection<YamlProxyRuleConfiguration> ruleConfigurations = new LinkedList<>();
        // 加载/conf/config-*.yaml
        for (File each : findRuleConfigurationFiles(configPath)) {
            loadRuleConfiguration(each, serverConfig).ifPresent(yamlProxyRuleConfiguration -> {
                // 同一个schema只能有一个配置文件，否则报错
                Preconditions.checkState(
                        schemaNames.add(yamlProxyRuleConfiguration.getSchemaName()), "...");
                ruleConfigurations.add(yamlProxyRuleConfiguration);
            });
        }
        Preconditions.checkState(!ruleConfigurations.isEmpty() || null != serverConfig.getOrchestration(), "...");
        Map<String, YamlProxyRuleConfiguration> ruleConfigurationMap = ruleConfigurations.stream().collect(Collectors.toMap(YamlProxyRuleConfiguration::getSchemaName, each -> each));
        return new ShardingConfiguration(serverConfig, ruleConfigurationMap);
    }
}

3、初始化上下文

ShardingProxyContext全局单例，将server.yml中的授权配置和kv配置放入ShardingProxy全局上下文。同时ShardingProxyContext构造时将自身注册到了EventBus，如果配置了服务编排，可以动态更新（配置更新都是通过EventBus实现的）。

@Getter
public final class ShardingProxyContext {
    
    private static final ShardingProxyContext INSTANCE = new ShardingProxyContext();
    // server.yaml kv配置
    private ConfigurationProperties properties = new ConfigurationProperties(new Properties());
    // server.yaml 授权配置
    private Authentication authentication;
    // 断路器是否打开
    private boolean isCircuitBreak;
    
    private ShardingProxyContext() {
        ShardingOrchestrationEventBus.getInstance().register(this);
    }
    
    public static ShardingProxyContext getInstance() {
        return INSTANCE;
    }
    
    public void init(final Authentication authentication, final Properties props) {
        this.authentication = authentication;
        properties = new ConfigurationProperties(props);
    }
    
    @Subscribe
    public synchronized void renew(final PropertiesChangedEvent event) {
        ConfigurationLogger.log(event.getProps());
        properties = new ConfigurationProperties(event.getProps());
    }
    
    @Subscribe
    public synchronized void renew(final AuthenticationChangedEvent event) {
        ConfigurationLogger.log(event.getAuthentication());
        authentication = event.getAuthentication();
    }
    
    @Subscribe
    public synchronized void renew(final CircuitStateChangedEvent event) {
        isCircuitBreak = event.isCircuitBreak();
    }
}

4、初始化LogicSchemas

回顾一下sharding-jdbc中ShardingRuntimeContext的结构，如下：

ShardingRuntimeContext一个ShardingDataSource中是一个单例，基本可以认为是全局单例。持有下面这些实例：

• CachedDatabaseMetaData：数据库元信息
• ShardingTransactionManagerEngine：全局事务引擎
• ShardingSphereMetaData：sharding-jdbc数据源和数据表的元信息
• BaseRule：对应规则配置，如分片、加密、主从、影子
• DatabaseType：数据库类型，如MySQL
• ExecutorEngine：SQL执行引擎
• SQLParserEngine：SQL解析引擎

LogicSchemas全局单例，管理多个LogicSchema。LogicSchemas可以类比sharding-jdbc中的RuntimeContext。

@Getter
public final class LogicSchemas {
    // 全局单例
    private static final LogicSchemas INSTANCE = new LogicSchemas();
    // schema - LogicSchema
    private final Map<String, LogicSchema> logicSchemas = new ConcurrentHashMap<>();
    // 数据库类型
    private DatabaseType databaseType;
    // 注册到EventBus
    private LogicSchemas() {
        ShardingOrchestrationEventBus.getInstance().register(this);
    }
}

LogicSchema抽象类。

@Getter
public abstract class LogicSchema {
    // schema名称
    private final String name;
    // SQL解析引擎
    private final SQLParserEngine sqlParserEngine;
    // 持有 DataSource 事务引擎
    private JDBCBackendDataSource backendDataSource;
    
    public LogicSchema(final String name, final Map<String, YamlDataSourceParameter> dataSources) {
        this.name = name;
        sqlParserEngine = SQLParserEngineFactory.getSQLParserEngine(DatabaseTypes.getTrunkDatabaseTypeName(LogicSchemas.getInstance().getDatabaseType()));
        backendDataSource = new JDBCBackendDataSource(dataSources);
        ShardingOrchestrationEventBus.getInstance().register(this);
    }
}

LogicSchema构造时创建了SQL解析引擎，创建了JDBCBackendDataSource。JDBCBackendDataSource持有所有DataSource和全局事务引擎。

public final class JDBCBackendDataSource implements BackendDataSource, AutoCloseable {
    @Getter
    private Map<String, DataSource> dataSources;
    @Getter
    private ShardingTransactionManagerEngine shardingTransactionManagerEngine = new ShardingTransactionManagerEngine();
}

ShardingSchema分片配置的LogicSchema实现类，持有运行时的ShardingRule和ShardingSphereMetaData。

public final class ShardingSchema extends LogicSchema {
    // BaseRule 分片配置
    private ShardingRule shardingRule;
    // 数据库数据表 元数据信息
    private final ShardingSphereMetaData metaData;
}

可以看到LogicSchemas和ShardingRuntimeContext非常相似，前者比后者少了执行引擎，前者比后者多持有了数据源（通过持有JDBCBackendDataSource而间接持有了所有数据源）。

5、Netty启动

sharding-proxy-frontend模块负责处理网络通信、数据库协议。

Netty启动入口ShardingProxy::start。

public final class ShardingProxy {
    // 单例
    private static final ShardingProxy INSTANCE = new ShardingProxy();
    
    private EventLoopGroup bossGroup;
    
    private EventLoopGroup workerGroup;
    
    public static ShardingProxy getInstance() {
        return INSTANCE;
    }
    
    public void start(final int port) {
          ServerBootstrap bootstrap = new ServerBootstrap();
          // bossGroup 线程数量 1
          bossGroup = createEventLoopGroup();
          // workerGroup 线程数量 核数*2
          if (bossGroup instanceof EpollEventLoopGroup) {
              groupsEpoll(bootstrap);
          } else {
              groupsNio(bootstrap);
          }
          ChannelFuture future = bootstrap.bind(port).sync();
          future.channel().closeFuture().sync();
          
          // 省略try catch
    }
}

重点看一下Acceptor线程池和Worker线程池的配置。

对于Acceptor线程池，线程数为1。

private EventLoopGroup createEventLoopGroup() {
    return Epoll.isAvailable() ? new EpollEventLoopGroup(1) : new NioEventLoopGroup(1);
}

对于Worker线程池，线程数是2倍核数。

private void groupsNio(final ServerBootstrap bootstrap) {
    // 默认线程数 = 核数*2
    // Math.max(1, SystemPropertyUtil.getInt("io.netty.eventLoopThreads", NettyRuntime.availableProcessors() * 2));
    workerGroup = new NioEventLoopGroup();
    bootstrap.group(bossGroup, workerGroup)
            .channel(NioServerSocketChannel.class)
            .option(ChannelOption.SO_BACKLOG, 128)
            .option(ChannelOption.WRITE_BUFFER_WATER_MARK, new WriteBufferWaterMark(8 * 1024 * 1024, 16 * 1024 * 1024))
            // 使用PooledByteBufAllocator
            .option(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT)
            .childOption(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT)
            .childOption(ChannelOption.TCP_NODELAY, true)
            .handler(new LoggingHandler(LogLevel.INFO))
            // ChannelInitializer
            .childHandler(new ServerHandlerInitializer());
}

此外关注一下ServerHandlerInitializer，初始化ChannelPipeline时加入了哪些Handler。

@RequiredArgsConstructor
public final class ServerHandlerInitializer extends ChannelInitializer<SocketChannel> {
    
    @Override
    protected void initChannel(final SocketChannel socketChannel) {
        // 数据库协议引擎
        DatabaseProtocolFrontendEngine databaseProtocolFrontendEngine = DatabaseProtocolFrontendEngineFactory.newInstance(LogicSchemas.getInstance().getDatabaseType());
        ChannelPipeline pipeline = socketChannel.pipeline();
        // 针对不同数据库协议的编解码器
        pipeline.addLast(new PacketCodec(databaseProtocolFrontendEngine.getCodecEngine()));
        // 实际的业务处理器
        pipeline.addLast(new FrontendChannelInboundHandler(databaseProtocolFrontendEngine));
    }
}

• PacketCodec：针对不同数据库协议的编解码器。
• FrontendChannelInboundHandler：实际的业务处理器。

总结

• ShardingConfiguration是ShardingProxy的全局配置。配置可以分为两部分：
  • YamlProxyServerConfiguration：通常是/conf/server.yml，是sharding-proxy的服务端配置。它包括三部分：编排、授权、普通kv配置。
  • YamlProxyRuleConfiguration：通常是/conf/config-*.yaml，是sharding-jdbc的规则配置。它包括schema、数据源、规则配置。其中schema对应客户端jdbc.url中的数据库schema，如jdbc:mysql://127.0.0.1:3307/sharding_db?serverTimezone=UTC&useSSL=false。
• ShardingProxy的启动入口是sharding-proxy-bootstrap模块下的BootStrap#main。启动流程可以划分为五步：
  • 组件注册
  • 加载配置
  • 初始化上下文
  • 初始化LogicSchemas
  • Netty启动
• ShardingProxyContext持有sharding-proxy运行时的配置，并且支持动态更新。
• LogicSchemas可以类比sharding-jdbc中的ShardingRuntimeContext，是运行时上下文。包含数据源、分片配置、事务引擎、SQL解析引擎等。