都lettuce就别再pipeline了

1,072 阅读3分钟

1.背景

前段时间线上有一些超时报警。看了下日志有一些异常堆栈让我非常的困惑,异常现实无法获取到连接,我寻思着lettuce不是共享连接,怎么还有这种情况。

7F5C0929-F2F8-4692-A775-D1348CD80708.png 再认真观察了一下日志,发现这些堆栈都和pipeline相关.

E6E7C897-1154-45A5-8101-24AE743AD347.png

两个问题

1.lettuce的pipeline会特殊处理?

2.业务同学使用pipeline主要是为了减少for循环带来的网络IO。如果是lettuce我们是否还需要考虑这一点?

2.看源码

带着两个问题,我翻出了lettuce的源码一探究竟。因为我们项目使用的spring-data-redis。所以我从spring源码开始。

RedisTemplate#execute

@Nullable
public <T> T execute(RedisCallback<T> action, boolean exposeConnection, boolean pipeline) {
   RedisConnectionFactory factory = getRequiredConnectionFactory();
   RedisConnection conn = RedisConnectionUtils.getConnection(factory, enableTransactionSupport);
   try {

      boolean existingConnection = TransactionSynchronizationManager.hasResource(factory);
      RedisConnection connToUse = preProcessConnection(conn, existingConnection);

      RedisConnection connToExpose = (exposeConnection ? connToUse : createRedisConnectionProxy(connToUse));
      T result = action.doInRedis(connToExpose);

      return postProcessResult(result, connToUse, existingConnection);
   } finally {
      RedisConnectionUtils.releaseConnection(conn, factory, enableTransactionSupport);
   }
}
// 没用的代码都被删掉了

所有redis的操作执行都会调用这个方法。

1.获取工厂。这里就是我们配置的lettuce的工厂。

2.调用RedisConnectionUtils.getConnection方法获取连接。

3.拿到连接调用action.doInRedis。对于pipeline的这个action参考下文executePipelined源码逻辑

我们继续关注获取连接的逻辑.

LettuceConnectionFactory#getConnection

public RedisConnection getConnection() {
   assertInitialized();
   if (isClusterAware()) {
      return getClusterConnection();
   }

   LettuceConnection connection;
   connection = doCreateLettuceConnection(getSharedConnection(), connectionProvider, getTimeout(), getDatabase());
   connection.setConvertPipelineAndTxResults(convertPipelineAndTxResults);
   return connection;
}

protected StatefulRedisConnection<byte[], byte[]> getSharedConnection() {
   return shareNativeConnection && !isClusterAware()
         ? (StatefulRedisConnection) getOrCreateSharedConnection().getConnection()
         : null;
}

protected LettuceConnection doCreateLettuceConnection(
      @Nullable StatefulRedisConnection<byte[], byte[]> sharedConnection, LettuceConnectionProvider connectionProvider,
      long timeout, int database) {

   LettuceConnection connection = new LettuceConnection(sharedConnection, connectionProvider, timeout, database);
   connection.setPipeliningFlushPolicy(this.pipeliningFlushPolicy);

   return connection;
}

1.调用getSharedConnection获取SharedConnection(第一次会创建)

SharedConnection是LettuceConnectionFactory的一个内部类。他的getConnection可以获取到共享的连接。第一次会调用getNativeConnection创建一个。成功之后这个共享链接会一直被使用。并不会被释放。除非调用LettuceConnectionFactorydestroy方法才会reset此连接。 代码如下

StatefulConnection<E, E> getConnection() {
   synchronized (this.connectionMonitor) {
      if (this.connection == null) {
         this.connection = getNativeConnection();
      }
      if (getValidateConnection()) {
         validateConnection();
      }
      return this.connection;
   }
}

doCreateLettuceConnection

每次执行都会new一个LettuceConnection对象。入参为sharedConnection和一个connectionProvider。这两个对象都保存在LettuceConnectionFactory中。

sharedConnection就是上面的共享连接。connectionProvider使用创建真正连接的Provider。LettuceConnection会将sharedConnection存储到全局变量以备使用。

如果是普通redis请求,直接通过sharedConnection获取对应连接处理即可。但是对于pipeline却非如此。剧透一下,对于pipeline会通过connectionProvider获取一个asyncDedicatedConn

executePipelined

public List<Object> executePipelined(RedisCallback<?> action, @Nullable RedisSerializer<?> resultSerializer) {

   return execute((RedisCallback<List<Object>>) connection -> {
      connection.openPipeline();
      boolean pipelinedClosed = false;
      try {
         Object result = action.doInRedis(connection);
         if (result != null) {
            throw new InvalidDataAccessApiUsageException(
                  "Callback cannot return a non-null value as it gets overwritten by the pipeline");
         }
         List<Object> closePipeline = connection.closePipeline();
         pipelinedClosed = true;
         return deserializeMixedResults(closePipeline, resultSerializer, hashKeySerializer, hashValueSerializer);
      } finally {
         if (!pipelinedClosed) {
            connection.closePipeline();
         }
      }
   });
}

上面的action.doInRedis其实就是这块代码

1.开启connection.openPipeline。

2.调用pipeline的逻辑代码action.doInRedis

3.closePipeline。其实就是聚集返回的结果然后反序列化返回。

LettuceConnection#openPipeline

public void openPipeline() {

   if (!isPipelined) {
      isPipelined = true;
      ppline = new ArrayList<>();
      flushState = this.pipeliningFlushPolicy.newPipeline();
      flushState.onOpen(this.getOrCreateDedicatedConnection());
   }
}

这个LettuceConnection对象就是上面代码new出来的。对象里面有一个SharedConnection。

LettuceConnection#getOrCreateDedicatedConnection

这个方法会调用doGetAsyncDedicatedConnection创建一个StatefulConnection保存在LettuceConnection的asyncDedicatedConn全局变量。具体创建逻辑和共享连接一摸一样。

private StatefulConnection<byte[], byte[]> getOrCreateDedicatedConnection() {

   if (asyncDedicatedConn == null) {
      asyncDedicatedConn = doGetAsyncDedicatedConnection();
   }
   return asyncDedicatedConn;
}

protected StatefulConnection<byte[], byte[]> doGetAsyncDedicatedConnection() {
   StatefulConnection connection = connectionProvider.getConnection(StatefulConnection.class);
   if (customizedDatabaseIndex()) {
      potentiallySelectDatabase(dbIndex);
   }
   return connection;
}

前面都是连接的初始化过程。

命令执行

List list = redisTemplate.executePipelined((RedisCallback<Object>) connection -> {
   connection.stringCommands().set("11".getBytes(), "22".getBytes());
   return null;
});

最终会调用LettuceStringCommands的set方法

public Boolean set(byte[] key, byte[] value) {

   return connection.invoke().from(RedisStringAsyncCommands::set, key, value)
         .get(Converters.stringToBooleanConverter());
}
LettuceInvoker invoke() {
   return invoke(getAsyncConnection());
}

这里其实包装了一下invoke,通过getAsyncConnection获取连接

RedisClusterAsyncCommands<byte[], byte[]> getAsyncConnection() {

   if (isQueueing() || isPipelined()) {
      return getAsyncDedicatedConnection();
   }

   if (asyncSharedConn != null) {

      if (asyncSharedConn instanceof StatefulRedisConnection) {
         return ((StatefulRedisConnection<byte[], byte[]>) asyncSharedConn).async();
      }
      if (asyncSharedConn instanceof StatefulRedisClusterConnection) {
         return ((StatefulRedisClusterConnection<byte[], byte[]>) asyncSharedConn).async();
      }
   }
   return getAsyncDedicatedConnection();
}

对于pipeline直接获取LettuceConnection的dedicatedConnection返回。其实到这里已经很清楚。

释放连接:RedisConnectionUtils.releaseConnection

会调用LettuceConnection的close方法。

public void close() throws DataAccessException {
   super.close();
   if (isClosed) {
      return;
   }
   isClosed = true;
   if (asyncDedicatedConn != null) {
      try {
         if (customizedDatabaseIndex()) {
            potentiallySelectDatabase(defaultDbIndex);
         }
         connectionProvider.release(asyncDedicatedConn);
      } catch (RuntimeException ex) {
         throw convertLettuceAccessException(ex);
      }
   }
   if (subscription != null) {
      if (subscription.isAlive()) {
         subscription.doClose();
      }
      subscription = null;
   }

   this.dbIndex = defaultDbIndex;
}

如果asyncDedicatedConn不为空则release调用连接,对于SharedConnection不进行释放。到这里已经水落石出了。

3.总结

对于普通请求,直接通过SharedConnection共享执行。线程之间共享链接互不影响。

但是对于pipeline,则会从连接池获取。如果没有空闲连接,则会去创建。如果最大连接数太小,就会出现上面的异常。并且只有pipeline执行完成之后链接才会被释放,所以lettuce的pipeline和jedis的模式没什么区别,阻塞式调用。