lettuce 连接Redis 报 protected-mode 错误,redis恢复后仍然报protected-mode 错误

118 阅读3分钟

环境

redis 3.2.1

lettuce 5.2.1.RELEASE

<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-data-redis</artifactId>  
   <version>2.2.1.RELEASE</version>
</dependency>

现象

时刻动作或现象
T1lettuce 连接上Redis正常
T2使用了错误的配置文件重启redis
T3lettuce 报错 redis开启了保护 模式
T4使用正常的文件重启redis
T5lettuce 仍然报错 redis开启了保护 模式

奇怪的是redis已经恢复正常了,可是lettuce 还是报错,导致应用无法恢复,在应用重启后恢复正常。

T5时刻我使用wireshake 抓包发现未发送请求到redis server。

lettuce 报错如下

 Caused by: io.lettuce.core.RedisException: io.lettuce.core.RedisConnectionException: DENIED Redis is running in protected mode because protected mode is enabled, no bind address was specified, no authentication password is requested to clients. In this mode connections are only accepted from the loopback interface. If you want to connect from external computers to Redis you may adopt one of the following solutions: 1) Just disable protected mode sending the command 'CONFIG SET protected-mode no' from the loopback interface by connecting to Redis from the same host the server is running, however MAKE SURE Redis is not publicly accessible from internet if you do so. Use CONFIG REWRITE to make this change permanent. 2) Alternatively you can just disable the protected mode by editing the Redis configuration file, and setting the protected mode option to 'no', and then restarting the server. 3) If you started the server manually just for testing, restart it with the '--protected-mode no' option. 4) Setup a bind address or an authentication password. NOTE: You only need to do one of the above things in order for the server to start accepting connections from the outside.
        at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:129)
        at io.lettuce.core.FutureSyncInvocationHandler.handleInvocation(FutureSyncInvocationHandler.java:69)
        at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
        at com.sun.proxy.$Proxy127.hvals(Unknown Source)
        at org.springframework.data.redis.connection.lettuce.LettuceHashCommands.hVals(LettuceHashCommands.java:369)
        ... 79 common frames omitted
Caused by: io.lettuce.core.RedisConnectionException: DENIED Redis is running in protected mode because protected mode is enabled, no bind address was specified, no authentication password is requested to clients. In this mode connections are only accepted from the loopback interface. If you want to connect from external computers to Redis you may adopt one of the following solutions: 1) Just disable protected mode sending the command 'CONFIG SET protected-mode no' from the loopback interface by connecting to Redis from the same host the server is running, however MAKE SURE Redis is not publicly accessible from internet if you do so. Use CONFIG REWRITE to make this change permanent. 2) Alternatively you can just disable the protected mode by editing the Redis configuration file, and setting the protected mode option to 'no', and then restarting the server. 3) If you started the server manually just for testing, restart it with the '--protected-mode no' option. 4) Setup a bind address or an authentication password. NOTE: You only need to do one of the above things in order for the server to start accepting connections from the outside.
        at io.lettuce.core.protocol.CommandHandler.onProtectedMode(CommandHandler.java:751)
        at io.lettuce.core.protocol.CommandHandler.decode(CommandHandler.java:597)
        at io.lettuce.core.protocol.CommandHandler.channelRead(CommandHandler.java:556)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:794)
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollRdHupReady(AbstractEpollChannel.java:437)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:431)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        ... 1 common frames omitted


报错message

DENIED Redis is running in protected mode because protected mode is enabled, no bind address was specified, no authentication password is requested to clients. In this mode connections are only accepted from the loopback interface. If you want to connect from external computers to Redis you may adopt one of the following solutions: 1) Just disable protected mode sending the command 'CONFIG SET protected-mode no' from the loopback interface by connecting to Redis from the same host the server is running, however MAKE SURE Redis is not publicly accessible from internet if you do so. Use CONFIG REWRITE to make this change permanent. 2) Alternatively you can just disable the protected mode by editing the Redis configuration file, and setting the protected mode option to 'no', and then restarting the server. 3) If you started the server manually just for testing, restart it with the '--protected-mode no' option. 4) Setup a bind address or an authentication password. NOTE: You only need to do one of the above things in order for the server to start accepting connections from the outside.

翻译结果

DENIED Redis 正在保护模式下运行,因为启用了保护模式,没有指定绑定地址,也没有向客户端请求身份验证密码。 在此模式下,仅接受来自环回接口的连接。 如果您想从外部计算机连接到 Redis,您可以采用以下解决方案之一:

  1. 只需从环回接口发送命令“CONFIG SET protected-mode no”,通过从服务器的同一主机连接到 Redis 来禁用保护模式 正在运行,但是如果您这样做,请确保 Redis 不能从互联网上公开访问。 使用 CONFIG REWRITE 使此更改永久生效。
  2. 或者,您可以通过编辑Redis配置文件并将protected mode设置为“no”来禁用保护模式,然后重新启动服务器。
  3. 如果您手动启动服务器只是为了测试,请使用“--protected-mode no”选项重新启动它。
  4. 设置绑定地址或验证密码。 注意:您只需执行上述操作之一即可使服务器开始接受来自外部的连接。

该报错如何解决

redis.conf 配置

bind

requirepass

protected-mode

只有 当requirepass和bind注释掉,并且protected-mode为yes的时候,保护模式才开启。

# requirepass
# bind
protected-mode yes

你只要设置bind和requirepass 就会让保护模式失效。

为啥lettuce没有发送请求而是直接报错了

我debug 深入源码看了下发送请求的代码

Redis发送请求的堆栈

write:144, DefaultEndpoint (io.lettuce.core.protocol)
write:112, CommandExpiryWriter (io.lettuce.core.protocol)
dispatch:187, RedisChannelHandler (io.lettuce.core)
dispatch:152, StatefulRedisConnectionImpl (io.lettuce.core)
dispatch:467, AbstractRedisAsyncCommands (io.lettuce.core)
lrange:871, AbstractRedisAsyncCommands (io.lettuce.core)
invoke:-1, GeneratedMethodAccessor32 (sun.reflect)
invoke:43, DelegatingMethodAccessorImpl (sun.reflect)
invoke:498, Method (java.lang.reflect)
handleInvocation:57, FutureSyncInvocationHandler (io.lettuce.core)
invoke:80, AbstractInvocationHandler (io.lettuce.core.internal)
lrange:-1, $Proxy83 (com.sun.proxy)
lRange:181, LettuceListCommands (org.springframework.data.redis.connection.lettuce)
lRange:594, DefaultedRedisConnection (org.springframework.data.redis.connection)
lambda$range$6:171, DefaultListOperations (org.springframework.data.redis.core)
doInRedis:-1, 1489095802 (org.springframework.data.redis.core.DefaultListOperations$$Lambda$582)
execute:228, RedisTemplate (org.springframework.data.redis.core)
execute:188, RedisTemplate (org.springframework.data.redis.core)
execute:96, AbstractOperations (org.springframework.data.redis.core)
range:171, DefaultListOperations (org.springframework.data.redis.core)
printHello:35, Hello1Controller (org.example.springboot.learn.controller)

io.lettuce.core.protocol.DefaultEndpoint#write(io.lettuce.core.protocol.RedisCommand<K,V,T>)

debug 发现isConnected()方法判断为false,走了writeToDisconnectedBuffer方法

public <K, V, T> RedisCommand<K, V, T> write(RedisCommand<K, V, T> command) {

    LettuceAssert.notNull(command, "Command must not be null");

    try {
        sharedLock.incrementWriters();

        validateWrite(1);

        if (autoFlushCommands) {
          
            if (isConnected()) {
                writeToChannelAndFlush(command);
            } else {
                // 如果未连接
                writeToDisconnectedBuffer(command);
            }

        } else {
            writeToBuffer(command);
        }
    } finally {
        sharedLock.decrementWriters();
        if (debugEnabled) {
            logger.debug("{} write() done", logPrefix());
        }
    }

    return command;
}

io.lettuce.core.protocol.DefaultEndpoint#writeToDisconnectedBuffer(io.lettuce.core.protocol.RedisCommand)

可以从源码中看到直接获取了之前的连接错误,直接返回了。

private void writeToDisconnectedBuffer(RedisCommand<?, ?, ?> command) {

    if (connectionError != null) {
        if (debugEnabled) {
            logger.debug("{} writeToDisconnectedBuffer() Completing command {} due to connection error", logPrefix(),
                    command);
        }
        command.completeExceptionally(connectionError);

        return;
    }

    if (debugEnabled) {
        logger.debug("{} writeToDisconnectedBuffer() buffering (disconnected) command {}", logPrefix(), command);
    }

    disconnectedBuffer.add(command);
}

为啥没有重连?

T3 时刻Redis第一次连接保protected-mode错误时的 log

 io.lettuce.core.protocol.CommandHandler  : [channel=0x14a79e33, /10.0.21.50:64857 -> /10.12.1.233:6379, chid=0x4] channelActive() done
 i.l.core.protocol.ReconnectionHandler    : Reconnected to 10.12.1.233:6379, Channel channel=0x14a79e33, /10.0.21.50:64857 -> /10.12.1.233:6379
 i.l.core.protocol.ConnectionWatchdog     : [channel=0x14a79e33, /10.0.21.50:64857 -> /10.12.1.233:6379, last known addr=/10.12.1.233:6379] userEventTriggered(ctx, io.lettuce.core.ConnectionEvents$Activated@238b0ba0)
 io.lettuce.core.protocol.CommandHandler  : [channel=0x14a79e33, /10.0.21.50:64857 -> /10.12.1.233:6379, chid=0x4] Received: 1024 bytes, 1 commands in the stack
 io.lettuce.core.protocol.CommandHandler  : [channel=0x14a79e33, /10.0.21.50:64857 -> /10.12.1.233:6379, chid=0x4] Stack contains: 1 commands
 i.l.core.protocol.RedisStateMachine      : Decode AsyncCommand [type=AUTH, output=StatusOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command]
 i.l.core.protocol.RedisStateMachine      : Decoded AsyncCommand [type=AUTH, output=StatusOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command], empty stack: false
 io.lettuce.core.protocol.CommandHandler  : [channel=0x14a79e33, /10.0.21.50:64857 -> /10.12.1.233:6379, chid=0x4] Received: 77 bytes, 1 commands in the stack
 io.lettuce.core.protocol.CommandHandler  : [channel=0x14a79e33, /10.0.21.50:64857 -> /10.12.1.233:6379, chid=0x4] Stack contains: 1 commands
 i.l.core.protocol.RedisStateMachine      : Decode AsyncCommand [type=AUTH, output=StatusOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command]
 i.l.core.protocol.RedisStateMachine      : Decoded AsyncCommand [type=AUTH, output=StatusOutput [output=null, error='DENIED Redis is running in protected mode because protected mode is enabled, no bind address was specified, no authentication password is requested to clients. In this mode connections are only accepted from the loopback interface. If you want to connect from external computers to Redis you may adopt one of the following solutions: 1) Just disable protected mode sending the command 'CONFIG SET protected-mode no' from the loopback interface by connecting to Redis from the same host the server is running, however MAKE SURE Redis is not publicly accessible from internet if you do so. Use CONFIG REWRITE to make this change permanent. 2) Alternatively you can just disable the protected mode by editing the Redis configuration file, and setting the protected mode option to 'no', and then restarting the server. 3) If you started the server manually just for testing, restart it with the '--protected-mode no' option. 4) Setup a bind address or an authentication password. NOTE: You only need to do one of the above things in order for the server to start accepting connections from the outside.'], commandType=io.lettuce.core.protocol.Command], empty stack: true
 io.lettuce.core.protocol.CommandHandler  : [channel=0x14a79e33, /10.0.21.50:64857 -> /10.12.1.233:6379, chid=0x4] channelInactive()
 i.lettuce.core.protocol.DefaultEndpoint  : [channel=0x14a79e33, /10.0.21.50:64857 -> /10.12.1.233:6379, epid=0x1] deactivating endpoint handler
 i.lettuce.core.protocol.DefaultEndpoint  : [channel=0x14a79e33, /10.0.21.50:64857 -> /10.12.1.233:6379, epid=0x1] notifyQueuedCommands adding 0 command(s) to buffer
 io.lettuce.core.protocol.CommandHandler  : [channel=0x14a79e33, /10.0.21.50:64857 -> /10.12.1.233:6379, chid=0x4] channelInactive() done
 i.l.core.protocol.ConnectionWatchdog     : [channel=0x14a79e33, /10.0.21.50:64857 -> /10.12.1.233:6379, last known addr=/10.12.1.233:6379] channelInactive()
 i.l.core.protocol.ConnectionWatchdog     : [channel=0x14a79e33, /10.0.21.50:64857 -> /10.12.1.233:6379, last known addr=/10.12.1.233:6379] Reconnect scheduling disabled
 io.lettuce.core.protocol.CommandHandler  : [channel=0x14a79e33, /10.0.21.50:64857 -> /10.12.1.233:6379, chid=0x4] channelUnregistered()

可以看到类ConnectionWatchdog打印了 Reconnect scheduling disabled 这句 定时重连关闭了。

io.lettuce.core.protocol.DefaultEndpoint#notifyException

可以看到如果是连接错误并且是 保护模式,那么就不重试了

public void notifyException(Throwable t) {
  
    if (t instanceof RedisConnectionException && RedisConnectionException.isProtectedMode(t.getMessage())) {

        connectionError = t;

        if (connectionWatchdog != null) {
            connectionWatchdog.setListenOnChannelInactive(false);
            // 
            connectionWatchdog.setReconnectSuspended(false);
        }

        doExclusive(this::drainCommands).forEach(cmd -> cmd.completeExceptionally(t));
    }

    if (!isConnected()) {
        connectionError = t;
    }
}

总结

碰到此种情况只能重启应用,我GitHub 已经提了相关issue,still throw protected-mode yes error after redis set protected-mode no,看看作者是否考虑到此种情况。

作者已经根据我提的问题 修复了,开启了重连,你可以从issue中看到相关的代码改动。