rabbitmq 消费者消费消息异常,无限重试导致日志过大,磁盘被占用满

749 阅读2分钟

问题的排查以及解决方式

消费者模块日志查看

org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Listener threw exception
2020/1/19 上午3:13:38	at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.wrapToListenerExecutionFailedExceptionIfNeeded(AbstractMessageListenerContainer.java:1506) ~[spring-rabbit-2.0.5.RELEASE.jar!/:2.0.5.RELEASE]
2020/1/19 上午3:13:38	at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1417) ~[spring-rabbit-2.0.5.RELEASE.jar!/:2.0.5.RELEASE]
2020/1/19 上午3:13:38	at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.actualInvokeListener(AbstractMessageListenerContainer.java:1337) ~[spring-rabbit-2.0.5.RELEASE.jar!/:2.0.5.RELEASE]
2020/1/19 上午3:13:38	at sun.reflect.GeneratedMethodAccessor446.invoke(Unknown Source) ~[na:na]
2020/1/19 上午3:13:38	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_111-internal]
2020/1/19 上午3:13:38	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111-internal]
2020/1/19 上午3:13:38	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) ~[spring-aop-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
2020/1/19 上午3:13:38	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) ~[spring-aop-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
2020/1/19 上午3:13:38	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
2020/1/19 上午3:13:38	at brave.spring.rabbit.TracingRabbitListenerAdvice.invoke(TracingRabbitListenerAdvice.java:84) ~[brave-instrumentation-spring-rabbit-5.1.0.jar!/:na]
2020/1/19 上午3:13:38	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) ~[spring-aop-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
2020/1/19 上午3:13:38	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) ~[spring-aop-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
2020/1/19 上午3:13:38	at org.springframework.amqp.rabbit.listener.$Proxy220.invokeListener(Unknown Source) ~[na:2.0.5.RELEASE]
2020/1/19 上午3:13:38	at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:1324) ~[spring-rabbit-2.0.5.RELEASE.jar!/:2.0.5.RELEASE]
2020/1/19 上午3:13:38	at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:1303) ~[spring-rabbit-2.0.5.RELEASE.jar!/:2.0.5.RELEASE]
2020/1/19 上午3:13:38	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:817) [spring-rabbit-2.0.5.RELEASE.jar!/:2.0.5.RELEASE]
2020/1/19 上午3:13:38	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:801) [spring-rabbit-2.0.5.RELEASE.jar!/:2.0.5.RELEASE]
2020/1/19 上午3:13:38	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$700(SimpleMessageListenerContainer.java:77) [spring-rabbit-2.0.5.RELEASE.jar!/:2.0.5.RELEASE]
2020/1/19 上午3:13:38	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1042) [spring-rabbit-2.0.5.RELEASE.jar!/:2.0.5.RELEASE]
2020/1/19 上午3:13:38	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111-internal]
2020/1/19 上午3:13:38Caused by: org.springframework.amqp.AmqpException: No method found for class java.lang.String
2020/1/19 上午3:13:38	at org.springframework.amqp.rabbit.listener.adapter.DelegatingInvocableHandler.getHandlerForPayload(DelegatingInvocableHandler.java:147) ~[spring-rabbit-2.0.5.RELEASE.jar!/:2.0.5.RELEASE]
2020/1/19 上午3:13:38	at org.springframework.amqp.rabbit.listener.adapter.DelegatingInvocableHandler.getMethodNameFor(DelegatingInvocableHandler.java:250) ~[spring-rabbit-2.0.5.RELEASE.jar!/:2.0.5.RELEASE]
2020/1/19 上午3:13:38	at org.springframework.amqp.rabbit.listener.adapter.HandlerAdapter.getMethodAsString(HandlerAdapter.java:70) ~[spring-rabbit-2.0.5.RELEASE.jar!/:2.0.5.RELEASE]
2020/1/19 上午3:13:38	at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:190) ~[spring-rabbit-2.0.5.RELEASE.jar!/:2.0.5.RELEASE]
2020/1/19 上午3:13:38	at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:120) ~[spring-rabbit-2.0.5.RELEASE.jar!/:2.0.5.RELEASE]

针对上述日志中产生的问题的解决方案

  1. 在消费者的类上面删除掉注解
@RabbitListener(queues = OfficeQueue.OFFICE_QUEUE)
  1. 在消费者的方法上面加上注解
@RabbitListener(queues = OfficeQueue.OFFICE_QUEUE)
  1. 在消费者的方法上面去掉注解
@RabbitHandler

上面的方法处理完后又出现异常(忘记记录)

  1. 总的意思是在说,消费者在队列中获取这条消息时,生产者那边生产的 message 是空的;

问题排查

对服务里面的各个模块生产消息的地方进行代码检查:

  1. 发现有几处生产消息的地方,message 发送出去时,内容是空的;

解决方式

  1. 将所有生产者 message 为空的地方作出修改,不允许消息为空还发出消息的情况;

本次异常引申出的一些问题

  1. 日志非常非常的大,本来服务器 180G 的磁盘空间,已经被占满了 152G
  2. 由于磁盘空间越来越少,导致其他服务器再发布时,无法发布成功