记录一下实际开发中出现的问题
问题描述
真实环境中需要部署新功能，需要重启之前的项目，但在重启过程中发现大量的端口处于 time wait ，导致端口被占用，项目启动失败。重新修改端口后启动成功并注册成功，但出现了 Failed to invoke the method * in the * 。
环境
- java: 8
- dubbo: 2.7.1
- nacos: docker 中最新版本
日志如下:
org.apache.dubbo.rpc.RpcException: Failed to invoke the method all in the service managercenter.service.IDevTargetMService.
Tried 3 times of the providers [192.168.0.138:21882] (1/1)
from the registry 127.0.0.1:8848 on the consumer 192.168.0.138
using the dubbo version 2.7.1. Last error is: Failed to invoke
remote method: all, provider: dubbo://192.168.0.138:21882/managercenter.service.IDevTargetMService?anyhost=true&application=all&bean.name=providers:dubbo:managercenter.service.IDevTargetMService&category=providers&check=false&default.check=false&default.deprecated=false&default.dynamic=false&default.generic=false&default.lazy=false&default.reference.filter=consumerHttpContextFilter&default.register=true&default.service.filter=providerHttpContextFilter&default.sticky=false&default.timeout=3000000&deprecated=false&dubbo=2.0.2&dynamic=false&generic=false&interface=managercenter.service.IDevTargetMService&lazy=false&methods=findDevTarget,findByIdNullable,save,deleteUpdate,update,insert,findCopyToObjects,delete,insertOrUpdate,download,find,findById,setConfiguration,findDevTargets,all,findPagedList,findCopyToDatas,saveCopyDatas,devImport,checkDataIsExist,findDevInvalidTargets,findPeriodType,findByIdNonNull,findPaseTargets,deleteById,queryDirectory,deletePermanent,insertOrUpdateByIdIsNull&path=managercenter.service.IDevTargetMService&pid=42184&protocol=dubbo&qos.accept.foreign.ip=false&qos.enable=false®ister=true®ister.ip=192.168.0.138&release=2.7.1&remote.application=all&remote.timestamp=1662691258034&revision=1.0.1&side=consumer&sticky=false×tamp=1662691217628, cause: message can not send, because channel is closed . url:dubbo://192.168.0.138:21882/control.service.IEntityDataCService?anyhost=true&application=all&bean.name=providers:dubbo:control.service.IEntityDataCService&category=providers&check=false&codec=dubbo&default.check=false&default.deprecated=false&default.dynamic=false&default.generic=false&default.lazy=false&default.reference.filter=consumerHttpContextFilter&default.register=true&default.service.filter=providerHttpContextFilter&default.sticky=false&default.timeout=3000000&deprecated=false&dubbo=2.0.2&dynamic=false&generic=false&heartbeat=60000&interface=control.service.IEntityDataCService&lazy=false&methods=all,findByIdNullable,deleteAll,save,updateAll,update,insert,findByInIds,delete,insertOrUpdate,find,findByIdNonNull,findById,deleteById,deletePermanent,insertAll,insertOrUpdateByIdIsNull,batchDeleteAll&path=control.service.IEntityDataCService&pid=42184&protocol=dubbo&qos.accept.foreign.ip=false&qos.enable=false®ister=true®ister.ip=192.168.0.138&release=2.7.1&remote.application=all&remote.timestamp=1662690025452&revision=1.0.1&side=consumer&sticky=false×tamp=1662691214667
at org.apache.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:111) ~[dubbo-2.7.1.jar:2.7.1]
at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:242) ~[dubbo-2.7.1.jar:2.7.1]
at org.apache.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:76) ~[dubbo-2.7.1.jar:2.7.1]
at org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:57) ~[dubbo-2.7.1.jar:2.7.1]
at org.apache.dubbo.common.bytecode.proxy19.all(proxy19.java) ~[dubbo-2.7.1.jar:2.7.1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_272]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_272]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_272]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_272]
at org.apache.dubbo.config.spring.beans.factory.annotation.ReferenceAnnotationBeanPostProcessor$ReferenceBeanInvocationHandler.invoke(ReferenceAnnotationBeanPostProcessor.java:165) ~[dubbo-2.7.1.jar:2.7.1]
at com.sun.proxy.$Proxy190.all(Unknown Source) ~[?:?]
at warning.scheduler.DevTargetInfoSchedule.run(DevTargetInfoSchedule.java:47) [grape-component-warning-sharding-1.0.1.jar:1.0.1]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:783) [spring-boot-2.0.9.RELEASE.jar:2.0.9.RELEASE]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:773) [spring-boot-2.0.9.RELEASE.jar:2.0.9.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) [spring-boot-2.0.9.RELEASE.jar:2.0.9.RELEASE]
at org.grape.GrapeApplication.initSpring(GrapeApplication.java:138) [grape-container-1.1.3.jar:1.1.3]
at org.grape.GrapeApplication.main(GrapeApplication.java:83) [grape-container-1.1.3.jar:1.1.3]
排查思路
1、确认 dubbo 服务是否注册成功，通过 host:8848/nacos 页面中确认了服务已经注册。
2、并测试其中的服务是否正常（自行搜索一下dubbo 服务测试），唯独 IDevTargetMService 无法调用成功。
3、( 问题解决 ) 根据问题描述中 出现大量 time wait， 猜想之前的服务并是否没有被成功 kill 掉，所以导致服务一直调用到之前未启动成功的项目中？通过排查 Java 进程，查询出有许多重复的项目未被关闭。如下（只是模拟一下场景，排查过程中出现了 3 - 6 重复的项目）。kill 重复项目后，服务正常
查看 java 进程脚本（JDK8，openJDK 缺少命令没法使用）
file=$(jps |awk '{print $1}');
num=1;
for v in $file
do
fielname[$num]=$v;
echo " $num : $v "
num=`expr $num + 1`
done
select=-1;
while :
do
if read -t 60 -p "Enter the program to java the project,Y withdraw from :" #limited time 5s
then
case $REPLY in
Y|y)
RECOVERY_Recovery=1;
echo -e "\n Update Recovery\n"
break
;;
R|r)
num=1;
echo -e "\n";
for v in $(jps |awk '{print $1}')
do
fielname[$num]=$v;
echo " $num : $v "
num=`expr $num + 1`
done
continue
;;
*)
select=$REPLY
cd /proc/${fielname[$REPLY]}
ls -l | sed -n "11, 1p" | awk '{print $11}'
while :
do
if read -t 60 -n 1 -p "Enter the program to kill the project,Y withdraw from :" #limited time 5s
then
case $REPLY in
Y|y)
RECOVERY_Recovery=1;
echo -e "\n kill start\n"
kill -9 ${fielname[$select]}
break
;;
*)
break
esac
else
RECOVERY_Recovery=0;
echo -e "\n Not kill\n"
break
fi
done
continue
esac
else
RECOVERY_Recovery=0;
echo -e "\n Not java cmd \n"
break
fi
done
