开启掘金成长之旅！这是我参与「掘金日新计划 · 12 月更文挑战」的第7天，点击查看活动详情
环境迁移导致的小问题
最近有同事在迁移环境，发现有个很老的应用启动不了。并且相关负责人都已经离职，无辜的我只能再次躺枪。
拿着报错相关的日志，发现还是Dubbo相关的异常。现在公司内部的应用已经不用Dubbo，所以也不是很熟悉，只能先从日志看起来了。(无奈重启大法没有生效，没人维护的应用在很老的项目还在使用，遇到问题实在不想去填坑，第一步就是重启几次....)。
大概日志如下：
601 INFO - DubboRegistry - [DUBBO] Reconnect to registry dubbo://nacos.svc.cluster.local:8848/org.apache.dubbo.registry.RegistryService?address=nacos.svc.cluster.local:8848&application=app&callbacks=10000&connect.timeout=10000&dubbo=2.0.2&interface=org.apache.dubbo.registry.RegistryService&lazy=true&methods=lookup,subscribe,unsubscribe,unregister,register&pid=1&reconnect=false&release=2.7.3&sticky=true&subscribe.1.callback=true&timeout=10000×tamp=1670915197560&unsubscribe.1.callback=false, dubbo version: 2.7.3, current host: 10.244.18.35
Last error is: Invoke remote method timeout. method: subscribe, provider: dubbo://nacos.svc.cluster.local:8848/org.apache.dubbo.registry.RegistryService?address=nacos.svc.cluster.local:8848&application=app&callbacks=10000&check=false&connect.timeout=10000&dubbo=2.0.2&interface=org.apache.dubbo.registry.RegistryService&lazy=true&methods=lookup,subscribe,unsubscribe,unregister,register&pid=1&reconnect=false&release=2.7.3&remote.application=app&sticky=true&subscribe.1.callback=true&timeout=10000×tamp=1670915197560&unsubscribe.1.callback=false, cause: org.apache.dubbo.remoting.TimeoutException: Waiting server-side response timeout by scan timer
通过日志，发现错误信息和注册中心相关，并且注册中心的地址前缀是dubbo://。因为项目使用的是nacos做注册中心，前缀应该是nacos://。于是又找人看了下配置的环境变量，果然是因为没有配置前缀，Dubbo会提供默认值dubbo://。
当前这个应用作为Provider，在项目启动的时候就会去进行暴露服务的的过程，主要的操作就是进行服务的远程暴露或者本地暴露，以及暴露的服务信息注册到注册中心提供给Consumer。于是趁着这个机会阅读一下Dubbo中相关的源码。
源码分析
可以直接从下面的demo入手：
private static void startWithBootstrap() {
ServiceConfig<DemoServiceImpl> service = new ServiceConfig<>();
service.setInterface(DemoService.class);
service.setRef(new DemoServiceImpl());
DubboBootstrap bootstrap = DubboBootstrap.getInstance();
bootstrap.application(new ApplicationConfig("dubbo-demo-api-provider"))
.registry(new RegistryConfig("zookeeper://127.0.0.1:2181"))
.service(service)
.start()
.await();
}
上面指定了需要暴露的服务,DemoService以及使用的注册中心，这里是zookeeper。顺着这个代码就可以找到ServiceConfig中的doExport方法，然后调用doExportUrls：
protected synchronized void doExport() {
if (unexported) {
throw new IllegalStateException("The service " + interfaceClass.getName() + " has already unexported!");
}
if (exported) {
return;
}
exported = true;
if (StringUtils.isEmpty(path)) {
path = interfaceName;
}
doExportUrls();
bootstrap.setReady(true);
}
private void doExportUrls() {
ServiceRepository repository = ApplicationModel.getServiceRepository();
ServiceDescriptor serviceDescriptor = repository.registerService(getInterfaceClass());
repository.registerProvider(
getUniqueServiceName(),
ref,
serviceDescriptor,
this,
serviceMetadata
);
List<URL> registryURLs = ConfigValidationUtils.loadRegistries(this, true);
int protocolConfigNum = protocols.size();
for (ProtocolConfig protocolConfig : protocols) {
String pathKey = URL.buildKey(getContextPath(protocolConfig)
.map(p -> p + "/" + path)
.orElse(path), group, version);
// In case user specified path, register service one more time to map it to path.
repository.registerService(pathKey, interfaceClass);
doExportUrlsFor1Protocol(protocolConfig, registryURLs, protocolConfigNum);
}
}
上面的代码中,loadRegistries方法就是在获取注册中心信息，比如demo中配置了new RegistryConfig("zookeeper://127.0.0.1:2181")那么则会返回：
registry://127.0.0.1:2181/org.apache.dubbo.registry.RegistryService?application=dubbo-demo-api-provider&dubbo=2.0.2&pid=8519®istry=zookeeper×tamp=1671071648130
可以看到URL中registry=zookeeper。我们就是因为配置的环境变量有误，导致配置变为RegistryConfig("127.0.0.1:2181")。此时dubbo会给一个默认的协议，导致URL中registry=dubbo ，具体代码如下：
if (!map.containsKey(PROTOCOL_KEY)) {
map.put(PROTOCOL_KEY, DUBBO_PROTOCOL);
}
url = URLBuilder.from(url)
.addParameter(REGISTRY_KEY, url.getProtocol())
.setProtocol(extractRegistryType(url))
.build();
registry://127.0.0.1:2181/org.apache.dubbo.registry.RegistryService?application=dubbo-demo-api-provider&dubbo=2.0.2&pid=9223®istry=dubbo×tamp=1671071848144
继续看暴露服务操作的方法doExportUrlsFor1Protocol，这个方法内容较多，这里只看主要的部分
private void doExportUrlsFor1Protocol(ProtocolConfig protocolConfig, List<URL> registryURLs, int protocolConfigNum) {
...省略部分代码
// export service,这里会获取到当前服务要暴露的地址和端口号,端口号默认是20880
String host = findConfigedHosts(protocolConfig, registryURLs, map);
Integer port = findConfigedPorts(protocolConfig, name, map, protocolConfigNum);
URL url = new URL(name, host, port, getContextPath(protocolConfig).map(p -> p + "/" + path).orElse(path), map);
// You can customize Configurator to append extra parameters
if (ExtensionLoader.getExtensionLoader(ConfiguratorFactory.class)
.hasExtension(url.getProtocol())) {
url = ExtensionLoader.getExtensionLoader(ConfiguratorFactory.class)
.getExtension(url.getProtocol()).getConfigurator(url).configure(url);
}
//上面获取到的服务url
//dubbo://10.109.0.14:20880/org.apache.dubbo.demo.GreetingService?anyhost=true&application=dubbo-demo-api-provider&bind.ip=10.109.0.14&bind.port=20880&default=true&deprecated=false&dubbo=2.0.2&dynamic=true&generic=false&interface=org.apache.dubbo.demo.GreetingService&methods=hello&pid=22385&release=&service.name=ServiceBean:/org.apache.dubbo.demo.GreetingService&side=provider×tamp=1671075564338
String scope = url.getParameter(SCOPE_KEY);
// don't export when none is configured
if (!SCOPE_NONE.equalsIgnoreCase(scope)) {
// export to local if the config is not remote (export to remote only when config is remote)
if (!SCOPE_REMOTE.equalsIgnoreCase(scope)) {
exportLocal(url);
}
// export to remote if the config is not local (export to local only when config is local)
if (!SCOPE_LOCAL.equalsIgnoreCase(scope)) {
if (CollectionUtils.isNotEmpty(registryURLs)) {
for (URL registryURL : registryURLs) {
if (SERVICE_REGISTRY_PROTOCOL.equals(registryURL.getProtocol())) {
url = url.addParameterIfAbsent(SERVICE_NAME_MAPPING_KEY, "true");
}
//if protocol is only injvm ,not register
if (LOCAL_PROTOCOL.equalsIgnoreCase(url.getProtocol())) {
continue;
}
url = url.addParameterIfAbsent(DYNAMIC_KEY, registryURL.getParameter(DYNAMIC_KEY));
URL monitorUrl = ConfigValidationUtils.loadMonitor(this, registryURL);
if (monitorUrl != null) {
url = url.addParameterAndEncoded(MONITOR_KEY, monitorUrl.toFullString());
}
if (logger.isInfoEnabled()) {
if (url.getParameter(REGISTER_KEY, true)) {
logger.info("Register dubbo service " + interfaceClass.getName() + " url " + url + " to registry " +
registryURL);
} else {
logger.info("Export dubbo service " + interfaceClass.getName() + " to url " + url);
}
}
// For providers, this is used to enable custom proxy to generate invoker
String proxy = url.getParameter(PROXY_KEY);
if (StringUtils.isNotEmpty(proxy)) {
registryURL = registryURL.addParameter(PROXY_KEY, proxy);
}
//代理
Invoker<?> invoker = PROXY_FACTORY.getInvoker(ref, (Class) interfaceClass,
registryURL.addParameterAndEncoded(EXPORT_KEY, url.toFullString()));
DelegateProviderMetaDataInvoker wrapperInvoker = new DelegateProviderMetaDataInvoker(invoker, this);
//服务暴露和注册
Exporter<?> exporter = PROTOCOL.export(wrapperInvoker);
exporters.add(exporter);
}
} else {
if (logger.isInfoEnabled()) {
logger.info("Export dubbo service " + interfaceClass.getName() + " to url " + url);
}
Invoker<?> invoker = PROXY_FACTORY.getInvoker(ref, (Class) interfaceClass, url);
DelegateProviderMetaDataInvoker wrapperInvoker = new DelegateProviderMetaDataInvoker(invoker, this);
Exporter<?> exporter = PROTOCOL.export(wrapperInvoker);
exporters.add(exporter);
}
MetadataUtils.publishServiceDefinition(url);
}
}
}
通过上面的代码可以了解到，如果同时注册多个服务，默认都会使用一个暴露的端口:20880，netty服务端启用会用到(这块不深入了)。
主要来看一下Exporter<?> exporter = PROTOCOL.export(wrapperInvoker);这一行。
首先来看一下这个PROTOCOL到底是哪个实现类，先来看看Protocol这个接口的定义：
private static final Protocol PROTOCOL = ExtensionLoader.getExtensionLoader(Protocol.class).getAdaptiveExtension();
@SPI("dubbo")
public interface Protocol {
/**
* Get default port when user doesn't config the port.
*
* @return default port
*/
int getDefaultPort();
/**
* Export service for remote invocation: <br>
* 1. Protocol should record request source address after receive a request:
* RpcContext.getContext().setRemoteAddress();<br>
* 2. export() must be idempotent, that is, there's no difference between invoking once and invoking twice when
* export the same URL<br>
* 3. Invoker instance is passed in by the framework, protocol needs not to care <br>
*
* @param <T> Service type
* @param invoker Service invoker
* @return exporter reference for exported service, useful for unexport the service later
* @throws RpcException thrown when error occurs during export the service, for example: port is occupied
*/
@Adaptive
<T> Exporter<T> export(Invoker<T> invoker) throws RpcException;
...//省略
}
这里有两个注解需要注意，一个是在类上面 @SPI("dubbo") ，一个是在方法上面@Adaptive。
这里是Dubbo自己的SPI实现机制。有兴趣的可以了解一下，在被Adaptive注解标注的方法，Dubbo会基于AdaptiveCompiler生成一个类：
public class Protocol$Adaptive implements org.apache.dubbo.rpc.Protocol {
public org.apache.dubbo.rpc.Invoker refer(java.lang.Class arg0, org.apache.dubbo.common.URL arg1) throws org.apache.dubbo.rpc.RpcException {
if (arg1 == null) throw new IllegalArgumentException("url == null");
org.apache.dubbo.common.URL url = arg1;
String extName = ( url.getProtocol() == null ? "dubbo" : url.getProtocol() );
if(extName == null) throw new IllegalStateException("Failed to get extension (org.apache.dubbo.rpc.Protocol) name from url (" + url.toString() + ") use keys([protocol])");
org.apache.dubbo.rpc.Protocol extension = (org.apache.dubbo.rpc.Protocol)ExtensionLoader.getExtensionLoader(org.apache.dubbo.rpc.Protocol.class).getExtension(extName);
return extension.refer(arg0, arg1);
}
public java.util.List getServers() {
throw new UnsupportedOperationException("The method public default java.util.List org.apache.dubbo.rpc.Protocol.getServers() of interface org.apache.dubbo.rpc.Protocol is not adaptive method!");
}
public org.apache.dubbo.rpc.Exporter export(org.apache.dubbo.rpc.Invoker arg0) throws org.apache.dubbo.rpc.RpcException {
if (arg0 == null) throw new IllegalArgumentException("org.apache.dubbo.rpc.Invoker argument == null");
if (arg0.getUrl() == null) throw new IllegalArgumentException("org.apache.dubbo.rpc.Invoker argument getUrl() == null");
org.apache.dubbo.common.URL url = arg0.getUrl();
String extName = ( url.getProtocol() == null ? "dubbo" : url.getProtocol() );
if(extName == null) throw new IllegalStateException("Failed to get extension (org.apache.dubbo.rpc.Protocol) name from url (" + url.toString() + ") use keys([protocol])");
org.apache.dubbo.rpc.Protocol extension = (org.apache.dubbo.rpc.Protocol)ExtensionLoader.getExtensionLoader(org.apache.dubbo.rpc.Protocol.class).getExtension(extName);
return extension.export(arg0);
}
public void destroy() {
throw new UnsupportedOperationException("The method public abstract void org.apache.dubbo.rpc.Protocol.destroy() of interface org.apache.dubbo.rpc.Protocol is not adaptive method!");
}
public int getDefaultPort() {
throw new UnsupportedOperationException("The method public abstract int org.apache.dubbo.rpc.Protocol.getDefaultPort() of interface org.apache.dubbo.rpc.Protocol is not adaptive method!");
}
}
通过上面生成的代码我们就可以看到export这个方法执行逻辑，通过url.getProtocol()的内容去获取一遍实现类：这里extName=url.getProtocol()，获取到的是registry。
org.apache.dubbo.rpc.Protocol extension = (org.apache.dubbo.rpc.Protocol)ExtensionLoader.getExtensionLoader(org.apache.dubbo.rpc.Protocol.class).getExtension(extName);
因此META-INF/dubbo/internal/org.apache.dubbo.rpc.Protocol可以根据当前文件内容找到具体的实现类RegistryProtocol。
然后就可以看看这个RegistryProtocol实现类中的具体实现了：
public <T> Exporter<T> export(final Invoker<T> originInvoker) throws RpcException {
URL registryUrl = getRegistryUrl(originInvoker);
// url to export locally
URL providerUrl = getProviderUrl(originInvoker);
// Subscribe the override data
// FIXME When the provider subscribes, it will affect the scene : a certain JVM exposes the service and call
// the same service. Because the subscribed is cached key with the name of the service, it causes the
// subscription information to cover.
final URL overrideSubscribeUrl = getSubscribedOverrideUrl(providerUrl);
final OverrideListener overrideSubscribeListener = new OverrideListener(overrideSubscribeUrl, originInvoker);
overrideListeners.put(overrideSubscribeUrl, overrideSubscribeListener);
providerUrl = overrideUrlWithConfig(providerUrl, overrideSubscribeListener);
// export invoker ，这里继续暴露服务
final ExporterChangeableWrapper<T> exporter = doLocalExport(originInvoker, providerUrl);
//下面是服务注册的逻辑
// url to registry
final Registry registry = getRegistry(originInvoker);
final URL registeredProviderUrl = getUrlToRegistry(providerUrl, registryUrl);
// decide if we need to delay publish
boolean register = providerUrl.getParameter(REGISTER_KEY, true);
if (register) {
registry.register(registeredProviderUrl);
}
// register stated url on provider model
registerStatedUrl(registryUrl, registeredProviderUrl, register);
exporter.setRegisterUrl(registeredProviderUrl);
exporter.setSubscribeUrl(overrideSubscribeUrl);
// Deprecated! Subscribe to override rules in 2.6.x or before.
registry.subscribe(overrideSubscribeUrl, overrideSubscribeListener);
notifyExport(exporter);
//Ensure that a new exporter instance is returned every time export
return new DestroyableExporter<>(exporter);
}
上面代码中的doLocalExport其实就是继续在进行服务暴露的流程，因为指定了@SPI("dubbo")，所以最终会走到DubboProtocol的export方法中了，在这个方法中维护了exporterMap，管理暴露的服务信息，以及通过openServer方法来基于netty启动服务端了等待消费者的请求了，然后从exporterMap中获取到Invoker并执行invokerMethod方法来完成调用(本篇就不继续扩展了)。
public <T> Exporter<T> export(Invoker<T> invoker) throws RpcException {
URL url = invoker.getUrl();
// export service.
String key = serviceKey(url);
DubboExporter<T> exporter = new DubboExporter<T>(invoker, key, exporterMap);
exporterMap.addExportMap(key, exporter);
//export an stub service for dispatching event
Boolean isStubSupportEvent = url.getParameter(STUB_EVENT_KEY, DEFAULT_STUB_EVENT);
Boolean isCallbackservice = url.getParameter(IS_CALLBACK_SERVICE, false);
if (isStubSupportEvent && !isCallbackservice) {
String stubServiceMethods = url.getParameter(STUB_EVENT_METHODS_KEY);
if (stubServiceMethods == null || stubServiceMethods.length() == 0) {
if (logger.isWarnEnabled()) {
logger.warn(new IllegalStateException("consumer [" + url.getParameter(INTERFACE_KEY) +
"], has set stubproxy support event ,but no stub methods founded."));
}
}
}
openServer(url);
optimizeSerialization(url);
return exporter;
}
在完成上面的步骤之后，就可以进行服务的注册了。demo中使用的zookeeper，那么最终会走到ZookeeperRegistry方法中：
public void doRegister(URL url) {
try {
zkClient.create(toUrlPath(url), url.getParameter(DYNAMIC_KEY, true));
} catch (Throwable e) {
throw new RpcException("Failed to register " + url + " to zookeeper " + getUrl() + ", cause: " + e.getMessage(), e);
}
}
总结
之前一直没有阅读过Dubbo相关的代码，这次也是为了验证一下配置错误导致的问题原因。版本是2.7.x。在这个过程中发现一个优秀的RPC框架每个模块都可以拿出来好好学习一下。后面有时间会在学习一下其SPI，服务调用，底层通信等相关的模块。
