tomcat系列（三） 连接器(上)

在上一章中，我们过了tomcat整体启动的流程。也留写了很多坑，在后续的章节中，作者会一步一步的填坑。今天的这一章，我们将看到tomcat连接池的各种细节。

什么是连接池

在讲连接器之前，我们需要知道tomcat的连接器是什么，为什么要出现连接器这样的一个东西。

我们知道tomcat的本质其实就是一个serverSocket,我们指定socket需要监听的ip和端口，在有请求到来的时候socket会捕捉到请求的具体信息 然后生成对应的servletRequest,servletResponse。给对应的servlet来调用service方法。

连接器就是做解析servletRequest的，讲解析请求和响应抽离出来。交给一个模块统一的进行处理，这样子tomcat就可以支持多种协议的解析，可以解析http的请求头，cookie，具体参数等。在这里只是做简单的赘述，关于连接器详细的描述可以看书（深入剖析tomcat），这里引入书中的原话

官网提供可选的连接器

• http 常规http请求，这里支持bio(8.0之前默认) nio（8.0之后默认） aio
• AJP 这个实际开发中没使用过，有兴趣的朋友可以自行了解

connector连接器

因为之前讲过tomcat启动流程，所以这里直接上connector的代码了

super.initInternal();

// Initialize adapter
adapter = new CoyoteAdapter(this);
protocolHandler.setAdapter(adapter);

// Make sure parseBodyMethodsSet has a default
if (null == parseBodyMethodsSet) {
    setParseBodyMethods(getParseBodyMethods());
}

if (protocolHandler.isAprRequired() && !AprLifecycleListener.isInstanceCreated()) {
    throw new LifecycleException(sm.getString("coyoteConnector.protocolHandlerNoAprListener",
            getProtocolHandlerClassName()));
}
if (protocolHandler.isAprRequired() && !AprLifecycleListener.isAprAvailable()) {
    throw new LifecycleException(sm.getString("coyoteConnector.protocolHandlerNoAprLibrary",
            getProtocolHandlerClassName()));
}
if (AprLifecycleListener.isAprAvailable() && AprLifecycleListener.getUseOpenSSL() &&
        protocolHandler instanceof AbstractHttp11JsseProtocol) {
    AbstractHttp11JsseProtocol<?> jsseProtocolHandler =
            (AbstractHttp11JsseProtocol<?>) protocolHandler;
    if (jsseProtocolHandler.isSSLEnabled() &&
            jsseProtocolHandler.getSslImplementationName() == null) {
        // OpenSSL is compatible with the JSSE configuration, so use it if APR is available
        jsseProtocolHandler.setSslImplementationName(OpenSSLImplementation.class.getName());
    }
}

try {
    protocolHandler.init();
} catch (Exception e) {
    throw new LifecycleException(
            sm.getString("coyoteConnector.protocolHandlerInitializationFailed"), e);
}

protocolHandler.init()

if (getLog().isInfoEnabled()) {
    getLog().info(sm.getString("abstractProtocolHandler.init", getName()));
}

if (oname == null) {
    // Component not pre-registered so register it
    oname = createObjectName();
    if (oname != null) {
        Registry.getRegistry(null, null).registerComponent(this, oname, null);
    }
}

if (this.domain != null) {
    ObjectName rgOname = new ObjectName(domain + ":type=GlobalRequestProcessor,name=" + getName());
    this.rgOname = rgOname;
    Registry.getRegistry(null, null).registerComponent(
            getHandler().getGlobal(), rgOname, null);
}

String endpointName = getName();
endpoint.setName(endpointName.substring(1, endpointName.length()-1));
endpoint.setDomain(domain);

endpoint.init();

endpoint.init()

这里用了模板方法，bind()方法留给子类实现，看到这里的继承关系，也验证了上面的一句话。tomcat8.0之后默认的是nio的连接方式，之前bio的实现已经移除了

nioEndPoint.bind()

if (!getUseInheritedChannel()) {
    serverSock = ServerSocketChannel.open();
    socketProperties.setProperties(serverSock.socket());
    InetSocketAddress addr = (getAddress()!=null?new InetSocketAddress(getAddress(),getPort()):new InetSocketAddress(getPort()));
    serverSock.socket().bind(addr,getAcceptCount());
} else {
    // Retrieve the channel provided by the OS
    Channel ic = System.inheritedChannel();
    if (ic instanceof ServerSocketChannel) {
        serverSock = (ServerSocketChannel) ic;
    }
    if (serverSock == null) {
        throw new IllegalArgumentException(sm.getString("endpoint.init.bind.inherited"));
    }
}
serverSock.configureBlocking(true); //mimic APR behavior

// Initialize thread count defaults for acceptor, poller
if (acceptorThreadCount == 0) {
    // FIXME: Doesn't seem to work that well with multiple accept threads
    acceptorThreadCount = 1;
}
if (pollerThreadCount <= 0) {
    //minimum one poller thread
    pollerThreadCount = 1;
}
setStopLatch(new CountDownLatch(pollerThreadCount));

// Initialize SSL if needed
initialiseSsl();

selectorPool.open();

在这一步我们终于看到了serverSocket的相关内容，serverSocket绑定了对应的端口号和IP，到了这一步，连假期的初始化就完成了。

connector.start()

// Validate settings before starting
if (getPort() < 0) {
    throw new LifecycleException(sm.getString(
            "coyoteConnector.invalidPort", Integer.valueOf(getPort())));
}

setState(LifecycleState.STARTING);

try {
    protocolHandler.start();
} catch (Exception e) {
    throw new LifecycleException(
            sm.getString("coyoteConnector.protocolHandlerStartFailed"), e);
}

protocolHandler.start()

if (getLog().isInfoEnabled()) {
    getLog().info(sm.getString("abstractProtocolHandler.start", getName()));
}

endpoint.start();

// Start timeout thread
asyncTimeout = new AsyncTimeout();
Thread timeoutThread = new Thread(asyncTimeout, getNameInternal() + "-AsyncTimeout");
int priority = endpoint.getThreadPriority();
if (priority < Thread.MIN_PRIORITY || priority > Thread.MAX_PRIORITY) {
    priority = Thread.NORM_PRIORITY;
}
timeoutThread.setPriority(priority);
timeoutThread.setDaemon(true);
timeoutThread.start();

这里有关键的两步

• endpoint的启动
• 创建了一个监听异步超时的守护线程

endpoint.start()

if (!running) {
    running = true;
    paused = false;

    processorCache = new SynchronizedStack<>(SynchronizedStack.DEFAULT_SIZE,
            socketProperties.getProcessorCache());
    eventCache = new SynchronizedStack<>(SynchronizedStack.DEFAULT_SIZE,
                    socketProperties.getEventCache());
    nioChannels = new SynchronizedStack<>(SynchronizedStack.DEFAULT_SIZE,
            socketProperties.getBufferPool());

    // Create worker collection
    if (getExecutor() == null) {
        createExecutor();
    }

    initializeConnectionLatch();

    // Start poller threads
    pollers = new Poller[getPollerThreadCount()];
    for (int i=0; i<pollers.length; i++) {
        pollers[i] = new Poller();
        Thread pollerThread = new Thread(pollers[i], getName() + "-ClientPoller-"+i);
        pollerThread.setPriority(threadPriority);
        pollerThread.setDaemon(true);
        pollerThread.start();
    }

    startAcceptorThreads();
}

这里做了一下的几步

• 设置对应的缓存栈
• 创建连接工作线程池
• 初始化连接门栓，用于限制请求的并发
• 启动nio的poller线程
• 启动acceptor线程，默认是一个

acceptor.run()

int errorDelay = 0;
long pauseStart = 0;

// Loop until we receive a shutdown command
while (running) {
    // 如果运行中停止，则会进行自选
     while (paused && running) {
         if (state != AcceptorState.PAUSED) {
             pauseStart = System.nanoTime();
             // Entered pause state
             state = AcceptorState.PAUSED;
         }
         if ((System.nanoTime() - pauseStart) > 1_000_000) {
             // Paused for more than 1ms
             try {
                 if ((System.nanoTime() - pauseStart) > 10_000_000) {
                     Thread.sleep(10);
                 } else {
                     Thread.sleep(1);
                 }
             } catch (InterruptedException e) {
                 // Ignore
             }
         }
    }

    // 停止则推出循环
    if (!running) {
        break;
    }
    state = AcceptorState.RUNNING;

    try {
        // 如果请求并发数到达最大限制，则会进行限流。底层是limitLatch实现了AQS
        countUpOrAwaitConnection();

        SocketChannel socket = null;
        try {
            // serverSocket核心代码
            socket = serverSock.accept();
        } catch (IOException ioe) {
            // We didn't get a socket
            countDownConnection();
            if (running) {
                // Introduce delay if necessary
                errorDelay = handleExceptionWithDelay(errorDelay);
                // re-throw
                throw ioe;
            } else {
                break;
            }
        }
        // Successful accept, reset the error delay
        errorDelay = 0;

        // Configure the socket
        if (running && !paused) {
            // 将socket中获取到的请求以事件的形式传递给poller
            if (!setSocketOptions(socket)) {
                closeSocket(socket);
            }
        } else {
            closeSocket(socket);
        }
    } catch (Throwable t) {
        ExceptionUtils.handleThrowable(t);
        log.error(sm.getString("endpoint.accept.fail"), t);
    }
}
state = AcceptorState.ENDED;

这里做了以下几个关键步骤

• 检查连接数是否到达最大限制数，没有则+1。到达了则等待
• socket.accept
• 将获取到的socket连接以事件的方式通知poller

setSocketOptions()

// Process the connection
try {
    //disable blocking, APR style, we are gonna be polling it
    socket.configureBlocking(false);
    Socket sock = socket.socket();
    socketProperties.setProperties(sock);

    NioChannel channel = nioChannels.pop();
    if (channel == null) {
        SocketBufferHandler bufhandler = new SocketBufferHandler(
                socketProperties.getAppReadBufSize(),
                socketProperties.getAppWriteBufSize(),
                socketProperties.getDirectBuffer());
        if (isSSLEnabled()) {
            channel = new SecureNioChannel(socket, bufhandler, selectorPool, this);
        } else {
            channel = new NioChannel(socket, bufhandler);
        }
    } else {
        channel.setIOChannel(socket);
        channel.reset();
    }
    getPoller0().register(channel);
} catch (Throwable t) {
    ExceptionUtils.handleThrowable(t);
    try {
        log.error("",t);
    } catch (Throwable tt) {
        ExceptionUtils.handleThrowable(tt);
    }
    // Tell to close the socket
    return false;
}
return true;

这个方法主要三步操作

• 从channel栈中弹出一个可用的channel，如果没有则新建一个
• getPoller0获取一个poller进行注册
• 如果注册成功则返回true,否则返回false。然后再上一层方法中关闭socket连接

getPoller0()

int idx = Math.abs(pollerRotater.incrementAndGet()) % pollers.length;
return pollers[idx];

随机获取一个poller线程进行工作

poller.register()

public void register(final NioChannel socket) {
    socket.setPoller(this);
    NioSocketWrapper ka = new NioSocketWrapper(socket, NioEndpoint.this);
    socket.setSocketWrapper(ka);
    ka.setPoller(this);
    ka.setReadTimeout(getSocketProperties().getSoTimeout());
    ka.setWriteTimeout(getSocketProperties().getSoTimeout());
    ka.setKeepAliveLeft(NioEndpoint.this.getMaxKeepAliveRequests());
    ka.setReadTimeout(getConnectionTimeout());
    ka.setWriteTimeout(getConnectionTimeout());
    PollerEvent r = eventCache.pop();
    ka.interestOps(SelectionKey.OP_READ);//this is what OP_REGISTER turns into.
    if ( r==null) {
        r = new PollerEvent(socket,ka,OP_REGISTER);
    } else {
        r.reset(socket,ka,OP_REGISTER);
    }
    addEvent(r);
   
    private void addEvent(PollerEvent event) {
        events.offer(event);
        if (wakeupCounter.incrementAndGet() == 0) {
            selector.wakeup();
        }
    }

poller.run

// Loop until destroy() is called
while (true) {

    boolean hasEvents = false;

    try {
        if (!close) {
            // 执行pollerEvent的run方法
            hasEvents = events();
            if (wakeupCounter.getAndSet(-1) > 0) {
                // If we are here, means we have other stuff to do
                // Do a non blocking select
                keyCount = selector.selectNow();
            } else {
                keyCount = selector.select(selectorTimeout);
            }
            wakeupCounter.set(0);
        }
        if (close) {
            events();
            timeout(0, false);
            try {
                selector.close();
            } catch (IOException ioe) {
                log.error(sm.getString("endpoint.nio.selectorCloseFail"), ioe);
            }
            break;
        }
    } catch (Throwable x) {
        ExceptionUtils.handleThrowable(x);
        log.error("",x);
        continue;
    }
    // Either we timed out or we woke up, process events first
    if (keyCount == 0) {
        hasEvents = (hasEvents | events());
    }

    // 获取当前选择器中所有注册的“选择键(已就绪的监听事件)
    Iterator<SelectionKey> iterator =
        keyCount > 0 ? selector.selectedKeys().iterator() : null;
    // 对已经准备好的key进行处理
    while (iterator != null && iterator.hasNext()) {
        SelectionKey sk = iterator.next();
        iterator.remove();
        NioSocketWrapper socketWrapper = (NioSocketWrapper) sk.attachment();
        // Attachment may be null if another thread has called
        // cancelledKey()
        if (socketWrapper != null) {
            // 真正处理请求
            processKey(sk, socketWrapper);
        }
    }

    // Process timeouts
    timeout(keyCount,hasEvents);
}

getStopLatch().countDown();

poller.events()

public boolean events() {
    boolean result = false;

    PollerEvent pe = null;
    for (int i = 0, size = events.size(); i < size && (pe = events.poll()) != null; i++ ) {
        result = true;
        try {
            pe.run();
            pe.reset();
            if (running && !paused) {
                eventCache.push(pe);
            }
        } catch ( Throwable x ) {
            log.error("",x);
        }
    }

    return result;
}

这里有几个关键的步骤

• 如果events队列里面有元素会先执行完event然后再进行操作
• 对select返回的SelectionKey进行处理，由于在PollerEvent中注册通道时带上了NioSocketWrapper附件，因此这里可以用SelectionKey的attachment方法得到，接着调用processKey去处理已连接套接字通道。

processKey()

try {
    if (close) {
        cancelledKey(sk);
    } else if ( sk.isValid() && attachment != null ) {
        if (sk.isReadable() || sk.isWritable() ) {
            if ( attachment.getSendfileData() != null ) {
                processSendfile(sk,attachment, false);
            } else {
                unreg(sk, attachment, sk.readyOps());
                boolean closeSocket = false;
                // Read goes before write
                if (sk.isReadable()) {
                    if (!processSocket(attachment, SocketEvent.OPEN_READ, true)) {
                        closeSocket = true;
                    }
                }
                if (!closeSocket && sk.isWritable()) {
                    if (!processSocket(attachment, SocketEvent.OPEN_WRITE, true)) {
                        closeSocket = true;
                    }
                }
                if (closeSocket) {
                    cancelledKey(sk);
                }
            }
        }
    } else {
        // Invalid key
        cancelledKey(sk);
    }
} catch (CancelledKeyException ckx) {
    cancelledKey(sk);
} catch (Throwable t) {
    ExceptionUtils.handleThrowable(t);
    log.error("",t);
}

这里重要的方法就processSocket(),我们可以看到，他会根据是读事件或者写事件进行区分

processSocket()

try {
    if (socketWrapper == null) {
        return false;
    }
    SocketProcessorBase<S> sc = processorCache.pop();
    if (sc == null) {
        sc = createSocketProcessor(socketWrapper, event);
    } else {
        sc.reset(socketWrapper, event);
    }
    Executor executor = getExecutor();
    if (dispatch && executor != null) {
        executor.execute(sc);
    } else {
        sc.run();
    }
} catch (RejectedExecutionException ree) {
    getLog().warn(sm.getString("endpoint.executor.fail", socketWrapper) , ree);
    return false;
} catch (Throwable t) {
    ExceptionUtils.handleThrowable(t);
    // This means we got an OOM or similar creating a thread, or that
    // the pool and its queue are full
    getLog().error(sm.getString("endpoint.process.fail"), t);
    return false;
}
return true;