TCP连接耗尽的优化方案

报错类型一

org.apache.http.conn.HttpHostConnectException: Connect to 0.0.0.x:80 failed: Cannot assign requested address (connect failed)

报错类型二

Connection reset by peer

完整报错信息

org.apache.http.conn.HttpHostConnectException: Connect to 192.168.132.20:80 [/29.16.132.20] failed: Cannot assign requested address (connect failed)
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374)
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
        at com.x.x.HttpUtil.postForRsp(HttpUtil.java:13)
        at com.x.x.HttpUtil.postForRsp(HttpUtil.java:7)

这两种情况大概率对应这服务器的tcp连接耗尽，具体情况具体分析。

问题原因

TCP连接的断开需要发送四个包，所以称为四次挥手。

第四次挥手 ([ACK], ACK = w + 1) 客户端接收到服务端的关闭请求，再发送ACK标记的确认包，进入TIME_WAIT状态，等待服务端可能请求重传的ACK包。服务端接收到ACK包后，关闭连接，进入CLOSED状态。客户端在等待固定时间(两个最大段生命周期)后，没有接收到服务的ACK包，认为服务器已关闭连接，自己也关闭连接，进入CLOSED状态。这里一点需要注意是：主动关闭连接的，才有 TIME_WAIT 状态。

可以看到，TIME_WAIT 是「主动关闭方」断开连接时的最后一个状态，该状态会持续 2MSL(Maximum Segment Lifetime) 时长，之后进入CLOSED 状态。

本次原因是发声了java发起http客户端调用，并发过高，导致close的资源跟不上，从而引起tcp连接数不够用。

解决方案

经过实践，优化方案有两种

方案一：调整linux内核参数

linux内核中存在两个参数：
    #表示开启重用。允许将TIME-WAIT sockets重新用于新的TCP连接，默认为0，表示关闭；
    net.ipv4.tcp_tw_reuse = 1
    #表示开启TCP连接中TIME-WAIT sockets的快速回收，默认为0，表示关闭。
    net.ipv4.tcp_tw_recycle = 1
在/etc/sysctl.conf文件中加入上述参数，然后执行/sbin/sysctl -p让参数生效。

    #当然也可以调整调低端口释放后的等待时间，默认为60s，修改为15~30s，原则是非必要不调整。
    sysctl -w net.ipv4.tcp_fin_timeout=30

方案二：将HttpClient池化调用

public class HttpUtil {
	public static String timeout = SystemConfig.getInstance().getProperties().getProperty("http.client.timeout","5000");
	public static RequestConfig config = RequestConfig.custom().setSocketTimeout(Integer.parseInt(timeout)).setConnectTimeout(Integer.parseInt(timeout)).setConnectionRequestTimeout(Integer.parseInt(timeout)).build();
	public static HttpClientBuilder clientBuilder = HttpClients.custom().setDefaultRequestConfig(config).setKeepAliveStrategy(new DefaultConnectionKeepAliveStrategy());
	public static PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
	public static CloseableHttpClient httpClient;

	private static CloseableHttpClient getHttpClient(){

		if(Objects.isNull(httpClient)){
			synchronized (HttpUtil.class) {
				if(Objects.isNull(httpClient)) {
					Properties props = SystemConfig.getInstance().getProperties();
					int maxConnTotal = Integer.parseInt(props.getProperty("http.client.maxConnTotal", "200"));
					int maxConnPerRoute = Integer.parseInt(props.getProperty("http.client.maxConnPerRoute", "20"));
					connectionManager.setMaxTotal(maxConnTotal);
					connectionManager.setDefaultMaxPerRoute(maxConnPerRoute);
					httpClient = clientBuilder.setConnectionManager(connectionManager).build();
				}
			}
		}
		return httpClient;
	}


	/**
	 *
	 * @param url 请求地址
	 * @param urlParams 请求参数
	 * @param body 数据流
	 * @return
	 * @throws Exception
	 */
	public static Rsp postForRsp(String url, Map<String, Object> urlParams, String body, String encoding,
								 String proxyHost, Integer proxyPort, String proxyUser, String proxyPassword) throws Exception {
		URIBuilder uriBuilder = new URIBuilder(url).setCharset(Charset.forName(encoding));

		if (urlParams != null) {
			for (Map.Entry<String, Object> e : urlParams.entrySet()) {
				uriBuilder.addParameter(e.getKey(), e.getValue().toString());
			}
		}
		URI uri = uriBuilder.build();
		getHttpClient();
		HttpPost post = new HttpPost(uri);
		StringEntity stringEntity = new StringEntity(body, ContentType.APPLICATION_JSON);
		post.addHeader("Authorization","Basic "+ AuthConfig.getInstance().getEsAuth());
		post.setEntity(stringEntity);
		try {
			long start = System.currentTimeMillis();
			CloseableHttpResponse response = httpClient.execute(post);
		    long end = System.currentTimeMillis();
			int rspCode = response.getStatusLine().getStatusCode();
			String rsp = getRspAsString(response, encoding);
			log.warn("[完成http请求] uri:{}, 耗时:{}ms, 返回信息:{}", uri, end - start, rsp);
			log.warn("连接池connectionManager:{}",connectionManager.getTotalStats().toString());
			return new Rsp(rspCode, rsp);
		}catch (Exception e){
			log.error("[http请求异常]", e);
			return new Rsp(408, "");
		}finally {
			connectionManager.closeExpiredConnections();
			post.releaseConnection();
		}
	}
}

博客地址

参考：

被微信问懵了：既然开启 net.ipv4.tcp_tw_reuse 可以复用连接，为什么内核不默认开启呢？

本文由博客一文多发平台 OpenWrite 发布！

TCP连接耗尽优化,Cannot assign requested address问题总结

报错类型一

报错类型二

问题原因

解决方案