导言:
项目中使用了flink+mybatis+druid来进行持久化,但是目前的使用方法很粗糙:每个算子实例都有一个sqlSessionFactory,实际上造成了资源的很大浪费, 初始化SqlSessionFactory 的方法类似于SqlSessionFactory initSqlSessionFactory(Set mapperXmls, DataSource dataSource). 实际上这样子flink的每个算子实例(每一个并行度算子实例)都有一个连接池,处理数据的时候都是从自己的连接池中取连接,并没有共享这个连接池创建的连接,利用率很低.
于是决定趁此机会好好了解下连接池druid,进一步思考下怎么优化目前在flink上的使用.
一、Druid + H2数据库使用例子
本文将结合H2数据库 ,先简单介绍用Druid连接池如何实现数据库的简单操作。test1()是利用druid连接池来insert到数据库中,test2()是用原生的jdbc来insert。可以看到其实大体使用上没有差别很多。主要都是几步:1)初始化密码连接等 2)获取connection 3)利用connection获取preparestatement。 4)执行statement 5)statement关闭 6)connection关闭
public class TestDruidDataSourceH2DB {
public static String insertSql = "INSERT INTO TEST VALUES(\'%s\', 'World')";
public static void test1() throws SQLException {
DruidDataSource dataSource = new DruidDataSource();
String pwd = "";
String user = "sa";
String url = "jdbc:h2:tcp://localhost/~/test";
String driverName = "org.h2.Driver";
//配置数据库基本四项,创建连接,初始化连接池容器
dataSource.setDriverClassName(driverName);
dataSource.setUrl(url);
dataSource.setUsername(user);
dataSource.setPassword(pwd);
// 指定初始化个数
dataSource.setInitialSize(5);
// 指定最大个数
dataSource.setMaxActive(100);
// 指定低峰期个数
dataSource.setMinIdle(20);
// 指定等待时间
dataSource.setMaxWait(3000);
// 从连接池中获得连接对象
Connection connection = dataSource.getConnection();
System.out.println(connection);
insert(connection);
// 归还到连接池
connection.close(); // 此对象是druid提供的子类,实现了对close方法的增强,不再是销毁对象,而是归还到连接池
}
public static void insert(Connection connection) throws SQLException {
String id = "id";
String id_numeric = String.valueOf(System.currentTimeMillis());
System.out.println("description :" + id_numeric);
String provinceId = "provinceId";
String cityName = "cityName";
String insert = String.format(insertSql, id_numeric);
// 获取statement
PreparedStatement preparedStatement = connection.prepareStatement(insert);
preparedStatement.execute();
preparedStatement.close();
}
public static void test2() {
Connection conn = null;
Statement stmt = null;
String pwd = "";
String user = "sa";
String url = "jdbc:h2:tcp://localhost/~/test";
String driverName = "org.h2.Driver";
try {
Class.forName(driverName);
conn = DriverManager.getConnection(url, user, pwd);
insert(conn);
conn.close();
} catch (Exception exception) {
exception.printStackTrace();
}
}
public static void main(String[] args) throws SQLException {
test1();
}
}
二、DruidDataSource的主要方法和属性
Druid核心的逻辑在类DruidDataSource中,从上面的两个例子也可以看出来,大部分代码都是相同的,就是从获取connection的方式那里开始不一样而已。Druid获取连接的方法:
Connection connection = dataSource.getConnection();
下图是DruidDataSource主要类继承结构及成员变量和方法。
| 类名或属性名 | 描述 |
|---|---|
| ExceptionSorter | 用于判断SQLException对象是否致命异常 |
| ValidConnectionChecker | 用于校验指定连接对象是否有效 |
| CreateConnectionThread | DruidDataSource的内部类,用于异步创建连接对象 |
| notEmpty | 调用notEmpty.await()时,当前线程进入等待;当连接创建完成或者回收了连接,会调用notEmpty.signal()时,将等待线程唤醒; |
| empty | 调用empty.await()时,CreateConnectionThread进入等待;调用empty.signal()时,CreateConnectionThread被唤醒,并进入创建连接; |
| DestroyConnectionThread | DruidDataSource的内部类,用于异步检验连接对象,包括校验空闲连接的phyTimeoutMillis、minEvictableIdleTimeMillis,以及校验借出连接的removeAbandonedTimeoutMillis |
| LogStatsThread | DruidDataSource的内部类,用于异步记录统计信息 |
| connections | 用于存放所有连接对象 |
| evictConnections | 用于存放需要丢弃的连接对象 |
| keepAliveConnections | 用于存放需要keepAlive的连接对象 |
| activeConnections | 用于存放需要进行removeAbandoned的连接对象 |
| poolingCount | 空闲连接对象的数量 |
| activeCount | 借出连接对象的数量 |
另外,DruidDatasource类总体的工作流程大致如下:
三、Druid获取连接,执行sql流程
下面会针对上面用到的测试例子代讲解下DruidDataSource的大概运行原理
public static void test1() throws SQLException {
DruidDataSource dataSource = new DruidDataSource();
String pwd = "";
String user = "sa";
String url = "jdbc:h2:tcp://localhost/~/test";
String driverName = "org.h2.Driver";
//配置数据库基本四项,创建连接,初始化连接池容器
dataSource.setDriverClassName(driverName);
dataSource.setUrl(url);
dataSource.setUsername(user);
dataSource.setPassword(pwd);
// 指定初始化个数
//dataSource.setInitialSize(0);
// 指定最大个数
dataSource.setMaxActive(100);
// 指定低峰期个数
dataSource.setMinIdle(20);
// 指定等待时间
dataSource.setMaxWait(3000);
//设定初始化连接数量
dataSource.setInitialSize(2);
// 从连接池中获得连接对象
Connection connection = dataSource.getConnection();
System.out.println(connection);
insert(connection);
// 归还到连接池
connection.close(); // 此对象是druid提供的子类,实现了对close方法的增强,不再是销毁对象,而是归还到连接池
}
总的来说只是初始化变量多一点。后面会补充一些重要参数的作用说明。
3.1 DruidDataSource的构造函数
// 客户端新建一个DruidDataSource实例
DruidDataSource dataSource = new DruidDataSource();
//DruidDataSource构造函数
public DruidDataSource(){
this(false);
}
public DruidDataSource(boolean fairLock){
super(fairLock);
// 从System配置中初始化一些变量
configFromPropety(System.getProperties());
}
super(fairLock)调用的是父类的构造方法
public DruidAbstractDataSource(boolean lockFair){
//初始化
lock = new ReentrantLock(lockFair);
//初始化两个Condition类。用于同步CreateConnectionThread 和 DestroyConnectionThread
notEmpty = lock.newCondition();
empty = lock.newCondition();
}
3.2 从连接池中获取连接对象
// 从连接池中获得连接对象
Connection connection = dataSource.getConnection();
@Override
public DruidPooledConnection getConnection() throws SQLException {
return getConnection(maxWait);
}
@Override
public DruidPooledConnection getConnection() throws SQLException {
return getConnection(maxWait);
}
public DruidPooledConnection getConnection(long maxWaitMillis) throws SQLException {
init();
if (filters.size() > 0) {
FilterChainImpl filterChain = new FilterChainImpl(this);
return filterChain.dataSource_connect(this, maxWaitMillis);
} else {
return getConnectionDirect(maxWaitMillis);
}
}
3.2.1 init() 方法
public void init() throws SQLException {
if (inited) {
return;
}
// 注册驱动driver
DruidDriver.getInstance();
// 获取锁
final ReentrantLock lock = this.lock;
try {
lock.lockInterruptibly();
} catch (InterruptedException e) {
throw new SQLException("interrupt", e);
}
boolean init = false;
try {
if (inited) {
return;
}
initStackTrace = Utils.toString(Thread.currentThread().getStackTrace());
// 这里使用了AtomicLongFieldUpdater来进行原子更新,保证了写的安全和读的高效
this.id = DruidDriver.createDataSourceId();
if (this.id > 1) {
long delta = (this.id - 1) * 100000;
this.connectionIdSeedUpdater.addAndGet(this, delta);
this.statementIdSeedUpdater.addAndGet(this, delta);
this.resultSetIdSeedUpdater.addAndGet(this, delta);
this.transactionIdSeedUpdater.addAndGet(this, delta);
}
// 根据url前缀,确定dbType
if (this.jdbcUrl != null) {
this.jdbcUrl = this.jdbcUrl.trim();
// 针对druid自定义的一种url格式,进行解析
// jdbc:wrap-jdbc:开头,可设置driver、name、jmx等
initFromWrapDriverUrl();
}
// 初始化过滤器。后面会讲解如何使用filter进行????
for (Filter filter : filters) {
filter.init(this);
}
if (this.dbTypeName == null || this.dbTypeName.length() == 0) {
this.dbTypeName = JdbcUtils.getDbType(jdbcUrl, null);
}
DbType dbType = DbType.of(this.dbTypeName);
if (dbType == DbType.mysql
|| dbType == DbType.mariadb
|| dbType == DbType.oceanbase
|| dbType == DbType.ads) {
boolean cacheServerConfigurationSet = false;
if (this.connectProperties.containsKey("cacheServerConfiguration")) {
cacheServerConfigurationSet = true;
} else if (this.jdbcUrl.indexOf("cacheServerConfiguration") != -1) {
cacheServerConfigurationSet = true;
}
if (cacheServerConfigurationSet) {
this.connectProperties.put("cacheServerConfiguration", "true");
}
}
//参数校验
if (maxActive <= 0) {
throw new IllegalArgumentException("illegal maxActive " + maxActive);
}
if (maxActive < minIdle) {
throw new IllegalArgumentException("illegal maxActive " + maxActive);
}
if (getInitialSize() > maxActive) {
throw new IllegalArgumentException("illegal initialSize " + this.initialSize + ", maxActive " + maxActive);
}
if (timeBetweenLogStatsMillis > 0 && useGlobalDataSourceStat) {
throw new IllegalArgumentException("timeBetweenLogStatsMillis not support useGlobalDataSourceStat=true");
}
if (maxEvictableIdleTimeMillis < minEvictableIdleTimeMillis) {
throw new SQLException("maxEvictableIdleTimeMillis must be grater than minEvictableIdleTimeMillis");
}
if (keepAlive && keepAliveBetweenTimeMillis <= timeBetweenEvictionRunsMillis) {
throw new SQLException("keepAliveBetweenTimeMillis must be grater than timeBetweenEvictionRunsMillis");
}
//参数校验
if (this.driverClass != null) {
this.driverClass = driverClass.trim();
}
//采用SPI机制加载过滤器,这部分过滤器除了放入filters,还会放入autoFilters
initFromSPIServiceLoader();
resolveDriver();
initCheck();
initExceptionSorter();
initValidConnectionChecker();
validationQueryCheck();
if (isUseGlobalDataSourceStat()) {
dataSourceStat = JdbcDataSourceStat.getGlobal();
if (dataSourceStat == null) {
dataSourceStat = new JdbcDataSourceStat("Global", "Global", this.dbTypeName);
JdbcDataSourceStat.setGlobal(dataSourceStat);
}
if (dataSourceStat.getDbType() == null) {
dataSourceStat.setDbType(this.dbTypeName);
}
} else {
// 用于监控sql。但是有内存泄漏的风险 参考:https://segmentfault.com/a/1190000021636834
dataSourceStat = new JdbcDataSourceStat(this.name, this.jdbcUrl, this.dbTypeName, this.connectProperties);
}
dataSourceStat.setResetStatEnable(this.resetStatEnable);
connections = new DruidConnectionHolder[maxActive];
evictConnections = new DruidConnectionHolder[maxActive];
keepAliveConnections = new DruidConnectionHolder[maxActive];
SQLException connectError = null;
if (createScheduler != null && asyncInit) {
for (int i = 0; i < initialSize; ++i) {
submitCreateTask(true);
}
} else if (!asyncInit) {
// init connections
while (poolingCount < initialSize) {
try {
PhysicalConnectionInfo pyConnectInfo = createPhysicalConnection();
DruidConnectionHolder holder = new DruidConnectionHolder(this, pyConnectInfo);
connections[poolingCount++] = holder;
} catch (SQLException ex) {
LOG.error("init datasource error, url: " + this.getUrl(), ex);
if (initExceptionThrow) {
connectError = ex;
break;
} else {
Thread.sleep(3000);
}
}
}
if (poolingCount > 0) {
poolingPeak = poolingCount;
poolingPeakTime = System.currentTimeMillis();
}
}
createAndLogThread();
createAndStartCreatorThread();
createAndStartDestroyThread();
initedLatch.await();
init = true;
initedTime = new Date();
registerMbean();
if (connectError != null && poolingCount == 0) {
throw connectError;
}
if (keepAlive) {
// async fill to minIdle
if (createScheduler != null) {
for (int i = 0; i < minIdle; ++i) {
submitCreateTask(true);
}
} else {
this.emptySignal();
}
}
} catch (SQLException e) {
LOG.error("{dataSource-" + this.getID() + "} init error", e);
throw e;
} catch (InterruptedException e) {
throw new SQLException(e.getMessage(), e);
} catch (RuntimeException e){
LOG.error("{dataSource-" + this.getID() + "} init error", e);
throw e;
} catch (Error e){
LOG.error("{dataSource-" + this.getID() + "} init error", e);
throw e;
} finally {
inited = true;
lock.unlock();
if (init && LOG.isInfoEnabled()) {
String msg = "{dataSource-" + this.getID();
if (this.name != null && !this.name.isEmpty()) {
msg += ",";
msg += this.name;
}
msg += "} inited";
LOG.info(msg);
}
}
}
初始化Filter
还可以利用SPI机制来加载filter。注意到还需要有AutoLoad注解才行,另外还需要配置load.spifilter.skip=false
private void initFromSPIServiceLoader() {
if (loadSpifilterSkip) {
return;
}
if (autoFilters == null) {
List<Filter> filters = new ArrayList<Filter>();
ServiceLoader<Filter> autoFilterLoader = ServiceLoader.load(Filter.class);
for (Filter filter : autoFilterLoader) {
AutoLoad autoLoad = filter.getClass().getAnnotation(AutoLoad.class);
if (autoLoad != null && autoLoad.value()) {
filters.add(filter);
}
}
autoFilters = filters;
}
for (Filter filter : autoFilters) {
if (LOG.isInfoEnabled()) {
LOG.info("load filter from spi :" + filter.getClass().getName());
}
addFilter(filter);
}
}
用了反射的方法加载Filter
private S nextService() {
if (!hasNextService())
throw new NoSuchElementException();
String cn = nextName;
nextName = null;
Class<?> c = null;
try {
// 加载类
c = Class.forName(cn, false, loader);
} catch (ClassNotFoundException x) {
fail(service,
"Provider " + cn + " not found");
}
if (!service.isAssignableFrom(c)) {
fail(service,
"Provider " + cn + " not a subtype");
}
try {
// 实例化
S p = service.cast(c.newInstance());
providers.put(cn, p);
return p;
} catch (Throwable x) {
fail(service,
"Provider " + cn + " could not be instantiated",
x);
}
throw new Error(); // This cannot happen
}
初始化Driver
利用反射加载配置的DriverClass,并且调用DriverManager.registerDriver(driver)注册。
protected void resolveDriver() throws SQLException {
if (this.driver == null) {
if (this.driverClass == null || this.driverClass.isEmpty()) {
this.driverClass = JdbcUtils.getDriverClassName(this.jdbcUrl);
}
if (MockDriver.class.getName().equals(driverClass)) {
driver = MockDriver.instance;
} else if ("com.alibaba.druid.support.clickhouse.BalancedClickhouseDriver".equals(driverClass)) {
Properties info = new Properties();
info.put("user", username);
info.put("password", password);
info.putAll(connectProperties);
driver = new BalancedClickhouseDriver(jdbcUrl, info);
} else {
if (jdbcUrl == null && (driverClass == null || driverClass.length() == 0)) {
throw new SQLException("url not set");
}
driver = JdbcUtils.createDriver(driverClassLoader, driverClass);
}
} else {
if (this.driverClass == null) {
this.driverClass = driver.getClass().getName();
}
}
}
public static Driver createDriver(ClassLoader classLoader, String driverClassName) throws SQLException {
Class<?> clazz = null;
if (classLoader != null) {
try {
clazz = classLoader.loadClass(driverClassName);
} catch (ClassNotFoundException e) {
// skip
}
}
if (clazz == null) {
try {
ClassLoader contextLoader = Thread.currentThread().getContextClassLoader();
if (contextLoader != null) {
clazz = contextLoader.loadClass(driverClassName);
}
} catch (ClassNotFoundException e) {
// skip
}
}
if (clazz == null) {
try {
// 加载配置的Driver Class
clazz = Class.forName(driverClassName);
} catch (ClassNotFoundException e) {
throw new SQLException(e.getMessage(), e);
}
}
try {
// 实例化
return (Driver) clazz.newInstance();
} catch (IllegalAccessException e) {
throw new SQLException(e.getMessage(), e);
} catch (InstantiationException e) {
throw new SQLException(e.getMessage(), e);
}
}
校验参数
// 校验maxActive、minIdle、initialSize、timeBetweenLogStatsMillis、useGlobalDataSourceStat、maxEvictableIdleTimeMillis、minEvictableIdleTimeMillis等配置是否合法
// ·······
// 针对oracle和DB2,需要校验validationQuery
initCheck();
// 当开启了testOnBorrow/testOnReturn/testWhileIdle,判断是否设置了validationQuery,没有的话会打印错误信息
validationQueryCheck();
初始化ExceptionSorter、ValidConnectionChecker、JdbcDataSourceStat
根据不同的数据驱动实例化不同的ExceptionSorter。
ExceptionSorter接口如下,从中可以看出这个类是用来判断异常是不是致命"Fatal"异常
public interface ExceptionSorter {
/**
* Returns true or false whether or not the exception is fatal.
*
* @param e the exception
* @return true or false if the exception is fatal.
*/
boolean isExceptionFatal(SQLException e);
void configFromProperties(Properties properties);
}
private void initExceptionSorter() {
if (exceptionSorter instanceof NullExceptionSorter) {
if (driver instanceof MockDriver) {
return;
}
} else if (this.exceptionSorter != null) {
return;
}
for (Class<?> driverClass = driver.getClass();;) {
String realDriverClassName = driverClass.getName();
if (realDriverClassName.equals(JdbcConstants.MYSQL_DRIVER) //
|| realDriverClassName.equals(JdbcConstants.MYSQL_DRIVER_6)) {
this.exceptionSorter = new MySqlExceptionSorter();
this.isMySql = true;
} else if (realDriverClassName.equals(JdbcConstants.ORACLE_DRIVER)
|| realDriverClassName.equals(JdbcConstants.ORACLE_DRIVER2)) {
this.exceptionSorter = new OracleExceptionSorter();
} else if (realDriverClassName.equals(JdbcConstants.OCEANBASE_DRIVER)) { // 写一个真实的 TestCase
if (JdbcUtils.OCEANBASE_ORACLE.name().equalsIgnoreCase(dbTypeName)) {
this.exceptionSorter = new OceanBaseOracleExceptionSorter();
} else {
this.exceptionSorter = new MySqlExceptionSorter();
}
} else if (realDriverClassName.equals("com.informix.jdbc.IfxDriver")) {
this.exceptionSorter = new InformixExceptionSorter();
} else if (realDriverClassName.equals("com.sybase.jdbc2.jdbc.SybDriver")) {
this.exceptionSorter = new SybaseExceptionSorter();
} else if (realDriverClassName.equals(JdbcConstants.POSTGRESQL_DRIVER)
|| realDriverClassName.equals(JdbcConstants.ENTERPRISEDB_DRIVER)
|| realDriverClassName.equals(JdbcConstants.POLARDB_DRIVER)) {
this.exceptionSorter = new PGExceptionSorter();
} else if (realDriverClassName.equals("com.alibaba.druid.mock.MockDriver")) {
this.exceptionSorter = new MockExceptionSorter();
} else if (realDriverClassName.contains("DB2")) {
this.exceptionSorter = new DB2ExceptionSorter();
} else {
Class<?> superClass = driverClass.getSuperclass();
if (superClass != null && superClass != Object.class) {
driverClass = superClass;
continue;
}
}
break;
}
}
// 根据不同的driver class 用不同的连接检查器。比如,当testOnBorrow设置为true的时候就会调用ValidConnectionChecker去检查连接是否有效
private void initValidConnectionChecker() {
if (this.validConnectionChecker != null) {
return;
}
String realDriverClassName = driver.getClass().getName();
if (JdbcUtils.isMySqlDriver(realDriverClassName)) {
this.validConnectionChecker = new MySqlValidConnectionChecker();
} else if (realDriverClassName.equals(JdbcConstants.ORACLE_DRIVER)
|| realDriverClassName.equals(JdbcConstants.ORACLE_DRIVER2)) {
this.validConnectionChecker = new OracleValidConnectionChecker();
} else if (realDriverClassName.equals(JdbcConstants.SQL_SERVER_DRIVER)
|| realDriverClassName.equals(JdbcConstants.SQL_SERVER_DRIVER_SQLJDBC4)
|| realDriverClassName.equals(JdbcConstants.SQL_SERVER_DRIVER_JTDS)) {
this.validConnectionChecker = new MSSQLValidConnectionChecker();
} else if (realDriverClassName.equals(JdbcConstants.POSTGRESQL_DRIVER)
|| realDriverClassName.equals(JdbcConstants.ENTERPRISEDB_DRIVER)
|| realDriverClassName.equals(JdbcConstants.POLARDB_DRIVER)) {
this.validConnectionChecker = new PGValidConnectionChecker();
}
}
private void validationQueryCheck() {
if (!(testOnBorrow || testOnReturn || testWhileIdle)) {
return;
}
if (this.validConnectionChecker != null) {
return;
}
if (this.validationQuery != null && this.validationQuery.length() > 0) {
return;
}
String errorMessage = "";
if (testOnBorrow) {
errorMessage += "testOnBorrow is true, ";
}
if (testOnReturn) {
errorMessage += "testOnReturn is true, ";
}
if (testWhileIdle) {
errorMessage += "testWhileIdle is true, ";
}
LOG.error(errorMessage + "validationQuery not set");
}
初始化3个连接对象封装类
connections = new DruidConnectionHolder[maxActive];
evictConnections = new DruidConnectionHolder[maxActive];
keepAliveConnections = new DruidConnectionHolder[maxActive];
DruidConnectionHolder类构造不复杂,主要属性和方法有下面这些:
初始化连接,数量为参数initialSize
创建的方式有两种,一种是同步一种是异步
// init connections
while (poolingCount < initialSize) {
try {
// 创建连接。用DruidConnectionHolder来封装
PhysicalConnectionInfo pyConnectInfo = createPhysicalConnection();
DruidConnectionHolder holder = new DruidConnectionHolder(this, pyConnectInfo);
connections[poolingCount++] = holder;
} catch (SQLException ex) {
LOG.error("init datasource error, url: " + this.getUrl(), ex);
if (initExceptionThrow) {
connectError = ex;
break;
} else {
Thread.sleep(3000);
}
}
}
创建连接的过程
前面大部分代码可以暂时不用关注,主要是createPhysicalConnection、initPhysicalConnection方法
public PhysicalConnectionInfo createPhysicalConnection() throws SQLException {
String url = this.getUrl();
Properties connectProperties = getConnectProperties();
String user;
if (getUserCallback() != null) {
user = getUserCallback().getName();
} else {
user = getUsername();
}
String password = getPassword();
PasswordCallback passwordCallback = getPasswordCallback();
if (passwordCallback != null) {
if (passwordCallback instanceof DruidPasswordCallback) {
DruidPasswordCallback druidPasswordCallback = (DruidPasswordCallback) passwordCallback;
druidPasswordCallback.setUrl(url);
druidPasswordCallback.setProperties(connectProperties);
}
char[] chars = passwordCallback.getPassword();
if (chars != null) {
password = new String(chars);
}
}
Properties physicalConnectProperties = new Properties();
if (connectProperties != null) {
physicalConnectProperties.putAll(connectProperties);
}
if (user != null && user.length() != 0) {
physicalConnectProperties.put("user", user);
}
if (password != null && password.length() != 0) {
physicalConnectProperties.put("password", password);
}
Connection conn = null;
long connectStartNanos = System.nanoTime();
long connectedNanos, initedNanos, validatedNanos;
Map<String, Object> variables = initVariants
? new HashMap<String, Object>()
: null;
Map<String, Object> globalVariables = initGlobalVariants
? new HashMap<String, Object>()
: null;
createStartNanosUpdater.set(this, connectStartNanos);
creatingCountUpdater.incrementAndGet(this);
try {
// 在这里开始调用驱动。参考后面的介绍
conn = createPhysicalConnection(url, physicalConnectProperties);
connectedNanos = System.nanoTime();
if (conn == null) {
throw new SQLException("connect error, url " + url + ", driverClass " + this.driverClass);
}
initPhysicalConnection(conn, variables, globalVariables);
initedNanos = System.nanoTime();
validateConnection(conn);
validatedNanos = System.nanoTime();
setFailContinuous(false);
setCreateError(null);
} catch (SQLException ex) {
setCreateError(ex);
JdbcUtils.close(conn);
throw ex;
} catch (RuntimeException ex) {
setCreateError(ex);
JdbcUtils.close(conn);
throw ex;
} catch (Error ex) {
createErrorCountUpdater.incrementAndGet(this);
setCreateError(ex);
JdbcUtils.close(conn);
throw ex;
} finally {
long nano = System.nanoTime() - connectStartNanos;
createTimespan += nano;
creatingCountUpdater.decrementAndGet(this);
}
return new PhysicalConnectionInfo(conn, connectStartNanos, connectedNanos, initedNanos, validatedNanos, variables, globalVariables);
}
DruidAbstractDataSource类的创建连接的方法
具体的创建连接过程由相应的Driver的connect方法来执行。其实说白了就是返回一个封装了Socket的Connect类
public Connection createPhysicalConnection(String url, Properties info) throws SQLException {
Connection conn;
if (getProxyFilters().size() == 0) {
// getDriver 返回驱动类。这里就是我们注册的org.h2.Driver。
conn = getDriver().connect(url, info);
} else {
conn = new FilterChainImpl(this).connection_connect(info);
}
createCountUpdater.incrementAndGet(this);
return conn;
}
这里一h2数据库的驱动org.h2.Driver来讲解下“创建连接”是怎么一回事。这个驱动源码也十分简单,但所谓麻雀虽小,五脏俱全,通过看源码能充分了解连接过程。
//org.h2.Driver 创建连接过程
public Connection connect(String url, Properties info) throws SQLException {
try {
if (info == null) {
info = new Properties();
}
if (!acceptsURL(url)) {
return null;
}
if (url.equals(DEFAULT_URL)) {
return DEFAULT_CONNECTION.get();
}
// 这里看上去是更新的。进去就返回了,先不用管。
Connection c = DbUpgrade.connectOrUpgrade(url, info);
if (c != null) {
return c;
}
// 在这里创建连接
return new JdbcConnection(url, info);
} catch (Exception e) {
throw DbException.toSQLException(e);
}
}
//JdbcConnection构造方法
public JdbcConnection(ConnectionInfo ci, boolean useBaseDir)
throws SQLException {
try {
if (useBaseDir) {
String baseDir = SysProperties.getBaseDir();
if (baseDir != null) {
ci.setBaseDir(baseDir);
}
}
// this will return an embedded or server connection 在这里创建会话。里面封装了Socket信息
session = new SessionRemote(ci).connectEmbeddedOrServer(false);
trace = session.getTrace();
int id = getNextId(TraceObject.CONNECTION);
setTrace(trace, TraceObject.CONNECTION, id);
this.user = ci.getUserName();
if (isInfoEnabled()) {
trace.infoCode("Connection " + getTraceObjectName()
+ " = DriverManager.getConnection("
+ quote(ci.getOriginalURL()) + ", " + quote(user)
+ ", \"\");");
}
this.url = ci.getURL();
scopeGeneratedKeys = ci.getProperty("SCOPE_GENERATED_KEYS", false);
closeOld();
watcher = CloseWatcher.register(this, session, keepOpenStackTrace);
} catch (Exception e) {
throw logAndConvert(e);
}
}
JdbcConnection类主要构造:
可以看到JdbcConnection类有一个属性是SessionInterface,它的实现类是SessionRemote。SessionRemote类有一个Transfer类的List(顺便说下这里用list是用来针对多个数据库的主机的)。重点是Transfer类里就有一个属性Socket,还有DataInputInputStream和DataOutputStream等用来通信的class
// 初始化Transfer类。 这里可以看到设置了用户名,密码等,与数据库主机连接
private Transfer initTransfer(ConnectionInfo ci, String db, String server)
throws IOException {
Socket socket = NetUtils.createSocket(server,
Constants.DEFAULT_TCP_PORT, ci.isSSL());
Transfer trans = new Transfer(this, socket);
trans.setSSL(ci.isSSL());
trans.init();
trans.writeInt(Constants.TCP_PROTOCOL_VERSION_MIN_SUPPORTED);
trans.writeInt(Constants.TCP_PROTOCOL_VERSION_MAX_SUPPORTED);
trans.writeString(db);
trans.writeString(ci.getOriginalURL());
// 用户名
trans.writeString(ci.getUserName());
// 密码
trans.writeBytes(ci.getUserPasswordHash());
trans.writeBytes(ci.getFilePasswordHash());
String[] keys = ci.getKeys();
trans.writeInt(keys.length);
for (String key : keys) {
trans.writeString(key).writeString(ci.getProperty(key));
}
try {
done(trans);
clientVersion = trans.readInt();
trans.setVersion(clientVersion);
if (clientVersion >= Constants.TCP_PROTOCOL_VERSION_14) {
if (ci.getFileEncryptionKey() != null) {
trans.writeBytes(ci.getFileEncryptionKey());
}
}
trans.writeInt(SessionRemote.SESSION_SET_ID);
trans.writeString(sessionId);
done(trans);
if (clientVersion >= Constants.TCP_PROTOCOL_VERSION_15) {
autoCommit = trans.readBoolean();
} else {
autoCommit = true;
}
return trans;
} catch (DbException e) {
trans.close();
throw e;
}
}
详细的创建过程这里就不展开介绍了,毕竟这不是本文介绍的重点,感兴趣的读者可以自行下载h2的源码来学习,另外作者在工作中也看过公司某自研数据库的java客户端驱动。大概流程也是类似,都是Connection类封装了Socket信息。然后发送和接受消息就用往socket中写入和读取(类似于socket和ServerSocket通信的流程)。
DruidAbstractDataSource的validateConnection方法
在createPhysicalConnection方法创建完连接之后,Druid会验证连接是否有效。dataSource可以通过方法dataSource.setValidationQuery()设置用于验证的语句。通常我们可以设置validationQuery = select 1之类。
public void validateConnection(Connection conn) throws SQLException {
// 返回属性validationQuery
String query = getValidationQuery();
if (conn.isClosed()) {
throw new SQLException("validateConnection: connection closed");
}
// 如果validConnectionChecker不为null则用validConnectionChecker来检查连接是否有效。针对Mysql,PG,Oracle等有特定的
// validConnectionChecker
if (validConnectionChecker != null) {
boolean result;
Exception error = null;
try {
result = validConnectionChecker.isValidConnection(conn, validationQuery, validationQueryTimeout);
if (result && onFatalError) {
lock.lock();
try {
if (onFatalError) {
onFatalError = false;
}
} finally {
lock.unlock();
}
}
} catch (SQLException ex) {
throw ex;
} catch (Exception ex) {
result = false;
error = ex;
}
if (!result) {
SQLException sqlError = error != null ? //
new SQLException("validateConnection false", error) //
: new SQLException("validateConnection false");
throw sqlError;
}
return;
}
// 如果有设置validationQuery,则会去执行该查询语句。没有返回结果的话就会抛出异常
if (null != query) {
Statement stmt = null;
ResultSet rs = null;
try {
stmt = conn.createStatement();
if (getValidationQueryTimeout() > 0) {
stmt.setQueryTimeout(getValidationQueryTimeout());
}
rs = stmt.executeQuery(query);
if (!rs.next()) {
throw new SQLException("validationQuery didn't return a row");
}
if (onFatalError) {
lock.lock();
try {
if (onFatalError) {
onFatalError = false;
}
}
finally {
lock.unlock();
}
}
} finally {
JdbcUtils.close(rs);
JdbcUtils.close(stmt);
}
}
}
Druid将Connection封装成PhysicalConnectionInfo类返回
PhysicalConnectionInfo的主要属性如下:
private Connection connection;
private long connectStartNanos;
private long connectedNanos;
private long initedNanos;
private long validatedNanos;
private Map<String, Object> vairiables;
private Map<String, Object> globalVairiables;
DruidConnectionHolder将DataSource,PhysicalConnectionInfo封装在一起
DruidConnectionHolder[] connections 将连接添加到末尾
// init connections
while (poolingCount < initialSize) {
try {
// 创建连接。用DruidConnectionHolder来封装
PhysicalConnectionInfo pyConnectInfo = createPhysicalConnection();
// DruidConnectionHolder将DataSource,PhysicalConnectionInfo封装在一起
DruidConnectionHolder holder = new DruidConnectionHolder(this, pyConnectInfo);
connections[poolingCount++] = holder;
} catch (SQLException ex) {
LOG.error("init datasource error, url: " + this.getUrl(), ex);
if (initExceptionThrow) {
connectError = ex;
break;
} else {
Thread.sleep(3000);
}
}
}
// DruidConnectionHolder的构造器。
public DruidConnectionHolder(DruidAbstractDataSource dataSource, PhysicalConnectionInfo pyConnectInfo)
throws SQLException{
this(dataSource,
pyConnectInfo.getPhysicalConnection(),
pyConnectInfo.getConnectNanoSpan(),
pyConnectInfo.getVairiables(),
pyConnectInfo.getGlobalVairiables());
}
调用createAndLogThread(), createAndStartCreatorThread(), createAndStartDestroyThread()方法创建线程
// 启动监控数据记录线程
createAndLogThread();
// 启动连接创建线程
createAndStartCreatorThread();
// 启动连接检测线程
createAndStartDestroyThread();
// 主线程用CountDownLatch 来等待上面的CreateConnectionThread和DestroyConnectionThread创建完才继续
initedLatch.await();
createAndStartCreatorThread类的run方法
public void run() {
initedLatch.countDown();
long lastDiscardCount = 0;
int errorCount = 0;
for (;;) {
// addLast
try {
lock.lockInterruptibly();
} catch (InterruptedException e2) {
break;
}
long discardCount = DruidDataSource.this.discardCount;
boolean discardChanged = discardCount - lastDiscardCount > 0;
lastDiscardCount = discardCount;
try {
boolean emptyWait = true;
if (createError != null
&& poolingCount == 0
&& !discardChanged) {
emptyWait = false;
}
if (emptyWait
&& asyncInit && createCount < initialSize) {
emptyWait = false;
}
if (emptyWait) {
// 必须存在线程等待,才创建连接
// poolingCount是已经有的连接,notEmptyWaitThreadCount是因为连接不够正在等待创建连接的线程的数量
if (poolingCount >= notEmptyWaitThreadCount //
&& (!(keepAlive && activeCount + poolingCount < minIdle))
&& !isFailContinuous()
) {
empty.await();
}
// 防止创建超过maxActive数量的连接
if (activeCount + poolingCount >= maxActive) {
empty.await();
continue;
}
}
} catch (InterruptedException e) {
lastCreateError = e;
lastErrorTimeMillis = System.currentTimeMillis();
if ((!closing) && (!closed)) {
LOG.error("create connection Thread Interrupted, url: " + jdbcUrl, e);
}
break;
} finally {
lock.unlock();
}
// 满足条件后可以创建连接了
PhysicalConnectionInfo connection = null;
try {
connection = createPhysicalConnection();
} catch (SQLException e) {
LOG.error("create connection SQLException, url: " + jdbcUrl + ", errorCode " + e.getErrorCode()
+ ", state " + e.getSQLState(), e);
errorCount++;
if (errorCount > connectionErrorRetryAttempts && timeBetweenConnectErrorMillis > 0) {
// fail over retry attempts
setFailContinuous(true);
if (failFast) {
lock.lock();
try {
notEmpty.signalAll();
} finally {
lock.unlock();
}
}
if (breakAfterAcquireFailure) {
break;
}
try {
Thread.sleep(timeBetweenConnectErrorMillis);
} catch (InterruptedException interruptEx) {
break;
}
}
} catch (RuntimeException e) {
LOG.error("create connection RuntimeException", e);
setFailContinuous(true);
continue;
} catch (Error e) {
LOG.error("create connection Error", e);
setFailContinuous(true);
break;
}
if (connection == null) {
continue;
}
boolean result = put(connection);
if (!result) {
JdbcUtils.close(connection.getPhysicalConnection());
LOG.info("put physical connection to pool failed.");
}
errorCount = 0; // reset errorCount
if (closing || closed) {
break;
}
}
}
DestroyConnectionThread的run方法
public void run() {
initedLatch.countDown();
for (;;) {
// 从前面开始删除
try {
if (closed || closing) {
break;
}
if (timeBetweenEvictionRunsMillis > 0) {
Thread.sleep(timeBetweenEvictionRunsMillis);
} else {
Thread.sleep(1000); //
}
if (Thread.interrupted()) {
break;
}
destroyTask.run();
} catch (InterruptedException e) {
break;
}
}
}
3.2.2getConnectionDirect()
init() 方法后,就是直接获取连接
public DruidPooledConnection getConnectionDirect(long maxWaitMillis) throws SQLException {
int notFullTimeoutRetryCnt = 0;
for (;;) {
// handle notFullTimeoutRetry
DruidPooledConnection poolableConnection;
try {
// 获取连接
poolableConnection = getConnectionInternal(maxWaitMillis);
} catch (GetConnectionTimeoutException ex) {
if (notFullTimeoutRetryCnt <= this.notFullTimeoutRetryCount && !isFull()) {
notFullTimeoutRetryCnt++;
if (LOG.isWarnEnabled()) {
LOG.warn("get connection timeout retry : " + notFullTimeoutRetryCnt);
}
continue;
}
throw ex;
}
// 如果设置了testOnBorrow为true。则在获取连接的时候会去检查该连接是否可用(用validConnectionChecker或validationQuery)
if (testOnBorrow) {
boolean validate = testConnectionInternal(poolableConnection.holder, poolableConnection.conn);
if (!validate) {
if (LOG.isDebugEnabled()) {
LOG.debug("skip not validate connection.");
}
discardConnection(poolableConnection.holder);
continue;
}
} else {
if (poolableConnection.conn.isClosed()) {
discardConnection(poolableConnection.holder); // 传入null,避免重复关闭
continue;
}
if (testWhileIdle) {
final DruidConnectionHolder holder = poolableConnection.holder;
long currentTimeMillis = System.currentTimeMillis();
long lastActiveTimeMillis = holder.lastActiveTimeMillis;
long lastExecTimeMillis = holder.lastExecTimeMillis;
long lastKeepTimeMillis = holder.lastKeepTimeMillis;
if (checkExecuteTime
&& lastExecTimeMillis != lastActiveTimeMillis) {
lastActiveTimeMillis = lastExecTimeMillis;
}
if (lastKeepTimeMillis > lastActiveTimeMillis) {
lastActiveTimeMillis = lastKeepTimeMillis;
}
long idleMillis = currentTimeMillis - lastActiveTimeMillis;
long timeBetweenEvictionRunsMillis = this.timeBetweenEvictionRunsMillis;
if (timeBetweenEvictionRunsMillis <= 0) {
timeBetweenEvictionRunsMillis = DEFAULT_TIME_BETWEEN_EVICTION_RUNS_MILLIS;
}
if (idleMillis >= timeBetweenEvictionRunsMillis
|| idleMillis < 0 // unexcepted branch
) {
boolean validate = testConnectionInternal(poolableConnection.holder, poolableConnection.conn);
if (!validate) {
if (LOG.isDebugEnabled()) {
LOG.debug("skip not validate connection.");
}
discardConnection(poolableConnection.holder);
continue;
}
}
}
}
if (removeAbandoned) {
StackTraceElement[] stackTrace = Thread.currentThread().getStackTrace();
poolableConnection.connectStackTrace = stackTrace;
poolableConnection.setConnectedTimeNano();
poolableConnection.traceEnable = true;
activeConnectionLock.lock();
try {
activeConnections.put(poolableConnection, PRESENT);
} finally {
activeConnectionLock.unlock();
}
}
if (!this.defaultAutoCommit) {
poolableConnection.setAutoCommit(false);
}
return poolableConnection;
}
}
在调用**getConnectionInternal(long maxWait)**方法获取连接的时候,会传入maxWait参数。这个参数的表示:从连接池中获取连接的最大等待时间,单位ms,默认-1,即会一直等待下去
if (maxWait > 0) {
holder = pollLast(nanos);
} else {
holder = takeLast();
}
下面是pollLast方法:
利用了Condition的awaitNanos(long nanosTimeout)方法来阻塞线程。如果连接池内没有连接了,则调用empty.signal(),通知CreateThread创建连接,并且等待指定的时间,被唤醒之后再去查看是否有可用连接。
并且,当wait的时间到了之后就会直接返回null,不会一直等待下去
if (poolingCount == 0) {
if (estimate > 0) {
continue;
}
waitNanosLocal.set(nanos - estimate);
return null;
}
private DruidConnectionHolder pollLast(long nanos) throws InterruptedException, SQLException {
long estimate = nanos;
for (;;) {
if (poolingCount == 0) {
emptySignal(); // send signal to CreateThread create connection
if (failFast && isFailContinuous()) {
throw new DataSourceNotAvailableException(createError);
}
if (estimate <= 0) {
waitNanosLocal.set(nanos - estimate);
return null;
}
notEmptyWaitThreadCount++;
if (notEmptyWaitThreadCount > notEmptyWaitThreadPeak) {
notEmptyWaitThreadPeak = notEmptyWaitThreadCount;
}
try {
long startEstimate = estimate;
estimate = notEmpty.awaitNanos(estimate); // signal by
// recycle or
// creator
notEmptyWaitCount++;
notEmptyWaitNanos += (startEstimate - estimate);
if (!enable) {
connectErrorCountUpdater.incrementAndGet(this);
if (disableException != null) {
throw disableException;
}
throw new DataSourceDisableException();
}
} catch (InterruptedException ie) {
notEmpty.signal(); // propagate to non-interrupted thread
notEmptySignalCount++;
throw ie;
} finally {
notEmptyWaitThreadCount--;
}
if (poolingCount == 0) {
if (estimate > 0) {
continue;
}
waitNanosLocal.set(nanos - estimate);
return null;
}
}
decrementPoolingCount();
DruidConnectionHolder last = connections[poolingCount];
connections[poolingCount] = null;
long waitNanos = nanos - estimate;
last.setLastNotEmptyWaitNanos(waitNanos);
return last;
}
}
下面是takeLast()方法:
可以看到takeLast调用的是notEmpty.await()方法,会一直调用下去。
maxWait默认是不超时,即如果连接池没有空闲连接,则会一直等待下去
DruidConnectionHolder takeLast() throws InterruptedException, SQLException {
try {
while (poolingCount == 0) {
emptySignal(); // send signal to CreateThread create connection
if (failFast && isFailContinuous()) {
throw new DataSourceNotAvailableException(createError);
}
notEmptyWaitThreadCount++;
if (notEmptyWaitThreadCount > notEmptyWaitThreadPeak) {
notEmptyWaitThreadPeak = notEmptyWaitThreadCount;
}
try {
notEmpty.await(); // signal by recycle or creator
} finally {
notEmptyWaitThreadCount--;
}
notEmptyWaitCount++;
if (!enable) {
connectErrorCountUpdater.incrementAndGet(this);
if (disableException != null) {
throw disableException;
}
throw new DataSourceDisableException();
}
}
} catch (InterruptedException ie) {
notEmpty.signal(); // propagate to non-interrupted thread
notEmptySignalCount++;
throw ie;
}
decrementPoolingCount();
DruidConnectionHolder last = connections[poolingCount];
connections[poolingCount] = null;
return last;
}
testOnBorrow、testWhileIdle、removeAbandoned参数
- 如果testOnBorrow为true,则进行对连接进行校验,校验失败则进行清理并重新进入循环,否则跳到下一步
- 如果testWhileIdle为true,距离上次激活时间超过timeBetweenEvictionRunsMillis,则进行清理
- 如果removeAbandoned为true,则会把连接存放在activeConnections中,清理线程会对其定期进行处理
CreateThread和向线程池取连接的消费者线程形成了一个消费者,生产者模型
3.2.3Connection.close() 连接回收
调用的是DruidPooledConnection类的close()方法。代码如下:
实际调用的是回收recycle方法
public void close() throws SQLException {
if (this.disable) {
return;
}
DruidConnectionHolder holder = this.holder;
if (holder == null) {
if (dupCloseLogEnable) {
LOG.error("dup close");
}
return;
}
DruidAbstractDataSource dataSource = holder.getDataSource();
boolean isSameThread = this.getOwnerThread() == Thread.currentThread();
if (!isSameThread) {
dataSource.setAsyncCloseConnectionEnable(true);
}
if (dataSource.isAsyncCloseConnectionEnable()) {
syncClose();
return;
}
if (!CLOSING_UPDATER.compareAndSet(this, 0, 1)) {
return;
}
try {
for (ConnectionEventListener listener : holder.getConnectionEventListeners()) {
listener.connectionClosed(new ConnectionEvent(this));
}
List<Filter> filters = dataSource.getProxyFilters();
if (filters.size() > 0) {
FilterChainImpl filterChain = new FilterChainImpl(dataSource);
filterChain.dataSource_recycle(this);
} else {
// 实际调用的是回收recycle方法
recycle();
}
} finally {
CLOSING_UPDATER.set(this, 0);
}
this.disable = true;
}
recycle()方法,已省略部分代码:
protected void recycle(DruidPooledConnection pooledConnection) throws SQLException {
......
// 如果连接已经close掉了就把closeCount加1后返回
if (physicalConnection.isClosed()) {
lock.lock();
try {
if (holder.active) {
activeCount--;
holder.active = false;
}
closeCount++;
} finally {
lock.unlock();
}
return;
}
......
// 如果设置了testOnReturn参数则会调用testConnectionInternal(DruidConnectionHolder holder, Connection conn)方法检查连接是否有效,可参考前面讲的validateConnection(Connection conn)方法
if (testOnReturn) {
boolean validate = testConnectionInternal(holder, physicalConnection);
if (!validate) {
JdbcUtils.close(physicalConnection);
destroyCountUpdater.incrementAndGet(this);
lock.lock();
try {
if (holder.active) {
activeCount--;
holder.active = false;
}
closeCount++;
} finally {
lock.unlock();
}
return;
}
}
..........
// 最后获取锁,并把连接放回到connections数组中
lock.lock();
try {
if (holder.active) {
activeCount--;
holder.active = false;
}
closeCount++;
result = putLast(holder, currentTimeMillis);
recycleCount++;
} finally {
lock.unlock();
}
}
boolean putLast(DruidConnectionHolder e, long lastActiveTimeMillis) {
if (poolingCount >= maxActive || e.discard || this.closed) {
return false;
}
e.lastActiveTimeMillis = lastActiveTimeMillis;
connections[poolingCount] = e;
incrementPoolingCount();
if (poolingCount > poolingPeak) {
poolingPeak = poolingCount;
poolingPeakTime = lastActiveTimeMillis;
}
notEmpty.signal();
notEmptySignalCount++;
return true;
}