今日启动虚拟机CDH（有好长时间没有启动了）,几分钟后Cloudera manager主页面没有出来，发现cloudera-scm-server启动有问题，排查发现对应服务器上没有对应的7180端口。打开相应日志cloudera-scm-server.log，继续追溯问题产生原因，解决该问题。
错误日志信息如下：
2023-02-20 16:41:44,927 WARN C3P0PooledConnectionPoolManager[identityToken->1hge2xxauuilxwtwas5ok|62e7dffa]-HelperThread-#1:com.mchange.v2.resourcepool.BasicResourcePool: Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@5fad41be is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests.
2023-02-20 16:41:44,938 WARN main:org.springframework.context.support.GenericApplicationContext: Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.cloudera.server.cmf.TrialState': Cannot resolve reference to bean 'entityManagerFactoryBean' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is org.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection
2023-02-20 16:41:44,938 ERROR main:com.cloudera.server.cmf.Main: Server failed.
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.cloudera.server.cmf.TrialState': Cannot resolve reference to bean 'entityManagerFactoryBean' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is org.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:634)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1198)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1100)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:511)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
at com.cloudera.server.cmf.Main.bootstrapSpringContext(Main.java:421)
at com.cloudera.server.cmf.Main.<init>(Main.java:263)
at com.cloudera.server.cmf.Main.main(Main.java:236)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is org.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:185)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1640)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:254)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
... 17 more
Caused by: org.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:47)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:102)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:129)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getConnectionForTransactionManagement(LogicalConnectionManagedImpl.java:247)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:254)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:203)
at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:56)
at com.cloudera.enterprise.AbstractWrappedEntityManager.beginForRollbackAndReadonly(AbstractWrappedEntityManager.java:89)
at com.cloudera.enterprise.dbutil.DbUtil.isInnoDbEnabled(DbUtil.java:558)
at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.checkMysqlTableEngineType(EntityManagerFactoryBean.java:187)
at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:166)
at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:75)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:178)
... 22 more
Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database!
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:118)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:689)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:140)
at org.hibernate.c3p0.internal.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:73)
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:35)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:99)
... 33 more
Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source.
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1418)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:606)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:526)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:755)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:682)
... 37 more
初步判断和cdh源数据库mysql有关系
初步判断mysql登陆有问题 找到 cloudera-manager-server节点，scm配置文件 /etc/cloudera-scm-server/db.properties 查看配置mysql信息 ：
cat /etc/cloudera-scm-server/db.properties
com.cloudera.cmf.db.type=mysql com.cloudera.cmf.db.host=192.168.10.103:3306 com.cloudera.cmf.db.name=scm com.cloudera.cmf.db.user=scm com.cloudera.cmf.db.password=Example@2022 com.cloudera.cmf.db.setupType=EXTERNAL
到192.168.10.103 服务器上面分别执行下面命令
mysql--------------------------直接使用“mysql”命令，不带主机名参数； mysql -h localhost------------使用带了主机名“localhost”参数的命令 mysql -h 127.0.0.1-------------使用带了主机名“127.0.0.1”参数的命令
均报错 ： ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
【错误解析】：
这是由于我们连接数据库使用的主机名参数为“localhost”，或者未使用主机名参数、服务器默认使用“localhost”做为主机名。 使用主机名参数为“localhost”连接mysql服务端时，mysql客户端会认为是连接本机， 所以会尝试以socket文件方式进行连接(socket文件连接方式，比“ip：端口”方式效率更高)，这时根据配置文件“/etc/mysql.cnf”的路径，未找到相应的socket文件，就会引发此错误。
【解决思路】 1、看mysql服务是否在运行 由于“socket”文件是由mysql服务运行时创建的，如果提示“ERROR 2002 (HY000): Can’t connect to local MySQL server through socket ‘***’ (2)”， 找不到“socket”文件，我们首先要确认的是mysql服务是否正在运行。
1.1、端口是否打开 lsof -i:3306
1.2、mysqld服务是否正在运行 service mysqld status
【备注】：若mysql未运行，重启mysql服务，一般可以解决。
2、确定“socket”文件正确位置： 确定mysql服务正常运行后，产生此错误的原因只剩下“socket”文件路径不正确了， 我们可以使用“find”命令或者“lsof”命令来确定socket文件的正确路径：
lsof -c mysqld|grep sock
find / -name ‘*.sock’ /storage/db/mysql/mysql.sock
【备注】：若mysql运行，而且mysql.sock也存在，那么就是指向问题
【解决方案】
1、【第一种情况】若mysql未运行，重启mysql，再次尝试连接
使用 service 启动： service mysqld restart service mysql restart
【扩展信息】： --使用service启动 service mysqld start service mysql start --使用service停止 service mysqld stop
2、【第二种情况-方案一】若mysql运行，mysql.sock,路径和报错信息中的路径不一致。
修改“/etc/my.cnf”配置文件，在/etc/php.ini文件中"[MySQL]“项下找到"mysql.default_socket”， 并设置其值指向正确的mysql服务socket文件即可， 在配置文件中添加“[client]”选项和“[mysql]”选项， 并使用这两个选项下的“socket”参数值，与“[mysqld]”选项下的“socket”参数值，指向的socket文件路径完全一致。 如下：
[mysqld] datadir=/storage/db/mysql socket=/storage/db/mysql/mysql.sock …省略n行 [client] default-character-set=utf8 socket=/storage/db/mysql/mysql.sock [mysql] default-character-set=utf8 socket=/storage/db/mysql/mysql.sock
修改完后，重启mysqld服务，即可解决此问题。
3、【第二种情况-方案二】 使用“ln -s /storage/db/mysql/mysql.sock /var/lib/mysql/mysql.sock”命令， 将正确的socket文件位置，软链接到提示错误的socket文件路径位置，即可解决此问题：
[root@lam7 opt]# ls /var/lib/mysql/mysql.sock ls: 无法访问/var/lib/mysql/mysql.sock: 没有那个文件或目录
[root@lam7 opt]# ln -s /storage/db/mysql/mysql.sock /var/lib/mysql/mysql.sock [root@lam7 opt]# ls /var/lib/mysql/mysql.sock /var/lib/mysql/mysql.sock
