本文已参与「新人创作礼」活动，一起开启掘金创作之路。
Couldn't acquire the DB log notification lock because we reached the maximu
背景
集群遇到了个问题，Hadoop 3版本的，有20个5分钟级任务，一个分钟级任务，这些任务都会创建分钟分区，现在在创建分区的时候卡那，看元数据库是锁那了，有什么解决办法，重试hiveserver2，开始好事一会又不行了，这些任务也没做啥复杂处理，就是先判断有没有分区，没有就建分区，然后再执行文件拷贝到分区目录下
sql脚本
为了防止重复分区，先drop再add
alter table hi_cloudjnt.int_loc_sigvolte_calling_hang_up_min drop if exists partition {date_no_=${batchNo?substnng(0,8)},hour_no_=${batchNo?substnng(0,10)},minute_no_=${batchNo?substring(0,12)});
alter table hi_cloudjnt.int_loc_sigvolte_calling_hang_up_min add if not exists partition (date_no_=${batchNo?substnng(0,8)},hour_no_=${batchNo?substnng(0,10)},minute_no_=${batchNo?substring(0,12)});
报错
查看hive报错 Hive DDL 间歇性失败并出现错误 - 无法获取数据库日志通知锁，因为我们达到了最大重试次数：10 次重试
2022-04-19 04:36:27 [pool-1-thread-2] [ERROR] Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Couldn't acquire the DB log notification lock because we reached the maximum # of retries: 10 retries. If this happens too often, then is recommended to increase the maximum number of retries on the hive.notification.sequence.lock.max.retries configuration :: Error executing SQL query "select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update".)
解决
看来，该集群中的元存储操作很慢，因此并发写入/DDL 操作无法锁定行以进行更新。
目前，重试之间的睡眠间隔是通过配置hive.notification.sequence.lock.retry.sleep.interval指定的。默认值是 500 毫秒，这似乎太小了。我们可以为睡眠间隔和重试次数设置更高的值吗？
hive.notification.sequence.lock.retry.sleep.interval=10s
hive.notification.sequence.lock.max.retries=10 修改hive.notification.sequence.lock.retry.sleep.interval=改成10s 之前是 hive.notification.sequence.lock.retry.sleep.interval=500ms
参考
问题描述
Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Couldn't acquire the DB log notification lock because we reached the maximum # of retries: 5 retries. If this happens too often, then is recommended to increase the maximum number of retries on the hive.notification.sequence.lock.max.retries configuration :: Error executing SQL query "select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update".)
2018-08-28 01:17:56,808|INFO|MainThread|machine.py:183 - run()||GUID=94e6ff4d-5db8-45eb-8654-76f546e7f0b3|java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Couldn't acquire the DB log notification lock because we reached the maximum # of retries: 5 retries. If this happens too often, then is recommended to increase the maximum number of retries on the hive.notification.sequence.lock.max.retries configuration :: Error executing SQL query "select "NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update".)
