CDH集群缓存清理

各部分缓存查看

1. Hadoop

hadoop fs -rm -r .Trash

1. cloudera manager

位于cm机器的/var/lib下

du -sh /var/lib/*

du -sh /var/lib/cloudera-host-monitor/ts/*


du -sh /var/lib/cloudera-service-monitor/ts/*

1. yarn

位于子节点的/yarn/nm下

du -sh /yarn/nm/usercache/af/filecache/*

1. Spark2

hdfs dfs -ls /user/spark/spark2ApplicationHistory

指令整合

"/root/bin/cleancache"

#!/bin/bash
hdfs dfs -rm -r -f /user/spark/spark2ApplicationHistory


rm -rf /var/lib/cloudera-host-monitor/ts/*/partition*/*
rm -rf /var/lib/cloudera-service-monitor/ts/*/partition*/*


rm -rf /var/log/cloudera-scm-eventserver/*out*
rm -rf /var/log/cloudera-scm-firehose/*out*


rm -rf /var/log/hadoop-hdfs/*out*
rm -rf /var/log/hadoop-httpfs/*out*
rm -rf /var/log/hadoop-kms/*out*
rm -rf /var/log/hadoop-mapreduce/*out*


rm -rf /var/log/zookeeper/*log*


hadoop fs -rm -r .Trash


for i in ljw2 ljw3
	do
		echo "---$i---"
		ssh $i "rm -rf /yarn/nm/usercache/af/filecache/*"

定时启动

crontab -e
# 每周五下午19点
00 19 * * 5 cleancache

不确定的情况下，请先备份，再开始行动。