Hadoop之配置日志的聚集

日志聚集概念：应用运行完成以后，将程序运行日志信息上传到 HDFS 系统上。

日志聚集功能好处：可以方便的查看到程序运行详情，方便开发调试。

注意：开启日志聚集功能，需要重新启动 NodeManager 、ResourceManager 和 HistoryServer。

开启日志聚集功能具体步骤如下：

配置yarn-site.xml

[muyi@hadoop102 hadoop]$ vim yarn-site.xml 


<!-- 开启日志聚集功能 -->

<property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
</property>
<!-- 设置日志聚集服务器地址 -->
<property>
    <name>yarn.log.server.url</name>
    <value>http://hadoop102:19888/jobhistory/logs</value>
</property>
<!-- 设置日志保留时间为 7 天 -->
<property>
    <name>yarn.log-aggregation.retain-seconds</name>
    <value>604800</value>
</property>

分发配置

[muyi@hadoop102 hadoop]$ xsync yarn-site.xml 
==================== hadoop102 ====================
sending incremental file list

sent 64 bytes  received 12 bytes  152.00 bytes/sec
total size is 1,598  speedup is 21.03
==================== hadoop103 ====================
sending incremental file list
yarn-site.xml

sent 1,013 bytes  received 47 bytes  2,120.00 bytes/sec
total size is 1,598  speedup is 1.51
==================== hadoop104 ====================
sending incremental file list
yarn-site.xml

sent 1,013 bytes  received 47 bytes  2,120.00 bytes/sec
total size is 1,598  speedup is 1.51
[muyi@hadoop102 hadoop]$ 

关闭 NodeManager 、ResourceManager 和 HistoryServer

[muyi@hadoop102 hadoop]$ jps
5522 JobHistoryServer
6373 DataNode
5323 NodeManager
8188 Jps
6206 NameNode
[muyi@hadoop102 hadoop]$ mapred --daemon stop historyserver
[muyi@hadoop102 hadoop]$ jps
8258 Jps
6373 DataNode
5323 NodeManager
6206 NameNode
[muyi@hadoop102 hadoop]$ 


[muyi@hadoop103 hadoop-3.1.3]$ sbin/stop-yarn.sh 
Stopping nodemanagers
hadoop103: WARNING: nodemanager did not stop gracefully after 5 seconds: Trying to kill with kill -9
Stopping resourcemanager
[muyi@hadoop103 hadoop-3.1.3]$ sbin/start-yarn.sh


[muyi@hadoop102 hadoop]$ mapred --daemon start historyserver

测试

[muyi@hadoop102 hadoop-3.1.3]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /wcinput /output2