YARN上运行MapReduce 程序

398 阅读1分钟
  1. 配置yarn-env.sh

    export JAVA_HOME=/opt/module/jdk1.7.0_79

  2. 配置yarn-site.xml

    <!-- reducer获取数据的方式 -->
    <property>
     <name>yarn.nodemanager.aux-services</name>
     <value>mapreduce_shuffle</value>
    </property>
    
    <!-- 指定YARN的ResourceManager的地址 -->
    <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>hadoop100</value>
    </property>
    

  3. 配置:mapred-env.sh

    export JAVA_HOME=/opt/module/jdk1.7.0_79

  4. 配置: (对mapred-site.xml.template重新命名为) mapred-site.xml

    <!-- 指定mr运行在yarn上 -->
    	<property>
    		<name>mapreduce.framework.name</name>
    		<value>yarn</value>
    	</property>
    

  5. 启动resourcemanager

    sbin/yarn-daemon.sh start resourcemanager

  6. 启动nodemanager

    sbin/yarn-daemon.sh start nodemanager

  7. yarn的浏览器页面查看

    http://10.1.100.100:8088/cluster

  8. 删除文件系统上的output文件

    bin/hdfs dfs -rm -R /user/wangxiaofan/mapreduce/wordcount/output

  9. 执行mapreduce程序

    bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount /user/wangxiaofan/mapreduce/wordcount/input  /user/wangxiaofan/mapreduce/wordcount/output

  10. 查看运行结果

    bin/hdfs dfs -cat /user/wangxiaofan/mapreduce/wordcount/output/*