hadoop完全分布式搭建（学不会干我），从三流大数据开发外包到秒杀阿里P7

网上学习资料一大堆，但如果学到的知识不成体系，遇到问题时只是浅尝辄止，不再深入研究，那么很难做到真正的技术提升。

需要这份系统化资料的朋友，可以戳这里获取

一个人可以走的很快，但一群人才能走的更远！不论你是正从事IT行业的老鸟或是对IT行业感兴趣的新人，都欢迎加入我们的的圈子（技术交流、学习资源、职场吐槽、大厂内推、面试辅导），让我们一起学习成长！

<property>
    <name>hadoop.http.staticuser.user</name>
    <value>root</value>
</property>

###   vim hadoop-env.sh

export JAVA_HOME=/opt/java

![](https://p9-xtjj-sign.byteimg.com/tos-cn-i-73owjymdk6/4550c5fc607c48e6b0c127451c552316~tplv-73owjymdk6-jj-mark-v1:0:0:0:0:5o6Y6YeR5oqA5pyv56S-5Yy6IEAg55So5oi3MzM5MTQ5MjgwNjA=:q75.awebp?rk3s=f64ab15b&x-expires=1772525228&x-signature=LxD3LeKTDB0uJIYCdnRK%2FkrVJ1Y%3D)


###  vim hdfs-site.xml

dfs.namenode.http-address

master:9870

dfs.namenode.secondary.http-address

slave1:9868

### vim workers


![](https://p9-xtjj-sign.byteimg.com/tos-cn-i-73owjymdk6/3b275aba2d384162b02e24baf97c5562~tplv-73owjymdk6-jj-mark-v1:0:0:0:0:5o6Y6YeR5oqA5pyv56S-5Yy6IEAg55So5oi3MzM5MTQ5MjgwNjA=:q75.awebp?rk3s=f64ab15b&x-expires=1772525228&x-signature=iKz%2BT4ZYEtm8aHxRFe7T41qC%2F8M%3D)


### vim yarn-site.xml

<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>master</value>

yarn.resourcemanager.webapp.address master:8088 指定 ResourceManager 的 Web 应用连接地址 yarn.resourcemanager.address master:8032 yarn.resourcemanager.scheduler.address 指定 ResourceManager 的调度器连接地址 master:8030 yarn.resourcemanager.resource-tracker.address master:8031 yarn.nodemanager.aux-services mapreduce_shuffle

yarn.nodemanager.pmem-check-enabled false

<property>
   <name>yarn.nodemanager.vmem-check-enabled</name>
   <value>false</value>
</property>
<!-- 环境变量的继承 -->
<property>

    <name>yarn.nodemanager.env-whitelist</name>
    <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>

<property>
	<name>yarn.log-aggregation-enable</name>
	<value>true</value>
</property>
<!-- 设置日志聚集服务器地址 -->
<property>  

	<name>yarn.log.server.url</name>  
	<value>http://master:19888/jobhistory/logs</value>
</property>
<!-- 设置日志保留时间为7天 -->
<property>
	<name>yarn.log-aggregation.retain-seconds</name>
	<value>604800</value>
</property>

### vim mapred-site.xml

mapreduce.framework.name yarn mapreduce.jobhistory.address master:10020

<!-- 历史服务器web端地址 -->
<property>
	<name>mapreduce.jobhistory.webapp.address</name>
	<value>master:19888</value>
</property>

###  cd /opt/hadoop/sbin


### Start-dfs.sh


HDFS\_NAMENODE\_USER=root


HDFS\_DATANODE\_USER=root


HDFS\_SECONDARYNAMENODE\_USER=root


### 


### Stop-dfs.sh


HDFS\_NAMENODE\_USER=root


HDFS\_DATANODE\_USER=root


HDFS\_SECONDARYNAMENODE\_USER=root



Start-yarn.sh:


YARN\_RESOURCEMANAGER\_USER=root


YARN\_NODEMANAGER\_USER=root



Stop-yarn.sh:


YARN\_RESOURCEMANAGER\_USER=root


YARN\_NODEMANAGER\_USER=root


### 




## 接下来就是分发个各个集群

[root@master opt]# scp -r ./hadoop slave1:pwd

[root@master opt]# scp -r ./hadoop slave2:pwd

### 格式化hdfs

hdfs namenode -format

![](https://p9-xtjj-sign.byteimg.com/tos-cn-i-73owjymdk6/53faaebb8a884669bb23acd98514e09a~tplv-73owjymdk6-jj-mark-v1:0:0:0:0:5o6Y6YeR5oqA5pyv56S-5Yy6IEAg55So5oi3MzM5MTQ5MjgwNjA=:q75.awebp?rk3s=f64ab15b&x-expires=1772525228&x-signature=WNY2XMhyA%2BK3EvwsCZ7XyLeH968%3D)


启动集群 

start-all.sh

jps查看集群是否启动


master：


![](https://p9-xtjj-sign.byteimg.com/tos-cn-i-73owjymdk6/19cf5d0c37d6466c91903e161e43f79c~tplv-73owjymdk6-jj-mark-v1:0:0:0:0:5o6Y6YeR5oqA5pyv56S-5Yy6IEAg55So5oi3MzM5MTQ5MjgwNjA=:q75.awebp?rk3s=f64ab15b&x-expires=1772525228&x-signature=0A3GX5lRKS1hNFym9uNx4eRkZyY%3D)


slave1：


![](https://p9-xtjj-sign.byteimg.com/tos-cn-i-73owjymdk6/38ff91a663124977831cc9a333b10e54~tplv-73owjymdk6-jj-mark-v1:0:0:0:0:5o6Y6YeR5oqA5pyv56S-5Yy6IEAg55So5oi3MzM5MTQ5MjgwNjA=:q75.awebp?rk3s=f64ab15b&x-expires=1772525228&x-signature=YGPP9xZ81%2F9V3O8PdY42xND7m0A%3D)


slave2：


![](https://p9-xtjj-sign.byteimg.com/tos-cn-i-73owjymdk6/4093ebb5be9d443399176aee4ca35235~tplv-73owjymdk6-jj-mark-v1:0:0:0:0:5o6Y6YeR5oqA5pyv56S-5Yy6IEAg55So5oi3MzM5MTQ5MjgwNjA=:q75.awebp?rk3s=f64ab15b&x-expires=1772525228&x-signature=Cuqqz3YvPuenOP83KaONBCUGnMk%3D)


浏览器中输入master:9870  和 master:8088看看是否有web界面，有的话就搭建成功了


![](https://p9-xtjj-sign.byteimg.com/tos-cn-i-73owjymdk6/23ff9ba28094430c9966b2fd657ae066~tplv-73owjymdk6-jj-mark-v1:0:0:0:0:5o6Y6YeR5oqA5pyv56S-5Yy6IEAg55So5oi3MzM5MTQ5MjgwNjA=:q75.awebp?rk3s=f64ab15b&x-expires=1772525228&x-signature=5DGNxsGhwdi%2B4mGJBwWFHrWBYtU%3D)


![](https://p9-xtjj-sign.byteimg.com/tos-cn-i-73owjymdk6/950c2e3a71b84fb59cb263bef6f79ced~tplv-73owjymdk6-jj-mark-v1:0:0:0:0:5o6Y6YeR5oqA5pyv56S-5Yy6IEAg55So5oi3MzM5MTQ5MjgwNjA=:q75.awebp?rk3s=f64ab15b&x-expires=1772525228&x-signature=8EiidNtHVfjLP0OHqVGbZVPohUo%3D)



## 部分错误处理


![img](https://p9-xtjj-sign.byteimg.com/tos-cn-i-73owjymdk6/c703eff4ea154939acc109b0f2739532~tplv-73owjymdk6-jj-mark-v1:0:0:0:0:5o6Y6YeR5oqA5pyv56S-5Yy6IEAg55So5oi3MzM5MTQ5MjgwNjA=:q75.awebp?rk3s=f64ab15b&x-expires=1772525228&x-signature=5qKI5OgDpE0GafyRApMYESIgk3A%3D)
![img](https://p9-xtjj-sign.byteimg.com/tos-cn-i-73owjymdk6/333b23f6e8a14837b624b45e1488d626~tplv-73owjymdk6-jj-mark-v1:0:0:0:0:5o6Y6YeR5oqA5pyv56S-5Yy6IEAg55So5oi3MzM5MTQ5MjgwNjA=:q75.awebp?rk3s=f64ab15b&x-expires=1772525228&x-signature=vAOghT3Vftkx8L8SG4lxuFvRZqY%3D)

**网上学习资料一大堆，但如果学到的知识不成体系，遇到问题时只是浅尝辄止，不再深入研究，那么很难做到真正的技术提升。**

**[需要这份系统化资料的朋友，可以戳这里获取](https://gitee.com/vip204888)**


**一个人可以走的很快，但一群人才能走的更远！不论你是正从事IT行业的老鸟或是对IT行业感兴趣的新人，都欢迎加入我们的的圈子（技术交流、学习资源、职场吐槽、大厂内推、面试辅导），让我们一起学习成长！**