环境准备

安装JAVA
安装MySQL
配置服务器hostname

vim /etc/hosts
172.16.5.133 service01
172.16.4.149 service02
172.16.5.172 service03

配置服务器互信

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
ssh-copy-id service01
ssh-copy-id service02
ssh-copy-id service03

搭建 hadoop 集群

下载hadoop: https://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.10.2/hadoop-2.10.2.tar.gz
进入路径 /root/olap/hadoop-2.10.2/etc/hadoop，修改配置文件：

core-site.xml

<configuration>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/usr/hadoop/tmp</value>
    <description>A base for other temprary directories</description>
  </property>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://172.16.5.133:9000</value>
  </property>
</configuration>

hdfs-site.xml

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>2</value>
  </property>
  <property>
    <name>dfs.http.address</name>
    <value>172.16.5.133:50070</value>
  </property>
  <property>
    <name>dfs.namenode.secondary.http-address</name>
    <value>172.16.4.149:50090</value>
  </property>
  <property>
    <name>dfs.webhdfs.enabled</name>
    <value>true</value>
  </property>
</configuration>

mapred-site.xml

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>

yarn-site.xml

<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>172.16.5.133</value>
  </property>
</configuration>

slaves

service01
service02
service03

在/etc/profile.d/下新建文件hadoop.sh，并编辑以下内容，执行source /etc/profile.d/hadoop.sh

export HADOOP_HOME=/root/olap/hadoop-2.10.22
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin

将/root/olap/hadoop-2.10.22发送到另外两台服务器

scp -r /root/olap/hadoop-2.10.22 root@service02:/root/olap
scp -r /root/olap/hadoop-2.10.22 root@service03:/root/olap

格式化和启动

bin/hdfs namenode -format

22/11/08 13:33:25 INFO common.Storage: Storage directory /usr/hadoop/tmp/dfs/name has been successfully formatted.
22/11/08 13:33:25 INFO namenode.FSImageFormatProtobuf: Saving image file /usr/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
22/11/08 13:33:25 INFO namenode.FSImageFormatProtobuf: Image file /usr/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 323 bytes saved in 0 seconds .
22/11/08 13:33:25 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
22/11/08 13:33:25 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid = 0 when meet shutdown.
22/11/08 13:33:25 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at service01/127.0.0.1
************************************************************/

启动hdfs

/sbin/start-dfs.sh

Starting namenodes on [172.16.5.133]
172.16.5.133: starting namenode, logging to /root/olap/hadoop-2.10.2/logs/hadoop-root-namenode-service01.out
service02: starting datanode, logging to /root/olap/hadoop-2.10.2/logs/hadoop-root-datanode-service02.out
service03: starting datanode, logging to /root/olap/hadoop-2.10.2/logs/hadoop-root-datanode-service03.out
service01: starting datanode, logging to /root/olap/hadoop-2.10.2/logs/hadoop-root-datanode-service01.out
Starting secondary namenodes [service02]
service02: starting secondarynamenode, logging to /root/olap/hadoop-2.10.2/logs/hadoop-root-secondarynamenode-service02.out

访问service01:50070，若正常则hadoop部署成功。

Screenshot 2022-11-08 at 13.49.39.png

问题排查：
Q: datanode启动报错Incompatible clusterIDs in /usr/hadoop/tmp/dfs/data: namenode clusterID = CID-f375d97c-0c1d-4582-8b81-553f07ddc403; datanode clusterID = CID-e1e4265d-ca12-4475-bcd1-49a2f1caa25b
A: 将/usr/hadoop/tmp/dfs下的data和name文件夹删除，再执行hdfs namenode -format

Q:报错There appears to be a gap in the edit log. We expected txid 1
A: hadoop namenode -recover 选择y和c

启动yarn

/sbin/start-yarn.sh

starting yarn daemons
starting resourcemanager, logging to /root/olap/hadoop-2.10.2/logs/yarn-root-resourcemanager-service01.out
service02: starting nodemanager, logging to /root/olap/hadoop-2.10.2/logs/yarn-root-nodemanager-service02.out
service01: starting nodemanager, logging to /root/olap/hadoop-2.10.2/logs/yarn-root-nodemanager-service01.out
service03: starting nodemanager, logging to /root/olap/hadoop-2.10.2/logs/yarn-root-nodemanager-service03.out

Screenshot 2022-11-08 at 14.02.34.png

部署分布式 Hadoop 集群

环境准备

搭建 hadoop 集群