生产环境 HDFS 配置 hadoop-3.2.1

1. 准备三台vps主机

2. 检查三台主机内网ip,并修改hosts文件，使其可以通过主机名访问

10.111.222.121 hadoop-1
10.111.222.122 hadoop-2
10.111.222.123 hadoop-3

创建 SSH Key

> ssh-keygen -b 4096

配置好三台主机间的ssh无密码访问

> cat ~/.ssh/id_rsa.pub
> cat ~/.ssh/authorized_keys

检查java版本和JAVA_HOME

/usr/lib/jvm/java-1.8.0-openjdk
> java -version
openjdk version "1.8.0_232"
OpenJDK Runtime Environment (build 1.8.0_232-b09)
OpenJDK 64-Bit Server VM (build 25.232-b09, mixed mode)

下载和安装hadoop

> wget http://apache.mirrors.tds.net/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz 
> tar zxvf hadoop-3.2.1.tar.gz
> mv hadoop-3.2.1 ~/tool/


> sudo ln -s /home/centos/tool/hadoop-3.2.1 /opt/hadoop-3.2.1
> sudo ln -s /opt/hadoop-3.2.1 /opt/hadoop

增加系统配置

> cat ~/.bash_profile
PATH=$PATH:/opt/hadoop/sbin:/opt/hadoop/bin
export JAVA_HOME="/usr/lib/jvm/java-1.8.0-openjdk"
> . ~/.bash_profile
> hdfs version
Hadoop 3.2.1

配置hadoop

> cd /opt/hadoop
> vi etc/hadoop/hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
> vi etc/hadoop/core-site.xml
<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://hadoop-1:9000</value>
        </property>
        <property>
                <name>hadoop.http.filter.initializers</name>
                <value>org.apache.hadoop.security.AuthenticationFilterInitializer</value>
        </property>
        <property>
                <name>hadoop.http.authentication.type</name>
                <value>simple</value>
        </property>
        <property>
                <name>hadoop.http.authentication.token.validity</name>
                <value>12000</value>
        </property>
        <property>
                <name>hadoop.http.authentication.simple.anonymous.allowed</name>
                <value>false</value>
        </property>
        <property>
                <name>hadoop.http.authentication.signature.secret.file</name>
                <value>/opt/hadoop/hadoop-http-auth-signature-secret</value>
        </property>
        <property>
                <name>hadoop.http.authentication.cookie.domain</name>
                <value></value>
        </property>
</configuration>

设置password

> vi hadoop-http-auth-signature-secret
adminpassword

修改hdfs-siter.xml

> vi etc/hadoop/hdfs-site.xml
<configuration>
        <property>
            <name>dfs.namenode.name.dir</name>
            <value>/opt/hadoop/data/nameNode</value>
        </property>
        <property>
            <name>dfs.datanode.data.dir</name>
            <value>/opt/hadoop/data/dataNode</value>
        </property>
        <property>
                <name>dfs.replication</name>
                <value>1</value>
        </property>
        <property>
                <name>dfs.permissions</name>
                <value>false</value>
        </property>
        <property>
                <name>dfs.namenode.http-address</name>
                <value>0.0.0.0:9870</value>
        </property>
        <property>
                <name>dfs.datanode.http.address</name>
                <value>0.0.0.0:9864</value>
        </property>
        <property>
                <name>dfs.datanode.address</name>
                <value>0.0.0.0:9866</value>
        </property>
</configuration>

增加 Slaves

> vi etc/hadoop/workers
hadoop-2
hadoop-3

到worker文件夹，增加一些软链接

> cd ~/tool/
> scp -r hadoop-1:/opt/hadoop-3.2.1 ./hadoop-3.2.1
> sudo ln -s /home/centos/tool/hadoop-3.2.1 /opt/hadoop-3.2.1
> sudo ln -s /opt/hadoop-3.2.1 /opt/hadoop

在 master 机器上

> hdfs namenode -format

在master上启动dfs

> sbin/start-dfs.sh

在master上检查进程

> jps
234 NameNode
235 Jps
236 SecondaryNameNode

在 Slave 上检查进程

> jps
234 DataNode
235 Jps

停止服务

> sbin/stop-dfs.sh

没有授权访问的时候

http://hadoop-1:9870/ 

包含密码访问

http://hadoop-1:9870/explorer.html?user.name=adminpassword

测试上传一个文件

> hdfs dfs -mkdir -p /demo
> hdfs dfs -put ./test.txt /demo
File is there now
> hdfs dfs -ls /demo
Found 1 items
-rw-r--r--   1 centos supergroup       3952 2020-10-08 11:29 /demo/test.txt