环境准备
| 服务器 | 内存 | CPU | 系统 | 内网ip |
|---|---|---|---|---|
| k8s-master | 4G | 2核 | Centos 7.6 | 192.168.0.6 |
| k8s-node1 | 4G | 2核 | Centos 7.6 | 192.168.0.47 |
| k8s-node2 | 4G | 2核 | Centos 7.6 | 192.168.0.154 |
配置hosts
192.168.0.6 k8s-master
192.168.0.47 k8s-node1
192.168.0.154 k8s-node2
配置免密登录
1.生成秘钥
ssh-keygen -t rsa
2.将公钥复制到对应免密登录机器上
ssh-copy-id -i ~/.ssh/id_rsa.pub k8s-master
ssh-copy-id -i ~/.ssh/id_rsa.pub k8s-node1
ssh-copy-id -i ~/.ssh/id_rsa.pub k8s-node2
3.测试免密登录
ssh k8s-master
ssh k8s-node1
ssh k8s-node2
安装JDK
1. 查看可用jdk版本
yum search jdk
2. 安装jdk
yum install -y java-1.8.0-openjdk java-1.8.0-openjdk-devel
3. 卸载jdk
yum remove java-1.8.0-openjdk java-1.8.0-openjdk-devel -y
安装hadoop
1.访问hadoop官网下载地址
https://hadoop.apache.org/releases.html
2.选用下载版本hadoop-3.2.1、 二进制下载Binary download
在k8s-master上执行
mkdir /usr/local/bigdata
cd /usr/local/bigdata
wget https://mirror.bit.edu.cn/apache/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz
tar -zxvf hadoop-3.2.1.tar.gz
mv hadoop-3.2.1 hadoop
vi readme.md
haddoop 版本:3.2.1
环境变量配置:
修改core-site.xml
修改hdfs-site.xml
修改mapred-site.xml
修改yarn-site.xml
3.配置环境变量
vi /etc/profile
快捷命令 G o shift+insert
export HADOOP_HOME=/usr/local/bigdata/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
source /etc/profile
检测环境变量是否配置成功
java -version
cd $HADOOP_HOME
4.修改配置文件
core-site.xml
cd /usr/local/bigdata/hadoop/etc/hadoop
vi core-site.xml
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/usr/local/bigdata/hadoop/tmp</value>
<description>Abase for other temporary directories.</description>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://k8s-master:9000</value>
</property>
</configuration>
hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/local/bigdata/hadoop/hdfs/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/local/bigdata/hadoop/hdfs/data</value>
</property>
</configuration>
mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>$HADOOP_HOME/share/hadoop/mapreduce/*:$HADOOP_HOME/share/hadoop/mapreduce/lib/*</value>
</property>
</configuration>
yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>k8s-master</value>
</property>
<property>
<name>yarn.nodemanager.env-whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME</value>
</property>
</configuration>
workers
vi workers
k8s-master
k8s-node1
k8s-node2
配置含义见官网：Configuration
hadoop.apache.org/docs/stable…
5.配置namenode
yum install zip
zip -r hadoop.zip hadoop/
scp hadoop.zip root@k8s-node1:/usr/local/hadoop/
scp hadoop.zip root@k8s-node2:/usr/local/hadoop/
在k8s-node1 k8s-node2 重复安装JDK配置,环境变量步骤
cd /usr/local/hadoop
unzip hadoop.zip
6.启动集群
格式化hdfs
bin/hadoop namenode -format
启动集群
sbin/start-all.sh
7.配置root用户启动报错
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [k8s-master]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
Starting resourcemanager
ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
Starting nodemanagers
ERROR: Attempting to operate on yarn nodemanager as root
ERROR: but there is no YARN_NODEMANAGER_USER defined. Aborting operation.
解决:
sbin/start-dfs.sh stop-dfs.sh 添加
HDFS_DATANODE_USER=root
HADOOP_SECURE_DN_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
sbin/start-yarn.sh stop-yarn.sh 添加
YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root
8.k8s-master: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
没有配置自身的免密的登录:ssh-copy-id -i ~/.ssh/id_rsa.pub k8s-master
9.JAVA_HOME 未设置报错
k8s-master: ERROR: JAVA_HOME is not set and could not be found.
k8s-node2: ERROR: JAVA_HOME is not set and could not be found.
k8s-node1: ERROR: JAVA_HOME is not set and could not be found
三台节点执行
cd /usr/local/bigdata/hadoop/etc/hadoop
vi hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
10.解决报错 再次启动集群
sbin/start-all.sh
日志打印:
WARNING: HADOOP_SECURE_DN_USER has been replaced by HDFS_DATANODE_SECURE_USER. Using value of HADOOP_SECURE_DN_USER.
Starting namenodes on [k8s-master]
Last login: Wed Jan 20 23:34:59 CST 2021 on pts/0
Starting datanodes
Last login: Wed Jan 20 23:40:33 CST 2021 on pts/0
k8s-node1: WARNING: /usr/local/bigdata/hadoop/logs does not exist. Creating.
k8s-node2: WARNING: /usr/local/bigdata/hadoop/logs does not exist. Creating.
Starting secondary namenodes [k8s-master]
Last login: Wed Jan 20 23:40:35 CST 2021 on pts/0
Starting resourcemanager
Last login: Wed Jan 20 23:40:38 CST 2021 on pts/0
resourcemanager is running as process 21204. Stop it first.
Starting nodemanagers
Last login: Wed Jan 20 23:40:44 CST 2021 on pts/0
可以查看 /usr/local/bigdata/hadoop/logs 文件夹下日志
ssh k8s-node1
tail -f hadoop-root-datanode-k8s-node1.log
报错: Retrying connect to server: k8s-master/192.168.0.6:9000
查看k8s-master 9000端口是否存在进程
lsof -i:9000
java 23342 root 277u IPv4 57218 0t0 TCP localhost:cslistener (LISTEN)
java 23342 root 287u IPv4 57703 0t0 TCP localhost:cslistener->localhost:40644 (ESTABLISHED)
java 23518 root 315u IPv4 58624 0t0 TCP localhost:40644->localhost:cslistener (ESTABLISHED)
进程存在
vi /etc/hosts
注释掉 #127.0.0.1 k8s-master
#127.0.0.1 k8s-node1
#127.0.0.1 k8s-node2
11.查看集群状态 hadoop dfsadmin -report
WARNING: Use of this script to execute dfsadmin is deprecated.
WARNING: Attempting to execute replacement "hdfs dfsadmin" instead.
Configured Capacity: 84280803328 (78.49 GB)
Present Capacity: 72120131584 (67.17 GB)
DFS Remaining: 72120082432 (67.17 GB)
DFS Used: 49152 (48 KB)
DFS Used%: 0.00%
Replicated Blocks:
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
Low redundancy blocks with highest priority to recover: 0
Pending deletion blocks: 0
Erasure Coded Block Groups:
Low redundancy block groups: 0
Block groups with corrupt internal blocks: 0
Missing block groups: 0
Low redundancy blocks with highest priority to recover: 0
Pending deletion blocks: 0
-------------------------------------------------
Live datanodes (2):
Name: 192.168.0.154:9866 (k8s-node2)
Hostname: k8s-node2
Decommission Status : Normal
Configured Capacity: 42140401664 (39.25 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 4114890752 (3.83 GB)
DFS Remaining: 35861278720 (33.40 GB)
DFS Used%: 0.00%
DFS Remaining%: 85.10%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Wed Jan 20 23:56:42 CST 2021
Last Block Report: Wed Jan 20 23:53:06 CST 2021
Num of Blocks: 0
Name: 192.168.0.47:9866 (k8s-node1)
Hostname: k8s-node1
Decommission Status : Normal
Configured Capacity: 42140401664 (39.25 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 3717365760 (3.46 GB)
DFS Remaining: 36258803712 (33.77 GB)
DFS Used%: 0.00%
DFS Remaining%: 86.04%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Wed Jan 20 23:56:42 CST 2021
Last Block Report: Wed Jan 20 23:53:06 CST 2021
Num of Blocks: 0
登录华为云开放安全组端口 9870 和 8088
访问 xxx.xxx.xxx.xxx:9870/ 替换成公网ip
访问 xxx.xxx.xxx.xxx:8088/ 替换公网ip
参考：
hadoop.apache.org/docs/stable…
下一篇
Zookeeper+Hbase安装 : juejin.cn/post/692065…
