Linux服务器使用Docker部署Hadoop

search hadoop

查找合适的Hadoop镜像

[root@administrator ~]# docker search hadoop
NAME                             DESCRIPTION                                     STARS     OFFICIAL   AUTOMATED
sequenceiq/hadoop-docker         An easy way to try Hadoop                       663                  [OK]
uhopper/hadoop                   Base Hadoop image with dynamic configuration…   103                  [OK]
harisekhon/hadoop                Apache Hadoop (HDFS + Yarn, tags 2.2 - 2.8)     67                   [OK]
bde2020/hadoop-namenode          Hadoop namenode of a hadoop cluster             52                   [OK]
bde2020/hadoop-datanode          Hadoop datanode of a hadoop cluster             41                   [OK]

拉取镜像

[root@administrator ~]# docker pull sequenceiq/hadoop-docker

创建启动容器

docker run -dit --name hadoop --privileged=true -p 50070:50070 -p 8088:8088  -p 9000:9000 sequenceiq/hadoop-docker /etc/bootstrap.sh -bash

进入容器

docker exec -it hadoop /bin/bash

使用Hadoop命令

bash-4.1# hadoop fs -ls
bash: hadoop: command not found

hadoop: command not found，添加环境变量配置信息

PATH=$PATH:/usr/local/hadoop/bin/

再次使用Hadoop命令

bash-4.1# PATH=$PATH:/usr/local/hadoop/bin/
bash-4.1# hadoop fs -ls
Found 1 items
drwxr-xr-x   - root supergroup          0 2015-07-22 11:17 input
bash-4.1# 

bash-4.1# hadoop version
Hadoop 2.7.0
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r d4c8d4d4d203c934e8074b31289a28724c0842cf
Compiled by jenkins on 2015-04-10T18:40Z
Compiled with protoc 2.5.0
From source with checksum a9e90912c37a35c3195d23951fd18f
This command was run using /usr/local/hadoop-2.7.0/share/hadoop/common/hadoop-common-2.7.0.jar
bash-4.1# 

访问WebUI

查看集群状态：IP:8088

浏览HDFS文件：IP:50070

HDFS的API操作测试

    @Test
    public void listFile() throws Exception {
        FileSystem fileSystem = FileSystem.get(new URI("hdfs://IP:9000"), new Configuration());
        //获取所有的文件或者文件夹; 指定遍历的路径，指定是否要递归遍历
        RemoteIterator<LocatedFileStatus> locatedFileStatusRemoteIterator = fileSystem.listFiles(new Path("/"), true);
        while (locatedFileStatusRemoteIterator.hasNext()) {
            // 获取得到每一个文件详细信息
            LocatedFileStatus fileStatus = locatedFileStatusRemoteIterator.next();
            // 获取每一个文件存储路径 名称
            System.out.println(fileStatus.getPath());
            System.out.println(fileStatus.getPath().getName());
        }
        fileSystem.close();
    }
}