【Kylin】安装

格格步入
376 阅读4分钟

这是我参与11月更文挑战的第26天，活动详情查看：2021最后一次更文挑战

一、依赖环境

需要软件如下：

软件版本
Hive2.3.7
Hadoop2.9.2
HBase1.3.1
Zookeeper3.4.14
Kafka1.0.2
Spark2.4.5

集群规划： 2021-05-0120-19-40.png

注意: 要求 hbase 的 hbase.zookeeper.quorum 值必须只能是 host1、host2、...。不允许出现 host:2181。

hbase-site.xml 文件，如下:

<configuration>
    <!-- 指定hbase在HDFS上存储的路径 -->
    <property>
        <name>hbase.rootdir</name>
        <value>hdfs://linux121:9000/hbase</value>
    </property>
    
    <!-- 指定hbase是分布式的 -->
    <property>
        <name>hbase.cluster.distributed</name>
        <value>true</value>
    </property>
    
    <!-- 指定zk的地址,多个用“,”分割 -->
    <property>
        <name>hbase.zookeeper.quorum</name>
        <value>linux121,linux122,linux123</value>
    </property>
</configuration>

二、Kylin 安装配置

下载：archive.apache.org/dist/kylin/…

  1. 解压缩软件 (apache-kylin-3.1.1-bin-hbase1x.tar.gz), 并移动到 /opt/lagou/servers 目录下
# linux122 机子下
cd /opt/lagou/software
tar zxvf apache-kylin-3.1.1-bin-hbase1x.tar.gz
mv apache-kylin-3.1.1-bin-hbase1x/ ../servers/kylin-3.1.1
cd ../servers/kylin-3.1.1
  1. 添加环境变量, 并使之生效
vi /etc/profile

# 增加以下内容
export KYLIN_HOME=/opt/lagou/servers/kylin-3.1.1
export PATH=$PATH:$KYLIN_HOME/bin

# 使生效
source /etc/profile
  1. 增加 kylin 依赖组件的配置
cd $KYLIN_HOME/conf
ln -s $HADOOP_HOME/etc/hadoop/hdfs-site.xml hdfs-site.xml
ln -s $HADOOP_HOME/etc/hadoop/core-site.xml core-site.xml
ln -s $HBASE_HOME/conf/hbase-site.xml hbase-site.xml
ln -s $HIVE_HOME/conf/hive-site.xml hive-site.xml
ln -s $SPARK_HOME/conf/spark-defaults.conf spark-defaults.conf
  1. 修改 kylin.sh
cd $KYLIN_HOME/bin
vim kylin.sh

# 在 kylin.sh 文件头部添加
# 在 set verbose=true to print more logs during start up 后面添加即可

export HADOOP_HOME=/opt/lagou/servers/hadoop-2.9.2
export HIVE_HOME=/opt/lagou/servers/hive-2.3.7
export HBASE_HOME=/opt/lagou/servers/hbase-1.3.1
export SPARK_HOME=/opt/lagou/servers/spark-2.4.5
  1. 检查依赖
$KYLIN_HOME/bin/check-env.sh

# 实操如下：
[root@linux122 bin]# ./check-*
Retrieving hadoop conf dir...
...................................................[PASS]
KYLIN_HOME is set to /opt/lagou/servers/kylin-3.1.1
Checking HBase
...................................................[PASS]
Checking hive
...................................................[PASS]
Checking hadoop shell
...................................................[PASS]
Checking hdfs working dir
mkdir: End of File Exception between local host is: "linux122/172.16.64.122"; destination host is: "linux121":9000; : java.io.EOFException; For more details see:  http://wiki.apache.org/hadoop/EOFException
...................................................[FAIL]
Failed to create /kylin. Please make sure the user has right to access /kylin



# 最后一个 FAIL 是因为 hadoop 端口 9000 与 clickhouse 相冲突了。需要把 clickhouse 停止
# 对应停止 ck 的命令如下：systemctl stop clickhouse-server


# kylin 中执行的：
# 对应执行的命令：hadoop ${hadoop_conf_param} fs -mkdir -p $WORKING_DIR
# 对应执行的命令：hadoop --config /opt/lagou/servers/hadoop-2.9.2/etc/hadoop fs -mkdir -p /kylin

错误处理:

/opt/lagou/servers/kylin-3.1.1/bin/check-port-availability.sh: line 30: netstat: command not found

# 解决:
yum install net-tools

三、启动集群

  1. 启动 zookeeper (linux121)
[root@linux121 shells]# pwd
/root/shells

# 启动
[root@linux121 shells]# zk.sh start
start zookeeper server...
ZooKeeper JMX enabled by default
Using config: /opt/lagou/servers/zookeeper-3.4.14/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
ZooKeeper JMX enabled by default
Using config: /opt/lagou/servers/zookeeper-3.4.14/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
ZooKeeper JMX enabled by default
Using config: /opt/lagou/servers/zookeeper-3.4.14/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

# 关闭
[root@linux121 shells]# zk.sh stop
  1. 启动 HDFS(linux121)
[root@linux121 ~]# start-dfs.sh 
Starting namenodes on [linux121]
linux121: starting namenode, logging to /opt/lagou/servers/hadoop-2.9.2/logs/hadoop-root-namenode-linux121.out
linux121: starting datanode, logging to /opt/lagou/servers/hadoop-2.9.2/logs/hadoop-root-datanode-linux121.out
linux122: starting datanode, logging to /opt/lagou/servers/hadoop-2.9.2/logs/hadoop-root-datanode-linux122.out
linux123: starting datanode, logging to /opt/lagou/servers/hadoop-2.9.2/logs/hadoop-root-datanode-linux123.out
Starting secondary namenodes [linux123]
linux123: starting secondarynamenode, logging to /opt/lagou/servers/hadoop-2.9.2/logs/hadoop-root-secondarynamenode-linux123.out
  1. 启动 YARN (linux123)
[root@linux123 ~]# start-yarn.sh 
starting yarn daemons
starting resourcemanager, logging to /opt/lagou/servers/hadoop-2.9.2/logs/yarn-root-resourcemanager-linux123.out
linux122: starting nodemanager, logging to /opt/lagou/servers/hadoop-2.9.2/logs/yarn-root-nodemanager-linux122.out
linux121: starting nodemanager, logging to /opt/lagou/servers/hadoop-2.9.2/logs/yarn-root-nodemanager-linux121.out
linux123: starting nodemanager, logging to /opt/lagou/servers/hadoop-2.9.2/logs/yarn-root-nodemanager-linux123.out
  1. 启动 HBase (linux121)
[root@linux121 ~]# start-hbase.sh 
starting master, logging to /opt/lagou/servers/hbase-1.3.1/logs/hbase-root-master-linux121.out
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
linux121: starting regionserver, logging to /opt/lagou/servers/hbase-1.3.1/bin/../logs/hbase-root-regionserver-linux121.out
linux122: starting regionserver, logging to /opt/lagou/servers/hbase-1.3.1/bin/../logs/hbase-root-regionserver-linux122.out
linux123: starting regionserver, logging to /opt/lagou/servers/hbase-1.3.1/bin/../logs/hbase-root-regionserver-linux123.out
linux121: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
linux121: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
linux122: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
linux122: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
linux123: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
linux123: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
linux122: starting master, logging to /opt/lagou/servers/hbase-1.3.1/bin/../logs/hbase-root-master-linux122.out
linux122: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
linux122: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
  1. 启动 metastore(linux123)
$ nohup hive --service metastore &

# 实操如下：
[root@linux123 ~]# nohup hive --service metastore &
[1] 2722
[root@linux123 ~]# nohup: ignoring input and appending output to ‘nohup.out’
  1. 启动 Yarn history server(linux121)
$ mr-jobhistory-daemon.sh start historyserver

# 实操如下：
[root@linux121 ~]# mr-jobhistory-daemon.sh start historyserver
starting historyserver, logging to /opt/lagou/servers/hadoop-2.9.2/logs/mapred-root-historyserver-linux121.out

# 查看是否存在：
$ jps
  1. 启动 kylin
$ kylin.sh start

# 实操如下：
[root@linux122 bin]# pwd
/opt/lagou/servers/kylin-3.1.1/bin

[root@linux122 bin]# kylin.sh start
Retrieving hadoop conf dir...
...................................................[PASS]
KYLIN_HOME is set to /opt/lagou/servers/kylin-3.1.1
Checking HBase
...................................................[PASS]
Checking hive
...................................................[PASS]
Checking hadoop shell
...................................................[PASS]
Checking hdfs working dir --config /opt/lagou/servers/hadoop-2.9.2/etc/hadoop , /kylin
...................................................[PASS]
Retrieving Spark dependency...
Optional dependency spark not found, if you need this; set SPARK_HOME, or run bin/download-spark.sh
...................................................[PASS]
Retrieving Flink dependency...
Optional dependency flink not found, if you need this; set FLINK_HOME, or run bin/download-flink.sh
...................................................[PASS]
Retrieving kafka dependency...
Couldn't find kafka home. If you want to enable streaming processing, Please set KAFKA_HOME to the path which contains kafka dependencies.
...................................................[PASS]
/opt/lagou/servers/kylin-3.1.1/bin/check-port-availability.sh: line 30: netstat: command not found

Checking environment finished successfully. To check again, run 'bin/check-env.sh' manually.
Using cached dependency...
skip spark_dependency
  1. 登录 Kylin Web 界面
URLhttp://linux122:7070/kylin
默认用户名ADMIN
默认密码KYLIN

备注: 用户名和密码都必须是大写

如下图所示： 2021-05-0917-07-23.png

avatar
文章
阅读
粉丝