HIve安装踩坑手册

363

「这是我参与2022首次更文挑战的第7天,活动详情查看:2022首次更文挑战」。 大家好,我是王老狮,hive是基于Hadoop的一个数据仓库工具,所以我们要安装hive首先要安装Hadoop。本文hadoop和hive的安装方式和我踩的一些坑记录起来分享给大家

一、安装环境

JDK 1.8

二、安装Hadoop

1、下载hadoop

 mirror.bit.edu.cn/apache/hado… 选择合适的版本

下载hadoop

wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-3.3.0/hadoop-3.3.0.tar.gz

执行 进行解压,为了方便使用吗,mv进行修改名称

tar -xzvf hadoop-3.3.0.tar.gz 
mv hadoop-3.3.0.tar.gz  hadoop 

2、修改环境变量

将hadoop环境信息写入环境变量中

vim /etc/profile

export HADOOP_HOME=/opt/hadoop

export PATH=$HADOOP_HOME/bin:$PATH

执行source etc/profile使其生效

3、修改配置文件

修改hadoop-env.sh文件,vim etc/hadoop/hadoop-env.sh修改JAVA_HOME信息


export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64

执行hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.0.jar grep input output 'dfs[a-z]',hadoop自带的例子,验证hadoop是否安装成功

三、安装hive

1、下载hive

wget mirror.bit.edu.cn/apache/hive…

解压tar -zxvf apache-hive-3.1.2-bin.tar.gz 

修改名称 mv apache-hive-3.1.2-bin hive

2、修改环境变量

vim /etc/profile

export HIVE_HOME=/opt/hive

export PATH=$MAVEN_HOME/bin:$HIVE_HOME/bin:$HADOOP_HOME/bin:$PATH

source etc/profile

3、修改hivesite 配置


<!-- WARNING!!! This file is auto generated  for   documentation purposes ONLY! -->

<!-- WARNING!!! Any changes you make to  this   file will be ignored by Hive.   -->

<!-- WARNING!!! You must make your changes in hive-site.xml instead.         -->

<!-- Hive Execution Parameters -->



       <!-- 以下配置原配置都有,搜索之后进行修改或者删除后在统一位置添加 -->

   <property>

       <name>javax.jdo.option.ConnectionUserName</name>用户名

       <value>root</value>

   </property>

   <property>

       <name>javax.jdo.option.ConnectionPassword</name>密码

       <value> 123456 </value>

   </property>

  <property>

       <name>javax.jdo.option.ConnectionURL</name>mysql

       <value>jdbc:mysql: //127.0.0.1:3306/hive</value>

   </property>

   <property>

       <name>javax.jdo.option.ConnectionDriverName</name>mysql驱动程序

       <value>com.mysql.jdbc.Driver</value>

   </property>

<property>

   <name>hive.exec.script.wrapper</name>

   <value/>

   <description/>

</property>

复制mysql的驱动程序到hive/lib下面,然后进入/hive/bin 目录执行

 schematool -dbType mysql -initSchema

4、验证是否安装成功

hive --version查看当前版本

hive 看是否进入hive命令操作行,进去的话说明成功

四、错误记录

1、配置文件中存在异常字符

根据指定的

Logging initialized using configuration in jar:file:/opt/hive/lib/hive-common- 3.1 . 2 .jar!/hive-log4j2.properties Async:  true

Exception in thread  "main"   java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D

     at org.apache.hadoop.fs.Path.initialize(Path.java: 263 )

     at org.apache.hadoop.fs.Path.<init>(Path.java: 221 )

     at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java: 710 )

     at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java: 627 )

     at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java: 591 )

     at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java: 747 )

     at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java: 683 )

     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: 62 )

     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java: 43 )

     at java.lang.reflect.Method.invoke(Method.java: 498 )

     at org.apache.hadoop.util.RunJar.run(RunJar.java: 323 )

     at org.apache.hadoop.util.RunJar.main(RunJar.java: 236 )

Caused by: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D

     at java.net.URI.checkPath(URI.java: 1823 )

     at java.net.URI.<init>(URI.java: 745 )

     at org.apache.hadoop.fs.Path.initialize(Path.java: 260 )

     ...  12   more

解决方式:

找到指定的配置文件行数,将描述进行删除

 <property>
    <name>hive.exec.scratchdir</name>
    <value>/tmp/hive</value>
    <description>HDFS root scratch dir for Hive jobs which gets created with write all (733) permission. For each connecting user, an HDFS scratch dir: ${hive.exec.scratchdir}/&lt;username&gt; is created, with ${hive.scratch.dir.permission}.</description>
  </property>

  <property>
    <name>hive.exec.local.scratchdir</name>
    <value>/tmp/hive/local</value>
    <description>Local scratch space for Hive jobs</description>
  </property>

  <property>
    <name>hive.downloaded.resources.dir</name>
    <value>/tmp/hive/resources</value>
    <description>Temporary local directory for added resources in the remote file system.</description>
  </property>

二、guava版本不一致

Exception in thread  "main"   java.lang.RuntimeException: com.ctc.wstx.exc.WstxParsingException: Illegal character entity: expansion character (code  0x8

  at [row,col,system-id]: [ 3215 , 96 , "file:/opt/hive/conf/hive-site.xml" ]

     at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java: 3051 )

     at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java: 3000 )

     at org.apache.hadoop.conf.Configuration.getProps(Configuration.java: 2875 )

     at org.apache.hadoop.conf.Configuration.get(Configuration.java: 1484 )

     at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java: 4996 )

     at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java: 5069 )

     at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java: 5156 )

     at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java: 5104 )

     at org.apache.hive.beeline.HiveSchemaTool.<init>(HiveSchemaTool.java: 96 )

     at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java: 1473 )

     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: 62 )

     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java: 43 )

     at java.lang.reflect.Method.invoke(Method.java: 498 )

     at org.apache.hadoop.util.RunJar.run(RunJar.java: 323 )

     at org.apache.hadoop.util.RunJar.main(RunJar.java: 236 )

Caused by: com.ctc.wstx.exc.WstxParsingException: Illegal character entity: expansion character (code  0x8

  at [row,col,system-id]: [ 3215 , 96 , "file:/opt/hive/conf/hive-site.xml" ]

     at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java: 621 )

     at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java: 491 )

     at com.ctc.wstx.sr.StreamScanner.reportIllegalChar(StreamScanner.java: 2456 )

     at com.ctc.wstx.sr.StreamScanner.validateChar(StreamScanner.java: 2403 )

     at com.ctc.wstx.sr.StreamScanner.resolveCharEnt(StreamScanner.java: 2369 )

     at com.ctc.wstx.sr.StreamScanner.fullyResolveEntity(StreamScanner.java: 1515 )

     at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java: 2828 )

     at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java: 1123 )

     at org.apache.hadoop.conf.Configuration$Parser.parseNext(Configuration.java: 3347 )

     at org.apache.hadoop.conf.Configuration$Parser.parse(Configuration.java: 3141 )

     at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java: 3034 )

     ...  15   more

解决办法:

1、com.google.common.base.Preconditions.checkArgument这个类所在的jar包为:guava.jar

2、hadoop-3.2.1(路径:hadoop\share\hadoop\common\lib)中该jar包为  guava-27.0-jre.jar;而hive-3.1.2(路径:hive/lib)中该jar包为guava-19.0.1.jar

3、将jar包变成一致的版本:删除hive中低版本jar包,将hadoop中高版本的复制到hive的lib中。

再次启动问题得到解决!