搭建Hive所遇过的坑

1,723 阅读2分钟

1.启动hive时报错

java.lang.ExceptionInInitializerError
      at java.lang.Class.forName0(Native Method)
      at java.lang.Class.forName(Class.java:190)
      at org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher.init(JDBCStatsPublisher.java:265)
      at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:412)
  Caused by: java.lang.SecurityException: sealing violation: package org.apache.derby.impl.jdbc.authentication is sealed
      at java.net.URLClassLoader.getAndVerifyPackage(URLClassLoader.java:388)
      at java.net.URLClassLoader.defineClass(URLClassLoader.java:417)

解决方案:

将mysql-connector-java-5.1.6-bin.jar包导入到$HIVE_HOME/lib目录下

2.启动hive时报错:

[ERROR] Terminal initialization failed; falling back to unsupported
  java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected
      at jline.TerminalFactory.create(TerminalFactory.java:101)
      at jline.TerminalFactory.get(TerminalFactory.java:158)
  Exception in thread "main" java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected
      at jline.console.ConsoleReader.<init>(ConsoleReader.java:230)
      at jline.console.ConsoleReader.<init>(ConsoleReader.java:221)

解决方案:

将当前hive版本的$HIVE_HOME/lib目录下的jline-2.12.jar包拷贝到$HADOOP_HOME/share/hadoop/yarn/lib目录下, 并将旧版本的Hive的Jline包从$HADOOP_HOME/etc/hadoop/yarn/lib目录下删除

3.启动hive时报错

Exception in thread "main" java.lang.RuntimeException: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
  Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
        at org.apache.hadoop.fs.Path.initialize(Path.java:206)
        at org.apache.hadoop.fs.Path.<init>(Path.java:172)
  Caused by: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
        at java.net.URI.checkPath(URI.java:1804)
        at java.net.URI.<init>(URI.java:752)
        at org.apache.hadoop.fs.Path.initialize(Path.java:203)
        ... 11 more

解决方案:

1.查看hive-site.xml配置,会看到配置值含有"system:java.io.tmpdir"的配置项
  2.新建文件夹${HIVE_HOME}/hive/logs
  3.将含有"system:java.io.tmpdir"的配置项的值修改为${HIVE_HOME}/hive/logs
  即: 新添属性为
  <property>
      <name>hive.exec.local.scratchdir</name>
      <value>${HIVE_HOME}/logs/HiveJobsLog</value>
      <description>Local scratch space for Hive jobs</description>
  </property>
  <property>
      <name>hive.downloaded.resources.dir</name>
      <value>${HIVE_HOME}/logs/ResourcesLog</value>
      <description>Temporary local directory for added resources in the remote file system.</description>
  </property>
  <property>
      <name>hive.querylog.location</name>
      <value>${HIVE_HOME}/logs/HiveRunLog</value>
      <description>Location of Hive run time structured log file</description>
  </property>
  <property>
      <name>hive.server2.logging.operation.log.location</name>
      <value>${HIVE_HOME}/logs/OpertitionLog</value>
      <description>Top level directory where operation logs are stored if logging functionality is enabled</description>
  </property>

4.启动hive时报错

Caused by: java.sql.SQLException: Access denied for user 'root'@'master' (using password: YES)
      at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:946)
      at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2870)
      at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:812)
      at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:3269)

解决方案:

 mysql密码不正确, 查看hive-site.xml配置与mysql的密码是否一致

5.操作表数据时(向表中导入数据), 报错:

FAILED: RuntimeException org.apache.hadoop.security.AccessControlException: Permission denied: user=services02, access=EXECUTE, inode="/tmp":services01:supergroup:drwx------

解决方案:

  user=services02与inode="/tmp":services01:supergroup不同时,说明hive登录的主机与HDFS的active状态的主机不一样
  应把user=services02的主机变为HDFS的active状态的主机.

二.扩展Parquet功能:

  • 1.创建存储格式为parquet时,建表语句为: (Hive版本为0.12)
create table parquet_test(x int, y string) 
  row format serde 'parquet.hive.serde.ParquetHiveSerDe'    
  stored as inputformat 'parquet.hive.DeprecatedParquetInputFormat'                    
  outputformat 'parquet.hive.DeprecatedParquetOutputFormat';

报错:

FAILED: SemanticException [Error 10055]: Output Format must implement 
  HiveOutputFormat, otherwise it should be either IgnoreKeyTextOutputFormat or 
  SequenceFileOutputFormat

*解决方案

因为parquet.hive.DeprecatedParquetOutputFormat类并没有在Hive的CLASSPATH中配置
  单独下载parquet-hive-1.2.5.jar包(此类属于$IMPALA_HOME/lib目录下), 在$HIVE_HOME/lib目录下建立个软链就可以了
  cd $HIVE_HOME/lib
  ln -s $/home/hadoop/soft/gz.zip/parquet-hive-1.2.5.jar

2.继续提交创表语句: (Hive版本为0.12)

create table parquet_test(x int, y string) 
 row format serde 'parquet.hive.serde.ParquetHiveSerDe'    
 stored as inputformat 'parquet.hive.DeprecatedParquetInputFormat'                    
 outputformat 'parquet.hive.DeprecatedParquetOutputFormat';

报错:

Exception in thread "main" java.lang.NoClassDefFoundError: parquet/hadoop/api/WriteSupport
 at java.lang.Class.forName0(Native Method)
 at java.lang.Class.forName(Class.java:247)

解决方案:

通过yum下载Parquet
sodu yum -y install parquet
下载parquet的jar包在/usr/lib/parquet目录下, 将/usr/lib/parquet目录下的所有jar(除javadoc.jar和sources.jar外)拷贝到$HIVE_HOME/lib目录下.
若yum无法下载parquet资源包, 则这是需要配置yum源, 请自行百度寻找资料