Sqoop2的安装与使用

1,261 阅读1分钟
1.下载安装
解压: tar -zxvf sqoop-1.99.7-bin-hadoop200.tar.gz

2.配置依赖:
在目录server/conf创建文件catalina.properties.
文件内容:
common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/usr/hadoop3/share/hadoop/common/*.jar,/usr/hadoop3/share/hadoop/common/lib/*.jar,/usr/hadoop3/share/hadoop/hdfs/*.jar,/usr/hadoop3/share/hadoop/hdfs/lib/*.jar,/usr/hadoop3/share/hadoop/mapreduce/*.jar,/usr/hadoop3/share/hadoop/mapreduce/lib/*.jar,/usr/hadoop3/share/hadoop/tools/lib/*.jar,/usr/hadoop3/share/hadoop/yarn/*.jar,/usr/hadoop3/share/hadoop/yarn/lib/*.jar,/usr/hadoop3/share/hadoop/httpfs/tomcat/lib/*.jar
配置环境变量:
#一定要配置,否则会找不到jar
$HADOOP_HOME/share/hadoop/common
$HADOOP_HOME/share/hadoop/hdfs
$HADOOP_HOME/share/hadoop/mapreduce
$HADOOP_HOME/share/hadoop/yarn

export SQOOP_HOME=/usr/sqoop
export LOGDIR=$SQOOP_HOME/logs/
export CATALINA_BASE=/usr/sqoop/server
export SQOOP_SERVER_EXTRA_LIB=$SQOOP_HOME/extra
export PATH=$SQOOP_HOME/bin:$PATH

conf/sqoop.properties配置文件修改:
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/usr/hadoop3/etc/hadoop/

conf/sqoop.properties 使用mysql持久化数据迁移信息
org.apache.sqoop.repository.jdbc.handler=org.apache.sqoop.repository.mysql.MySqlRepositoryHandler
org.apache.sqoop.repository.jdbc.transaction.isolation=READ_COMMITTED
org.apache.sqoop.repository.jdbc.maximum.connections=10
org.apache.sqoop.repository.jdbc.url=jdbc:mysql://192.168.172.103:3306/sqoop2
org.apache.sqoop.repository.jdbc.driver=com.mysql.jdbc.Driver
org.apache.sqoop.repository.jdbc.user=root
org.apache.sqoop.repository.jdbc.password=123456

hadop要开启history
<property>
<name>mapreduce.jobhistory.address</name>
<value>ngmm-hadoop-master:10020</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>

3.下载jdbc驱动:
下载 central.maven.org/maven2/mysq… 到server/lib下

4.启动服务
./sqoop.sh server start

./sqoop.sh client

verify
检查数据:show version -all


sqoop2-tool upgrade
sqoop2-tool verify

日志
set option --name verbose --value true


创建任务
create job -f "mysql-link" -t "hdfs-link”

执行任务
start job -n job1