实战：ubuntu 18 离线安装ambari HDPubuntu 18 离线安装ambari HDP 基础环境修改h

ubuntu 18 离线安装ambari HDP

基础环境

修改hosts

修改

 ulimit -n 10000

做免密

sshScript.sh

#! /bin/bash
ssh-keygen -t rsa
# 192.168.111 改为集群所在网段（和hosts中配置对应）
ALL_CLIENTS=`cat /etc/hosts| grep "192.168.111" | awk '{print $2}'`
for client in $ALL_CLIENTS
do
        echo "=============copy-ssh-id $client============="
        ssh-copy-id $client
done

修改UMASK

#为当前登录会话设置umask：
umask 0022
# check umask
umask
#永久设置umask
echo umask 0022 >> /etc/profile

关闭防火墙

sudo ufw disable
sudo iptables -X
sudo iptables -t nat -F
sudo iptables -t nat -X
sudo iptables -t mangle -F
sudo iptables -t mangle -X
sudo iptables -P INPUT ACCEPT
sudo iptables -P FORWARD ACCEPT
sudo iptables -P OUTPUT ACCEPT

安装jdk

installJDK.sh

#! /bin/bash
echo "******** JDK ********"
path=/opt/
userAndGroup=ocdp:ocdp
tar -xvf ${path}jdk-8u201-linux-x64.tar.gz -C ${path}
chown -R ${userAndGroup} ${path}jdk1.8.0_201
ln -s ${path}jdk1.8.0_201 ${path}jdk
chown -R ${userAndGroup} ${path}jdk
# 我这里是没有配置过环境变量的新机器，所以可以直接在/etc/profile 后追加
# 如果已经有环境变量可根据具体情况使用sed -i等命令配置
echo "export JAVA_HOME=${path}jdk" >> /etc/profile
echo "export PATH=\$PATH:\$JAVA_HOME/bin" >> /etc/profile
echo "export CLASSPATH=.:\$JAVA_HOME/lib/dt.jar:\$JAVA_HOME/lib/tools.jar" >> /etc/profile
rm -rf /usr/bin/java
rm -rf /usr/bin/javac
#hostname=`"hostname"`
#ssh $hostname "source /etc/profile;java -version;exit"
source /etc/profile
echo "******* JDK installation completed ********"

修改时区

timedatectl set-timezone Asia/Shanghai

build-essential（基础包）

sudo apt-get  -o Acquire::http::proxy="http://10.41.99.41:808/" install build-essential

python2.7

sudo apt-get  -o Acquire::http::proxy="http://10.41.99.41:808/" install python2.7
sudo apt-get  -o Acquire::http::proxy="http://10.41.99.41:808/" install python-pip

apache2与hdp源

安装apache2

sudo apt-get  -o Acquire::http::proxy="http://10.41.99.41:808/" install apache2

配置apache2

vim  /etc/apache2/sites-available/000-default.confDocumentRoot
#DocumentRoot /hadoop/var/www/html
sudo vim  /etc/apache2/apache2.conf
<Directory /hadoop/var/www/>
        Options Indexes FollowSymLinks
        AllowOverride None
        Require all granted
</Directory>
cp /var/www/html/index.html /hadoop/var/www/html/
#重启apache2
systemctl restart apache2

解压相关tar包


tar -zxvf ambari-2.7.4.0-ubuntu18.tar.gz -C /hadoop/var/www/html/

tar -zxvf HDP-3.1.4.0-ubuntu18-deb.tar.gz -C /hadoop/var/www/html/

tar -zxvf HDP-GPL-3.1.4.0-ubuntu18-gpl.tar.gz -C /hadoop/var/www/html/

tar -zxvf HDP-UTILS-1.1.0.22-ubuntu18.tar.gz -C /hadoop/var/www/html/

配置amabri源

/etc/apt/sources.list.d/ambari.list

deb http://10.41.99.212/ambari/ubuntu18/2.7.4.0-118/ Ambari main

mysql-client

sudo apt-get  -o Acquire::http::proxy="http://10.41.99.41:808/" install mysql-client
sudo apt-get  -o Acquire::http::proxy="http://10.41.99.41:808/" install libmysqlclient-dev

mysql ambari，hive，ranger元数据库创建

-- 创建ambari专用的数据库
CREATE DATABASE `ambari`;
CREATE USER 'ambari'@'%' IDENTIFIED BY 'TBbigdata2023';
grant all privileges on ambari.* to 'ambari'@'%';
-- 刷新权限
flush privileges;

-- 创建hive专用的数据库
CREATE DATABASE `hive`;
CREATE USER 'hive'@'%' IDENTIFIED BY 'TBhive20@#';
grant all privileges on hive.* to 'hive'@'%';
-- 刷新权限
flush privileges;

-- 创建ranger专用的数据库
CREATE DATABASE `ranger`;
CREATE USER 'ranger'@'%' IDENTIFIED BY 'TBranger20@#';
grant all privileges on ranger.* to 'ranger'@'%';
-- 刷新权限
flush privileges;

安装mabri

#安装
apt-get install ambari-server
#设置数据库驱动
ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar
#初始化mysql ambari库
mysql -h 10.41.115.138 -uambari -p -D ambari
mysql> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
#启动
ambari-server start

HDP镜像源

在/etc/apt/sources.list.d/路径下分别添加 ambari.list 和 ambari-hdp.list

cd /etc/apt/sources.list.d
# ambari.list
deb http://10.41.99.212/ambari/ubuntu18/2.7.4.0-118/ Ambari main
# ambari-hdp.list
deb http://192.168.72.17/HDP/ubuntu18/3.1.0.0-78/ HDP main
deb http://192.168.72.17/HDP-GPL/ubuntu18/3.1.0.0-78/ HDP-GPL main
deb http://192.168.72.17/HDP-UTILS/ubuntu18/1.1.0.22/ HDP-UTILS main

需要添加公钥不然apt-get update 会报错

## 报错
Get:1 http://10.41.99.212/ambari/ubuntu18/2.7.4.0-118 Ambari InRelease [7,393 B]
Err:1 http://10.41.99.212/ambari/ubuntu18/2.7.4.0-118 Ambari InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY B9733A7A07513CAD
Reading package lists... Done
W: GPG error: http://10.41.99.212/ambari/ubuntu18/2.7.4.0-118 Ambari InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY B9733A7A07513CAD
E: The repository 'http://10.41.99.212/ambari/ubuntu18/2.7.4.0-118 Ambari InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

因为是离线环境，我们可以编辑一个hdp.pgp添加

#有网可以
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys B9733A7A07513CAD
### 编辑一个hdp.pgp添加
sudo apt-key add hdp.pgp
apt-get update

hdp.pgp

ubuntu18 的ambari 安装依赖postgresql

sudo apt-get  -o Acquire::http::proxy="http://10.41.115.204:808/" install postgresql

安装

sudo apt-get update
sudo apt-get install ambari-server

问题

ambari-server连接mysql报错

连接mysql报错，说连接过多，实际是ssl引起的问题需要修改/etc/ambari-server/conf/ambari.properties

WARN [C3P0PooledConnectionPoolManager[identityToken->2ti5xsaz1uhjd2o14fjqcx|6f10d5b6]-HelperThread-#1] BasicResourcePool:223 - com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@77b62b6b -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:
java.sql.SQLException: null,  message from server: "Host '10.41.115.144' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'"
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:964)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:897)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:886)
        at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1040)
        at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2253)
        at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2284)
        at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2083)
        at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:806)
        at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
        at sun.reflect.GeneratedConstructorAccessor46.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
......

修改ambari.properties

server.jdbc.url=jdbc:mysql://10.41.115.138:3306/ambari?useUnicode=true&characterEncoding=utf8&useSSL=false

同上因为useSSL=false导致ranger admin setup失败

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/RANGER/package/scripts/ranger_admin.py", line 258, in <module>
    RangerAdmin().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/RANGER/package/scripts/ranger_admin.py", line 114, in start
    self.configure(env, upgrade_type=upgrade_type, setup_db=params.stack_supports_ranger_setup_db_on_start)
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/RANGER/package/scripts/ranger_admin.py", line 149, in configure
    setup_ranger_xml.setup_ranger_db()
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/RANGER/package/scripts/setup_ranger_xml.py", line 286, in setup_ranger_db
    user=params.unix_user,
  File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
    returns=self.resource.returns)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'ambari-python-wrap /usr/hdp/current/ranger-admin/db_setup.py' returned 1. 2023-11-21 15:17:12,056  [I] DB FLAVOR :MYSQL
2023-11-21 15:17:12,057  [I] --------- Verifying Ranger DB connection ---------
2023-11-21 15:17:12,057  [I] Checking connection..
2023-11-21 15:17:12,057  [JISQL] /hadoop/jdk1.8.0_361/bin/java  -cp /usr/hdp/current/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://10.41.115.138/ranger -u 'ranger' -p '********' -noheader -trim -c ;  -query "select 1;"
Tue Nov 21 15:17:12 CST 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
SQLException : SQL state: 08S01 com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

解决

根据报错信息修改/usr/hdp/current/ranger-admin/db_setup.py

#这个参数没有加?useSSL=false
 -cstring jdbc:mysql://10.41.115.138/ranger

在db_setup.py中发现是用get_jisql_cmd方法获取最后执行的cmd命令找到，发现db_ssl_param这个拼接参数为空，直接加上启动成功

#在第737行修改
db_ssl_param='?useSSL=false'

注意

我这里是mysql unix 所以在这个位置修改，如果是orcle或者windos请找到对应位置修改