超详细的Apache Doris 集群部署方案

# 安装JDK

a) 卸载系统自带的OpenJDK以及相关的java文件
[root@localhost ~]# java -version   # 若有OpenJDK，则卸载
[root@localhost ~]# rpm -qa | grep openjdk
[root@localhost ~]# rpm -qa | grep openjdk | xargs rpm -e --nodeps

b) 上传并解压源码包
[root@localhost ~]# mkdir /usr/java
[root@localhost ~]# tar -xzvf jdk-8u211-linux-x64.tar.gz -C /usr/java/

c) 设置jdk环境变量
[root@localhost ~]# vim /etc/profile  
# 进入文档增加如下内容
export JAVA_HOME=/usr/java/jdk1.8.0_211
export JRE_HOME=/usr/java/jdk1.8.0_211/jre
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$PATH

[root@localhost ~]# source /etc/profile
[root@localhost ~]# java -version

集群时钟保持一致

a) 安装ntp服务
[root@localhost ~]# yum -y install ntp ntpdate

b) 设置时区为上海
[root@localhost ~]# rm -rf /etc/localtime
[root@localhost ~]# ln -s /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

c) 配置时间同步作业
[root@localhost ~]# crontab -e
*/10 * * * * /usr/sbin/ntpdate 172.16.102.2  # 生产环境IP待定

d) 手动同步一下时间
/usr/sbin/ntpdate 172.16.102.2

e) 临时更新机器时间
[root@localhost ~]# date -s "2021-11-03 17:23:45"  # 更改系统时间
[root@localhost ~]# hwclock --show  # 显示硬件时间
[root@localhost ~]# hwclock  -w     # 根据系统时间设置硬件时间

设置打开最大文件数限制

# 参考网页：https://www.e-learn.cn/topic/2959291
[root@localhost ~]# vim /etc/security/limits.conf
# 在末尾增加如下四行
* soft nproc 65535
* hard nproc 65535
* soft nofile 65535
* hard nofile 65535

[root@localhost ~]# vim /etc/sysctl.conf
# 在末尾增加如下一行
fs.file-max = 65535

在/etc/security/limits.d/目录下找是否有“xx-nproc.conf”的文件，一般是“20-nproc.conf”或“90-nproc.conf”
[root@localhost ~]# vim /etc/security/limits.d/20-nproc.conf
# 在末尾增加如下两行
* soft nproc 65535
* hard nproc 65535

# 用户退出，重新登录是配置生效

添加用户：doris/doris

a) 添加新用户
[root@localhost ~]# adduser doris
[root@localhost ~]# passwd doris

b) 增加sudo权限
• 修改文件权限，文件默认为只读
[root@localhost ~]# ll /etc/sudoers
[root@localhost ~]# chmod u+w /etc/sudoers 
• 修改文件，在如下位置增加一行
[root@localhost ~]# vim /etc/sudoers
root      ALL=(ALL)   ALL                # 已有行
doris     ALL=(ALL)   NOPASSWD: ALL      # 执行sudo命令（不需要输入密码）
• 将文件权限还原回只读
[root@localhost ~]# chmod u-w /etc/sudoers
[root@localhost ~]# ll /etc/sudoers

安装MySQL客户端（所有节点）

a）将安装包和JDBC驱动上传，共计5个
01_mysql-community-common-5.7.16-1.el7.x86_64.rpm
02_mysql-community-libs-5.7.16-1.el7.x86_64.rpm
03_mysql-community-libs-compat-5.7.16-1.el7.x86_64.rpm
04_mysql-community-client-5.7.16-1.el7.x86_64.rpm
mysql-connector-java-5.1.27-bin.jar

b) 卸载自带的Mysql-libs（如果之前安装过mysql，要全都卸载掉）
[doris@localhost ~]$ sudo rpm -qa | grep -i -E mysql\|mariadb | xargs -n1 sudo rpm -e --nodeps

c) 安装MySQL依赖和客户端
[doris@localhost ~]$ sudo rpm -ivh 01_mysql-community-common-5.7.16-1.el7.x86_64.rpm
[doris@localhost ~]$ sudo rpm -ivh 02_mysql-community-libs-5.7.16-1.el7.x86_64.rpm
[doris@localhost ~]$ sudo rpm -ivh 03_mysql-community-libs-compat-5.7.16-1.el7.x86_64.rpm
[doris@localhost ~]$ sudo rpm -ivh 04_mysql-community-client-5.7.16-1.el7.x86_64.rpm

安装MySQL的ODBC Driver

# 安装mysql-connector-odbc（所有节点）
[doris@localhost ~]$ sudo yum install mysql-connector-odbc.x86_64
# 上传并解压 mysql-connector-odbc-5.3.13-linux-glibc2.12-x86-64bit.tar.gz
[doris@localhost ~]$ tar -zxvf mysql-connector-odbc-5.3.13-linux-glibc2.12-x86-64bit.tar.gz
[doris@localhost ~]$ mv mysql-connector-odbc-5.3.13-linux-glibc2.12-x86-64bit mysql-connector-odbc-5.3.13
# 复制文件到指定目录
[doris@localhost ~]$ cd mysql-connector-odbc-5.3.13
[doris@localhost mysql-connector-odbc-5.3.13]$ sudo cp -r bin/* /usr/local/bin/
[doris@localhost mysql-connector-odbc-5.3.13]$ sudo cp -r lib/* /usr/local/lib/
# 远程拷贝文件到其它各个节点
[doris@localhost mysql-connector-odbc-5.3.13]$ scp -r bin/*  root@172.16.102.142:/usr/local/bin/
[doris@localhost mysql-connector-odbc-5.3.13]$ scp -r lib/*  root@172.16.102.142:/usr/local/lib/

# 执行myodbc-installer（所有节点）
[doris@localhost mysql-connector-odbc-5.3.13]$ cd /usr/local/bin/
[doris@localhost bin]$ sudo ./myodbc-installer -a -d -n "MySQL ODBC 5.3 Driver" -t "Driver=/usr/local/lib/libmyodbc5w.so"
[doris@localhost bin]$ sudo ./myodbc-installer -a -d -n "MySQL ODBC 5.3" -t "Driver=/usr/local/lib/libmyodbc5a.so"
# 查看是否安装成功
[doris@localhost bin]$ myodbc-installer -d -l
PostgreSQL
MySQL
MySQL ODBC 5.3 Driver
MySQL ODBC 5.3

部署规划

测试环境：

服务	172.16.102.20	172.16.102.21	172.16.102.22	172.16.102.23	172.16.102.24
Doris	FE(Leader)	FE(Follower)	FE(Follower)
Doris	BE	BE	BE	BE	BE
Supervisor	Supervisor	Supervisor	Supervisor	Supervisor	Supervisor

生产环境：

服务	101.188.98.29	101.188.98.30	101.188.98.31	101.188.98.32	101.188.98.33
Doris	FE(Follower)	FE(Follower)	FE(Leader)
Doris	BE	BE	BE	BE	BE
Supervisor	Supervisor	Supervisor	Supervisor	Supervisor	Supervisor

创建目录并赋权（所有节点）

0) 查看看磁盘情况，若有数据盘，则修改对应的路径参数！！！

a) 创建软件安装目录
[doris@localhost ~]$ sudo mkdir -p /opt/soft/
[doris@localhost ~]$ sudo chmod 777 /opt/soft/

b) 创建Doris运行时目录
[doris@localhost ~]$ sudo mkdir /data
[doris@localhost ~]$ sudo chmod 777 /data

[doris@localhost ~]$ mkdir /data/doris-meta
[doris@localhost ~]$ mkdir /data/doris-data
[doris@localhost ~]$ mkdir -p /data/logs/doris/audit
[doris@localhost ~]$ mkdir -p /data/logs/doris/sys

c) 创建Supervisor需要的目录
[doris@localhost ~]$ mkdir -p /data/supervisor/conf.d
[doris@localhost ~]$ mkdir -p /data/logs/supervisor/

安装Doris服务

a) 任一节点上上传PALO-0.14.13.1-release-binary.tar.gz安装包
b) 解压到指定安装目录
[doris@localhost ~]$ tar -zxvf PALO-0.14.13.1-release-binary.tar.gz -C /opt/soft/
c) 修改目录名称
[doris@localhost ~]$ cd /opt/soft/
[doris@redis_01 soft]$ mv PALO-0.14.13.1-release-binary/ doris-palo-0.14.13.1
d) 复制到其它所有节点
[doris@localhost ~]$ scp -r doris-palo-0.14.13.1/ 172.16.102.142:$PWD

e) 修改配置文件
• 修改FE节点的配置
[doris@localhost ~]$ vim /opt/soft/doris-palo-0.14.13.1/fe/conf/fe.conf
LOG_DIR = /data/logs/doris               # 25行
meta_dir = /data/doris-meta              # 42行
priority_networks = 172.16.102.141/24    # 55行
sys_log_dir = /data/logs/doris/sys       # 59行
audit_log_dir = /data/logs/doris/audit   # 62行
• 修改BE节点的配置
[doris@localhost ~]$ vim /opt/soft/doris-palo-0.14.13.1/be/conf/be.conf
PPROF_TMPDIR="/data/logs/doris"          # 18行
priority_networks = 172.16.102.141/24    # 35行
storage_root_path = /data/doris-data     # 50行
sys_log_dir = /data/logs/doris/sys       # 53行
• 修改BE节点的配置
[doris@localhost ~]$ vim /opt/soft/doris-palo-0.14.13.1/be/conf/odbcinst.ini
# Driver from the mysql-connector-odbc package
# Setup from the unixODBC package
[MySQL ODBC 5.3 Unicode Driver]
Description     = ODBC for MySQL
Driver          = /usr/local/lib/libmyodbc5a.so
FileUsage       = 1

安装Supervisor服务

a) 任一节点上上传supervisor.tar.gz安装包并解压
[doris@localhost ~]$ tar -zxvf supervisor.tar.gz
[doris@localhost ~]$ cd supervisor/
[doris@localhost supervisor]$ unzip setuptools-39.1.0.zip
[doris@localhost supervisor]$ tar -zxvf meld3-0.6.10.tar.gz
[doris@localhost supervisor]$ tar -zxvf supervisor-3.3.4.tar.gz

b) 复制到其它所有节点
[doris@localhost ~]$ cd ~/
[doris@localhost ~]$ scp -r supervisor/ 172.16.102.249:$PWD

c) 安装依赖-setuptools（所有节点）
[doris@localhost ~]$ cd /home/doris/supervisor/setuptools-39.1.0/
[doris@localhost setuptools-39.1.0]$ sudo python setup.py build
[doris@localhost setuptools-39.1.0]$ sudo python setup.py install

d) 安装依赖-meld3（所有节点）
[doris@localhost ~]$ cd /home/doris/supervisor/meld3-0.6.10/
[doris@localhost meld3-0.6.10]$ sudo python setup.py install

e) 安装-supervisor（所有节点）
[doris@localhost ~]$ cd /home/doris/supervisor/supervisor-3.3.4/
[doris@localhost supervisor-3.3.4]$ sudo python setup.py install

配置Supervisor

新建Supervisor配置文件（所有节点）

[doris@localhost ~]$ vim /data/supervisor/supervisord.conf
# 配置一下内容
[supervisord]
logfile = /data/logs/supervisor/supervisord.log
logfile_maxbytes = 50MB
logfile_backups=10
loglevel = info
pidfile = /data/supervisor/supervisord.pid
nodaemon = false
#nodaemon = true
minfds = 1024
minprocs = 200
umask = 022
user = doris
identifier = supervisor
directory = /data/supervisor/
nocleanup = true
childlogdir = /data/logs/supervisor/
strip_ansi = false
[supervisorctl]
serverurl = unix:///data/supervisor/supervisord.sock
prompt = supervisor
chmod = 0777
[unix_http_server]
file=/data/supervisor/supervisord.sock
chmod=0700
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[include]
files=conf.d/*.conf

新建Supervisor监控BE进程的配置文件（BE节点）

[doris@localhost ~]$ vim /data/supervisor/conf.d/doris_be.conf
# 配置以下内容
[program:doris_be]
command=/opt/soft/doris-palo-0.14.13.1/be/bin/start_be.sh
process_name=%(program_name)s
numprocs=1
directory=/opt/soft/doris-palo-0.14.13.1/be
umask=022
priority=999
autostart=true
autorestart=true
startsecs=10
startretries=20
exitcodes=0,2
stopsignal=TERM
stopasgroup=true
killasgroup=true
stopwaitsecs=10
user=doris
redirect_stderr=true
stdout_logfile=/data/logs/supervisor/%(program_name)s.log

新建Supervisor监控FE进程的配置文件（FE节点）

[doris@localhost ~]$ vim /data/supervisor/conf.d/doris_fe.conf
# 配置以下内容
[program:doris_fe]
command=/opt/soft/doris-palo-0.14.13.1/fe/bin/start_fe.sh
process_name=%(program_name)s
numprocs=1
directory=/opt/soft/doris-palo-0.14.13.1/fe
umask=022
priority=999
autostart=true
autorestart=true
startsecs=10
startretries=20
exitcodes=0,2
stopsignal=TERM
stopasgroup=true
killasgroup=true
stopwaitsecs=10
user=doris
redirect_stderr=true
stdout_logfile=/data/logs/supervisor/%(program_name)s.log

# 备注：第二行在“FE Follower”节点修改为如下：
# command=/opt/soft/doris-palo-0.14.13.1/fe/bin/start_fe.sh --helper ${FE Leader IP}:9010 
# 并在集群启动完成后需要删除--helper参数

启动Supervisor和Doris

启动Supervisor服务

a) 先启动Supervisor服务（每个节点）
[doris@localhost ~]$ supervisord -c /data/supervisor/supervisord.conf

b) 使用MySQL客户端连接FE Leader，添加节点，初始化集群
[doris@localhost ~]$ mysql -h172.16.102.141 -P9030 -uroot
mysql> ALTER SYSTEM ADD BACKEND "${BE IP}:9050";
mysql> SHOW PROC '/backends';

mysql> ALTER SYSTEM ADD FOLLOWER "${FE Follower IP}:9010";
mysql> SHOW PROC '/frontends';

c) 观察各节点的状态
mysql> SHOW PROC '/backends';
mysql> SHOW PROC '/frontends';

d) 设置参数
mysql> show variables;  -- 查看所有参数
mysql> show variables like '%mem_limit%';  -- 查看BE内存限制
mysql> set global exec_mem_limit = 26843545600;  -- 25G，按实际情况分配
mysql> set global enable_insert_strict=true;     -- 设置完全无法容忍错误
mysql> set global query_timeout=1800;            -- 设置查询超时时间为30分钟
mysql> set global forward_to_master=true;        -- 设置follower指向master，这样连每个FE效果都一样

收尾操作

a) 删除FE Follower节点/data/supervisor/conf.d/doris_fe.conf中“--helper”参数

b) FE Follower节点supervisorctl命令行停止进程
[doris@localhost ~]$ supervisorctl -c /data/supervisor/supervisord.conf
supervisor> stop doris_be
supervisor> stop doris_fe

c) FE Follower节点doris进程是否已关闭
[doris@localhost ~]$ ps -ef | grep doris

d) FE Follower节点supervisorctl命令行重载进程
[doris@localhost ~]$ supervisorctl -c /data/supervisor/supervisord.conf
supervisor> reload
supervisor> status

e) Supervisor服务加入开机启动
echo '/usr/bin/su - doris -c "/usr/bin/supervisord -c /data/supervisor/supervisord.conf"' >> /etc/rc.local
chmod +x /etc/rc.d/rc.local