ElasticSearch + SearchGuard 集群搭建

SherLock_OD
2,289 阅读3分钟

背景介绍

业务原有ES集群因证书到期，需要进行证书更换操作。由于更新证书后需重启整个集群后才能生效，且业务敏感度较高，无法容忍整个集群停止对外服务。

解决方案

  • OP负责部署一套新ES集群，版本配置同原集群一致

  • RD负责开发程序

    • 将历史数据导入新集群
    • 业务服务实现双写两套ES集群功能

  • 两套集群数据一致后，OP和RD配合切换业务流量

  • OP将原有集群更新证书

ElasticSearch 集群

准备

  • ES版本 5.5.1
  • Search Guard
  • JAVA版本 jdk1.8.0_161

集群配置

3台 Master Node (56核/128G/3.7T普通硬盘)
10.90.104.133
10.90.105.133
10.90.106.133

5台 Data Node (56核/128G/3.7T普通硬盘)
10.90.107.132
10.90.108.132
10.90.109.133
10.90.110.133
10.90.111.133

端口: 9201/9301

系统调整

临时生效
$ sudo sysctl -w vm.max_map_count=262144

永久生效
$ grep vm.max_map_count /etc/sysctl.conf
$ echo vm.max_map_count=262144 >> /etc/sysctl.conf

or 
$ vi /etc/sysctl.conf
vm.max_map_count=262144

安装JAVA

下载 jdk1.8.0_161(若链接404，自行注册账号下载)
$ wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u161-b12/2f38c3b165be4555a1fa6e98c45e0808/jdk-8u161-linux-x64.tar.gz

创建目录, 解压文件
$ mkdir -p /usr/local/java/jdk1.8.0_161/
$ cp jdk-8u161-linux-x64.tar.gz /usr/local/java/jdk1.8.0_161/ 
$ cd /usr/local/java/jdk1.8.0_161/ 
$ tar zxf jdk-8u161-linux-x64.tar.gz && rm -rf jdk-8u161-linux-x64.tar.gz

安装 java，使用 alternatives 切换默认的 java 命令 (command --install <link> <name> <path> <priority>)
$ alternatives --install /usr/bin/java java $JAVA_18_161/bin/java 2
$ alternatives --config java

There are 4 programs which provide 'java'.

  Selection    Command
-----------------------------------------------
   1           /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91-2.6.2.3.el7.x86_64/jre/bin/java
*+ 2           /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.65-3.b17.el7.x86_64/jre/bin/java
   3           /usr/lib/jvm/jre-1.6.0-openjdk.x86_64/bin/java
   4           /usr/local/java/jdk1.8.0_161/bin/java

Enter to keep the current selection[+], or type selection number: 4

安装 jar , javac 
alternatives --install /usr/bin/jar jar /usr/local/java/jdk1.8.0_161/bin/jar 2
alternatives --install /usr/bin/javac javac /usr/local/java/jdk1.8.0_161/bin/javac 2
alternatives --set jar /usr/local/java/jdk1.8.0_161/bin/jar
alternatives --set javac /usr/local/java/jdk1.8.0_161/bin/javac

设置环境变量
$ vim /etc/bashrc
export JAVA_HOME=/usr/local/java/jdk1.8.0_161
export JRE_HOME=/usr/local/java/jdk1.8.0_161/jre
export PATH=/root/perl5/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/java/jdk1.8.0_161/bin:/usr/local/java/jdk1.8.0_161/jre/bin

查看java版本
$ java -version
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)

安装ElasticSearch

创建用户
$ adduser -U -s /sbin/nologin elasticsearch

创建目录
$ mkdir -p /usr/local/elasticsearch
$ mkdir -p /data/elasticsearch/my-project
$ mkdir -p /data/logs/elasticsearch/my-project/{log,pid}

下载es 5.5.1
$ cd /usr/local/elasticsearch
$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.1.tar.gz

解压文件
$ tar zxf elasticsearch-5.5.1.tar.gz && rm -rf elasticsearch-5.5.1.tar.gz
$ mv elasticsearch-5.5.1 my-project-es

安装Search Guard插件

下载插件
$ cd {PROJECT_NAME}
$ wget http://search.maven.org/remotecontent?filepath=com/floragunn/search-guard-5/5.5.1-15/search-guard-5-5.5.1-15.zip

安装插件
$ bin/elasticsearch-plugin install search-guard-5-5.5.1-15.zip

生成证书

下载工具(路径没有特殊要求)
$ git clone https://github.com/floragunncom/search-guard-ssl.git

生成证书
$ cd search-guard-ssl/example-pki-scripts/
$ ./clean.sh
$ ./gen_root_ca.sh kubernetes- kubernetes-                      # COMMAND <CA_PASS> <TS_PASS>
$ ./gen_node_cert.sh k8s-es-node kubernetes- kubernetes-        # COMMAND <NODE_NAME> <KS_PASS> <CA_PASS>
$ ./gen_client_node_cert.sh sgadmin kubernetes- kubernetes-     # COMMAND <CLIENT_NAME> <KS_PASS> <CA_PASS>

拷贝证书到 config 目录下
$ cp *jks /usr/local/elasticsearch/my-project/config/

配置ElasticSearch

备份配置
$ cd /usr/local/elasticsearch/my-project/config/
$ mv elasticsearch.yml{,bak}

Node Master (三台master都按此配置):
主配置: 不做存储，不开启 http
$ vim elasticsearch.yml

cluster.name: my-project-es

node.name: 10.90.104.133
node.master: true
node.data: false
node.ingest: false

path:
    data:
    - /data/elasticsearch/my-project
    logs: /data/logs/elasticsearch/my-project/log

thread_pool.index.queue_size: 1000
thread_pool.bulk.queue_size: 1000

bootstrap.memory_lock: true

network.host: 10.90.104.133
http.port: 9201
transport.tcp.port: 9301

http:
    enabled: false
    compression: true
    cors:
        enabled: true
        allow-origin: "*"
        allow-headers: Authorization

discovery.zen.ping.unicast.hosts: [10.90.104.133:9301,10.90.105.133:9301,10.90.106.133:9301]
discovery.zen.minimum_master_nodes: 2

searchguard.ssl.transport.keystore_filepath: node-k8s-es-node-keystore.jks
searchguard.ssl.transport.keystore_password: kubernetes-
searchguard.ssl.transport.truststore_filepath: truststore.jks
searchguard.ssl.transport.truststore_password: kubernetes-
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.authcz.admin_dn:
- CN=sgadmin,OU=client,O=client,L=test,C=DE

内存配置:
$ vim jvm.options
...
Xms20g
Xmx20g
...


Node Data (后5台数据节点都按此配置):
主配置: 存储数据并开启 http 访问
$ vim elasticsearch.yml

cluster.name: my-project-es

node.name: 10.90.107.132
node.master: false
node.data: true
node.ingest: true

path:
    data:
    - /data/elasticsearch/my-project
    logs: /data/logs/elasticsearch/my-project/log

thread_pool.index.queue_size: 1000
thread_pool.bulk.queue_size: 1000

bootstrap.memory_lock: true

network.host: 10.90.107.132
http.port: 9201
transport.tcp.port: 9301

http:
    enabled: true
    compression: true
    cors:
        enabled: true
        allow-origin: "*"
        allow-headers: Authorization

discovery.zen.ping.unicast.hosts: [10.90.104.133:9301,10.90.105.133:9301,10.90.106.133:9301]
discovery.zen.minimum_master_nodes: 2

searchguard.ssl.transport.keystore_filepath: node-k8s-es-node-keystore.jks
searchguard.ssl.transport.keystore_password: kubernetes-
searchguard.ssl.transport.truststore_filepath: truststore.jks
searchguard.ssl.transport.truststore_password: kubernetes-
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.authcz.admin_dn:
- CN=sgadmin,OU=client,O=client,L=test,C=DE

内存配置:
$ vim jvm.options
...
Xms63g
Xmx63g
...

用户权限及密码

$ cd /usr/local/elasticsearch/my-project/plugins/search-guard-5

生成密码
$ sh tools/hash.sh -p ICdsy1B3j68Tr4w2
$2a$12$BrsaHcr6q0SawxbJtjN46ORVcYsdLH1hQSwVTW4vbUAOwUckpWCbe

修改用户默认密码, 添加用户 myuser
$ vim sgconfig/sg_internal_users.yml

# This is the internal user database
# The hash value is a bcrypt hash and can be generated with plugin/tools/hash.sh
admin:
  hash: $2a$12$BrsaHcr6q0SawxbJtjN46ORVcYsdLH1hQSwVTW4vbUAOwUckpWCbe
  #password is: ICdsy1B3j68Tr4w2
logstash:
  hash: $2a$12$BrsaHcr6q0SawxbJtjN46ORVcYsdLH1hQSwVTW4vbUAOwUckpWCbe
  #password is: ICdsy1B3j68Tr4w2
kibanaserver:
  hash: $2a$12$BrsaHcr6q0SawxbJtjN46ORVcYsdLH1hQSwVTW4vbUAOwUckpWCbe
  #password is: ICdsy1B3j68Tr4w2
kibanaro:
  hash: $2a$12$BrsaHcr6q0SawxbJtjN46ORVcYsdLH1hQSwVTW4vbUAOwUckpWCbe
  #password is: ICdsy1B3j68Tr4w2
  roles:
    - kibanarole
readall:
  hash: $2a$12$BrsaHcr6q0SawxbJtjN46ORVcYsdLH1hQSwVTW4vbUAOwUckpWCbe
  #password is: ICdsy1B3j68Tr4w2
myuser:
  hash: $2a$12$BrsaHcr6q0SawxbJtjN46ORVcYsdLH1hQSwVTW4vbUAOwUckpWCbe
  # password is: ICdsy1B3j68Tr4w2
 
修改角色映射文件(分配admin权限)
$ vim sgconfig/sg_roles_mapping.yml

...

# 仅将新用户添加到该权限下，其他配置保持默认
sg_all_access:
  users:
    - admin
    - myuser
    
...

添加服务管理脚本

$ vim /usr/lib/systemd/system/es-my-project.service

[Unit]
Description=Elasticsearch
Documentation=http://www.elastic.co
Wants=network-online.target
After=network-online.target

[Service]
Environment=ES_HOME=/usr/local/elasticsearch/my-project
Environment=CONF_DIR=/usr/local/elasticsearch/my-project/config
Environment=PID_DIR=/data/logs/elasticsearch/my-project/pid
WorkingDirectory=/usr/local/elasticsearch/my-project

User=elasticsearch
Group=elasticsearch

ExecStartPre=/usr/local/elasticsearch/my-project/bin/elasticsearch-systemd-pre-exec

ExecStart=/usr/local/elasticsearch/my-project/bin/elasticsearch \
        -p \${PID_DIR}/elasticsearch.pid \
        -Edefault.path.conf=\${CONF_DIR}

StandardOutput=journal
StandardError=inherit
LimitNOFILE=65536
LimitNPROC=2048
LimitMEMLOCK=infinity
TimeoutStopSec=0
KillSignal=SIGTERM
KillMode=process
SendSIGKILL=no
SuccessExitStatus=143

[Install]
WantedBy=multi-user.target

修改用户

chown -R elasticsearch:elasticsearch /usr/local/elasticsearch
chown -R elasticsearch:elasticsearch /data/logs/elasticsearch
chown -R elasticsearch:elasticsearch /data/elasticsearch
chown -R elasticsearch:elasticsearch /usr/lib/systemd/system/es-my-project.service

启动服务

添加开机自启
sudo systemctl daemon-reload
sudo systemctl enable es-my-project.service
sudo systemctl start es-my-project

sguard 初始化密码

sguard配置生效，只需要在集群中的任意节点执行即可(推荐固定一台master操作)
更改sgconfig配置就需要执行本脚本，执行过程中不会重启ES集群。

$ cd /usr/local/elasticsearch/my-project/plugins/search-guard-5/tools
$ ./sgadmin.sh --hostname 10.90.104.133 --port 9301 \
    -cd ../sgconfig/ \
    -ks /usr/local/elasticsearch/my-project/config/sgadmin-keystore.jks \
    -kspass kubernetes- \
    -ts ./usr/local/elasticsearch/my-project/config/truststore.jks  \
    -tspass kubernetes- \
    -nhnv --diagnose -cn my-project-es

查看集群状态

$ curl -u myuser:ICdsy1B3j68Tr4w2 -XGET http://10.90.107.132:9201/_cluster/health?pretty

{
  "cluster_name" : "my-project-es",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 4,
  "active_shards" : 9,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Kibana 部署

Dockerfile

FROM docker.xxx.com/library/kibana-searchguard:5.5.1

ENV CLUSTER_NAME=my-project-es \
    SERVER_BASEPATH= \
    ELASTICSEARCH_URL="http://10.90.107.132:9201" \
    XPACK_SECURITY_ENABLED=false \
    XPACK_GRAPH_ENABLED=false \
    XPACK_ML_ENABLED=false \
    ELASTICSEARCH_USERNAME=kibanaserver \
    ELASTICSEARCH_PASSWORD=ICdsy1B3j68Tr4w2

# port is 5601

基于内部仓库镜像构建

构建镜像+生成容器

$ docker build -t my-project-kibana .
$ docker run -d my-project-kibana

登录 kibana

进行初始化设置

更换证书

新生成的证书文件*.jks拷贝到原集群config下，重启ES集群和search guard即可。

avatar
文章
阅读
粉丝
相关推荐
AndResGuard源码分析
578阅读
 · 
3点赞
Elasticsearch 可视化集群工具 Cerebro
1.5k阅读
 · 
8点赞
Rocky9 部署WireGuard
828阅读
 · 
0点赞
Easysearch 跨集群复制实战
4.2k阅读
 · 
2点赞
Elasticsearch 与 OpenSearch：揭开性能差距
1.9k阅读
 · 
9点赞
搭建Elasticsearch集群
3.6k阅读
 · 
22点赞
Vue3.3 + Vite+ Element-Plus + TypeScript 从0到1搭建企业级后台管理系统（前后端开源）
153k阅读
 · 
1.0k点赞
Elasticsearch（ELK）集群环境部署
2.1k阅读
 · 
2点赞
WireGuard 系列文章（一）：什么是 VPN
1.6k阅读
 · 
11点赞
基于 ChatGPT 和 React 搭建 JSON 转 TS 的 Web 应用
254k阅读
 · 
214点赞

友情链接：