Elasticsearch：使用 docker compose 来实现热温冷架构的 Elasticsearch 集群

Elasticsearch lv-5

在我们的许多练习中，我需要模拟索引生命管理周期的环境。在这样的集群中，我们需要模拟出热（hot），温（warm）及冷（cold）的架构。那么我们该如何进行模拟呢？

我们参考之前的文章 “Elasticsearch：使用 Docker compose 来一键部署 Elastic Stack 8.x”。我们使用 docker compose 来进行部署。

首先，我们按照如下的步骤来进行修改。

.env

.env 文件设置运行 docker-compose.yml 配置文件时使用的环境变量。 确保使用 ELASTIC_PASSWORD 和 KIBANA_PASSWORD 变量为 elastic 和 kibana_system 用户指定密码。 这些变量由 docker-compose.yml 文件引用。

`

1.  # Password for the 'elastic' user (at least 6 characters)
2.  ELASTIC_PASSWORD=password

4.  # Password for the 'kibana_system' user (at least 6 characters)
5.  KIBANA_PASSWORD=password

7.  # Version of Elastic products
8.  STACK_VERSION=8.5.1

10.  # Set the cluster name
11.  CLUSTER_NAME=docker-cluster

13.  # Set to 'basic' or 'trial' to automatically start the 30-day trial
14.  LICENSE=basic
15.  #LICENSE=trial

17.  # Port to expose Elasticsearch HTTP API to the host
18.  ES_PORT=9200
19.  #ES_PORT=127.0.0.1:9200

21.  # Port to expose Kibana to the host
22.  KIBANA_PORT=5601
23.  #KIBANA_PORT=80

25.  # Increase or decrease based on the available host memory (in bytes)
26.  MEM_LIMIT=1073741824

28.  # Project namespace (defaults to the current folder name if not set)
29.  #COMPOSE_PROJECT_NAME=myproject

`![](https://csdnimg.cn/release/blogv2/dist/pc/img/newCodeMoreWhite.png)
复制代码

在上面，我们设定 Elastic Stack 的版本为最新的 8.5.1。在这个文件中，我们同时也设定 elastic 这个超级用户的密码。

docker-compose.yml

这个 docker-compose.yml 文件创建了一个启用了身份验证和网络加密的三节点安全 Elasticsearch 集群，以及一个安全连接到它的 Kibana 实例。在我们的设置中，我们分别设置三个节点为 data_hot，data_warm 及 data_cold。有关 data tiers 的介绍，请阅读我的另外一篇文章 “Elastic：Data tiers 介绍及索引生命周期管理 - 7.10 之后版本”。

修改后的 docker-compose.yml 文件如下：

docker-compose.yml

`

1.  version: "2.2"

3.  services:
4.    setup:
5.      image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
6.      volumes:
7.        - certs:/usr/share/elasticsearch/config/certs
8.      user: "0"
9.      command: >
10.        bash -c '
11.          if [ x${ELASTIC_PASSWORD} == x ]; then
12.            echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
13.            exit 1;
14.          elif [ x${KIBANA_PASSWORD} == x ]; then
15.            echo "Set the KIBANA_PASSWORD environment variable in the .env file";
16.            exit 1;
17.          fi;
18.          if [ ! -f certs/ca.zip ]; then
19.            echo "Creating CA";
20.            bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
21.            unzip config/certs/ca.zip -d config/certs;
22.          fi;
23.          if [ ! -f certs/certs.zip ]; then
24.            echo "Creating certs";
25.            echo -ne \
26.            "instances:\n"\
27.            "  - name: es01\n"\
28.            "    dns:\n"\
29.            "      - es01\n"\
30.            "      - localhost\n"\
31.            "    ip:\n"\
32.            "      - 127.0.0.1\n"\
33.            "  - name: es02\n"\
34.            "    dns:\n"\
35.            "      - es02\n"\
36.            "      - localhost\n"\
37.            "    ip:\n"\
38.            "      - 127.0.0.1\n"\
39.            "  - name: es03\n"\
40.            "    dns:\n"\
41.            "      - es03\n"\
42.            "      - localhost\n"\
43.            "    ip:\n"\
44.            "      - 127.0.0.1\n"\
45.            > config/certs/instances.yml;
46.            bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
47.            unzip config/certs/certs.zip -d config/certs;
48.          fi;
49.          echo "Setting file permissions"
50.          chown -R root:root config/certs;
51.          find . -type d -exec chmod 750 \{\} \;;
52.          find . -type f -exec chmod 640 \{\} \;;
53.          echo "Waiting for Elasticsearch availability";
54.          until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
55.          echo "Setting kibana_system password";
56.          until curl -s -X POST --cacert config/certs/ca/ca.crt -u elastic:${ELASTIC_PASSWORD} -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
57.          echo "All done!";
58.        '
59.      healthcheck:
60.        test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
61.        interval: 1s
62.        timeout: 5s
63.        retries: 120

65.    es01:
66.      depends_on:
67.        setup:
68.          condition: service_healthy
69.      image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
70.      volumes:
71.        - certs:/usr/share/elasticsearch/config/certs
72.        - esdata01:/usr/share/elasticsearch/data
73.      ports:
74.        - ${ES_PORT}:9200
75.      environment:
76.        - node.name=es01
77.        - node.roles=data_hot,data_content,master,ingest
78.        - cluster.name=${CLUSTER_NAME}
79.        - cluster.initial_master_nodes=es01,es02,es03
80.        - discovery.seed_hosts=es02,es03
81.        - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
82.        - bootstrap.memory_lock=true
83.        - xpack.security.enabled=true
84.        - xpack.security.http.ssl.enabled=true
85.        - xpack.security.http.ssl.key=certs/es01/es01.key
86.        - xpack.security.http.ssl.certificate=certs/es01/es01.crt
87.        - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
88.        - xpack.security.http.ssl.verification_mode=certificate
89.        - xpack.security.transport.ssl.enabled=true
90.        - xpack.security.transport.ssl.key=certs/es01/es01.key
91.        - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
92.        - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
93.        - xpack.security.transport.ssl.verification_mode=certificate
94.        - xpack.license.self_generated.type=${LICENSE}
95.      mem_limit: ${MEM_LIMIT}
96.      ulimits:
97.        memlock:
98.          soft: -1
99.          hard: -1
100.      healthcheck:
101.        test:
102.          [
103.            "CMD-SHELL",
104.            "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
105.          ]
106.        interval: 10s
107.        timeout: 10s
108.        retries: 120

110.    es02:
111.      depends_on:
112.        - es01
113.      image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
114.      volumes:
115.        - certs:/usr/share/elasticsearch/config/certs
116.        - esdata02:/usr/share/elasticsearch/data
117.      environment:
118.        - node.name=es02
119.        - node.roles=data_warm,data_content,master,ingest
120.        - cluster.name=${CLUSTER_NAME}
121.        - cluster.initial_master_nodes=es01,es02,es03
122.        - discovery.seed_hosts=es01,es03
123.        - bootstrap.memory_lock=true
124.        - xpack.security.enabled=true
125.        - xpack.security.http.ssl.enabled=true
126.        - xpack.security.http.ssl.key=certs/es02/es02.key
127.        - xpack.security.http.ssl.certificate=certs/es02/es02.crt
128.        - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
129.        - xpack.security.http.ssl.verification_mode=certificate
130.        - xpack.security.transport.ssl.enabled=true
131.        - xpack.security.transport.ssl.key=certs/es02/es02.key
132.        - xpack.security.transport.ssl.certificate=certs/es02/es02.crt
133.        - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
134.        - xpack.security.transport.ssl.verification_mode=certificate
135.        - xpack.license.self_generated.type=${LICENSE}
136.      mem_limit: ${MEM_LIMIT}
137.      ulimits:
138.        memlock:
139.          soft: -1
140.          hard: -1
141.      healthcheck:
142.        test:
143.          [
144.            "CMD-SHELL",
145.            "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
146.          ]
147.        interval: 10s
148.        timeout: 10s
149.        retries: 120

151.    es03:
152.      depends_on:
153.        - es02
154.      image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
155.      volumes:
156.        - certs:/usr/share/elasticsearch/config/certs
157.        - esdata03:/usr/share/elasticsearch/data
158.      environment:
159.        - node.name=es03
160.        - node.roles=data_cold,data_content,master,ingest
161.        - cluster.name=${CLUSTER_NAME}
162.        - cluster.initial_master_nodes=es01,es02,es03
163.        - discovery.seed_hosts=es01,es02
164.        - bootstrap.memory_lock=true
165.        - xpack.security.enabled=true
166.        - xpack.security.http.ssl.enabled=true
167.        - xpack.security.http.ssl.key=certs/es03/es03.key
168.        - xpack.security.http.ssl.certificate=certs/es03/es03.crt
169.        - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
170.        - xpack.security.http.ssl.verification_mode=certificate
171.        - xpack.security.transport.ssl.enabled=true
172.        - xpack.security.transport.ssl.key=certs/es03/es03.key
173.        - xpack.security.transport.ssl.certificate=certs/es03/es03.crt
174.        - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
175.        - xpack.security.transport.ssl.verification_mode=certificate
176.        - xpack.license.self_generated.type=${LICENSE}
177.      mem_limit: ${MEM_LIMIT}
178.      ulimits:
179.        memlock:
180.          soft: -1
181.          hard: -1
182.      healthcheck:
183.        test:
184.          [
185.            "CMD-SHELL",
186.            "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
187.          ]
188.        interval: 10s
189.        timeout: 10s
190.        retries: 120

192.    kibana:
193.      depends_on:
194.        es01:
195.          condition: service_healthy
196.        es02:
197.          condition: service_healthy
198.        es03:
199.          condition: service_healthy
200.      image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
201.      volumes:
202.        # - ./kibana.yml:/usr/share/kibana/config/kibana.yml
203.        - certs:/usr/share/kibana/config/certs
204.        - kibanadata:/usr/share/kibana/data
205.      ports:
206.        - ${KIBANA_PORT}:5601
207.      environment:
208.        - SERVERNAME=kibana
209.        - ELASTICSEARCH_HOSTS=https://es01:9200
210.        - ELASTICSEARCH_USERNAME=kibana_system
211.        - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
212.        - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
213.      mem_limit: ${MEM_LIMIT}
214.      healthcheck:
215.        test:
216.          [
217.            "CMD-SHELL",
218.            "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
219.          ]
220.        interval: 10s
221.        timeout: 10s
222.        retries: 120

224.  volumes:
225.    certs:
226.      driver: local
227.    esdata01:
228.      driver: local
229.    esdata02:
230.      driver: local
231.    esdata03:
232.      driver: local
233.    kibanadata:
234.      driver: local

`![](https://csdnimg.cn/release/blogv2/dist/pc/img/newCodeMoreWhite.png)
复制代码

启动集群

在上面，我们已经创建好了如下的文件：



1.  $ pwd
2.  /Users/liuxg/data/elastic8
3.  $ ls -al
4.  total 40
5.  drwxr-xr-x    5 liuxg  staff   160 Oct 11 08:07 .
6.  drwxr-xr-x  166 liuxg  staff  5312 Nov 15 14:32 ..
7.  -rw-r--r--    1 liuxg  staff   728 Nov 17 07:13 .env
8.  -rw-r--r--    1 liuxg  staff  8321 Nov 17 07:30 docker-compose.yml


复制代码

我们使用如下的命令来启动 Elasticsearch 集群：

docker-compose up
复制代码

或者：

docker-compose up -d
复制代码

如果你想让 docker-compose 在后台运行的话。

直到我们看到如下的画面：

 

我们在浏览器中启动 Kibana：

 

在上面，我们输入密码 password 即可登录：

 

我们在 console 中打入如下的命令：

GET _cat/nodes
复制代码

我们可以看到如下的结果：



1.  172.21.0.5 43 100 10 0.33 0.89 0.66 cims - es03
2.  172.21.0.3 49  96  9 0.33 0.89 0.66 hims - es01
3.  172.21.0.4 72 100 10 0.33 0.89 0.66 imsw * es02


复制代码

 

如果你不想使用 Kibana，你也可以使用 curl 命令来查看：

curl -k -u elastic:password https://localhost:9200/_cat/nodes
复制代码


1.  $ curl -k -u elastic:password https://localhost:9200/_cat/nodes
2.  172.21.0.5 77 100 4 0.27 0.62 0.59 cims - es03
3.  172.21.0.3 64  97 4 0.27 0.62 0.59 hims - es01
4.  172.21.0.4 69 100 4 0.27 0.62 0.59 imsw * es02


复制代码

或者更为确切的显示：

GET _cat/nodes?v
复制代码

从上面的输出中，我们可以看到每个节点的 role。如果我们想了解上面的 cims 各个代表什么意义，请参阅另外一篇文章 “Elasticsearch：Node roles 介绍 - 7.9 之后版本”。字母的意思是：

  • c : cold node
  • d : data node
  • f : frozen node
  • h : hot node
  • i : ingest node
  • l : machine learning node
  • m : master eligible node
  • r : remote cluster client node
  • s : content node
  • t : transform node
  • v : voting-only node
  • w : warm node
    •  : coordinating node only

比如针对 es02 这个节点，imsw 代表的意思是：ingest node，master eligible node，content node，warm node。

这样我们就创建了我们所需要的热，温及冷架构的 Elasticsearch 集群。我们可以在这个集群上测试我们的 ILM（索引生命周期管理）练习。

分类：
后端
标签：
Elasticsearch
Elastic 社区首席布道师 @ Elastic
私信
获得点赞  926
文章被阅读  503,484