在我们的许多练习中,我需要模拟索引生命管理周期的环境。在这样的集群中,我们需要模拟出热(hot),温(warm)及冷(cold)的架构。那么我们该如何进行模拟呢?
我们参考之前的文章 “Elasticsearch:使用 Docker compose 来一键部署 Elastic Stack 8.x”。我们使用 docker compose 来进行部署。
首先,我们按照如下的步骤来进行修改。
.env
.env 文件设置运行 docker-compose.yml 配置文件时使用的环境变量。 确保使用 ELASTIC_PASSWORD 和 KIBANA_PASSWORD 变量为 elastic 和 kibana_system 用户指定密码。 这些变量由 docker-compose.yml 文件引用。
`
1. # Password for the 'elastic' user (at least 6 characters)
2. ELASTIC_PASSWORD=password
4. # Password for the 'kibana_system' user (at least 6 characters)
5. KIBANA_PASSWORD=password
7. # Version of Elastic products
8. STACK_VERSION=8.5.1
10. # Set the cluster name
11. CLUSTER_NAME=docker-cluster
13. # Set to 'basic' or 'trial' to automatically start the 30-day trial
14. LICENSE=basic
15. #LICENSE=trial
17. # Port to expose Elasticsearch HTTP API to the host
18. ES_PORT=9200
19. #ES_PORT=127.0.0.1:9200
21. # Port to expose Kibana to the host
22. KIBANA_PORT=5601
23. #KIBANA_PORT=80
25. # Increase or decrease based on the available host memory (in bytes)
26. MEM_LIMIT=1073741824
28. # Project namespace (defaults to the current folder name if not set)
29. #COMPOSE_PROJECT_NAME=myproject
`
复制代码
在上面,我们设定 Elastic Stack 的版本为最新的 8.5.1。在这个文件中,我们同时也设定 elastic 这个超级用户的密码。
docker-compose.yml
这个 docker-compose.yml 文件创建了一个启用了身份验证和网络加密的三节点安全 Elasticsearch 集群,以及一个安全连接到它的 Kibana 实例。在我们的设置中,我们分别设置三个节点为 data_hot,data_warm 及 data_cold。有关 data tiers 的介绍,请阅读我的另外一篇文章 “Elastic:Data tiers 介绍及索引生命周期管理 - 7.10 之后版本”。
修改后的 docker-compose.yml 文件如下:
docker-compose.yml
`
1. version: "2.2"
3. services:
4. setup:
5. image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
6. volumes:
7. - certs:/usr/share/elasticsearch/config/certs
8. user: "0"
9. command: >
10. bash -c '
11. if [ x${ELASTIC_PASSWORD} == x ]; then
12. echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
13. exit 1;
14. elif [ x${KIBANA_PASSWORD} == x ]; then
15. echo "Set the KIBANA_PASSWORD environment variable in the .env file";
16. exit 1;
17. fi;
18. if [ ! -f certs/ca.zip ]; then
19. echo "Creating CA";
20. bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
21. unzip config/certs/ca.zip -d config/certs;
22. fi;
23. if [ ! -f certs/certs.zip ]; then
24. echo "Creating certs";
25. echo -ne \
26. "instances:\n"\
27. " - name: es01\n"\
28. " dns:\n"\
29. " - es01\n"\
30. " - localhost\n"\
31. " ip:\n"\
32. " - 127.0.0.1\n"\
33. " - name: es02\n"\
34. " dns:\n"\
35. " - es02\n"\
36. " - localhost\n"\
37. " ip:\n"\
38. " - 127.0.0.1\n"\
39. " - name: es03\n"\
40. " dns:\n"\
41. " - es03\n"\
42. " - localhost\n"\
43. " ip:\n"\
44. " - 127.0.0.1\n"\
45. > config/certs/instances.yml;
46. bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
47. unzip config/certs/certs.zip -d config/certs;
48. fi;
49. echo "Setting file permissions"
50. chown -R root:root config/certs;
51. find . -type d -exec chmod 750 \{\} \;;
52. find . -type f -exec chmod 640 \{\} \;;
53. echo "Waiting for Elasticsearch availability";
54. until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
55. echo "Setting kibana_system password";
56. until curl -s -X POST --cacert config/certs/ca/ca.crt -u elastic:${ELASTIC_PASSWORD} -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
57. echo "All done!";
58. '
59. healthcheck:
60. test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
61. interval: 1s
62. timeout: 5s
63. retries: 120
65. es01:
66. depends_on:
67. setup:
68. condition: service_healthy
69. image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
70. volumes:
71. - certs:/usr/share/elasticsearch/config/certs
72. - esdata01:/usr/share/elasticsearch/data
73. ports:
74. - ${ES_PORT}:9200
75. environment:
76. - node.name=es01
77. - node.roles=data_hot,data_content,master,ingest
78. - cluster.name=${CLUSTER_NAME}
79. - cluster.initial_master_nodes=es01,es02,es03
80. - discovery.seed_hosts=es02,es03
81. - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
82. - bootstrap.memory_lock=true
83. - xpack.security.enabled=true
84. - xpack.security.http.ssl.enabled=true
85. - xpack.security.http.ssl.key=certs/es01/es01.key
86. - xpack.security.http.ssl.certificate=certs/es01/es01.crt
87. - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
88. - xpack.security.http.ssl.verification_mode=certificate
89. - xpack.security.transport.ssl.enabled=true
90. - xpack.security.transport.ssl.key=certs/es01/es01.key
91. - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
92. - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
93. - xpack.security.transport.ssl.verification_mode=certificate
94. - xpack.license.self_generated.type=${LICENSE}
95. mem_limit: ${MEM_LIMIT}
96. ulimits:
97. memlock:
98. soft: -1
99. hard: -1
100. healthcheck:
101. test:
102. [
103. "CMD-SHELL",
104. "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
105. ]
106. interval: 10s
107. timeout: 10s
108. retries: 120
110. es02:
111. depends_on:
112. - es01
113. image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
114. volumes:
115. - certs:/usr/share/elasticsearch/config/certs
116. - esdata02:/usr/share/elasticsearch/data
117. environment:
118. - node.name=es02
119. - node.roles=data_warm,data_content,master,ingest
120. - cluster.name=${CLUSTER_NAME}
121. - cluster.initial_master_nodes=es01,es02,es03
122. - discovery.seed_hosts=es01,es03
123. - bootstrap.memory_lock=true
124. - xpack.security.enabled=true
125. - xpack.security.http.ssl.enabled=true
126. - xpack.security.http.ssl.key=certs/es02/es02.key
127. - xpack.security.http.ssl.certificate=certs/es02/es02.crt
128. - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
129. - xpack.security.http.ssl.verification_mode=certificate
130. - xpack.security.transport.ssl.enabled=true
131. - xpack.security.transport.ssl.key=certs/es02/es02.key
132. - xpack.security.transport.ssl.certificate=certs/es02/es02.crt
133. - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
134. - xpack.security.transport.ssl.verification_mode=certificate
135. - xpack.license.self_generated.type=${LICENSE}
136. mem_limit: ${MEM_LIMIT}
137. ulimits:
138. memlock:
139. soft: -1
140. hard: -1
141. healthcheck:
142. test:
143. [
144. "CMD-SHELL",
145. "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
146. ]
147. interval: 10s
148. timeout: 10s
149. retries: 120
151. es03:
152. depends_on:
153. - es02
154. image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
155. volumes:
156. - certs:/usr/share/elasticsearch/config/certs
157. - esdata03:/usr/share/elasticsearch/data
158. environment:
159. - node.name=es03
160. - node.roles=data_cold,data_content,master,ingest
161. - cluster.name=${CLUSTER_NAME}
162. - cluster.initial_master_nodes=es01,es02,es03
163. - discovery.seed_hosts=es01,es02
164. - bootstrap.memory_lock=true
165. - xpack.security.enabled=true
166. - xpack.security.http.ssl.enabled=true
167. - xpack.security.http.ssl.key=certs/es03/es03.key
168. - xpack.security.http.ssl.certificate=certs/es03/es03.crt
169. - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
170. - xpack.security.http.ssl.verification_mode=certificate
171. - xpack.security.transport.ssl.enabled=true
172. - xpack.security.transport.ssl.key=certs/es03/es03.key
173. - xpack.security.transport.ssl.certificate=certs/es03/es03.crt
174. - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
175. - xpack.security.transport.ssl.verification_mode=certificate
176. - xpack.license.self_generated.type=${LICENSE}
177. mem_limit: ${MEM_LIMIT}
178. ulimits:
179. memlock:
180. soft: -1
181. hard: -1
182. healthcheck:
183. test:
184. [
185. "CMD-SHELL",
186. "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
187. ]
188. interval: 10s
189. timeout: 10s
190. retries: 120
192. kibana:
193. depends_on:
194. es01:
195. condition: service_healthy
196. es02:
197. condition: service_healthy
198. es03:
199. condition: service_healthy
200. image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
201. volumes:
202. # - ./kibana.yml:/usr/share/kibana/config/kibana.yml
203. - certs:/usr/share/kibana/config/certs
204. - kibanadata:/usr/share/kibana/data
205. ports:
206. - ${KIBANA_PORT}:5601
207. environment:
208. - SERVERNAME=kibana
209. - ELASTICSEARCH_HOSTS=https://es01:9200
210. - ELASTICSEARCH_USERNAME=kibana_system
211. - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
212. - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
213. mem_limit: ${MEM_LIMIT}
214. healthcheck:
215. test:
216. [
217. "CMD-SHELL",
218. "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
219. ]
220. interval: 10s
221. timeout: 10s
222. retries: 120
224. volumes:
225. certs:
226. driver: local
227. esdata01:
228. driver: local
229. esdata02:
230. driver: local
231. esdata03:
232. driver: local
233. kibanadata:
234. driver: local
`
复制代码
启动集群
在上面,我们已经创建好了如下的文件:
1. $ pwd
2. /Users/liuxg/data/elastic8
3. $ ls -al
4. total 40
5. drwxr-xr-x 5 liuxg staff 160 Oct 11 08:07 .
6. drwxr-xr-x 166 liuxg staff 5312 Nov 15 14:32 ..
7. -rw-r--r-- 1 liuxg staff 728 Nov 17 07:13 .env
8. -rw-r--r-- 1 liuxg staff 8321 Nov 17 07:30 docker-compose.yml
复制代码
我们使用如下的命令来启动 Elasticsearch 集群:
docker-compose up
复制代码
或者:
docker-compose up -d
复制代码
如果你想让 docker-compose 在后台运行的话。
直到我们看到如下的画面:
我们在浏览器中启动 Kibana:
在上面,我们输入密码 password 即可登录:
我们在 console 中打入如下的命令:
GET _cat/nodes
复制代码
我们可以看到如下的结果:
1. 172.21.0.5 43 100 10 0.33 0.89 0.66 cims - es03
2. 172.21.0.3 49 96 9 0.33 0.89 0.66 hims - es01
3. 172.21.0.4 72 100 10 0.33 0.89 0.66 imsw * es02
复制代码
如果你不想使用 Kibana,你也可以使用 curl 命令来查看:
curl -k -u elastic:password https://localhost:9200/_cat/nodes
复制代码
1. $ curl -k -u elastic:password https://localhost:9200/_cat/nodes
2. 172.21.0.5 77 100 4 0.27 0.62 0.59 cims - es03
3. 172.21.0.3 64 97 4 0.27 0.62 0.59 hims - es01
4. 172.21.0.4 69 100 4 0.27 0.62 0.59 imsw * es02
复制代码
或者更为确切的显示:
GET _cat/nodes?v
复制代码
从上面的输出中,我们可以看到每个节点的 role。如果我们想了解上面的 cims 各个代表什么意义,请参阅另外一篇文章 “Elasticsearch:Node roles 介绍 - 7.9 之后版本”。字母的意思是:
- c : cold node
- d : data node
- f : frozen node
- h : hot node
- i : ingest node
- l : machine learning node
- m : master eligible node
- r : remote cluster client node
- s : content node
- t : transform node
- v : voting-only node
- w : warm node
-
- : coordinating node only
- : coordinating node only
比如针对 es02 这个节点,imsw 代表的意思是:ingest node,master eligible node,content node,warm node。
这样我们就创建了我们所需要的热,温及冷架构的 Elasticsearch 集群。我们可以在这个集群上测试我们的 ILM(索引生命周期管理)练习。