Elasticsearch搜索一、索引生命周期管理

百炼000
2,193 阅读3分钟

遇到问题描述

  1. 第一次上线需要使用kibana或者curl建索引以及为索引关联模板，增加运维人员操作的复杂度。
  2. 使用Elasticsearch官方提供的Java High Level REST Client 创建policy和绑定template代码编写比较复杂，并且修改是不容易。
  3. 使用Java Low Level REST Client ，使用classpath:elastic/info-policy.json和classpath:elastic/info-template.json文件作为请求体，发送request请求创建滚动规则并绑定索引模板。

收获

1. 读取classpath文件

   final String filePath = ResourceUtils.getURL("classpath:elastic/info-policy.json").getFile();
        File file = new File(filePath);
        if (!file.exists()) {
            log.info("policy file [{}] is not exit ?", filePath);
            return;
        }

        final List<String> content = Files.readAllLines(file.toPath(), Charset.forName("UTF-8"));
        putPolicyRequest.setEntity(new NStringEntity(String.join("", content),

2. Elasticsearch提供两种方式的API

  1. Java Low Level REST Client 允许用HTTP协议与ES集群通信，是面向用户的。低级客户端的特效包括：

    1. 最小化依赖
    2. 所有节点(nodes)的负载均衡
    3. 某个节点发生故障会自动转移到其他节点
    4. 对于失败的链接给予乘法，也就是失败次数越多，client下次尝试重新链接的时间就越长，持续练级
    5. 跟踪(tracing)请求和相应的记录
    6. 可以自动发现集群中的节点(cluster nodes)

  2. Java high Level REST Client 基于低级客户端，面向开发者，开发者可以用管饭提供的各种API实现自己的功能

    1. Java High Level Client 运行在Java Low Level REST 客户端上，面向开发人员，高级客户端提供API的具体方法，这些方法以请求对象(Request Object)为输入，返回响应对象(Response Object)，每个API的方法都可以用同步或者异步的方式调用，异步方法返回的对象需要用监听器(listener)来接受。

    2. Java High Level Client 依赖于ES核心工程(core project)，因此与ES原生的Java API中的TransportClient有相同的输入和输出。由于8.0版本后，官方不在支持TransportClient，因此推荐把现有的用TransportClient写的代码移植到REST Client。目前，Java REST Client API中已经实现了原生API中的大部分方法，开发者可以在官方的github issue上提交暂时还未实现的方法。当然这些方法都可以用JSON请求Low Level REST Client手动实现。

3. 调整索引滚动策略刷新时间，默认为10m

PUT _cluster/settings
{
  "transient": {
    "indices.lifecycle.poll_interval":"1s"
  }
}

动手实践

1. 建立spring-boot工程，使用最新依赖

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.6.2</version>
    <relativePath/> <!-- lookup parent from repository -->
</parent>

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.projectlombok</groupId>
        <artifactId>lombok</artifactId>
        <version>1.18.16</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>

2. application.yml配置elasticsearch

spring:
  elasticsearch:
    connection-timeout: 5s
    uris: http://localhost:9200

3. 使用RestClient发送请求

  1. 通过RestHighLevelClient拿到RestClient客户端
    RestClient restClient;
    @Autowired
    RestHighLevelClient restHighLevelClient;
    @Autowired
    private ElasticsearchOperations elasticsearchOperations;

    @BeforeEach
    public void setUp() {
        restClient = restHighLevelClient.getLowLevelClient();
    }

    @AfterEach
    public void tearDown() {
        try {
            restClient.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
  1. 创建policy
    1. 检查policy是否存在，存在就直接返回
    2. 如果policy不存在，读取classpath:elastic/info-policy.json文件做为请求体，发送创建policy的请求
    @Test
    public void test_createPolicy() throws Exception {
        String policyName = "_ilm/policy/info-policy";
        Request policyExitsRequest = new Request("GET", policyName);
        boolean policyExits = false;
        try {
            Response response = restClient.performRequest(policyExitsRequest);
            if (response.getStatusLine().getStatusCode() == 200) {
                log.info("policy is already exits");
                policyExits = true;
            }
        } catch (ResponseException responseException) {
            log.info("policy is not exits [{}]", responseException.getMessage());
        }
        if (policyExits) {
            return;
        }
        Request putPolicyRequest = new Request("PUT", policyName);
        final String filePath = ResourceUtils.getURL("classpath:elastic/info-policy.json").getFile();
        File file = new File(filePath);
        if (!file.exists()) {
            log.info("policy file [{}] is not exit ?", filePath);
            return;
        }

        final List<String> content = Files.readAllLines(file.toPath(), Charset.forName("UTF-8"));
        putPolicyRequest.setEntity(new NStringEntity(String.join("", content), ContentType.APPLICATION_JSON));
        Response putTemplateResponse = restClient.performRequest(putPolicyRequest);
        if (putTemplateResponse.getStatusLine().getStatusCode() == 200) {
            log.info("create policy success");
            return;
        }
    }


    @Test
    public void test_createTemplate() throws Exception {
        String templateName = "_index_template/info-template";
        Request templateExitsRequest = new Request("GET", templateName);
        boolean templateExits = false;
        try {
            Response response = restClient.performRequest(templateExitsRequest);
            if (response.getStatusLine().getStatusCode() == 200) {
                log.info("template is already exits");
                templateExits = true;
            }
        } catch (ResponseException responseException) {
            log.info("template is not exits [{}]", responseException.getMessage());
        }
        if (templateExits) {
            return;
        }

        Request putTemplateRequest = new Request("PUT", templateName);
        final String filePath = ResourceUtils.getURL("classpath:elastic/info-template.json").getFile();
        File file = new File(filePath);
        if (!file.exists()) {
            log.info("template file [{}] is not exit ?", filePath);
            return;
        }

        final List<String> content = Files.readAllLines(file.toPath(), Charset.forName("UTF-8"));
        putTemplateRequest.setEntity(new NStringEntity(String.join("", content), ContentType.APPLICATION_JSON));
        Response putTemplateResponse = restClient.performRequest(putTemplateRequest);
        if (putTemplateResponse.getStatusLine().getStatusCode() == 200) {
            log.info("create template success");
            return;
        }
    }

3. 使用ElasticsearchOperations写入数据到Elasticsearch,查看索引滚动策略是否生效

  1. step1. 需要将索引滚动策略刷新时间调小一点，API
PUT _cluster/settings
{
  "transient": {
    "indices.lifecycle.poll_interval":"1s"
  }
}
  1. step2. 使用程序模拟数据
@Autowired
private ElasticsearchOperations elasticsearchOperations;

@BeforeEach
public void setUp() {
    restClient = restHighLevelClient.getLowLevelClient();
}

@AfterEach
public void tearDown() {
    try {
        restClient.close();
    } catch (IOException e) {
        e.printStackTrace();
    }
}
@Test
public void test_save() throws Exception {
    for (int i = 0; i < 1000; i++) {
        PersonDocument personDocument = new PersonDocument();
        personDocument.setName("name-" + i);
        personDocument.setId(i);
        elasticsearchOperations.save(personDocument);
        TimeUnit.SECONDS.sleep(2);
    }
}

效果

1.png

源码参考

avatar
文章
阅读
粉丝

友情链接：