最近有一个需求,索引中添加了2个字段,需要对历史数据进行割接,调查现网仅有800多条数据,开发环境600多条数据,所以代码中写死了size为2000条,一次割接完,但是转到测试环境出了问题,测试说有数据没割接,查询一下原来测试环境有2万多条数据,在网上查询了解到ES一次最多支持查询1万条记录,显然一次是割接不完的,即使一次可以割接完也是不行的,毕竟数据量太大,不适宜一次性加入到内存中,好在ES支持游标查询,下面是ES的游标用法
/**
* 通过游标查询数据
*
* indices 索引
* @throws IOException
*/
public static void searchByScroll(String indices) throws IOException {
SearchRequest request = new SearchRequest(indices);
SearchSourceBuilder build = new SearchSourceBuilder();
build.query(QueryBuilders.termQuery("age","21"));
// 设置每次查询的数据集大小
build.size(1);
build.timeout(new TimeValue(60, TimeUnit.SECONDS));
request.source(build);
// 游标时长
Scroll scroll = new Scroll(TimeValue.timeValueMillis(10l));
request.scroll(scroll);
SearchResponse response = restHighLevelClient.search(request,RequestOptions.DEFAULT);
SearchHit[] hit = response.getHits().getHits();
for (int i = 0;i<hit.length;i++){
System.out.println(hit[i].getSourceAsMap().get("name"));
}
String scrollId = response.getScrollId();
SearchScrollRequest searchScrollRequest;
while(hit != null && hit.length > 0) {
searchScrollRequest = new SearchScrollRequest(scrollId);
searchScrollRequest.scroll(scroll);
response = restHighLevelClient.scroll(searchScrollRequest,RequestOptions.DEFAULT);
hit = response.getHits().getHits();
for (int i = 0;i<hit.length;i++){
System.out.println(hit[i].getSourceAsMap().get("name"));
}
}
}