springboot中mongo查询大量数据的报错问题

1,175 阅读3分钟

mongo数据对于排序时允许使用的内存空间有限制,如果数据量比较大,会超出mongo允许的排序时内存容量限制,导致查询失败。

(1)

com.mongodb.MongoQueryException: Query failed with error code 96 and error message 'Executor error during find command :: caused by :: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit.' on server xx.x.xxx.xxx:27017

blog.csdn.net/cloume/arti…

没有索引时,mongo将所有数据读取到内存中排序

In MongoDB, sort operations can obtain the sort order by retrieving documents based on the ordering in an index. If the query planner cannot obtain the sort order from an index, it will sort the results in memory.Sort operations that use an index often have better performance than those that do not use an index. In addition, sort operations that do not use an index will abort when they use 32 megabytes of memory.

解决方案:为相关字段增加索引

(2)查询列表

com.mongodb.MongoCommandException: Command failed with error 16819 (Location16819): 'Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting.' on server xx.x.xxx.xxx:27017. The full response is {"ok": 0.0, "errmsg": "Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting.", "code": 16819, "codeName": "Location16819"}

www.jb51.net/article/119… 解决MongoDB 排序超过内存限制的问题

在MongoDB中,排序内存限制最大为100M,如果执行一个更大的排序,需要使用 allowDiskUse 选项来将数据写到临时文件来排序。

docs.mongodb.com/manual/core…

Memory Restrictions
Each individual pipeline stage has a limit of 100 megabytes of RAM. By default, if a stage exceeds this limit, MongoDB produces an error. For some pipeline stages you can allow pipeline processing to take up more space by using the allowDiskUse option to enable aggregation pipeline stages to write data to temporary files.

解决方案:

(1)去除不需要的字段:在展开和排序中间,增加project指定需要返回的字段

(2)增加allowDiskUse

Aggregation aggregationCount = Aggregation.newAggregation(
        ...
).withOptions(AggregationOptions.builder().allowDiskUse(true).build());

使排序操作使用到索引

1)为查询语句创建合适的索引如果查询中排序是单列排序,如sort({"Num":1}),那么只需添加为Num 列添加索引即可,排序的顺序无影响

## 例如索引为 {'Num':1},查询不管升/降序都可使用到索引排序db.data_test.find().sort({Num:1}) db.data_test.find().sort({Num:-1})

如果查询中排序是使用组合排序,如sort({"Num":1,"id":1}),那么需要建立对应的组合索引,如{"key" : {"Num" : 1, "_id" : 1} 或者 {"key" : {"Num" : -1, "_id" : -1}

## 例如索引为{"Num" : 1, "_id" : 1},可以用到索引排序的场景为db.data_test.find().sort({Num:1,_id:1})db.data_test.find().sort({Num:-1,_id:-1})

注意保持查询中组合排序的升降序和组合索引中的 方向 保持 全部相同 或 全部相反

2)注意前缀索引的使用

上文查询报错的案例分析已说明了组合索引每一个键的顺序非常重要,这将决定该组合索引在查询过程中能否被使用到,也将是MongoDB的索引及排序同样需遵循最左前缀原则。

  1. 聚合查询添加allowDiskUse选项

尽可能的保证查询语句的排序能够使用索引排序,但如果业务需要规避排序内存限制报错的问题,那么需要在代码中添加 {allowDiskUse : true} 参数。