阅读 91

大数据利器Elasticsearch相关词搜索之shingle相关词token过滤器

这是我参与8月更文挑战的第21天,活动详情查看:8月更文挑战
本Elasticsearch相关文章的版本为:7.4.2

shingle token过滤器的作用:可以实现短语搜索的功能同时保持上下文信息,并且不像短语查询那样需要所有词语都出现。

例如有以下文档:

POST /my_index/_doc/1
{"tittle": "ridingroad likes elasticsearch"}

POST /my_index/_doc/2
{"tittle": "elasticsearch likes ridingroad"}
复制代码

但是用户输入: handsome ridingroad likes elasticsearch, 如果我们使用短语查询:

GET /my_index/_search
{
  "query": {
    "match_phrase": {
      "title": "handsome ridingroad likes elasticsearch"
    }
  }
}
复制代码

返回的数据, 并没有命中任何文档,因为短语查询要求查询的内容需要全部出现在文档里:

{
  "took" : 748,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}
复制代码

那么,如果我们使用shingle token过滤器呢?那什么是shingle?
例如有这样的内容ridingroad likes elasticsearch, 如果拆分为以下长度为1,2的单词对:
一个长度的单词对(unigram):[ridingroad, likes, elasticsearch]
两个长度的单词对(bigrams):[ridingroad likes, likes elasticsearch]
那么,这些一个个的单词对就叫为shingle。

所以我们可以利用shingle保留了它们的先后顺序,因为ridingroad likes elasticsearchelasticsearch likes ridingroad的表达的意思是存在很大差异的。

那么要实现当我们搜索handsome ridingroad likes elasticsearch命中包含ridingroad likes elasticsearch的文档1需要做以下准备:

  1. 构造使用shingle的分词器;
  2. 把分词器应用到字段;
  3. 为了提高匹配度,既要使用unigram同时使用bigrams。
PUT /shingle_test_index
{
    "settings": {
        "number_of_shards": 1,  
        "analysis": {
            "filter": {
                "my_shingle_filter": {
                    "type":             "shingle",
                    "min_shingle_size": 2, 
                    "max_shingle_size": 2, 
                    "output_unigrams":  false   
                }
            },
            "analyzer": {
                "my_shingle_analyzer": {
                    "type":             "custom",
                    "tokenizer":        "standard",
                    "filter": [
                        "lowercase",
                        "my_shingle_filter" 
                    ]
                }
            }
        }
    },
    "mappings": {
      "properties": {
            "title": {
                "type": "text",
                "fields": {
                    "shingles": {
                        "type":     "text",
                        "analyzer": "my_shingle_analyzer"
                    }
                }
            }
        }
    }
}

POST /shingle_test_index/_doc/1
{"title": "ridingroad likes elasticsearch"}

POST /shingle_test_index/_doc/2
{"title": "elasticsearch likes ridingroad"}
复制代码

下面进行查询:

GET /shingle_test_index/_search
{
   "query": {
      "bool": {
         "must": {
            "match": {
               "title": "handsome ridingroad likes elasticsearch"
            }
         },
         "should": {
            "match": {
               "title.shingles": "handsome ridingroad likes elasticsearch"
            }
         }
      }
   }
}
复制代码

返回的数据:

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 1.933259,
    "hits" : [
      {
        "_index" : "shingle_test_index",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : 1.933259,
        "_source" : {
          "title" : "ridingroad likes elasticsearch"
        }
      },
      {
        "_index" : "shingle_test_index",
        "_type" : "_doc",
        "_id" : "2",
        "_score" : 0.5469647,
        "_source" : {
          "title" : "elasticsearch likes ridingroad"
        }
      }
    ]
  }
}
复制代码

从上面的查询结果中,可以看到我们需要文档1的得分比文档2的得分更高(通过should里面的shingle增加了相关性得分,文档1的 likes elasticsearch比文档2的likes ridingroad更符合查询语句的顺序),符合我们的需求。

文章分类
后端
文章标签