六、深入聚合分析

1. `Bucket` & `Metric` 聚合分析及嵌套聚合

1.1 `Bucket` & `Metric Aggregation` 与 `SQL` 的理解

1.2 `Aggregation`的语法

Aggregation属于Search的一部分。一般情况下，建议将其Size指定为0

这样就会只返回Aggregation的结果了(这里可以看后面的例子来体现)

1.3 `Metric Aggregation`

1.3.1 `Metric`聚合的理解分析

单值分析：只输出一个分析结果
- min、max、avg、sum
- Cardinality(类似distinct Count)
多值分析：输出多个分析结果
- stats、extended、stats
- percentile、percentile rank(对数据求百分位数的时候会用到)
- top hits(排在前面的示例)

1.3.2 `Metric`聚合的具体`Demo`

1.3.2.1 数据准备

# 定义一个employees索引的mapping
PUT /employees/
{
  "mappings": {
    "properties": {
      "age": {
        "type": "integer"
      },
      "gender": {
        "type": "keyword"
      },
      "job": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 50
          }
        }
      },
      "name": {
        "type": "keyword"
      },
      "salary": {
        "type": "integer"
      }
    }
  }
}

# 为employees索引加入一些数据
PUT /employees/_bulk
{ "index" : {  "_id" : "1" } }
{ "name" : "Emma","age":32,"job":"Product Manager","gender":"female","salary":35000 }
{ "index" : {  "_id" : "2" } }
{ "name" : "Underwood","age":41,"job":"Dev Manager","gender":"male","salary": 50000}
{ "index" : {  "_id" : "3" } }
{ "name" : "Tran","age":25,"job":"Web Designer","gender":"male","salary":18000 }
{ "index" : {  "_id" : "4" } }
{ "name" : "Rivera","age":26,"job":"Web Designer","gender":"female","salary": 22000}
{ "index" : {  "_id" : "5" } }
{ "name" : "Rose","age":25,"job":"QA","gender":"female","salary":18000 }
{ "index" : {  "_id" : "6" } }
{ "name" : "Lucy","age":31,"job":"QA","gender":"female","salary": 25000}
{ "index" : {  "_id" : "7" } }
{ "name" : "Byrd","age":27,"job":"QA","gender":"male","salary":20000 }
{ "index" : {  "_id" : "8" } }
{ "name" : "Foster","age":27,"job":"Java Programmer","gender":"male","salary": 20000}
{ "index" : {  "_id" : "9" } }
{ "name" : "Gregory","age":32,"job":"Java Programmer","gender":"male","salary":22000 }
{ "index" : {  "_id" : "10" } }
{ "name" : "Bryant","age":20,"job":"Java Programmer","gender":"male","salary": 9000}
{ "index" : {  "_id" : "11" } }
{ "name" : "Jenny","age":36,"job":"Java Programmer","gender":"female","salary":38000 }
{ "index" : {  "_id" : "12" } }
{ "name" : "Mcdonald","age":31,"job":"Java Programmer","gender":"male","salary": 32000}
{ "index" : {  "_id" : "13" } }
{ "name" : "Jonthna","age":30,"job":"Java Programmer","gender":"female","salary":30000 }
{ "index" : {  "_id" : "14" } }
{ "name" : "Marshall","age":32,"job":"Javascript Programmer","gender":"male","salary": 25000}
{ "index" : {  "_id" : "15" } }
{ "name" : "King","age":33,"job":"Java Programmer","gender":"male","salary":28000 }
{ "index" : {  "_id" : "16" } }
{ "name" : "Mccarthy","age":21,"job":"Javascript Programmer","gender":"male","salary": 16000}
{ "index" : {  "_id" : "17" } }
{ "name" : "Goodwin","age":25,"job":"Javascript Programmer","gender":"male","salary": 16000}
{ "index" : {  "_id" : "18" } }
{ "name" : "Catherine","age":29,"job":"Javascript Programmer","gender":"female","salary": 20000}
{ "index" : {  "_id" : "19" } }
{ "name" : "Boone","age":30,"job":"DBA","gender":"male","salary": 30000}
{ "index" : {  "_id" : "20" } }
{ "name" : "Kathy","age":29,"job":"DBA","gender":"female","salary": 20000}

1.3.2.2 查看最低的工资

# Metric聚合，找到最低的工资
POST employees/_search
{
  "size": 0,
  "aggs": {
    "min_salary": {
      "min": {
        "field": "salary"
      }
    }
  }
}

1.3.2.3 查看最高的工资

# Metric 聚合，找到最高的工资
POST employees/_search
{
  "size": 0, 
  "aggs": {
    "max_salary": {
      "max": {
        "field": "salary"
      }
    }
  }
}

1.3.2.4 一个聚合输出多个值

第一种方式就是下面这种

# 多个 Metric 聚合，找到最低和最高和平均工资
POST employees/_search
{
  "size": 0, 
  "aggs": {
    "max_salary": {
      "max": {
        "field": "salary"
      }
    },
    "min_salary": {
      "min": {
        "field": "salary"
      }
    },
    "avg_salary": {
      "avg": {
        "field": "salary"
      }
    }
  }
}

第二种方式

# 一个聚合，输出多值
POST employees/_search
{
  "size": 0,
  "aggs": {
    "stats_salary": {
      "stats": {
        "field": "salary"
      }
    }
  }
}

1.4 `Bucket Aggregation`

按照一定的规则，将文档分配到不同的桶中，从而达到分类的目的。ES提供的一些常见的Bucket Aggregation
- Term
- 数字类型
  - Range/Data Range
  - Histogram/Date Histogram
支持嵌套：也就在桶里再做分桶

1.4.1 `Terms Aggregation`

字段需要打开fielddata,才能进行Terms Aggregation
- keyword默认支持doc_values
- Text需要再Mapping中enable。会按照分词后的结果进行分

1.4.2 `Terms Aggregation` 的`Demo`

1.4.2.1 对`Job`和`job.keyword`进行聚合

# 对job.keyword进行聚合
POST employees/_search
{
  "size": 0,
  "aggs": {
    "jobs": {
      "terms": {
        "field": "job.keyword"
      }
    }
  }
}

# 对 Text 字段进行 terms 聚合查询，失败
POST employees/_search
{
  "size": 0, 
  "aggs": {
    "jobs": {
      "terms": {
        "field": "job"
      }
    }
  }
}

如果想要在Text类型的字段上进行聚合分析，那就需要在Mapping中开启fielddata才可以

# 对 Text 字段打开 fielddata,支持 terms aggregation
PUT employees/_mapping
{
  "properties": {
    "job": {
      "type": "text",
      "fielddata": true
    }
  }
}

# distinct 的使用
POST employees/_search
{
  "size": 0,
  "aggs": {
    "cardinate": {
      "cardinality": {
        "field": "job.keyword"
      }
    }
  }
}

1.4.2.2 对性别进行`Terms`聚合

# 对性别的keyword做分桶
POST employees/_search
{
  "size": 0,
  "aggs": {
    "gender": {
      "terms": {
        "field": "gender"
      }
    }
  }
}

1.4.2.3 指定`bucket size`

# 指定bucket size
POST employees/_search
{
  "size": 0,
  "aggs": {
    "ages_5": {
      "terms": {
        "field": "age",
        "size": 3
      }
    }
  }
}

分桶数就变为了3个

1.4.3 `Bucket Size` & `Top Hits` 的 `Demo`

应用场景：当获取分桶后，桶内最匹配的顶部文档列表
Size：按年龄分桶，找出指定数据量的分桶信息
Top Hits：查看各个工种中，年纪最大的3名员工

# 指定size，不同工种中，年纪最大的3个员工的具体信息
POST employees/_search
{
  "size": 0,
  "aggs": {
    "jobs": {
      "terms": {
        "field": "job.keyword"
      },
      "aggs": {
      "old_employee": {
        "top_hits": {
          "size": 3,
          "sort": [
              {
              "age": {
                "order": "desc"
                }
              }
            ]
          }
        }
      }
    }
  }
}

1.4.4 优化`Terms`聚合的性能

当聚合查询非常频繁，就打开这个配置，这是一个预加载的配置开关，可以很好的提升性能

1.4.5 `Range` & `Histogram`聚合

按照数字的范围，进行分桶
在 Range Aggregation中，可以自定义 Key
Demo:
- 按照工资的 Range 分桶
- 按照工资的间隔(Histogram)分桶

# Slary Ranges 分桶，可以自定义 Key
POST employees/_search
{
  "size": 0,
  "aggs": {
    "salary_range": {
      "range": {
        "field": "salary",
        "ranges": [
          {
            "to": 10000
          },
          {
            "from": 10000,
            "to": 20000
          },
          {
            "key": ">20000",
            "from": 20000
          }
        ]
      }
    }
  }
}

# Salary Histogram,工资0到10万，以5000一个区间进行分桶
POST employees/_search
{
  "size": 0,
  "aggs": {
    "salary_histrogram": {
      "histogram": {
        "field": "salary",
        "interval": 5000,
        "extended_bounds": {
          "min": 0,
          "max": 100000
        }
      }
    }
  }
}

1.5 `Bucket Aggregation` + `Metric Aggregation`

Bucket聚合分析允许通过添加子聚合分析来进一步分析，子聚合分析可以是
- Bucket
- Metric
Demo
- 按照工作类型进行分桶，并统计工资信息
- 先按照工作类型分桶，然后按性别分桶，并统计工资信息

1.5.1 嵌套聚合 `Demo`

# 嵌套聚合1，按照工作类型分桶，并统计工资信息
POST employees/_search
{
  "size": 0,
  "aggs": {
    "job": {
      "terms": {
        "field": "job.keyword"
      },
      "aggs": {
        "salary": {
          "stats": {
            "field": "salary"
          }
        }
      }
    }
  }
}

# 多次嵌套。根据工作类型分桶，然后按照性别分桶，计算工资的统计信息
POST employees/_search
{
  "size": 0,
  "aggs": {
    "job": {
      "terms": {
        "field": "job.keyword"
      },
      "aggs": {
        "gender": {
          "terms": {
            "field": "gender"
          },
          "aggs": {
            "stat_salary": {
              "stats": {
                "field": "salary"
              }
            }
          }
        }
      }
    }
  }
}

2. `Pipeline`聚合分析(对聚合再做一次聚合)

简单来说就是对聚合分析，再做一次聚合分析

2.1 例子：`Pipeline: min_bucket`

在员工数最多的工种里，找出平均工资最低的工种

bucket_path是用来指定关键字的，之前纠结那个大于号是什么意思，是自己没有好好听课。以后见到bucket_path那么这就是一个Pipeline聚合

2.2 `Pipeline`概念理解

管道(Pipeline)的概念: 支持对聚合分析的结果，再次进行聚合分析
Pipeline的分析结果会输出到原结果中，根据位置的不同，分为两类
- Sibling: 结果和现有分析结果同级（上面这个例子就是Sibling类型）
  - Max, Min , Avg & Sum Bucket
  - Stats, Extended Status Bucket
  - Percetiles Bucket
- Parent: 结果内嵌到现有的聚合分析结果之中
  - Derivate(求导)
  - Cumultive(累计求和)
  - Moving Function(滑动窗口)

2.3 例子

注意，下面的演示的例子的实验数据还是文章上面的准备数据

2.3.1 查看平均工资最低的工作类型

# 平均工资最低的工作类型
POST /employees/_search
{ 
  "size": 0, 
  "aggs": {
    "jobs": {
      "terms": {
        "field": "job.keyword",
        "size": 10
      },
      "aggs": {
        "avg_salary": {
          "avg": {
            "field": "salary"
          }
        }
      }
    },
    "min_salary_by_job": {
      "min_bucket": {
        "buckets_path": "jobs>avg_salary"
      }
    }
  }
}

2.3.2 平均工资的百分位数

# 平均工资的百分位数
POST employees/_search
{
  "size": 0,
  "aggs": {
    "jobs": {
      "terms": {
        "field": "job.keyword",
        "size": 10
      },
      "aggs": {
        "avg_salary": {
          "avg": {
            "field": "salary"
          }
        }
      }
    },
    "percentiles_salary_by_job": {
      "percentiles_bucket": {
        "buckets_path": "jobs>avg_salary"
      }
    }
  }
}

2.3.3 按照年龄对平均工资求导

# 按照年龄对平均工资求导
POST employees/_search
{
  "size": 0,
  "aggs": {
    "age": {
      //直方图
      "histogram": {
        "field": "age",
        "min_doc_count": 1, 
        "interval": 1
      },
      "aggs": {
        "avg_salary": {
          "avg": {
            "field": "salary"
          }
        },
        "derivative_avg_salary": {
          "derivative": {
            "buckets_path": "avg_salary"
          }
        }
      }
    }
  }
}

3. 作用范围与排序

之前我们在使用聚合操作的时候都是对整个数据集进行聚合，那么我们如果想在某个查询结构的结果集上做聚合呢？

这就是聚合的作用范围

ES聚合分析的默认作用范围是query的查询结果集
同时ES还支持以下方式改变聚合的作用范围
- Filter
- Post Filter
- Global

3.1 `Query`的作用范围

# Query的作用范围
POST employees/_search
{
  "size": 0,
  "query": {
    "range": {
      "age": {
        "gte": 40
      }
    }
  },
  "aggs": {
    "jobs": {
      "terms": {
        "field": "job.keyword"
      }
    }
  }
}

3.2 `Filter`的作用范围

Filter可以作用到某个具体的aggs查询中

# Filter
POST employees/_search
{
  "size": 0,
  "aggs": {
    "old_person": {
      "filter": {
        "range": {
          "age": {
            "from": 35  
          }
        }
      },
      "aggs": {
        "jobs": {
          "terms": {
            "field": "job.keyword"
          }
        }
      }
    },
    "all_jobs": {
      "terms": {
        "field": "job.keyword"
      }
    }
  }
}

3.3 `Post Filter`的作用范围

当我们聚合完后，如果想要展示符合条件的特定信息，就可以使用 Post Filter

# post field, 一条语句，找出所有的job类型。还能找到聚合后符合条件的结果
POST employees/_search
{
  "aggs": {
    "jobs": {
      "terms": {
        "field": "job.keyword"
      }
    }
  },
  "post_filter": {
    "match": {
      "job.keyword": "Dev Manager"
    }
  }
}

3.4 `Global`的作用范围

Global可以让我们的聚合忽略Query的限定

# global
POST employees/_search
{
    "size": 0,
    "query": {
        "range": {
            "age": {
                "gte": 40
            }
        }
    },
    "aggs": {
        "jobs": {
            "terms": {
                "field": "job.keyword"
            }
        },
        "all": {
            "global": {},
            "aggs": {
                "salary_avg": {
                    "avg": {
                        "field": "salary"
                    }
                }
            }
        }
    }
}

3.5 排序

# 排序 order
# count and key
POST employees/_search
{
    "size": 0,
    "query": {
        "range": {
            "age": {
                "gte": 20
            }
        }
    },
    "aggs": {
        "jobs": {
            "terms": {
                "field": "job.keyword",
                "order": [
                    {
                        "_count": "asc"
                    },
                    {
                        "_key": "desc"
                    }
                ]
            }
        }
    }
}

POST employees/_search
{
    "size": 0,
    "aggs": {
        "jobs": {
            "terms": {
                "field": "job.keyword",
                "order": [
                    {
                        "avg_salary": "desc"
                    }
                ]
            },
            "aggs": {
                "avg_salary": {
                    "avg": {
                        "field": "salary"
                    }
                }
            }
        }
    }
}

学习地址为极客时间《Elasticsearch核心技术与实战》，这只是我做的笔记，仅供参考；

Elasticsearch核心技术与实践五

六、深入聚合分析

1. Bucket & Metric 聚合分析及嵌套聚合

1.1 Bucket & Metric Aggregation 与 SQL 的理解

1.2 Aggregation的语法

1.3 Metric Aggregation

1.3.1 Metric聚合的理解分析

1.3.2 Metric聚合的具体Demo

1.3.2.1 数据准备

1.3.2.2 查看最低的工资

1.3.2.3 查看最高的工资

1.3.2.4 一个聚合输出多个值

1.4 Bucket Aggregation

1.4.1 Terms Aggregation

1.4.2 Terms Aggregation 的Demo

1.4.2.1 对Job和job.keyword进行聚合

1.4.2.2 对性别进行Terms聚合

1.4.2.3 指定bucket size

1.4.3 Bucket Size & Top Hits 的 Demo

1.4.4 优化Terms聚合的性能

1.4.5 Range & Histogram聚合

1.5 Bucket Aggregation + Metric Aggregation

1.5.1 嵌套聚合 Demo

2. Pipeline聚合分析(对聚合再做一次聚合)

2.1 例子：Pipeline: min_bucket

2.2 Pipeline概念理解

2.3 例子

2.3.1 查看平均工资最低的工作类型

2.3.2 平均工资的百分位数

2.3.3 按照年龄对平均工资求导

3. 作用范围与排序

3.1 Query的作用范围

3.2 Filter的作用范围

3.3 Post Filter的作用范围

3.4 Global的作用范围

3.5 排序

1. `Bucket` & `Metric` 聚合分析及嵌套聚合

1.1 `Bucket` & `Metric Aggregation` 与 `SQL` 的理解

1.2 `Aggregation`的语法

1.3 `Metric Aggregation`

1.3.1 `Metric`聚合的理解分析

1.3.2 `Metric`聚合的具体`Demo`

1.4 `Bucket Aggregation`

1.4.1 `Terms Aggregation`

1.4.2 `Terms Aggregation` 的`Demo`

1.4.2.1 对`Job`和`job.keyword`进行聚合

1.4.2.2 对性别进行`Terms`聚合

1.4.2.3 指定`bucket size`

1.4.3 `Bucket Size` & `Top Hits` 的 `Demo`

1.4.4 优化`Terms`聚合的性能

1.4.5 `Range` & `Histogram`聚合

1.5 `Bucket Aggregation` + `Metric Aggregation`

1.5.1 嵌套聚合 `Demo`

2. `Pipeline`聚合分析(对聚合再做一次聚合)

2.1 例子：`Pipeline: min_bucket`

2.2 `Pipeline`概念理解

3.1 `Query`的作用范围

3.2 `Filter`的作用范围

3.3 `Post Filter`的作用范围

3.4 `Global`的作用范围