Linux 命令行工具之 jq 最佳实践

CloudNativeOps
4,759 阅读6分钟

jq 介绍

jq 是 stedolan 开发的一个轻量级的和灵活的命令行 JSON 处理器。

它主要用于在命令行界面处理 JSON 输入，并使用给定的过滤条件来过滤符合条件的新的 JSON串。

通常在类 Unix 环境下，我们可以快速的使用 jq 来进行 JSON 数据格式化过滤和处理。

同时需要注意的是，该命令行工具和 awk/sed/grep 工具一样，属于系统的默认命令，如果系统没有该命令，可以尝试使用如下方式进行安装。

# Ubuntu 系列
$ sudo apt-get install jq 
# CentOS 系列
$ yum install jq

jq 基础使用

1. 基本语法


jq [options] <jq filter> [file...]
jq [options] --args <jq filter> [strings...]
jq [options] --jsonargs <jq filter> [JSON_TEXTS...]

# options 可选参数列表和说明
-c               将格式化json输出为紧凑的字符串格式;
-n               使用`null`作为单个输入值;
-e               根据输出设置退出状态代码;
-s               将所有输入读取（吸取）到数组中；应用过滤器;
-r               输出原始字符串，而不是JSON文本;
-R               读取原始字符串，而不是JSON文本;
-C               为JSON输出填充颜色;
-M               单色（不要为JSON着色）;
-S               在输出上排序对象的键;
--tab            使用制表符进行缩进;
--arg a v        将变量$a设置为value<v>;
--argjson a v    将变量$a设置为JSON value<v>;
--slurpfile a f  将变量$a设置为从<f>读取的JSON文本数组;
--rawfile a f    将变量$a设置为包含<f>内容的字符串;
--args           其余参数是字符串参数，而不是文件;
--jsonargs       其余的参数是JSON参数，而不是文件;
--               终止参数处理;

当然了，我们也可以使用 man jq 或者 jq --help 再或者官网文档 jq-doc 中查看更多使用指南。

2. 基础使用

2.1 基础字段解析

## 使用 . 参数默认格式化整个json数据
$ echo '{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/"}'  | jq .
{
  "Name": "CloudNativeOps",
  "Owner": "GoOps",
  "WebSite": "https://bgbiao.top/"
}

## 使用.$name 来获取指定filed
$ echo '{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/"}'  | jq .Name
"CloudNativeOps"

## 解析json 中的层级数据
$ echo '{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/", "Contact": {"Email":"weichuangxxb@qq.com","QQ":"371990778","WeChat":"GoOps"} ,"Skills": [ {"name":"Python","type":"dev" }, {"name":"Golang","type":"dev" },{"name":"Ansible","type":"ops" },{"name":"Kubernetes","type":"dev" },{"name":"ElasticSearch","type":"ops" }]}'  | jq .Contact
{
  "Email": "weichuangxxb@qq.com",
  "QQ": "371990778",
  "WeChat": "GoOps"
}
### 仅输出 Contact 中的 WeChat 
$ echo '{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/", "Contact": {"Email":"weichuangxxb@qq.com","QQ":"371990778","WeChat":"GoOps"} ,"Skills": [ {"name":"Python","type":"dev" }, {"name":"Golang","type":"dev" },{"name":"Ansible","type":"ops" },{"name":"Kubernetes","type":"dev" },{"name":"ElasticSearch","type":"ops" }]}'  | jq .Contact.WeChat
"GoOps"

## 获取多个字段 (使用'.filed1,.filed2' 可以获取两个字段)
$ echo '{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/", "Contact": {"Email":"weichuangxxb@qq.com","QQ":"371990778","WeChat":"GoOps"} ,"Skills": [ {"name":"Python","type":"dev" }, {"name":"Golang","type":"dev" },{"name":"Ansible","type":"ops" },{"name":"Kubernetes","type":"dev" },{"name":"ElasticSearch","type":"ops" }]}'  | jq ".Name,.Owner"
"CloudNativeOps"
"GoOps"

2.2 列表、迭代器、管道

## 解析json 中的数组或者列表
$ echo '{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/", "Contact": {"Email":"weichuangxxb@qq.com","QQ":"371990778","WeChat":"GoOps"} ,"Skills": [ {"name":"Python","type":"dev" }, {"name":"Golang","type":"dev" },{"name":"Ansible","type":"ops" },{"name":"Kubernetes","type":"dev" },{"name":"ElasticSearch","type":"ops" }]}'  | jq .Skills
[
  {
    "name": "Python",
    "type": "dev"
  },
  {
    "name": "Golang",
    "type": "dev"
  },
  {
    "name": "Ansible",
    "type": "ops"
  },
  {
    "name": "Kubernetes",
    "type": "dev"
  },
  {
    "name": "ElasticSearch",
    "type": "ops"
  }
]

### 同时也支持范围索引，比如 .Skills[1:3]
$ echo '{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/", "Contact": {"Email":"weichuangxxb@qq.com","QQ":"371990778","WeChat":"GoOps"} ,"Skills": [ {"name":"Python","type":"dev" }, {"name":"Golang","type":"dev" },{"name":"Ansible","type":"ops" },{"name":"Kubernetes","type":"dev" },{"name":"ElasticSearch","type":"ops" }]}'  | jq .Skills[1:3]
[
  {
    "name": "Golang",
    "type": "dev"
  },
  {
    "name": "Ansible",
    "type": "ops"
  }
]

## value 迭代器 .[]
### 前面我们在使用数组的时候通过.Skills 直接获取到了一个数组，而通过.Skills[n:m] 来通过指定索引范围获取一个子列表
### 而通过.[] 则可以直接迭代指定filed 里的values，这里不仅仅是列表了，比如可以通过 .Contact[] 将联系人的全部方式获取出来

$ echo '{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/", "Contact": {"Email":"weichuangxxb@qq.com","QQ":"371990778","WeChat":"GoOps"} ,"Skills": [ {"name":"Python","type":"dev" }, {"name":"Golang","type":"dev" },{"name":"Ansible","type":"ops" },{"name":"Kubernetes","type":"dev" },{"name":"ElasticSearch","type":"ops" }]}'  | jq .Contact[]
"weichuangxxb@qq.com"
"371990778"
"GoOps"

### 而如果是列表或者数组，则可以指定元素 (支持倒序获取，即.[-1]表示获取列表中的最后一个元素)
$ echo '{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/", "Contact": {"Email":"weichuangxxb@qq.com","QQ":"371990778","WeChat":"GoOps"} ,"Skills": [ {"name":"Python","type":"dev" }, {"name":"Golang","type":"dev" },{"name":"Ansible","type":"ops" },{"name":"Kubernetes","type":"dev" },{"name":"ElasticSearch","type":"ops" }]}'  | jq .Skills[0]
{
  "name": "Python",
  "type": "dev"
}

## 管道 (在jq 的表达式中，可以使用 管道符号 | 来对前面的表达式结果进行再次处理)
### 比如上面的列子中，我们使用.Skills[] 获取了子json，但是如果仅想获取子json中的某一个filed，就可以使用管道的能力
### 所以，一般管道的更多会和列表类型的迭代配合使用
$ echo '{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/", "Contact": {"Email":"weichuangxxb@qq.com","QQ":"371990778","WeChat":"GoOps"} ,"Skills": [ {"name":"Python","type":"dev" }, {"name":"Golang","type":"dev" },{"name":"Ansible","type":"ops" },{"name":"Kubernetes","type":"dev" },{"name":"ElasticSearch","type":"ops" }]}'  | jq '.Skills[] | .name '
"Python"
"Golang"
"Ansible"
"Kubernetes"
"ElasticSearch"

## 值计算 
echo '{"num":12}' | jq '(.num +2)*1024'
14336

注意1: 在上面的示例中，Skills 字段是一个 列表，我们需要注意， .Skills、.Skills[]、.Skills[N]以及.Skills[N:M] 的区别

  • .Skills: 普通的获取 Skills 字段的 values，由于是列表，返回为一个列表，为通用方式
  • .Skills[]: 迭代Skills 字段里的元素，因为Skills 为元素为子json的列表，因此返回全部的子json，为通用方式
  • .Skills[N]: 获取列表中索引为N的元素，仅用于在字段为列表类型时
  • .Skills[N:M]: 截取列表中索引从N到M元素为一个新的列表，仅用于字段为列表类型时

注意2: 在上述中对列表类型的值进行迭代后通过管道取值.Skills[] | .name 和 .Skills[].name 有着相同的作用

2.3 复杂数据类型构建

  • 列表(数组)构建[]: 可以将输出字符返回成一个列表(可以结合多字段使用)
  • 对象构建{}: 可以联合 () 构建新的json对象
    • {} 可以通过字段构建新的json结构，比如 {user, title: .titles[]} 表示将titles数组中的元素迭代出来后和user 字段重新组成多个json字串
    • () 可以用于直接将value 作为 key，比如{(.user): .titles} 可以直接构建无声明key 的json串
  • 递归下降..: 可以递归下降，将字符串的每一个value都迭代出来，用法和内置的 recurse 函数相似
## 数组构建 
$ echo '{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/", "Contact": {"Email":"weichuangxxb@qq.com","QQ":"371990778","WeChat":"GoOps"} ,"Skills": [ {"name":"Python","type":"dev" }, {"name":"Golang","type":"dev" },{"name":"Ansible","type":"ops" },{"name":"Kubernetes","type":"dev" },{"name":"ElasticSearch","type":"ops" }]}'  | jq '[.Skills[].name ]'
[  "Python",  "Golang",  "Ansible",  "Kubernetes",  "ElasticSearch"]

$ echo '{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/", "Contact": {"Email":"weichuangxxb@qq.com","QQ":"371990778","WeChat":"GoOps"} ,"Skills": [ {"name":"Python","type":"dev" }, {"name":"Golang","type":"dev" },{"name":"Ansible","type":"ops" },{"name":"Kubernetes","type":"dev" },{"name":"ElasticSearch","type":"ops" }]}'  | jq '[.Skills[].name,.Name]'
[  "Python",  "Golang",  "Ansible",  "Kubernetes",  "ElasticSearch",  "CloudNativeOps"]

$ echo '{"num": [1,2,3,4]}' | jq '[.num[] | . * 2 ]'
[  2,  4,  6,  8]


## 对象构建
### jq '{Name,Owner,skills: .Skills[].name}' 构建新的json串 
$ echo '{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/", "Contact": {"Email":"weichuangxxb@qq.com","QQ":"371990778","WeChat":"GoOps"} ,"Skills": [ {"name":"Python","type":"dev" }, {"name":"Golang","type":"dev" },{"name":"Ansible","type":"ops" },{"name":"Kubernetes","type":"dev" },{"name":"ElasticSearch","type":"ops" }]}' | jq '{Name,Owner,skills: .Skills[].name}'
{
  "Name": "CloudNativeOps",
  "Owner": "GoOps",
  "skills": "Python"
}
{
  "Name": "CloudNativeOps",
  "Owner": "GoOps",
  "skills": "Golang"
}
{
  "Name": "CloudNativeOps",
  "Owner": "GoOps",
  "skills": "Ansible"
}
{
  "Name": "CloudNativeOps",
  "Owner": "GoOps",
  "skills": "Kubernetes"
}
{
  "Name": "CloudNativeOps",
  "Owner": "GoOps",
  "skills": "ElasticSearch"
}

###  jq '{(.Name) : .Skills[].name}' 构建无声明key 的json串
$ echo '{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/", "Contact": {"Email":"weichuangxxb@qq.com","QQ":"371990778","WeChat":"GoOps"} ,"Skills": [ {"name":"Python","type":"dev" }, {"name":"Golang","type":"dev" },{"name":"Ansible","type":"ops" },{"name":"Kubernetes","type":"dev" },{"name":"ElasticSearch","type":"ops" }]}' | jq '{(.Name) : .Skills[].name}'
{
  "CloudNativeOps": "Python"
}
{
  "CloudNativeOps": "Golang"
}
{
  "CloudNativeOps": "Ansible"
}
{
  "CloudNativeOps": "Kubernetes"
}
{
  "CloudNativeOps": "ElasticSearch"
}


## 递归下降 使用 .. 可以将全部的子串解析出来，直到最后的叶子value
$ echo '[[{"a":1}]]' | jq '.. '
[  [    {      "a": 1    }  ]
]
[  {    "a": 1  }]
{
  "a": 1
}
1

### 递归下降获取具体的值 '.. | .filed'
#### 获取key 包含a的值
$ echo '[[{"a":1}]]' | jq '.. | .a?'
1

2.4 内置的操作符以及函数

  • +: 两个过滤器相加，支持 Numbers、Arrays、Strings、Objects 类型
  • -: 相减，用法同上
  • *,/,%: 乘除余运算
  • length: 获取不同类型值的长度，支持string,array,object,null(zero)
  • utf8bytelength: utf8 的字节长度
  • keys,keys_unsorted: 获取最外层的key以及排序之后的key，如果是列表或者数组，将返回索引
  • has(key): 返回json 中是否包含key，或者数组/列表中是否包含索引下标 【需要注意的是，如果需要解析解析数组/列表内部的子串是否包含某个key，需要先使用map 函数】
  • in(object): 判断json中是否包含object中给定的key
  • max(x),map_values(x): 使用过滤器x对输入数组的元素进行挨个计算，并返回新数组；后者是给值进行计算
  • path(path_expression): 输出给定路径表达式中的数组表示。
  • del(path_expression): 删除路径表达式中的filed
  • getpath(PATHS): 获取指定路径的索引
  • setpath(PATHS; VALUE): 给指定路径设置新的值
  • delpaths(PATHS): 删除指定路径
  • to_entries, from_entries, with_entries: 二次处理json实体，比如从[{key:k1,value: v1},{key:k2,value:v2} ]转换成 {k1:v1,k2:v2}
  • select(boolean_expression): 使用bool 表达式进行条件过滤
  • arrays, objects, iterables, booleans, numbers, normals, finites, strings, nulls, values, scalars: 分别只选择数组、对象、可迭代对象(数组或对象)、布尔值、数字、普通数字、有限数字、字符串、空值、非空值和不可迭代对象的输入。
  • add: 过滤器add接受一个数组作为输入，并将数组的元素加在一起作为输出。这可能意味着根据输入数组元素的类型进行求和、连接或合并——规则与上面描述的+运算符的规则相同。
  • any,all: 从数组或者列表中判断是否存在或者全部存在
  • range:对象生成器
  • floor: 输出数字的低阶值
  • sqrt: 求开方
  • tonumber: 字符串转数字
  • tostring: 数字转字符串
  • type: 获取元素类型
  • sort,sort_by(path_expression): 排序
  • unique, unique_by(path_exp): 去重
  • reverse: 反转
  • contains(element): 判断是否包含
  • startswith(str): 判断前缀
  • endswith(str): 判断后缀
  • split(str): 字符串转列表
  • join(str): 列表转字符串
  • while(cond; update): 条件判断
  • until(cond; next): 条件判断
  • \(foo): 引用foo的值
  • tojson,fromjson: 从原始字符串转到json或者从json转到原始字符串
  • @base64,@base64d: base64 编码和解码
  • @uri,@csv, 生成uri，以及表格格式
## 模版数据
$ testJson='{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/", "Contact": {"Email":"weichuangxxb@qq.com","QQ":"371990778","WeChat":"GoOps"} ,"Skills": [ {"name":"Python","type":"dev" }, {"name":"Golang","type":"dev" },{"name":"Ansible","type":"ops" },{"name":"Kubernetes","type":"dev" },{"name":"ElasticSearch","type":"ops" }]}'

### keys
$ echo ${testJson} | jq 'keys'
[
  "Contact",
  "Name",
  "Owner",
  "Skills",
  "WebSite"
]
$ echo ${testJson} | jq 'keys_unsorted'
[
  "Name",
  "Owner",
  "WebSite",
  "Contact",
  "Skills"
]

### has(key)
$ echo ${testJson} | jq '.Skills | map(has("name"))'
[
  true,
  true,
  true,
  true,
  true
]
$ echo ${testJson} | jq 'has("Name") '
true

### to_entries, from_entries, with_entries
$  echo ${testJson} | jq '.Contact | to_entries'
[
  {
    "key": "Email",
    "value": "weichuangxxb@qq.com"
  },
  {
    "key": "QQ",
    "value": "371990778"
  },
  {
    "key": "WeChat",
    "value": "GoOps"
  }
]

### select(bool_exp)
### 输出技能中包含 Ansible 的技能项
$ echo ${testJson} | jq '.Skills[] | select(.name == "Ansible")'
{
  "name": "Ansible",
  "type": "ops"
}

### tonumber
$  echo ${testJson} |jq '.Contact.QQ | tonumber '
371990778
$ echo ${testJson} |jq '.Contact.QQ '
"371990778"

### tostring/fromjson
$ echo ${testJson} |jq '.Contact | tostring'
"{\"Email\":\"weichuangxxb@qq.com\",\"QQ\":\"371990778\",\"WeChat\":\"GoOps\"}"

$ echo ${testJson} |jq '.Contact | tostring | fromjson'
{
  "Email": "weichuangxxb@qq.com",
  "QQ": "371990778",
  "WeChat": "GoOps"
}

### @base64/@base64d
$ echo ${testJson} |jq '.Contact | .QQ | @base64 | @base64d'
"371990778"
### @uri
$ echo ${testJson} | jq '.Skills[] | select(.name == "Ansible") | @uri "https://www.google.com/search?q=\(.name)"'
"https://www.google.com/search?q=Ansible"

2.5 示例

# 通过接口返回数据进行过滤查找
curl -s  http://goops.top:8080/vpc/api | jq '.returnData.detail[] | select(.ipType == 41)'
avatar
文章
阅读
粉丝