python使用正则表达式从json字符串中取出特定字段的值

惜鸟
3,025 阅读3分钟

这是我参与8月更文挑战的第5天，活动详情查看：8月更文挑战

一. 概述

最近在使用爬虫抓取一些数据，从获取到的 html 文本中无法直接通过 xpath 解析出对应的数据，需要从 js 脚本中解析一些类似于 json 的数据，数据格式如下：

<script src="//mat1.gtimg.com/pingjs/ext2020/configF2017/5df6e3b3.js" charset="utf-8"></script>
<script src="//mat1.gtimg.com/pingjs/ext2020/configF2017/5a978a31.js" charset="utf-8"></script>
<script>window.conf_dcom = apub_5a978a31 </script><!--[if !IE]>|xGv00|61438c491c69d576aec9846de884f28b<![endif]-->
<!--[if !IE]>|xGv00|038d6e161753081e56c192d04873c65c<![endif]-->
    <script>window.DATA = {
		"article_id": "20210808A034KU",
		"article_type": "0",
		"title": "突发！上海出台房贷重磅新规！你的贷款额度标准可能要变了",
		"abstract": null,
		"catalog1": "house",
		"catalog2": "house_zhengce",
		"media": "中国证券报",
		"media_id": "1368",
		"pubtime": "2021-08-08 13:45:25",
		"comment_id": "7221092095",
		"tags": "银行,上海,房贷,贷款,上海市房管局,二手房",
		"political": 0,
		"artTemplate": null,
		"FztCompetition": null,
		"FCompetitionName": null,
		"is_deleted": 0,
		"cms_id": "20210808A034KU00",
		"videoArr": []
}
      
    </script>
  </head>

二、通过正则表达式提取指定字段的值

需要将上面文本中指定的字段的值解析出来

分析字符串的结构可以看出需要匹配内容的结构为："key": "(.*)"

下面自己定义一个函数来实现此需求：

import re

def find_value_by_regex(key, text):
    """
    通过正则表达式提取json指定字段的值
    :param key: 字段名称
    :param text: 要提取的文本
    :return: 字段对应的值，不存在则返回空字符串
    """
    regex = r'"key": "(.*)"'.replace("key", key)
    match = re.compile(regex).findall(text)
    val = ''
    if match:
        val = match[0]
    return val


if __name__ == "__main__":
    text = '''
         {
                "article_id": "20210808A034KU",
                "article_type": "0",
                "title": "突发！上海出台房贷重磅新规！你的贷款额度标准可能要变了",
                "abstract": null,
                "catalog1": "house",
                "catalog2": "house_zhengce",
                "media": "中国证券报",
                "media_id": "1368",
                "pubtime": "2021-08-08 13:45:25",
                "comment_id": "7221092095",
                "tags": "银行,上海,房贷,贷款,上海市房管局,二手房"
        }

    '''
    title = find_value_by_regex("title", text)
    pubtime = find_value_by_regex("pubtime", text)
    print(title)
    print(pubtime)

输出结果如下：

突发！上海出台房贷重磅新规！你的贷款额度标准可能要变了
2021-08-08 13:45:25

上面定义的方法已经解决了这个需求，接下来将提取一些爬取网页的工具方法方便复用，详细代码如下所示：

#!/usr/bin/env python
# -*- coding:utf-8 -*-
import random
import requests
import urllib3


def get_random_agent():
    """
    随机获取 agent
    :return:
    """
    agent_list = [
        "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; AcooBrowser; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",
        "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Acoo Browser; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)",
        "Mozilla/4.0 (compatible; MSIE 7.0; AOL 9.5; AOLBuild 4337.35; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",
        "Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)",
        "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)",
        "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 1.0.3705; .NET CLR 1.1.4322)",
        "Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 3.0.04506.30)",
        "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN) AppleWebKit/523.15 (KHTML, like Gecko, Safari/419.3) Arora/0.3 (Change: 287 c9dfb30)",
        "Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/527+ (KHTML, like Gecko, Safari/419.3) Arora/0.6",
        "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2pre) Gecko/20070215 K-Ninja/2.1.1",
        "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9) Gecko/20080705 Firefox/3.0 Kapiko/3.0",
        "Mozilla/5.0 (X11; Linux i686; U;) Gecko/20070322 Kazehakase/0.4.5",
        "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko Fedora/1.9.0.8-1.fc10 Kazehakase/0.5.6",
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/535.20 (KHTML, like Gecko) Chrome/19.0.1036.7 Safari/535.20",
        "Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; fr) Presto/2.9.168 Version/11.52",
    ]
    agent = random.choice(agent_list)
    return agent


def get_headers():
    """
    发送HTTP请求时的HEAD信息
    :return:
    """
    headers = {
        'Connection': 'Keep-Alive',
        'Accept': 'text/html, application/xhtml+xml, */*',
        'Accept-Language': 'en-US,en;q=0.8,zh-Hans-CN;q=0.5,zh-Hans;q=0.3',
        'Accept-Encoding': 'gzip, deflate',
        'User-Agent': get_random_agent()
    }
    return headers


def get_html(url):
    """
    通过 url 获取网页 html 文本 
    :param url: 
    :return: 
    """
    # 设置禁用 https 警告
    urllib3.disable_warnings()
    response = requests.get(url, headers=get_headers(), verify=False)
    get_encoding = response.apparent_encoding  # 获取编码

    response.encoding = get_encoding
    return response.text
avatar
文章
阅读
粉丝