头条

今日目标：获取头条的新闻资讯

1.前端必备

1.1 基础

三元运算

v1 = 条件 ? 值A : 值B;

res = 1 === 1 ? 99 : 88;
console.log(res);  // 99

赋值和比较

v1 = 11 === (n=123)

console.log(n);  // 123
console.log(v1); // false

逻辑运算

v1 = true || true

v2 = 9 || 14             # v2=9
v3 = 0 || 15             # v3=15
v4 = 0 || 15 || "武沛齐"  # v4=15

最终结果取决那个值？结果就是那个值。

来一波案例

v1 = 11 || 22 ? 33 : 44
v1 = 11 ? 33 : 44
v1 = 33

v1 = 22 ? 33 : 44 || 55
v1 = 33

var o = (null === (n = window.byted_acrawler) || void 0 === n ? void 0 : null === (a = n.sign) || void 0 === a ? void 0 : a.call(n, i)) || "";

n = window.byted_acrawler             -> 确定不为空
a = window.byted_acrawler.sign        -> 确定不为空
window.byted_acrawler.sign.call(n, i) -> 确定不为空

var o = window.byted_acrawler.sign.call(n, i)

1.2 函数

执行函数

function sign(v1){
    // this
    console.log(v1);
}
// 执行，函数内部this=window
sign(123)

// 执行，函数内部 this=123
sign.call(123,456)

// n就会传递给call函数中this
// i当做参数传递
var o = window.byted_acrawler.sign.call(n,i)
var o = window.byted_acrawler.sign(i)

函数的参数

function sign(){
    console.log(arguments)
}

sign()
sign(11,22,33)
sign(11,22,44,55)

1.3 其他

ES5，语法100
ES6，语法100 + 50

v1 = { k1: 123 }
v2 = { k2:99, k3:888}

Object.assign(v1, v2)

console.log(v1) # {k1: 123, k2:99, k3:888 }

属性从一个或多个源对象复制到目标对象，返回修改后的对象。。

2.编译js代码

2.1 node.js编译代码

环境准备：node.js

v1.js

function func(arg) {
    return arg + 'i666';
}
let data = func("老铁");
console.log(data)

node编译执行

python执行执行本地命令：node v1.js

import os
import subprocess

# 根据自己的操作系统去修改（相当于python的sys.path，加载安装的模块）
os.environ["NODE_PATH"] = "/usr/local/lib/node_modules/"  

signature = subprocess.getoutput('node v1.js')

2.2 pyexecjs编译代码

本质还是调用node.js去编译代码。

准备环境：

node.js
pyexecjs模块
```
pip install pyexecjs
```

例如：

v2.js

function func(arg) {
    return arg + '666';
}

执行js代码

import execjs
import os

os.environ["NODE_PATH"] = "/usr/local/lib/node_modules/"
with open('v2.js', mode='r', encoding='utf-8') as f:
    js = f.read()

JS = execjs.compile(js)

sign = JS.call("func", "微信")
print(sign) # 微信666

本质上都是依赖node.js

电脑上安装上node.js之后（编译器，相当于装CPython解释器）
自动安装npm（第三方包管理器，相当于pip）

2.3 浏览器环境

环境准备：

node.js
jsdom（通过后端node+js代码实现伪造浏览器环境）
```
npm install node-gyp@latest sudo npm explore -g npm -- npm i node-gyp@latest

npm install jsdom -g
```
注意：上述安装成功后已可以模拟浏览器环境，由于今天的头条他的内容。
```
npm install canvas -g
```

2.3.1 方式1

const jsdom = require("jsdom");
const {JSDOM} = jsdom;

const resourceLoader = new jsdom.ResourceLoader({
    userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.55 Safari/537.36",
});

const html = `<!DOCTYPE html><p>Hello world</p>`;


const dom = new JSDOM(html, {
    url: "https://www.toutiao.com",
    referrer: "https://example.com/",
    contentType: "text/html",
    resources: resourceLoader,
});

console.log(dom.window.location)
console.log(dom.window.navigator.userAgent)
console.log(dom.window.document.referrer)

import os
import subprocess

# 根据自己的操作系统去修改（相当于python的sys.path，加载安装的模块）
os.environ["NODE_PATH"] = "/usr/local/lib/node_modules/"  

res = subprocess.getoutput('node v10.js')

2.3.2 方式2

const jsdom = require("jsdom");
const {JSDOM} = jsdom;

const resourceLoader = new jsdom.ResourceLoader({
    userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.55 Safari/537.36"
});

const html = `<!DOCTYPE html><p>Hello world</p>`;
const dom = new JSDOM(html, {
    url: "https://www.toutiao.com",
    referrer: "https://example.com/",
    contentType: "text/html",
    resources: resourceLoader,
});


//window = {}
window = global;

const params = {
    location: {
        hash: "",
        host: "www.toutiao.com",
        hostname: "www.toutiao.com",
        href: "https://www.toutiao.com",
        origin: "https://www.toutiao.com",
        pathname: "/",
        port: "",
        protocol: "https:",
        search: "",
    },
    navigator: {
        appCodeName: "Mozilla",
        appName: "Netscape",
        appVersion: "5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36",
        cookieEnabled: true,
        deviceMemory: 8,
        doNotTrack: null,
        hardwareConcurrency: 4,
        language: "zh-CN",
        languages: ["zh-CN", "zh"],
        maxTouchPoints: 0,
        onLine: true,
        platform: "MacIntel",
        product: "Gecko",
        productSub: "20030107",
        userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36",
        vendor: "Google Inc.",
        vendorSub: "",
        webdriver: false
    }
};

Object.assign(global,params);


//在下面如果你使用
location.href
navigator.appCodeName
window.location.href
window.appCodeName

注意：在nodejs中默认代码中会有一个global的关键字（全局变量）。

v1 = 123;
console.log(global);

global.v1 = 123
global.v2 = 123
global.navigator = {
	...
}
console.log(v1,v2);


navigator.userAgent

3.头条

3.1 分析请求

直接发送获取到结果：

import requests

res = requests.get(
    url="https://www.toutiao.com/api/pc/list/feed?channel_id=3189398957&max_behot_time=1642677189&category=pc_profile_channel&aid=24&app_name=toutiao_web&_signature=_02B4Z6wo00901DbrCdwAAIDByKd17DEKVtQ2zw1AAGxaLleFdEGNtgeXvdLnHqGQxe0n6aKnSt7I4BcTARJB9JJIAybeDkDFa-WH8Kfeq0u35BvzcA1tCRcQRb-.kBWC6eyn8KNWXHs271pf47",
    headers={
        "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36"
    }
)

print(res.text)

3.2 _signature

n=undefined;
i = {url:"https://www.toutiao.com/api/pc/list/feed?channel_id=3189398957&max_behot_time=1642685645&category=pc_profile_channel&aid=24&app_name=toutiao_web"}
var o = window.byted_acrawler.sign.call(n,i);

i = {url:"https://www.toutiao.com/api/pc/list/feed?channel_id=3189398957&max_behot_time=1642685645&category=pc_profile_channel&aid=24&app_name=toutiao_web"}
var o = window.byted_acrawler.sign(i);

找到sign算法，看看他是内部实现（走不通）。
应该有一个js，给全局变量中赋值，整体调用试试。

3.3 验证签名是否可用

import requests

res = requests.get(
    url="https://www.toutiao.com/api/pc/list/feed?channel_id=3189398957&max_behot_time=1642685645&category=pc_profile_channel&aid=24&app_name=toutiao_web&_signature=_02B4Z6wo00f01gfGxKAAAIDD-Yq4k6Rpx4oH6sAAAOAmr1zcfFgp6wmN73Rb5YObk9s-xG7dpC2shJNIvInOjEYhMJS1OV8iGNOVrpOl0wWoVDQEFxH3q9xU1Q.qAEgkZ0FNuc7otShtBXPe14",
    headers={
        "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36"
    }
)

print(res.text)

3.4 补环境运行

v20.js

const jsdom = require("jsdom");
const {JSDOM} = jsdom;

const resourceLoader = new jsdom.ResourceLoader({
    userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.55 Safari/537.36"
});

const html = `<!DOCTYPE html><p>Hello world</p>`;
const dom = new JSDOM(html, {
    url: "https://www.toutiao.com",
    referrer: "https://example.com/",
    contentType: "text/html",
    resources: resourceLoader,
});

/*
console.log(dom.window.location)
console.log(dom.window.navigator.userAgent)
console.log(dom.window.document.referrer)
*/


document = dom.window.document

window = global;

const params = {
    location: {
        hash: "",
        host: "www.toutiao.com",
        hostname: "www.toutiao.com",
        href: "https://www.toutiao.com",
        origin: "https://www.toutiao.com",
        pathname: "/",
        port: "",
        protocol: "https:",
        search: "",
    },
    navigator: {
        appCodeName: "Mozilla",
        appName: "Netscape",
        appVersion: "5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36",
        cookieEnabled: true,
        deviceMemory: 8,
        doNotTrack: null,
        hardwareConcurrency: 4,
        language: "zh-CN",
        languages: ["zh-CN", "zh"],
        maxTouchPoints: 0,
        onLine: true,
        platform: "MacIntel",
        product: "Gecko",
        productSub: "20030107",
        userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36",
        vendor: "Google Inc.",
        vendorSub: "",
        webdriver: false
    }
};

Object.assign(global, params)


// 浏览器执行，在函数内部会创建全局变量 byted_acrawler  ==》 global
window._$jsvmprt = function (b, e, f) {
    function a() {
        if ("undefined" == typeof Reflect || !Reflect.construct)
            return !1;
        if (Reflect.construct.sham)
            return !1;
        if ("function" == typeof Proxy)
            return !0;
        try {
            return Date.prototype.toString.call(Reflect.construct(Date, [], (function () {
                }
            ))),
                !0
        } catch (b) {
            return !1
        }
    }

    function d(b, e, f) {
        return (d = a() ? Reflect.construct : function (b, e, f) {
                var a = [null];
                a.push.apply(a, e);
                var d = new (Function.bind.apply(b, a));
                return f && c(d, f.prototype),
                    d
            }
        ).apply(null, arguments)
    }

    function c(b, e) {
        return (c = Object.setPrototypeOf || function (b, e) {
                return b.__proto__ = e,
                    b
            }
        )(b, e)
    }

    function n(b) {
        return function (b) {
            if (Array.isArray(b)) {
                for (var e = 0, f = new Array(b.length); e < b.length; e++)
                    f[e] = b[e];
                return f
            }
        }(b) || function (b) {
            if (Symbol.iterator in Object(b) || "[object Arguments]" === Object.prototype.toString.call(b))
                return Array.from(b)
        }(b) || function () {
            throw new TypeError("Invalid attempt to spread non-iterable instance")
        }()
    }

    for (var i = [], r = 0, t = [], o = 0, l = function (b, e) {
        var f = b[e++]
            , a = b[e]
            , d = parseInt("" + f + a, 16);
        if (d >> 7 == 0)
            return [1, d];
        if (d >> 6 == 2) {
            var c = parseInt("" + b[++e] + b[++e], 16);
            return d &= 63,
                [2, c = (d <<= 8) + c]
        }
        if (d >> 6 == 3) {
            var n = parseInt("" + b[++e] + b[++e], 16)
                , i = parseInt("" + b[++e] + b[++e], 16);
            return d &= 63,
                [3, i = (d <<= 16) + (n <<= 8) + i]
        }
    }, u = function (b, e) {
        var f = parseInt("" + b[e] + b[e + 1], 16);
        return f = f > 127 ? -256 + f : f
    }, s = function (b, e) {
        var f = parseInt("" + b[e] + b[e + 1] + b[e + 2] + b[e + 3], 16);
        return f = f > 32767 ? -65536 + f : f
    }, p = function (b, e) {
        var f = parseInt("" + b[e] + b[e + 1] + b[e + 2] + b[e + 3] + b[e + 4] + b[e + 5] + b[e + 6] + b[e + 7], 16);
        return f = f > 2147483647 ? 0 + f : f
    }, y = function (b, e) {
        return parseInt("" + b[e] + b[e + 1], 16)
    }, v = function (b, e) {
        return parseInt("" + b[e] + b[e + 1] + b[e + 2] + b[e + 3], 16)
    }, g = g || this || window, h = Object.keys || function (b) {
        var e = {}
            , f = 0;
        for (var a in b)
            e[f++] = a;
        return e.length = f,
            e
    }
             , m = (b.length,
            0), I = "", C = m; C < m + 16; C++) {
        var q = "" + b[C++] + b[C];
        q = parseInt(q, 16),
            I += String.fromCharCode(q)
    }
    if ("HNOJ@?RC" != I)
        throw new Error("error magic number " + I);
    m += 16;
    parseInt("" + b[m] + b[m + 1], 16);
    m += 8,
        r = 0;
    for (var w = 0; w < 4; w++) {
        var S = m + 2 * w
            , R = "" + b[S++] + b[S]
            , x = parseInt(R, 16);
        r += (3 & x) << 2 * w
    }
    m += 16,
        m += 8;
    var z = parseInt("" + b[m] + b[m + 1] + b[m + 2] + b[m + 3] + b[m + 4] + b[m + 5] + b[m + 6] + b[m + 7], 16)
        , O = z
        , E = m += 8
        , j = v(b, m += z);
    j[1];
    m += 4,
        i = {
            p: [],
            q: []
        };
    for (var A = 0; A < j; A++) {
        for (var D = l(b, m), T = m += 2 * D[0], $ = i.p.length, P = 0; P < D[1]; P++) {
            var U = l(b, T);
            i.p.push(U[1]),
                T += 2 * U[0]
        }
        m = T,
            i.q.push([$, i.p.length])
    }
    var _ = {
        5: 1,
        6: 1,
        70: 1,
        22: 1,
        23: 1,
        37: 1,
        73: 1
    }
        , k = {
        72: 1
    }
        , M = {
        74: 1
    }
        , H = {
        11: 1,
        12: 1,
        24: 1,
        26: 1,
        27: 1,
        31: 1
    }
        , J = {
        10: 1
    }
        , N = {
        2: 1,
        29: 1,
        30: 1,
        20: 1
    }
        , B = []
        , W = [];

    function F(b, e, f) {
        for (var a = e; a < e + f;) {
            var d = y(b, a);
            B[a] = d,
                a += 2;
            k[d] ? (W[a] = u(b, a),
                a += 2) : _[d] ? (W[a] = s(b, a),
                a += 4) : M[d] ? (W[a] = p(b, a),
                a += 8) : H[d] ? (W[a] = y(b, a),
                a += 2) : J[d] ? (W[a] = v(b, a),
                a += 4) : N[d] && (W[a] = v(b, a),
                a += 4)
        }
    }

    return K(b, E, O / 2, [], e, f);

    function G(b, e, f, a, c, l, m, I) {
        null == l && (l = this);
        var C, q, w, S = [], R = 0;
        m && (C = m);
        var x, z, O = e, E = O + 2 * f;
        if (!I)
            for (; O < E;) {
                var j = parseInt("" + b[O] + b[O + 1], 16);
                O += 2;
                var A = 3 & (x = 13 * j % 241);
                if (x >>= 2,
                A < 1) {
                    A = 3 & x;
                    if (x >>= 2,
                    A > 2)
                        (A = x) > 10 ? S[++R] = void 0 : A > 1 ? (C = S[R--],
                            S[R] = S[R] >= C) : A > -1 && (S[++R] = null);
                    else if (A > 1) {
                        if ((A = x) > 11)
                            throw S[R--];
                        if (A > 7) {
                            for (C = S[R--],
                                     z = v(b, O),
                                     A = "",
                                     P = i.q[z][0]; P < i.q[z][1]; P++)
                                A += String.fromCharCode(r ^ i.p[P]);
                            O += 4,
                                S[R--][A] = C
                        } else
                            A > 5 && (S[R] = h(S[R]))
                    } else if (A > 0) {
                        (A = x) > 8 ? (C = S[R--],
                            S[R] = typeof C) : A > 6 ? S[R] = --S[R] : A > 4 ? S[R -= 1] = S[R][S[R + 1]] : A > 2 && (q = S[R--],
                            (A = S[R]).x === G ? A.y >= 1 ? S[R] = K(b, A.c, A.l, [q], A.z, w, null, 1) : (S[R] = K(b, A.c, A.l, [q], A.z, w, null, 0),
                                A.y++) : S[R] = A(q))
                    } else {
                        if ((A = x) > 14)
                            z = s(b, O),
                                (U = function e() {
                                        var f = arguments;
                                        return e.y > 0 ? K(b, e.c, e.l, f, e.z, this, null, 0) : (e.y++,
                                            K(b, e.c, e.l, f, e.z, this, null, 0))
                                    }
                                ).c = O + 4,
                                U.l = z - 2,
                                U.x = G,
                                U.y = 0,
                                U.z = c,
                                S[R] = U,
                                O += 2 * z - 2;
                        else if (A > 12)
                            q = S[R--],
                                w = S[R--],
                                (A = S[R--]).x === G ? A.y >= 1 ? S[++R] = K(b, A.c, A.l, q, A.z, w, null, 1) : (S[++R] = K(b, A.c, A.l, q, A.z, w, null, 0),
                                    A.y++) : S[++R] = A.apply(w, q);
                        else if (A > 5)
                            C = S[R--],
                                S[R] = S[R] != C;
                        else if (A > 3)
                            C = S[R--],
                                S[R] = S[R] * C;
                        else if (A > -1)
                            return [1, S[R--]]
                    }
                } else if (A < 2) {
                    A = 3 & x;
                    if (x >>= 2,
                    A < 1) {
                        if ((A = x) > 9)
                            ;
                        else if (A > 7)
                            C = S[R--],
                                S[R] = S[R] & C;
                        else if (A > 5)
                            z = y(b, O),
                                O += 2,
                                S[R -= z] = 0 === z ? new S[R] : d(S[R], n(S.slice(R + 1, R + z + 1)));
                        else if (A > 3) {
                            z = s(b, O);
                            try {
                                if (t[o][2] = 1,
                                1 == (C = G(b, O + 4, z - 3, [], c, l, null, 0))[0])
                                    return C
                            } catch (m) {
                                if (t[o] && t[o][1] && 1 == (C = G(b, t[o][1][0], t[o][1][1], [], c, l, m, 0))[0])
                                    return C
                            } finally {
                                if (t[o] && t[o][0] && 1 == (C = G(b, t[o][0][0], t[o][0][1], [], c, l, null, 0))[0])
                                    return C;
                                t[o] = 0,
                                    o--
                            }
                            O += 2 * z - 2
                        }
                    } else if (A < 2) {
                        if ((A = x) > 12)
                            S[++R] = u(b, O),
                                O += 2;
                        else if (A > 10)
                            C = S[R--],
                                S[R] = S[R] << C;
                        else if (A > 8) {
                            for (z = v(b, O),
                                     A = "",
                                     P = i.q[z][0]; P < i.q[z][1]; P++)
                                A += String.fromCharCode(r ^ i.p[P]);
                            O += 4,
                                S[R] = S[R][A]
                        } else
                            A > 6 && (q = S[R--],
                                C = delete S[R--][q])
                    } else if (A < 3) {
                        (A = x) < 2 ? S[++R] = C : A < 4 ? (C = S[R--],
                            S[R] = S[R] <= C) : A < 11 ? (C = S[R -= 2][S[R + 1]] = S[R + 2],
                            R--) : A < 13 && (C = S[R],
                            S[++R] = C)
                    } else {
                        if ((A = x) > 12)
                            S[++R] = l;
                        else if (A > 5)
                            C = S[R--],
                                S[R] = S[R] !== C;
                        else if (A > 3)
                            C = S[R--],
                                S[R] = S[R] / C;
                        else if (A > 1) {
                            if ((z = s(b, O)) < 0) {
                                I = 1,
                                    F(b, e, 2 * f),
                                    O += 2 * z - 2;
                                break
                            }
                            O += 2 * z - 2
                        } else
                            A > -1 && (S[R] = !S[R])
                    }
                } else if (A < 3) {
                    A = 3 & x;
                    if (x >>= 2,
                    A > 2)
                        (A = x) > 7 ? (C = S[R--],
                            S[R] = S[R] | C) : A > 5 ? (z = y(b, O),
                            O += 2,
                            S[++R] = c["$" + z]) : A > 3 && (z = s(b, O),
                            t[o][0] && !t[o][2] ? t[o][1] = [O + 4, z - 3] : t[o++] = [0, [O + 4, z - 3], 0],
                            O += 2 * z - 2);
                    else if (A > 1) {
                        if ((A = x) < 2) {
                            for (z = v(b, O),
                                     C = "",
                                     P = i.q[z][0]; P < i.q[z][1]; P++)
                                C += String.fromCharCode(r ^ i.p[P]);
                            S[++R] = C,
                                O += 4
                        } else if (A < 4)
                            if (S[R--])
                                O += 4;
                            else {
                                if ((z = s(b, O)) < 0) {
                                    I = 1,
                                        F(b, e, 2 * f),
                                        O += 2 * z - 2;
                                    break
                                }
                                O += 2 * z - 2
                            }
                        else
                            A < 6 ? (C = S[R--],
                                S[R] = S[R] % C) : A < 8 ? (C = S[R--],
                                S[R] = S[R] instanceof C) : A < 15 && (S[++R] = !1)
                    } else if (A > 0) {
                        (A = x) < 1 ? S[++R] = g : A < 3 ? (C = S[R--],
                            S[R] = S[R] + C) : A < 5 ? (C = S[R--],
                            S[R] = S[R] == C) : A < 14 && (C = S[R - 1],
                            q = S[R],
                            S[++R] = C,
                            S[++R] = q)
                    } else {
                        (A = x) < 2 ? (C = S[R--],
                            S[R] = S[R] > C) : A < 9 ? (z = v(b, O),
                            O += 4,
                            q = R + 1,
                            S[R -= z - 1] = z ? S.slice(R, q) : []) : A < 11 ? (z = y(b, O),
                            O += 2,
                            C = S[R--],
                            c[z] = C) : A < 13 ? (C = S[R--],
                            S[R] = S[R] >> C) : A < 15 && (S[++R] = s(b, O),
                            O += 4)
                    }
                } else {
                    A = 3 & x;
                    if (x >>= 2,
                    A > 2)
                        (A = x) > 13 ? (S[++R] = p(b, O),
                            O += 8) : A > 11 ? (C = S[R--],
                            S[R] = S[R] >>> C) : A > 9 ? S[++R] = !0 : A > 7 ? (z = y(b, O),
                            O += 2,
                            S[R] = S[R][z]) : A > 0 && (C = S[R--],
                            S[R] = S[R] < C);
                    else if (A > 1) {
                        (A = x) > 10 ? (z = s(b, O),
                            t[++o] = [[O + 4, z - 3], 0, 0],
                            O += 2 * z - 2) : A > 8 ? (C = S[R--],
                            S[R] = S[R] ^ C) : A > 6 && (C = S[R--])
                    } else if (A > 0) {
                        if ((A = x) < 3) {
                            var D = 0
                                , T = S[R].length
                                , $ = S[R];
                            S[++R] = function () {
                                var b = D < T;
                                if (b) {
                                    var e = $[D++];
                                    S[++R] = e
                                }
                                S[++R] = b
                            }
                        } else
                            A < 5 ? (z = y(b, O),
                                O += 2,
                                C = c[z],
                                S[++R] = C) : A < 7 ? S[R] = ++S[R] : A < 9 && (C = S[R--],
                                S[R] = S[R] in C)
                    } else {
                        if ((A = x) > 13)
                            C = S[R],
                                S[R] = S[R - 1],
                                S[R - 1] = C;
                        else if (A > 4)
                            C = S[R--],
                                S[R] = S[R] === C;
                        else if (A > 2)
                            C = S[R--],
                                S[R] = S[R] - C;
                        else if (A > 0) {
                            for (z = v(b, O),
                                     A = "",
                                     P = i.q[z][0]; P < i.q[z][1]; P++)
                                A += String.fromCharCode(r ^ i.p[P]);
                            A = +A,
                                O += 4,
                                S[++R] = A
                        }
                    }
                }
            }
        if (I)
            for (; O < E;) {
                j = B[O];
                O += 2;
                A = 3 & (x = 13 * j % 241);
                if (x >>= 2,
                A < 1) {
                    var U;
                    A = 3 & x;
                    if (x >>= 2,
                    A < 1) {
                        if ((A = x) > 14)
                            z = W[O],
                                (U = function e() {
                                        var f = arguments;
                                        return e.y > 0 ? K(b, e.c, e.l, f, e.z, this, null, 0) : (e.y++,
                                            K(b, e.c, e.l, f, e.z, this, null, 0))
                                    }
                                ).c = O + 4,
                                U.l = z - 2,
                                U.x = G,
                                U.y = 0,
                                U.z = c,
                                S[R] = U,
                                O += 2 * z - 2;
                        else if (A > 12)
                            q = S[R--],
                                w = S[R--],
                                (A = S[R--]).x === G ? A.y >= 1 ? S[++R] = K(b, A.c, A.l, q, A.z, w, null, 1) : (S[++R] = K(b, A.c, A.l, q, A.z, w, null, 0),
                                    A.y++) : S[++R] = A.apply(w, q);
                        else if (A > 5)
                            C = S[R--],
                                S[R] = S[R] != C;
                        else if (A > 3)
                            C = S[R--],
                                S[R] = S[R] * C;
                        else if (A > -1)
                            return [1, S[R--]]
                    } else if (A < 2) {
                        (A = x) < 4 ? (q = S[R--],
                            (A = S[R]).x === G ? A.y >= 1 ? S[R] = K(b, A.c, A.l, [q], A.z, w, null, 1) : (S[R] = K(b, A.c, A.l, [q], A.z, w, null, 0),
                                A.y++) : S[R] = A(q)) : A < 6 ? S[R -= 1] = S[R][S[R + 1]] : A < 8 ? S[R] = --S[R] : A < 10 && (C = S[R--],
                            S[R] = typeof C)
                    } else if (A < 3) {
                        if ((A = x) > 11)
                            throw S[R--];
                        if (A > 7) {
                            for (C = S[R--],
                                     z = W[O],
                                     A = "",
                                     P = i.q[z][0]; P < i.q[z][1]; P++)
                                A += String.fromCharCode(r ^ i.p[P]);
                            O += 4,
                                S[R--][A] = C
                        } else
                            A > 5 && (S[R] = h(S[R]))
                    } else {
                        (A = x) < 1 ? S[++R] = null : A < 3 ? (C = S[R--],
                            S[R] = S[R] >= C) : A < 12 && (S[++R] = void 0)
                    }
                } else if (A < 2) {
                    A = 3 & x;
                    if (x >>= 2,
                    A > 2)
                        (A = x) > 12 ? S[++R] = l : A > 5 ? (C = S[R--],
                            S[R] = S[R] !== C) : A > 3 ? (C = S[R--],
                            S[R] = S[R] / C) : A > 1 ? O += 2 * (z = W[O]) - 2 : A > -1 && (S[R] = !S[R]);
                    else if (A > 1) {
                        (A = x) < 2 ? S[++R] = C : A < 4 ? (C = S[R--],
                            S[R] = S[R] <= C) : A < 11 ? (C = S[R -= 2][S[R + 1]] = S[R + 2],
                            R--) : A < 13 && (C = S[R],
                            S[++R] = C)
                    } else if (A > 0) {
                        if ((A = x) < 8)
                            q = S[R--],
                                C = delete S[R--][q];
                        else if (A < 10) {
                            for (z = W[O],
                                     A = "",
                                     P = i.q[z][0]; P < i.q[z][1]; P++)
                                A += String.fromCharCode(r ^ i.p[P]);
                            O += 4,
                                S[R] = S[R][A]
                        } else
                            A < 12 ? (C = S[R--],
                                S[R] = S[R] << C) : A < 14 && (S[++R] = W[O],
                                O += 2)
                    } else {
                        if ((A = x) < 5) {
                            z = W[O];
                            try {
                                if (t[o][2] = 1,
                                1 == (C = G(b, O + 4, z - 3, [], c, l, null, 0))[0])
                                    return C
                            } catch (m) {
                                if (t[o] && t[o][1] && 1 == (C = G(b, t[o][1][0], t[o][1][1], [], c, l, m, 0))[0])
                                    return C
                            } finally {
                                if (t[o] && t[o][0] && 1 == (C = G(b, t[o][0][0], t[o][0][1], [], c, l, null, 0))[0])
                                    return C;
                                t[o] = 0,
                                    o--
                            }
                            O += 2 * z - 2
                        } else
                            A < 7 ? (z = W[O],
                                O += 2,
                                S[R -= z] = 0 === z ? new S[R] : d(S[R], n(S.slice(R + 1, R + z + 1)))) : A < 9 && (C = S[R--],
                                S[R] = S[R] & C)
                    }
                } else if (A < 3) {
                    A = 3 & x;
                    if (x >>= 2,
                    A < 1)
                        (A = x) < 2 ? (C = S[R--],
                            S[R] = S[R] > C) : A < 9 ? (z = W[O],
                            O += 4,
                            q = R + 1,
                            S[R -= z - 1] = z ? S.slice(R, q) : []) : A < 11 ? (z = W[O],
                            O += 2,
                            C = S[R--],
                            c[z] = C) : A < 13 ? (C = S[R--],
                            S[R] = S[R] >> C) : A < 15 && (S[++R] = W[O],
                            O += 4);
                    else if (A < 2) {
                        (A = x) < 1 ? S[++R] = g : A < 3 ? (C = S[R--],
                            S[R] = S[R] + C) : A < 5 ? (C = S[R--],
                            S[R] = S[R] == C) : A < 14 && (C = S[R - 1],
                            q = S[R],
                            S[++R] = C,
                            S[++R] = q)
                    } else if (A < 3) {
                        if ((A = x) < 2) {
                            for (z = W[O],
                                     C = "",
                                     P = i.q[z][0]; P < i.q[z][1]; P++)
                                C += String.fromCharCode(r ^ i.p[P]);
                            S[++R] = C,
                                O += 4
                        } else
                            A < 4 ? S[R--] ? O += 4 : O += 2 * (z = W[O]) - 2 : A < 6 ? (C = S[R--],
                                S[R] = S[R] % C) : A < 8 ? (C = S[R--],
                                S[R] = S[R] instanceof C) : A < 15 && (S[++R] = !1)
                    } else {
                        (A = x) > 7 ? (C = S[R--],
                            S[R] = S[R] | C) : A > 5 ? (z = W[O],
                            O += 2,
                            S[++R] = c["$" + z]) : A > 3 && (z = W[O],
                            t[o][0] && !t[o][2] ? t[o][1] = [O + 4, z - 3] : t[o++] = [0, [O + 4, z - 3], 0],
                            O += 2 * z - 2)
                    }
                } else {
                    A = 3 & x;
                    if (x >>= 2,
                    A > 2)
                        (A = x) > 13 ? (S[++R] = W[O],
                            O += 8) : A > 11 ? (C = S[R--],
                            S[R] = S[R] >>> C) : A > 9 ? S[++R] = !0 : A > 7 ? (z = W[O],
                            O += 2,
                            S[R] = S[R][z]) : A > 0 && (C = S[R--],
                            S[R] = S[R] < C);
                    else if (A > 1) {
                        (A = x) > 10 ? (z = W[O],
                            t[++o] = [[O + 4, z - 3], 0, 0],
                            O += 2 * z - 2) : A > 8 ? (C = S[R--],
                            S[R] = S[R] ^ C) : A > 6 && (C = S[R--])
                    } else if (A > 0) {
                        if ((A = x) > 7)
                            C = S[R--],
                                S[R] = S[R] in C;
                        else if (A > 5)
                            S[R] = ++S[R];
                        else if (A > 3)
                            z = W[O],
                                O += 2,
                                C = c[z],
                                S[++R] = C;
                        else if (A > 1) {
                            D = 0,
                                T = S[R].length,
                                $ = S[R];
                            S[++R] = function () {
                                var b = D < T;
                                if (b) {
                                    var e = $[D++];
                                    S[++R] = e
                                }
                                S[++R] = b
                            }
                        }
                    } else {
                        if ((A = x) < 2) {
                            for (z = W[O],
                                     A = "",
                                     P = i.q[z][0]; P < i.q[z][1]; P++)
                                A += String.fromCharCode(r ^ i.p[P]);
                            A = +A,
                                O += 4,
                                S[++R] = A
                        } else
                            A < 4 ? (C = S[R--],
                                S[R] = S[R] - C) : A < 6 ? (C = S[R--],
                                S[R] = S[R] === C) : A < 15 && (C = S[R],
                                S[R] = S[R - 1],
                                S[R - 1] = C)
                    }
                }
            }
        return [0, null]
    }

    function K(b, e, f, a, d, c, n, i) {
        var r, t;
        null == c && (c = this),
        d && !d.d && (d.d = 0,
            d.$0 = d,
            d[1] = {});
        var o = {}
            , l = o.d = d ? d.d + 1 : 0;
        for (o["$" + l] = o,
                 t = 0; t < l; t++)
            o[r = "$" + t] = d[r];
        for (t = 0,
                 l = o.length = a.length; t < l; t++)
            o[t] = a[t];
        return i && !B[e] && F(b, e, 2 * f),
            B[e] ? G(b, e, f, 0, o, c, null, 1)[1] : G(b, e, f, 0, o, c, null, 0)[1]
    }
};

// (glb = "undefined" == typeof window ? global : window)
const v1 = "484e4f4a403f5243000d2d1aea78184c36c3d671";
window._$jsvmprt(v1, [, , void 0, void 0, void 0, "undefined" != typeof Object ? Object : void 0, void 0, "undefined" != typeof TypeError ? TypeError : void 0, "undefined" != typeof document ? document : void 0, void 0, void 0, "undefined" != typeof Date ? Date : void 0, "undefined" != typeof Math ? Math : void 0, "undefined" != typeof navigator ? navigator : void 0, "undefined" != typeof location ? location : void 0, "undefined" != typeof history ? history : void 0, "undefined" != typeof Image ? Image : void 0, "undefined" != typeof console ? console : void 0, "undefined" != typeof PluginArray ? PluginArray : void 0, "undefined" != typeof indexedDB ? indexedDB : void 0, "undefined" != typeof DOMException ? DOMException : void 0, "undefined" != typeof parseInt ? parseInt : void 0, "undefined" != typeof String ? String : void 0, "undefined" != typeof Array ? Array : void 0, "undefined" != typeof Error ? Error : void 0, "undefined" != typeof JSON ? JSON : void 0, "undefined" != typeof Promise ? Promise : void 0, "undefined" != typeof WebSocket ? WebSocket : void 0, "undefined" != typeof eval ? eval : void 0, "undefined" != typeof setTimeout ? setTimeout : void 0, "undefined" != typeof encodeURIComponent ? encodeURIComponent : void 0, "undefined" != typeof encodeURI ? encodeURI : void 0, "undefined" != typeof Request ? Request : void 0, "undefined" != typeof Headers ? Headers : void 0, "undefined" != typeof decodeURIComponent ? decodeURIComponent : void 0, "undefined" != typeof RegExp ? RegExp : void 0]);


let targetUrl = process.argv[2];
let sign = window.byted_acrawler.sign({url: targetUrl})
console.log(sign)

app.py

import os
import requests
import subprocess

# 根据自己的操作系统去修改（相当于python的sys.path，加载安装的模块）
os.environ["NODE_PATH"] = "/usr/local/lib/node_modules/"

url = "https://www.toutiao.com/api/pc/list/feed?channel_id=3189398957&max_behot_time=1642685645&category=pc_profile_channel&aid=24&app_name=toutiao_web"

signature = subprocess.getoutput(f'node v20.js "{url}"')

final_url = f"{url}&_signature={signature}"

res = requests.get(
    url=final_url,
    headers={
        "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36"
    }
)

print(res.text)

3.5 pyexecjs

import os
import requests
import subprocess
import execjs
import os

os.environ["NODE_PATH"] = "/usr/local/lib/node_modules/"
with open('v20.js', mode='r', encoding='utf-8') as f:
    js = f.read()
JS = execjs.compile(js)

url = "https://www.toutiao.com/api/pc/list/feed?channel_id=3189398957&max_behot_time=1642685645&category=pc_profile_channel&aid=24&app_name=toutiao_web"
signature = JS.call("get_sign", url)

final_url = f"{url}&_signature={signature}"

res = requests.get(
    url=final_url,
    headers={
        "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36"
    }
)

print(res.text)

python抓取网页头条内容

头条