Scrapy 中简单设置随机 User-Agent

安装scrapy-fake-useragent包

scrapy-fake-useragent包官网：pypi.org/project/scr… 安装： pip install scrapy-fake-useragent

使用：

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None, # 关闭默认方法
    'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400, # 开启
}

这里选用 httpbin.org/get 作为测试网站 测试结果：

import scrapy

# Scrapy 中设置随机 User-Agent
class RandomUaSpider(scrapy.Spider):
	name = 'random_ua'

	def __init__(self):
		self.test_url = 'http://httpbin.org/get'

	def start_requests(self):
		yield scrapy.Request(self.test_url, callback=self.parse)

	def parse(self, response):
		print('\n')
		print(response.request.headers['User-Agent'], '\n')