项目中使用了豆瓣的开放接口，最近发现豆瓣电影接口不能用了，所以想着要自己动手做一个爬取豆瓣电影数据的接口。
豆瓣热映电影
开发后的接口地址：zhaoyj.work/poetry/movi…
返回内容如下：
一、创建Django服务项目
二、在项目的应用目录miniprogram下创建spiderHot.py爬虫程序
1.在spiderHot.py中创建MovieBean电影实体类
2.在spiderHot.py中创建HotMovieSpider爬取热映电影类
-
1.编写
get_html_text方法，获取html页码内容def get_html_text(url, headers=None): time.sleep(random.uniform(0.5, 1.5)) response = requests.get(url, headers=ComHeaders.pc_headers, timeout=(5, 60)) html_text = '' if response.status_code == 200: html_text = response.text return html_text -
2.编写
get_info_from_movie方法，获取电影数据通过
xpath查找标签及文本内容，创建MovieBean对象添加到集合中并返回集合数据def get_info_from_movie(self, href): html_detail = self.get_html_text(href) com_html = etree.HTML(html_detail) movieLis = com_html.xpath('//*[@id="nowplaying"]/div[@class="mod-bd"]/ul/li') movieList = [] for index, movieItem in enumerate(movieLis): if index == 6: break subject = movieItem.xpath('./@data-subject')[0] title = movieItem.xpath('./@data-title')[0] director = movieItem.xpath('./@data-director')[0] actor = movieItem.xpath('./@data-actors')[0] area = movieItem.xpath('./@data-region')[0] duration = movieItem.xpath('./@data-duration')[0] score = movieItem.xpath('./@data-score')[0] image = movieItem.xpath('./ul/li[@class="poster"]/a/img/@src')[0] movieBean = MovieBean(subject, title, image, score, director, actor, duration, area) movieList.append(movieBean) return movieList -
3.编写
start方法，启动爬虫程序并返回爬取数据将上面
get_info_from_movie方法返回的电影集合数据转成json字符串<直接返回集合数据到接口中有问题>def start(self): movieList = self.get_info_from_movie(self.url) movieList_json = json.dumps(movieList, default=lambda obj: obj.__dict__, sort_keys=True, indent=4) return movieList_json
三、在项目的应用目录miniprogram下的views里面编写业务代码
1.在views包下创建movie.py文件并编写业务方法movie_hot获取热映电影数据并返回
将
HotMovieSpider程序中返回的json字符串转成JSON数据并返回
def movie_hot(request, format=None):
spider = HotMovieSpider()
movieList = spider.start()
movieListJson = json.loads(movieList)
return result.success(movieListJson)
2.在miniprogram的urls.py中添加接口地址
path(r'movie/showing', movie.movie_hot),
