爬了10000张NASA关于火星探索的图片,我发现了一个秘密

500 阅读6分钟

前言

最近,我使用爬虫技术,爬取了美国航空航天局,也就是你电影里经常见到的 NASA, 火星探索的相关图片,有 10000 张吧。

嗯嗯,小事情,小事情。

完事儿之后,有点小激动,于是就有了这篇文章,将有以下内容:

  1. 我为什么要爬取NASA的图片
  2. 我是如何爬取NASA图片的(超详细)
  3. 我得到了什么(高清大图)
  4. 我发现了什么秘密(超劲爆)

我为什么要爬NASA的图片

我已经超过35了,瑟瑟发抖啥时候被开了。

天天想着万一哪天失业了干点啥,想着玩个自媒体吧,天天给大家瞎白话。白话点历史谜团,宇宙奥秘啥的,于是我就盯上了 NASA

NASA 有各种宇宙探索任务,并且有相关的文章,访谈,图片,视频公开,这是不可多得的资源库。

我是如何爬取NASA图片的(超详细)

NASA 的网站是可以公开访问的,地址是

https://www.nasa.gov/

微信图片_20210630130517.png

打开以后,它首页长这样,可以看到各种内容。右上角还有个搜索框,我们输入 Mars 也就是 火星

稍等片刻,会展示出与 Mars 相关的各种内容,其中有一项 Mars Exploration 也就是 火星探索

点开之后,就到了一个新页面,然后找到 Images 图片,就到了我们爬取的目标页

https://www.nasa.gov/mission_pages/mars/images/index.html

微信图片_20210630130632.png

页面下拉,你将会看到有个大大的按钮,上面写着 MORE IMAGES, 点击试试你就发现:

页面的内容,并不是页面直接加载的,而是通过 api 请求后,异步渲染的

F12, 打开浏览器开发者模式,重新执行刚才的步骤,观察请求信息,发现都会有如下的情况

微信图片_20210630123102.png

看上去这个url地址非常重要,我们先看他的请求地址:

https://www.nasa.gov/api/2/ubernode/_search?size=24&from=24&sort=promo-date-time%3Adesc&q=((ubernode-type%3Aimage)%20AND%20(topics%3A3152))&_source_include=promo-date-time%2Cmaster-image%2Cnid%2Ctitle%2Ctopics%2Cmissions%2Ccollections%2Cother-tags%2Cubernode-type%2Cprimary-tag%2Csecondary-tag%2Ccardfeed-title%2Ctype%2Ccollection-asset-link%2Clink-or-attachment%2Cpr-leader-sentence%2Cimage-feature-caption%2Cattachments%2Curi

注意看里面的参数

size=24&from=24

很显然,size 就是每次请求图片的数量,from 经过试验,是查询初始位置,我们可以改它来获取其他内容

我们再来看看它的返回信息:

{
	"took": 3,
	"timed_out": false,
	"_shards": {
		"total": 5,
		"successful": 5,
		"skipped": 0,
		"failed": 0
	},
	"hits": {
		"total": 659,
		"max_score": null,
		"hits": [{
			"_index": "nasa-public",
			"_type": "ubernode",
			"_id": "450040",
			"_score": null,
			"_source": {
				"image-feature-caption": "Mars 2020 rover underwent an eye exam after several cameras were installed on the rover. ",
				"topics": ["3140", "3152"],
				"nid": "450040",
				"title": "NASA 'Optometrists' Verify Mars 2020 Rover's 20/20 Vision",
				"type": "ubernode",
				"uri": "/image-feature/jpl/nasa-optometrists-verify-mars-2020-rovers-2020-vision",
				"collections": ["4525", "5246"],
				"link-or-attachment": "link",
				"missions": ["6336"],
				"primary-tag": "6336",
				"cardfeed-title": "NASA 'Optometrists' Verify Mars 2020 Rover's 20/20 Vision",
				"promo-date-time": "2019-08-05T17:49:00-04:00",
				"secondary-tag": "3140",
				"master-image": {
					"fid": "603128",
					"alt": "Engineers test cameras on the top of the Mars 2020 rover’s mast and front chassis. ",
					"width": "1600",
					"id": "603128",
					"title": "Engineers test cameras on the top of the Mars 2020 rover’s mast and front chassis. ",
					"uri": "public://thumbnails/image/pia23314-16.jpg",
					"height": "900"
				},
				"ubernode-type": "image"
			},
			"sort": [1565041740000]
		}, {
			"_index": "nasa-public",
			"_type": "ubernode",
			"_id": "433172",
			"_score": null,
			"_source": {
				"image-feature-caption": "NASA still hasn't heard from the Opportunity rover, but at least we can see it again.",
				"topics": ["3152"],
				"nid": "433172",
				"title": "Opportunity Emerges in a Dusty Picture",
				"type": "ubernode",
				"uri": "/image-feature/opportunity-emerges-in-a-dusty-picture",
				"collections": ["7628"],
				"link-or-attachment": "link",
				"missions": ["3639"],
				"primary-tag": "3152",
				"cardfeed-title": "Opportunity Emerges in a Dusty Picture",
				"promo-date-time": "2018-09-26T12:39:00-04:00",
				"secondary-tag": "7628",
				"master-image": {
					"fid": "584263",
					"alt": "NASA's Opportunity rover appears as a blip in the center of this square",
					"width": "1400",
					"id": "584263",
					"title": "NASA's Opportunity rover appears as a blip in the center of this square",
					"uri": "public://thumbnails/image/pia22549-16.jpg",
					"height": "788"
				},
				"ubernode-type": "image"
			},
			"sort": [1537979940000]
		}]
	}
}

上面的 json 内容太长,我删除了一些重复的,实际上 hits 这个数组,也是24个,和页面上显示的图片数量是一样的。那基本可以断定,页面上的信息,就是从这个数组来的。

进而对比发现,master-image字段下,就是我们需要的信息,包括 图片地址, 图片尺寸, 图片标题

下面就上代码,组装请求URL, 获取内容,下载图片 三步走

我使用了 Dart 语言,你们随意

import 'dart:convert';
import 'package:dio/dio.dart';

main() async {
  // 每页数量是固定24个,改变初始值即可
  for (int from = 0; from < 24 * 100; from = from + 24) {
    await getPage(from);
  }
}

//获取每一页的信息并且下载
Future<void> getPage(int from) async {
  String url = 'https://www.nasa.gov/api/2/ubernode/_search?size=24&from=' +
      from.toString() +
      '&sort=promo-date-time%3Adesc&q=((ubernode-type%3Aimage)%20AND%20(topics%3A3152))&_source_include=promo-date-time%2Cmaster-image%2Cnid%2Ctitle%2Ctopics%2Cmissions%2Ccollections%2Cother-tags%2Cubernode-type%2Cprimary-tag%2Csecondary-tag%2Ccardfeed-title%2Ctype%2Ccollection-asset-link%2Clink-or-attachment%2Cpr-leader-sentence%2Cimage-feature-caption%2Cattachments%2Curi';

  //获取到内容
  var res = await Dio().get(url);
  var map = jsonDecode(res.toString());

  (map['hits']['hits'] as List<dynamic>).forEach((element) async {
    Uri fileUri = Uri.parse(getUri(element));
    String savePath = getSavePath(element);

    await Dio().downloadUri(fileUri, savePath);
    print('已下载: ' + savePath);
  });
}

//获取图片下载地址
String getUri(dynamic element) {
  String uri = element['_source']['master-image']['uri'].toString();

  uri = uri.replaceAll('public://',
      'https://www.nasa.gov/sites/default/files/styles/full_width_feature/public/');

  return uri;
}

//处理信息,并且返回图片保存地址
String getSavePath(dynamic element) {
  String id = element['_id'];
  String fid = element['_source']['master-image']['fid'].toString();
  String title = element['_source']['master-image']['title'].toString();
  String uri = element['_source']['master-image']['uri'].toString();

  String savePath =
      id + '_' + fid + '_' + title.trim() + '.' + uri.split('.').last;
  savePath = savePath.replaceAll('/', '');
  savePath = savePath.replaceAll('\\', '');
  savePath = savePath.replaceAll('"', '');
  savePath = 'images/' + savePath;

  return savePath;
}

上面的代码,还是很简单的,有经验的同学应该一看就懂。

走起来吧。

已下载: images/470436_643588_This is the third color image taken by NASA’s Ingenuity helicopter.jpg
已下载: images/470435_643587_This is the second color image taken by NASA’s Ingenuity helicopter.jpg
已下载: images/468546_639327_This is the first high-resolution, color image to be sent back by the Hazard Cameras (Hazcams).jpg
已下载: images/452007_605784_Danielson Crater on Mars.jpg
已下载: images/458478_615132_Gullies on Mars.jpg
已下载: images/469416_641582_A field of sand dunes occupies this frosty 5-kilometer diameter crater in the high-latitudes of the northern plains of Mars..jpeg
已下载: images/458075_614251_Mars 2020 With Sample Tubes (Artist's Concept).jpg
已下载: images/470381_643473_CME.jpg
已下载: images/458813_615896_Mars.jpg
已下载: images/467026_635309_Illustration of NASA’s Perseverance rover begins its descent through the Martian atmosphere.jpg
已下载: images/470438_643591_This black and white image was taken by NASA’s Ingenuity helicopter during its third flight on April 25, 2021.jpg
已下载: images/465488_631398_Cliffs in Ancient Ice on Mars.jpg
已下载: images/463659_626874_Avalanche on Mars.jpg
已下载: images/470251_643164_This image from NASA’s Perseverance rover shows the agency’s Ingenuity Mars Helicopter right after it successfully completed a high-speed spin-up test..jpeg
已下载: images/468636_639726_Mars' Jezero Crater.jpg

我得到了什么

这些图片

微信图片_20210630121020.png

还有这些 微信图片_20210630131701.png

图片,图片标题都有了, 够看一个月了我估计。

我发现了什么秘密

429059_580437_two images of mars, one clear, one fuzzy.png

这张图,我最喜欢了。一个如此清晰,一个如此浑浊,这是为什么? 火星人的裂缝产生器?

好吧,真正的秘密是:

NASA 的网站竟然没有防采集,不信你也试试。。。