一、爬虫基础与合规规范
1.1 爬虫定义与核心流程
网络爬虫是模拟浏览器请求、自动抓取并解析网页公开数据的Java程序，核心流程分为三步：发送网络请求、接收服务端响应、解析并持久化数据。
1.2 合法使用场景
-
Java爬虫技术实训、代码练习
-
公开新闻、政务、行业公开数据整理统计
-
个人离线学习数据采集，不商用、不分发
1.3 爬虫合规红线
-
遵守网站
robots.txt协议，不访问禁止目录 -
控制请求频率，规避高频访问导致的IP封禁
-
禁止爬取隐私、版权、涉密等非公开数据
-
采集数据仅限个人学习，禁止商用引流
二、开发环境与Maven配置
2.1 运行环境
JDK8 / JDK11、IDEA开发工具、内置Maven环境
2.2 项目创建
IDEA新建Maven项目，绑定对应JDK，等待依赖初始化完成即可。
2.3 完整可用Maven依赖
以下依赖整合爬虫所需所有工具，无冗余、可直接导入使用：
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.crawler</groupId>
<artifactId>java-crawler-demo</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<maven.compiler.source>8</maven.compiler.source>
<maven.compiler.target>8</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<!-- 网络请求工具 -->
<dependency>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>okhttp</artifactId>
<version>4.11.0</version>
</dependency>
<!-- HTML解析工具 -->
<dependency>
<groupId>org.jsoup</groupId>
<artifactId>jsoup</artifactId>
<version>1.17.2</version>
</dependency>
<!-- JSON解析工具 -->
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.15.2</version>
</dependency>
<!-- WebMagic爬虫框架 -->
<dependency>
<groupId>us.codecraft</groupId>
<artifactId>webmagic-core</artifactId>
<version>0.7.5</version>
</dependency>
<dependency>
<groupId>us.codecraft</groupId>
<artifactId>webmagic-extension</artifactId>
<version>0.7.5</version>
</dependency>
<!-- 日志框架 -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.7.36</version>
</dependency>
<!-- 动态页面爬虫 -->
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>4.10.0</version>
</dependency>
<!-- MySQL数据库连接 -->
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>8.0.33</version>
<scope>runtime</scope>
</dependency>
</dependencies>
</project>
三、核心工具库实战
本章提供爬虫三大核心工具可运行代码，修复乱码、空解析、请求异常问题。
3.1 OkHttp网络请求（防乱码、异常兜底）
通过UA伪装、强制UTF-8编码、全局异常捕获，解决基础网络请求报错与中文乱码。
import okhttp3.OkHttpClient;
import okhttp3.Request;
import okhttp3.Response;
import java.io.IOException;
public class OkHttpDemo {
public static void main(String[] args) {
OkHttpClient client = new OkHttpClient();
Request request = new Request.Builder()
.url("https://www.baidu.com")
.header("User-Agent","Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0.0.0")
.get()
.build();
try(Response response = client.newCall(request).execute()){
if(response.isSuccessful()){
String html = new String(response.body().bytes(),"UTF-8");
System.out.println("页面请求成功，源码片段：\n" + html.substring(0,500));
}else{
System.out.println("请求失败，状态码：" + response.code());
}
}catch (IOException e){
System.out.println("网络请求异常：" + e.getMessage());
}
}
}
3.2 Jsoup网页解析（空数据Bug修复）
实现HTML标签、文本、链接精准解析，规避空值解析异常。
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
public class JsoupDemo {
public static void main(String[] args) {
String html = "<html><body><h1>Java爬虫零基础教程</h1><a href=\"https://www.baidu.com\">百度首页</a></body></html>";
Document doc = Jsoup.parse(html);
String title = doc.select("h1").text();
System.out.println("页面标题：" + title);
Elements aList = doc.select("a");
for(Element a : aList){
System.out.println("链接文本：" + a.text() + " | 链接地址：" + a.attr("href"));
}
}
}
3.3 Jackson JSON解析
适配接口爬虫结构化数据解析，代码稳定无报错。
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.HashMap;
import java.util.Map;
public class JacksonDemo {
public static void main(String[] args) {
String json = "{\"name\":\"Java爬虫\",\"level\":\"入门\",\"author\":\"零基础教程\"}";
try{
ObjectMapper mapper = new ObjectMapper();
Map<String,String> map = mapper.readValue(json, HashMap.class);
System.out.println("教程名称：" + map.get("name"));
System.out.println("难度等级：" + map.get("level"));
}catch (Exception e){
e.printStackTrace();
}
}
}
四、原生静态爬虫实战
4.1 百度新闻爬取（解析+去重+本地存储）
适配新版百度新闻页面，解决拦截、空解析、乱码问题，实现完整爬取存储流程。
import okhttp3.OkHttpClient;
import okhttp3.Request;
import okhttp3.Response;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.FileWriter;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
public class NewsCrawler {
public static void main(String[] args) {
crawlNews("https://news.baidu.com/","news_data.txt");
}
public static void crawlNews(String url,String savePath){
OkHttpClient client = new OkHttpClient();
Request request = new Request.Builder()
.url(url)
.header("User-Agent","Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0.0.0")
.header("Accept","text/html,application/xhtml+xml")
.header("Referer","https://www.baidu.com/")
.build();
try(Response response = client.newCall(request).execute()){
if(!response.isSuccessful()){
System.out.println("页面请求失败，状态码：" + response.code());
return;
}
String html = new String(response.body().bytes(), StandardCharsets.UTF_8);
Document doc = Jsoup.parse(html);
Elements newsList = doc.select("a");
FileWriter writer = new FileWriter(savePath,false);
writer.write("百度新闻爬取结果（原生爬虫版）\r\n采集时间：" + System.currentTimeMillis() + "\r\n=====================\r\n");
int count = 0;
for(Element a : newsList){
String title = a.text().trim();
String href = a.attr("href").trim();
if(!title.isEmpty() && href.startsWith("http") && title.length() > 2){
count++;
String line = count + ". 标题：" + title + "\r\n 链接：" + href + "\r\n\r\n";
System.out.println(line);
writer.write(line);
}
}
writer.flush();
writer.close();
System.out.println("✅ 爬取完成！共采集有效新闻：" + count + " 条");
}catch (IOException e){
System.out.println("❌ 爬取异常：" + e.getMessage());
}
}
}
五、反爬机制与绕过方案
5.1 UA拦截绕过
现象：代码请求403、页面空白，浏览器可正常访问。解决方案：请求头携带正规浏览器UA，全文代码已内置配置。
5.2 频率封禁绕过
现象：单次请求正常，高频访问后IP封禁。解决方案：设置1-3秒随机延时，模拟人工浏览节奏。
5.3 中文乱码根治方案
禁止使用 response.body().string()，统一使用 new String(response.body().bytes(),"UTF-8") 强制编码。
5.4 完整伪装请求头（提升通过率）
.header("User-Agent","Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0.0.0")
.header("Accept","text/html,application/xhtml+xml,image/webp")
.header("Referer","https://www.baidu.com/")
.header("Cache-Control","no-cache")
.header("Connection","keep-alive")
六、常见报错与修复方案
6.1 页面解析为空排查步骤
-
打印网页源码，判断是否被网站拦截
-
区分静态/动态页面，动态页面需Selenium渲染
-
核对网页标签，适配页面更新
-
补全请求头，避免访问拦截
6.2 Maven依赖报错修复
切换阿里云Maven镜像、刷新项目、清理缓存、重启IDEA，重新粘贴纯净Pom配置即可解决绝大部分依赖问题。
6.3 程序中断容错说明
全文代码均添加全局异常捕获，单条数据异常不会终止整个爬虫程序，保证运行稳定性。
七、爬虫进阶核心功能
涵盖框架爬虫、接口爬取、动态页面渲染、数据库持久化核心能力。
7.1 WebMagic优化爬虫（去重+防爬+统计）
整合随机UA、随机延时、全局URL去重、自动数据统计，解决框架原生重复采集、易拦截问题。
import us.codecraft.webmagic.Page;
import us.codecraft.webmagic.Site;
import us.codecraft.webmagic.Spider;
import us.codecraft.webmagic.processor.PageProcessor;
import java.io.FileWriter;
import java.io.IOException;
import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
import java.util.HashSet;
import java.util.Random;
import java.util.Set;
public class WebMagicNewsCrawler implements PageProcessor {
private final String[] UA_POOL = {
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0.0.0",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) Edge/120.0.0.0",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36"
};
private final Site site = Site.me()
.setUserAgent(UA_POOL[new Random().nextInt(UA_POOL.length)])
.setRetryTimes(3)
.setTimeOut(10000);
private static final String SAVE_PATH = "webmagic_news_result.txt";
private static final Set<String> ALL_URL = new HashSet<>();
@Override
public void process(Page page) {
page.addTargetRequests(page.getHtml().links().regex(".*news\\.baidu\\.com.*").all());
page.getHtml().xpath("//a[@href]").nodes().forEach(node -> {
String title = node.xpath("text()").get() == null ? "" : node.xpath("text()").get().trim();
String link = node.attr("href").trim();
if(!title.isEmpty() && link.startsWith("http") && !ALL_URL.contains(link)){
ALL_URL.add(link);
String res = "【新闻标题】" + title + "\r\n【新闻链接】" + link + "\r\n----------------\r\n";
System.out.println(res);
appendFile(res);
}
});
}
public void initFile(){
try(FileWriter w = new FileWriter(SAVE_PATH,false)){
String time = LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss"));
w.write("WebMagic框架爬虫结果\r\n采集时间：" + time + "\r\n====================\r\n");
}catch (IOException e){
e.printStackTrace();
}
}
public void appendFile(String content){
try(FileWriter w = new FileWriter(SAVE_PATH,true)){
w.write(content);
}catch (IOException e){
e.printStackTrace();
}
}
public void writeTotal(int total){
try(FileWriter w = new FileWriter(SAVE_PATH,true)){
w.write("✅ 本次采集完成，有效不重复数据总量：" + total + " 条\r\n");
}catch (IOException e){
e.printStackTrace();
}
}
@Override
public Site getSite() {
return site.setSleepTime(new Random().nextInt(2000)+1000);
}
public static void main(String[] args) {
WebMagicNewsCrawler crawler = new WebMagicNewsCrawler();
crawler.initFile();
Spider.create(crawler)
.addUrl("https://news.baidu.com/")
.thread(1)
.run();
crawler.writeTotal(ALL_URL.size());
System.out.println("✅ 爬虫结束！总计采集：" + ALL_URL.size() + " 条有效不重复新闻");
}
}
7.2 WebMagic标准文件持久化
import us.codecraft.webmagic.ResultItems;
import us.codecraft.webmagic.Task;
import us.codecraft.webmagic.pipeline.FilePipeline;
public class CustomFilePipeline extends FilePipeline {
public CustomFilePipeline() {
super("webmagic_official_data");
}
@Override
public void process(ResultItems resultItems, Task task) {
super.process(resultItems, task);
}
}
7.3 JSON接口爬虫（企业主流方案）
直接解析后端接口结构化数据，解析快、反爬低、稳定性高。
import okhttp3.OkHttpClient;
import okhttp3.Request;
import okhttp3.Response;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.Map;
public class ApiCrawlerDemo {
public static void main(String[] args) {
OkHttpClient client = new OkHttpClient();
String url = "https://jsonplaceholder.typicode.com/todos/1";
Request request = new Request.Builder()
.url(url)
.header("User-Agent","Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0.0.0")
.build();
try(Response res = client.newCall(request).execute()){
String json = new String(res.body().bytes(),"UTF-8");
ObjectMapper mapper = new ObjectMapper();
Map<String,Object> data = mapper.readValue(json,Map.class);
System.out.println("用户ID：" + data.get("userId"));
System.out.println("任务ID：" + data.get("id"));
System.out.println("任务内容：" + data.get("title"));
System.out.println("完成状态：" + data.get("completed"));
}catch (Exception e){
e.printStackTrace();
}
}
}
7.4 Selenium动态页面爬虫
解决JS异步渲染页面数据空白问题，模拟真实浏览器加载页面。
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
public class SeleniumCrawlerDemo {
public static void main(String[] args) {
System.setProperty("webdriver.chrome.driver","chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.addArguments("--headless");
options.addArguments("--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0.0.0");
WebDriver driver = new ChromeDriver(options);
try {
driver.get("https://news.baidu.com/");
Thread.sleep(2000);
driver.findElements(By.tagName("a")).forEach(element -> {
String title = element.getText().trim();
String url = element.getAttribute("href");
if(!title.isEmpty() && url != null && url.startsWith("http")){
System.out.println("【动态新闻】" + title + " | " + url);
}
});
} catch (InterruptedException e) {
e.printStackTrace();
} finally {
driver.quit();
}
}
}
7.5 MySQL数据持久化
7.5.1 数据库建表语句
CREATE DATABASE IF NOT EXISTS crawler_data DEFAULT CHARACTER SET utf8mb4;
USE crawler_data;
CREATE TABLE IF NOT EXISTS news(
id INT PRIMARY KEY AUTO_INCREMENT COMMENT '主键ID',
title VARCHAR(500) NOT NULL COMMENT '新闻标题',
url VARCHAR(800) NOT NULL UNIQUE COMMENT '新闻链接（唯一去重）',
create_time DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT '采集时间'
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='爬虫新闻数据表';
7.5.2 通用数据入库工具类
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.SQLException;
public class CrawlerDbUtil {
private static final String URL = "jdbc:mysql://localhost:3306/crawler_data?useSSL=false&serverTimezone=UTC&characterEncoding=utf8";
private static final String USER = "root";
private static final String PASSWORD = "123456";
public static Connection getConnection(){
Connection conn = null;
try {
Class.forName("com.mysql.cj.jdbc.Driver");
conn = DriverManager.getConnection(URL,USER,PASSWORD);
} catch (Exception e) {
e.printStackTrace();
}
return conn;
}
public static int insertNews(String title,String url){
String sql = "INSERT INTO news(title,url) VALUES (?,?)";
try(Connection conn = getConnection();
PreparedStatement pstmt = conn.prepareStatement(sql)){
pstmt.setString(1,title);
pstmt.setString(2,url);
return pstmt.executeUpdate();
} catch (SQLException e) {
return 0;
}
}
public static void main(String[] args) {
int res = insertNews("Java爬虫零基础教程测试新闻","https://test.news.com/123");
System.out.println(res > 0 ? "✅ 数据入库成功" : "❌ 数据重复或入库失败");
}
}
7.6 爬虫学习路线
静态HTML爬虫 → JSON接口爬虫 → JS动态爬虫 → 反爬优化 → 多线程提速 → 数据库持久化 → WebMagic工程化开发
八、进阶完整能力拓展
涵盖零基础爬虫必备高阶知识点，完善工程化、容错处理、合规规范及高阶实战场景体系。
8.1 爬虫分类与适用场景
-
静态HTML爬虫：服务端渲染页面，数据在源码中，OkHttp+Jsoup解析，高效简单。
-
JSON接口爬虫：前后端分离项目，解析接口数据，企业主流方案。
-
动态JS爬虫：前端异步渲染页面，需Selenium浏览器渲染。
-
增量爬虫：仅爬取新增数据，自动去重，避免重复采集。
-
全量爬虫：首次全站合规数据初始化采集。
8.2 HTTP核心爬虫基础
8.2.1 GET/POST请求区别
GET：URL传参、无请求体、用于查询、可缓存；POST：请求体传参、无长度限制、用于提交、登录、搜索场景。
8.2.2 身份凭证作用
Cookie/Session/Token用于标识登录状态，携带凭证可爬取登录后私密权限页面。
8.2.3 POST请求完整实操
import okhttp3.MediaType;
import okhttp3.OkHttpClient;
import okhttp3.Request;
import okhttp3.RequestBody;
import okhttp3.Response;
import java.io.IOException;
public class OkHttpPostDemo {
private static final MediaType JSON = MediaType.get("application/json; charset=utf-8");
public static void main(String[] args) {
OkHttpClient client = new OkHttpClient();
String jsonParam = "{\"username\":\"test\",\"password\":\"123456\"}";
RequestBody body = RequestBody.create(jsonParam, JSON);
Request request = new Request.Builder()
.url("https://httpbin.org/post")
.header("User-Agent","Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0.0.0")
.post(body)
.build();
try(Response response = client.newCall(request).execute()){
if(response.isSuccessful()){
String result = new String(response.body().bytes(),"UTF-8");
System.out.println("POST请求响应结果：\n" + result);
}
}catch (IOException e){
System.out.println("POST请求异常：" + e.getMessage());
}
}
}
8.3 Cookie登录爬虫
携带登录Cookie模拟登录，爬取权限页面数据。
import okhttp3.OkHttpClient;
import okhttp3.Request;
import okhttp3.Response;
import java.io.IOException;
public class CookieCrawlerDemo {
public static void main(String[] args) {
OkHttpClient client = new OkHttpClient();
String cookie = "你的登录Cookie";
Request request = new Request.Builder()
.url("需要登录访问的目标地址")
.header("User-Agent","Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0.0.0")
.header("Cookie",cookie)
.get()
.build();
try(Response response = client.newCall(request).execute()){
if(response.isSuccessful()){
String html = new String(response.body().bytes(),"UTF-8");
System.out.println("登录后页面数据：\n" + html.substring(0,800));
}
}catch (IOException e){
System.out.println("请求异常：" + e.getMessage());
}
}
}
8.4 自动分页爬取
自动遍历页码、循环采集、延时防封，适配列表类数据。
import okhttp3.OkHttpClient;
import okhttp3.Request;
import okhttp3.Response;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.FileWriter;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
public class PageCrawlerDemo {
private static final String BASE_URL = "https://news.baidu.com/?tn=news&page=";
private static final int MAX_PAGE = 3;
public static void main(String[] args) {
for (int i = 1; i <= MAX_PAGE; i++) {
System.out.println("===== 正在爬取第" + i + "页 =====");
crawlPage(BASE_URL + i, "page_news_data.txt");
try {
Thread.sleep(2000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
System.out.println("✅ 全部分页爬取完成");
}
public static void crawlPage(String url,String savePath){
OkHttpClient client = new OkHttpClient();
Request request = new Request.Builder()
.url(url)
.header("User-Agent","Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0.0.0")
.build();
try(Response response = client.newCall(request).execute()){
String html = new String(response.body().bytes(), StandardCharsets.UTF_8);
Document doc = Jsoup.parse(html);
Elements newsList = doc.select("a");
FileWriter writer = new FileWriter(savePath,true);
for(Element a : newsList){
String title = a.text().trim();
String href = a.attr("href").trim();
if(!title.isEmpty() && href.startsWith("http") && title.length() > 2){
writer.write("标题：" + title + " 链接：" + href + "\r\n");
System.out.println("采集成功：" + title);
}
}
writer.flush();
writer.close();
}catch (IOException e){
System.out.println("分页爬取异常：" + e.getMessage());
}
}
}
8.5 标准化数据清洗工具
统一清理空格、换行、特殊符号、空值，标准化爬虫数据。
public class DataCleanUtil {
public static String cleanText(String text){
if(text == null || text.isEmpty()){
return "";
}
text = text.trim().replace("\n","").replace("\r","").replace("\t","");
text = text.replaceAll("\\s+"," ");
text = text.replaceAll("[^a-zA-Z0-9\u4e00-\u9fa5\\p{Punct}]","");
return text;
}
public static void main(String[] args) {
String rawText = " Java爬虫 零基础\n实战教程！！@@## ";
String cleanText = cleanText(rawText);
System.out.println("原始数据：" + rawText);
System.out.println("清洗后数据：" + cleanText);
}
}
8.6 异常重试容错机制
解决网络波动、超时导致的爬取中断，自动重试提升稳定性。
import okhttp3.OkHttpClient;
import okhttp3.Request;
import okhttp3.Response;
import java.io.IOException;
public class RetryCrawlerDemo {
private static final int RETRY_TIMES = 3;
public static void main(String[] args) {
String result = crawlWithRetry("https://news.baidu.com/");
System.out.println("最终响应结果：" + result.substring(0,500));
}
public static String crawlWithRetry(String url){
OkHttpClient client = new OkHttpClient();
for (int i = 0; i < RETRY_TIMES; i++) {
try{
Request request = new Request.Builder()
.url(url)
.header("User-Agent","Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0.0.0")
.build();
Response response = client.newCall(request).execute();
if(response.isSuccessful()){
return new String(response.body().bytes(),"UTF-8");
}
}catch (Exception e){
System.out.println("第" + (i+1) + "次请求失败，准备重试");
try {
Thread.sleep(1000);
} catch (InterruptedException ex) {
ex.printStackTrace();
}
}
}
return "请求失败，已达最大重试次数";
}
}
8.7 SSL证书报错终极解决方案
全局信任证书，解决HTTPS SSL握手失败、证书校验异常。
import okhttp3.OkHttpClient;
import okhttp3.Request;
import okhttp3.Response;
import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;
import java.io.IOException;
import java.security.SecureRandom;
import java.security.cert.X509Certificate;
public class SslSafeCrawler {
public static OkHttpClient getSslClient(){
try {
SSLContext sslContext = SSLContext.getInstance("TLS");
TrustManager[] trustManagers = new TrustManager[]{
new X509TrustManager() {
@Override
public void checkClientTrusted(X509Certificate[] x509Certificates, String s) {}
@Override
public void checkServerTrusted(X509Certificate[] x509Certificates, String s) {}
@Override
public X509Certificate[] getAcceptedIssuers() {
return new X509Certificate[0];
}
}
};
sslContext.init(null, trustManagers, new SecureRandom());
return new OkHttpClient.Builder()
.sslSocketFactory(sslContext.getSocketFactory(), (X509TrustManager) trustManagers[0])
.hostnameVerifier((hostname, session) -> true)
.build();
} catch (Exception e) {
return new OkHttpClient();
}
}
public static void main(String[] args) {
OkHttpClient client = getSslClient();
Request request = new Request.Builder()
.url("https://www.baidu.com")
.header("User-Agent","Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0.0.0")
.build();
try(Response response = client.newCall(request).execute()){
System.out.println("SSL安全请求成功，状态码：" + response.code());
}catch (IOException e){
System.out.println("请求异常：" + e.getMessage());
}
}
}
8.8 robots.txt合规实操
1. 网站根目录访问 /robots.txt 查看爬取规则；2. Disallow为禁止路径、Allow为允许路径；3. 爬虫代码主动过滤禁止目录，合规采集。
8.9 行业合规限速规范
-
小型静态网站：单线程、3-5秒随机延时
-
大型门户网站：单线程、1-2秒随机延时，禁止夜间高频爬取
-
禁止暴力多线程、无限极速请求
8.10 高阶拓展方向
代理IP池、分布式爬虫、Redis去重、定时爬虫、线程池优化、Excel批量存储、接口解密、验证码绕过。
