Java 爬虫 x Spring Boot

874 阅读1分钟

Porn Bot:Pornhub-downloader Link

java-1.8
license

介紹

  • 安装简单,所有相依的套件已包含,只需有Java环境。
  • 爬虫支援多Thread, 可24小时持续爬取
  • Spring boot x H2 Db 纪录已爬过的网站
  • 解决了该网站429防爬虫status

github連結 喜欢希望您能不吝给予星星, 笔者将会持续撰写更多优质的文章

Environment, Architecture

  • Java1.8

  • Crawler4j

  • Spring Boot x H2 Db

Run

java -jar PornBot.jar

Demo
h2_console

Database Description

http://localhost:8000/h2-console/

JDBC URL: jdbc:h2:~/porn/porn-db

User Name: sa

Password: empty

Record Table:

 Table_Name               :PORN_RECORD
 viewKey                  :The website's video unique key.
 imageUrl                 :Image url of video.
 linkUrl                  :Video jump to Website`s link
 videoUrl                 :Video adrress.
 videoTitle               :Title of video.
 videoDuration            :Video click count.
 videoQuality             :Defualt quality - 240, 480, 960, 1280p.
 download                 :Has been downloaded. True or false.
 createdTime              :The record created time.
 filePath                 :The video downloaded path.

更多設定請見 See Default Configuration