一个很不错的爬虫管理工具

A scalable admin ui for spider service

Features

• Manage your spiders from a dashboard. Schedule them to run automatically
• With a single click deploy the scrapy project
• Show spider running stats
• Provide api

Current Support spider service

• Scrapy

Screenshot

Getting Started

Installing

pip install spiderkeeper

Deployment

Usage: 

spiderkeeper [options]

Options:

  -h, --help          show this help message and exit
  --type=SERVER_TYPE  access spider server type, default:scrapyd
  --host=HOST         host, default:0.0.0.0
  --port=PORT         port, default:5000
  --server=SERVERS    servers, default:http://localhost:6800
  
 
Visit: 
- web ui : http://localhost:5000
- api swagger: http://localhost:5000/api.html

TODO

• User Authentication
• Collect & Show scrapy crawl stats
• Optimize load balancing

Versioning

We use SemVer for versioning. For the versions available, see the tags on this repository.

Authors

• Initial work - DormyMo

See also the list of contributors who participated in this project.

License

This project is licensed under the MIT License - see the LICENSE.md file for details

Contributing

Contributions are welcomed!