Ecosyste.ms: Repos
An open API service providing repository metadata for many open source software ecosystems.
GitHub / wangy8961 / python3-concurrency-pics-02
爬取 www.mzitu.com 全站图片，截至目前共5162个图集，16.5万多张美女图片，使用 asyncio 和 aiohttp 实现的异步版本只需要不到2小时就能爬取完成。按日期创建图集目录，保存更合理。控制台只显示下载的进度条，详细信息保存在日志文件中。支持异常处理，不会终止爬虫程序。失败的请求，下次再执行爬虫程序时会自动下载
JSON API: https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wangy8961%2Fpython3-concurrency-pics-02
Stars: 320
Forks: 122
Open Issues: 1
License: None
Language: Python
Repo Size: 113 KB
Dependencies:
15
Created: almost 6 years ago
Updated: 7 months ago
Last pushed: almost 5 years ago
Last synced: 7 months ago
Topics: aiohttp, asyncio, concurrency, coroutine, futures, progressbar, python3, requests, threadpool
Files
Dependencies
- aiofiles *
- aiohttp *
- beautifulsoup4 *
- lxml *
- progressbar2 *
- pymongo *
- requests *
- uvloop *
- aiofiles *
- aiohttp *
- beautifulsoup4 *
- lxml *
- progressbar2 *
- pymongo *
- requests *