Data

Tools

Indexes

Applications

Experiments

Repos

An open API service providing repository metadata for many open source software ecosystems.

GitHub / tonywangcn / scaleable-crawler-with-docker-cluster

a scaleable and efficient crawelr with docker cluster , crawl million pages in 2 hours with a single machine

JSON API: http://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tonywangcn%2Fscaleable-crawler-with-docker-cluster

Stars: 94
Forks: 27
Open issues: 1

License: None
Language: Python
Size: 7.81 KB
Dependencies parsed at: Pending

Created at: about 8 years ago
Updated at: over 2 years ago
Pushed at: over 3 years ago
Last synced at: about 2 years ago

Topics: celery, cluster, crawler, distributed, docker, python, rabbitmq, scaleable

    Loading...