Ecosyste.ms: Repos
An open API service providing repository metadata for many open source software ecosystems.
GitHub / TeamHG-Memex / domain-discovery-crawler
Broad crawler for domain discovery
JSON API: https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TeamHG-Memex%2Fdomain-discovery-crawler
Stars: 17
Forks: 10
Open Issues: 2
License: mit
Language: Python
Repo Size: 441 KB
Dependencies:
36
Created: almost 8 years ago
Updated: about 1 year ago
Last pushed: almost 6 years ago
Last synced: about 1 year ago
Files
Loading...
Readme
Loading...
Dependencies
Dockerfile
docker
- python 3.5 build
docker-compose.yml
docker
- dd-crawler latest
- redis 3.0
requirements.txt
pypi
- Formasaurus ==0.8
- autopager *
- bokeh ==0.12
- html_text ==0.2.1
- json_lines ==0.3.1
- numpy ==1.12.1
- pandas *
- proxy-middleware *
- pySmaz ==1.0.0
- scikit-learn ==0.18.1
- scipy ==0.18.1
- scrapy ==1.4.0
- scrapy-cdr >=0.4.0
- scrapy-redis ==0.6.8
- statsd *
- tldextract *
- vmprof *
setup.py
pypi
- Formasaurus *
- autologin-middleware *
- autopager *
- deepdeep *
- html_text >=0.2.1
- json-lines >=0.3.1
- numpy *
- pySmaz ==1.0.0
- scrapy >=1.1.0
- scrapy-cdr >=0.4.0
- scrapy-redis >=0.6.8
- tldextract *
- vmprof *
- pytest * test
- pytest-cov * test
- pytest-twisted * test