Ecosyste.ms: Repos
An open API service providing repository metadata for many open source software ecosystems.
GitHub / istresearch / scrapy-cluster
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
JSON API: https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/istresearch%2Fscrapy-cluster
Stars: 1,157
Forks: 324
Open Issues: 17
License: mit
Language: Python
Repo Size: 28 MB
Dependencies:
153
Created: about 9 years ago
Updated: about 1 month ago
Last pushed: 6 months ago
Last synced: 27 days ago
Commit Stats
Commits: 657
Authors: 31
Mean commits per author: 21.19
Development Distribution Score: 0.288
More commit stats: https://commits.ecosyste.ms/hosts/GitHub/repositories/istresearch/scrapy-cluster
Topics: distributed, kafka, python, redis, scraping, scrapy
Files
Dependencies
- ConcurrentLogHandler ==0.9.1
- PyDispatcher ==2.0.5
- PyYAML ==3.12
- Scrapy ==1.5.0
- Twisted ==18.4.0
- attrs ==18.1.0
- cffi ==1.11.5
- cryptography ==2.3
- cssselect ==1.0.3
- enum34 ==1.1.6
- funcsigs ==1.0.2
- future ==0.16.0
- idna ==2.6
- ipaddress ==1.0.22
- kafka-python ==1.4.3
- kazoo ==2.4.0
- lxml ==4.2.1
- mock ==2.0.0
- nose ==1.3.7
- parsel ==1.4.0
- pbr ==4.0.3
- pyOpenSSL ==18.0.0
- pyasn1 ==0.4.3
- pyasn1-modules ==0.2.1
- pycparser ==2.18
- python-json-logger ==0.1.8
- queuelib ==1.5.0
- redis >=3.0
- requests ==2.18.4
- requests-file ==1.4.3
- retrying ==1.3.3
- service-identity ==17.0.0
- six ==1.11.0
- testfixtures ==6.0.2
- tldextract ==2.2.0
- ujson ==1.35
- w3lib ==1.19.0
- zope.interface ==4.5.0
- ConcurrentLogHandler ==0.9.1
- PyYAML ==3.12
- funcsigs ==1.0.2
- future ==0.16.0
- idna ==2.6
- jsonschema ==2.6.0
- kafka-python ==1.4.3
- kazoo ==2.4.0
- mock ==2.0.0
- nose ==1.3.7
- pbr ==4.0.3
- python-json-logger ==0.1.8
- python-redis-lock ==3.2.0
- redis >=3.0
- requests ==2.18.4
- requests-file ==1.4.3
- retrying ==1.3.3
- six ==1.11.0
- testfixtures ==6.0.2
- tldextract ==2.2.0
- ujson ==1.35
- ConcurrentLogHandler ==0.9.1
- PyYAML ==3.12
- funcsigs ==1.0.2
- future ==0.16.0
- kafka-python ==1.4.3
- kazoo ==2.4.0
- mock ==2.0.0
- nose ==1.3.7
- pbr ==4.0.3
- python-json-logger ==0.1.8
- python-redis-lock ==3.2.0
- redis >=3.0
- retrying ==1.3.3
- six ==1.11.0
- testfixtures ==6.0.2
- ujson ==1.35
- ConcurrentLogHandler ==0.9.1
- Flask ==1.0.2
- Jinja2 ==2.10
- MarkupSafe ==1.0
- PyDispatcher ==2.0.5
- PyYAML ==3.12
- Scrapy ==1.5.0
- Twisted ==18.4.0
- Werkzeug ==0.14.1
- attrs ==18.1.0
- cffi ==1.11.5
- characteristic ==14.3.0
- click ==6.7
- coverage ==4.5.1
- cryptography ==2.3
- cssselect ==1.0.3
- enum34 ==1.1.6
- funcsigs ==1.0.2
- future ==0.16.0
- idna ==2.6
- ipaddress ==1.0.22
- itsdangerous ==0.24
- jsonschema ==2.6.0
- kafka-python ==1.4.3
- kazoo ==2.4.0
- lxml ==4.2.1
- mock ==2.0.0
- nose ==1.3.7
- parsel ==1.4.0
- pbr ==4.0.3
- pyOpenSSL ==18.0.0
- pyasn1 ==0.4.3
- pyasn1-modules ==0.2.1
- pycparser ==2.18
- python-json-logger ==0.1.8
- python-redis-lock ==3.2.0
- queuelib ==1.5.0
- redis >=3.0
- requests ==2.18.4
- requests-file ==1.4.3
- retrying ==1.3.3
- service-identity ==17.0.0
- six ==1.11.0
- testfixtures ==6.0.2
- tldextract ==2.2.0
- ujson ==1.35
- w3lib ==1.19.0
- zope.interface ==4.5.0
- ConcurrentLogHandler ==0.9.1
- Flask ==1.0.2
- Jinja2 ==2.10
- MarkupSafe ==1.1.0
- Werkzeug ==0.14.1
- click ==6.7
- funcsigs ==1.0.2
- future ==0.16.0
- itsdangerous ==0.24
- jsonschema ==2.6.0
- kafka-python ==1.4.3
- kazoo ==2.4.0
- mock ==2.0.0
- nose ==1.3.7
- pbr ==4.0.3
- python-json-logger ==0.1.8
- redis >=3.0
- requests ==2.18.4
- retrying ==1.3.3
- six ==1.11.0
- testfixtures ==6.0.2
- ujson ==1.35
- ConcurrentLogHandler >=0.9.1
- future >=0.16.0
- kazoo >=2.4.0
- mock >=2.0.0
- python-json-logger ==0.1.8
- redis >=3.0
- testfixtures >=6.0.2
- ujson >=1.35