Ecosyste.ms: Repos
An open API service providing repository metadata for many open source software ecosystems.
GitHub / aayu0795 / Automated-Web-Scraper
Automated web scraper using Selenium and Celery, allow you to scrape the website periodically based on some schedule. The scrapped data then cleaned and stored in our database.
JSON API: https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aayu0795%2FAutomated-Web-Scraper
Stars: 0
Forks: 0
Open Issues: 4
License: None
Language: Python
Repo Size: 9.55 MB
Dependencies:
57
Created: almost 4 years ago
Updated: almost 4 years ago
Last pushed: over 1 year ago
Last synced: about 1 year ago
Files
Loading...
Readme
Loading...
Dependencies
Pipfile
pypi
- Django ==3.0.7
- autopep8 ==1.4.4
- beautifulsoup4 *
- celery *
- django-celery-beat *
- django-debug-toolbar ==2.2
- django-environ ==0.4.5
- flower *
- pep8 ==1.7.1
- pycodestyle ==2.5.0
- python-decouple ==3.3
- pytz ==2020.1
- redis *
- requests *
- selenium *
- sqlparse ==0.3.1
requirements.txt
pypi
- Django ==3.0.5
- autopep8 ==1.4.4
- django-debug-toolbar ==2.2
- django-environ ==0.4.5
- pep8 ==1.7.1
- pycodestyle ==2.5.0
- python-decouple ==3.3
- pytz ==2020.1
- sqlparse ==0.3.1
Pipfile.lock
pypi
- amqp ==2.6.0
- asgiref ==3.2.7
- autopep8 ==1.4.4
- beautifulsoup4 ==4.9.1
- billiard ==3.6.3.0
- celery ==4.4.2
- certifi ==2020.4.5.1
- chardet ==3.0.4
- django ==3.0.7
- django-celery-beat ==2.0.0
- django-debug-toolbar ==2.2
- django-environ ==0.4.5
- django-timezone-field ==4.0
- flower ==0.9.4
- humanize ==0.5.1
- idna ==2.9
- kombu ==4.6.10
- pep8 ==1.7.1
- pycodestyle ==2.5.0
- python-crontab ==2.5.1
- python-dateutil ==2.8.1
- python-decouple ==3.3
- pytz ==2020.1
- redis ==3.5.2
- requests ==2.23.0
- selenium ==3.141.0
- six ==1.15.0
- soupsieve ==2.0.1
- sqlparse ==0.3.1
- tornado ==6.0.4
- urllib3 ==1.25.9
- vine ==1.3.0