Ecosyste.ms: Repos
An open API service providing repository metadata for many open source software ecosystems.
GitHub / p0dalirius / robotstester
This Python script can enumerate all URLs present in robots.txt files, and test whether they can be accessed or not.
JSON API: https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/p0dalirius%2Frobotstester
Stars: 39
Forks: 4
Open Issues: 1
License: gpl-3.0
Language: Python
Repo Size: 236 KB
Dependencies:
4
Created: almost 3 years ago
Updated: about 1 month ago
Last pushed: 8 months ago
Last synced: 25 days ago
Topics: bugbounty, crawler, pentesting, python, robots, tool
Funding links: https://github.com/sponsors/p0dalirius, https://patreon.com/Podalirius
Files
Dependencies
- requests *
- rich *
- setuptools *
- urllib3 *