GitHub / robotshell / robotScraper
RobotScraper is a simple tool written in Python to check each of the paths found in the robots.txt file and what HTTP response code they return.
JSON API: http://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robotshell%2FrobotScraper
PURL: pkg:github/robotshell/robotScraper
Stars: 15
Forks: 2
Open issues: 1
License: gpl-3.0
Language: Python
Size: 374 KB
Dependencies parsed at: Pending
Created at: about 4 years ago
Updated at: about 1 month ago
Pushed at: 11 months ago
Last synced at: 4 days ago
Topics: bounty-hunting-tools, bugbounty, hacking, infosec, python, robots, scraper, tool
Funding Links https://github.com/sponsors/robotshell