Ecosyste.ms: Repos
An open API service providing repository metadata for many open source software ecosystems.
GitHub / crwlrsoft / robots-txt
Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping
JSON API: https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crwlrsoft%2Frobots-txt
Stars: 8
Forks: 2
Open Issues: 0
License: mit
Language: PHP
Repo Size: 28.3 KB
Dependencies:
7
Created: over 2 years ago
Updated: 6 months ago
Last pushed: 7 months ago
Last synced: 3 days ago
Topics: hacktoberfest, robots-exclusion-protocol, robots-exclusion-standard, robots-txt, robots-txt-parser, web-crawling, web-scraping
Files
Dependencies
- friendsofphp/php-cs-fixer ^3.0 development
- mockery/mockery ^1.4 development
- phpstan/phpstan ^1.1 development
- phpunit/phpunit ^9.0 development
- sempro/phpunit-pretty-print ^1.4 development
- crwlr/url ^1.0
- php ^7.4|^8.0