GitHub topics: robots-txt-parser
jwmorley73/jwm.robotstxt
Provides python access to Googles parser for robot.txt files as used by their GoogleBot webscraper.
Language: Python - Size: 160 KB - Last synced at: about 14 hours ago - Pushed at: 11 months ago - Stars: 1 - Forks: 0
crwlrsoft/robots-txt
Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping
Language: PHP - Size: 32.2 KB - Last synced at: 26 days ago - Pushed at: 4 months ago - Stars: 11 - Forks: 2
bopoda/robots-txt-parser
PHP class for parse all directives from robots.txt files according to specifications
Language: DIGITAL Command Language - Size: 153 KB - Last synced at: 3 days ago - Pushed at: about 1 month ago - Stars: 45 - Forks: 17
arvid-berndtsson/robots-txt-analyzer
Modern robots.txt analyzer with instant analysis, security recommendations, and export capabilities. Built with Qwik and deployed on Cloudflare Pages.
Language: TypeScript - Size: 707 KB - Last synced at: about 2 months ago - Pushed at: about 2 months ago - Stars: 1 - Forks: 0
chrisakroyd/robots-txt-parser
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
Language: JavaScript - Size: 71.3 KB - Last synced at: about 20 hours ago - Pushed at: almost 2 years ago - Stars: 13 - Forks: 9
vxern/robots_txt
⚙️ A quality `robots.txt` ruleset parser to ensure your application follows the standard specification for the file.
Language: Dart - Size: 46.9 KB - Last synced at: 1 day ago - Pushed at: 6 months ago - Stars: 2 - Forks: 0
