GitHub topics: robots-txt-parser
arvid-berndtsson/robots-txt-analyzer
Modern robots.txt analyzer with instant analysis, security recommendations, and export capabilities. Built with Qwik and deployed on Cloudflare Pages.
Language: TypeScript - Size: 727 KB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 1 - Forks: 0
chrisakroyd/robots-txt-parser
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
Language: JavaScript - Size: 71.3 KB - Last synced at: 10 days ago - Pushed at: over 2 years ago - Stars: 15 - Forks: 9
jwmorley73/jwm.robotstxt
Provides python access to Googles parser for robot.txt files as used by their GoogleBot webscraper.
Language: Python - Size: 160 KB - Last synced at: about 2 months ago - Pushed at: over 1 year ago - Stars: 1 - Forks: 0
crwlrsoft/robots-txt
Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping
Language: PHP - Size: 32.2 KB - Last synced at: 6 months ago - Pushed at: 10 months ago - Stars: 11 - Forks: 2
bopoda/robots-txt-parser
PHP class for parse all directives from robots.txt files according to specifications
Language: DIGITAL Command Language - Size: 153 KB - Last synced at: 4 months ago - Pushed at: 7 months ago - Stars: 45 - Forks: 17
vxern/robots_txt
⚙️ A quality `robots.txt` ruleset parser to ensure your application follows the standard specification for the file.
Language: Dart - Size: 46.9 KB - Last synced at: 21 days ago - Pushed at: 12 months ago - Stars: 2 - Forks: 0
