GitHub topics: robots-exclusion-standard
andreburgaud/robotspy
Alternative robots parser module for Python
Language: Python - Size: 340 KB - Last synced at: 27 days ago - Pushed at: 27 days ago - Stars: 18 - Forks: 1
samclarke/robots-parser
NodeJS robots.txt parser with support for wildcard (*) matching.
Language: JavaScript - Size: 506 KB - Last synced at: 5 days ago - Pushed at: 9 months ago - Stars: 156 - Forks: 20
crwlrsoft/robots-txt
Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping
Language: PHP - Size: 32.2 KB - Last synced at: 2 months ago - Pushed at: 6 months ago - Stars: 11 - Forks: 2
chrisakroyd/robots-txt-parser
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
Language: JavaScript - Size: 71.3 KB - Last synced at: 4 days ago - Pushed at: about 2 years ago - Stars: 13 - Forks: 9
jonasjacek/robots.txt
Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.
Size: 135 KB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 87 - Forks: 38
AntoineGagne/robots
A parser for robots.txt with support for wildcards. See also RFC 9309.
Language: Erlang - Size: 30.3 KB - Last synced at: 26 days ago - Pushed at: over 1 year ago - Stars: 3 - Forks: 2
samclarke/robotstxt
Go robots.txt parser
Language: Go - Size: 17.6 KB - Last synced at: about 1 year ago - Pushed at: over 7 years ago - Stars: 16 - Forks: 7
toimik/RobotsProtocol
Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag
Language: C# - Size: 198 KB - Last synced at: 24 days ago - Pushed at: about 1 year ago - Stars: 0 - Forks: 0
0xIbra/robots-txt-component
Fully native robots.txt parsing component without any dependencies.
Language: JavaScript - Size: 19.5 KB - Last synced at: 5 months ago - Pushed at: almost 3 years ago - Stars: 0 - Forks: 0
hgruniaux/robotstxt Fork of google/robotstxt
The repository contains Google-based robots.txt parser and matcher as a C++ library (compliant to C++17).
Language: C++ - Size: 80.1 KB - Last synced at: about 2 years ago - Pushed at: almost 5 years ago - Stars: 0 - Forks: 0
