GitHub topics: robots-exclusion-standard
samclarke/robots-parser
NodeJS robots.txt parser with support for wildcard (*) matching.
Language: JavaScript - Size: 506 KB - Last synced at: about 1 month ago - Pushed at: about 1 year ago - Stars: 159 - Forks: 19
andreburgaud/robotspy
Alternative robots parser module for Python
Language: Python - Size: 342 KB - Last synced at: 19 days ago - Pushed at: 3 months ago - Stars: 20 - Forks: 1
chrisakroyd/robots-txt-parser
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
Language: JavaScript - Size: 71.3 KB - Last synced at: 13 days ago - Pushed at: over 2 years ago - Stars: 15 - Forks: 9
samclarke/robotstxt
Go robots.txt parser
Language: Go - Size: 17.6 KB - Last synced at: about 2 months ago - Pushed at: almost 8 years ago - Stars: 15 - Forks: 7
crwlrsoft/robots-txt
Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping
Language: PHP - Size: 32.2 KB - Last synced at: 6 months ago - Pushed at: 10 months ago - Stars: 11 - Forks: 2
jonasjacek/robots.txt
Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.
Size: 135 KB - Last synced at: 9 months ago - Pushed at: 9 months ago - Stars: 87 - Forks: 38
AntoineGagne/robots
A parser for robots.txt with support for wildcards. See also RFC 9309.
Language: Erlang - Size: 30.3 KB - Last synced at: 2 months ago - Pushed at: almost 2 years ago - Stars: 3 - Forks: 2
toimik/RobotsProtocol
Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag
Language: C# - Size: 198 KB - Last synced at: 3 months ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0
0xIbra/robots-txt-component
Fully native robots.txt parsing component without any dependencies.
Language: JavaScript - Size: 19.5 KB - Last synced at: 20 days ago - Pushed at: about 3 years ago - Stars: 0 - Forks: 0
hgruniaux/robotstxt Fork of google/robotstxt
The repository contains Google-based robots.txt parser and matcher as a C++ library (compliant to C++17).
Language: C++ - Size: 80.1 KB - Last synced at: over 2 years ago - Pushed at: about 5 years ago - Stars: 0 - Forks: 0
