GitHub topics: robots-exclusion-protocol
andreburgaud/robotspy
Alternative robots parser module for Python
Language: Python - Size: 337 KB - Last synced at: 3 days ago - Pushed at: 3 months ago - Stars: 18 - Forks: 1
bnomei/kirby3-robots-txt
Manage the robots.txt from the Kirby config file
Language: PHP - Size: 189 KB - Last synced at: 20 days ago - Pushed at: 21 days ago - Stars: 20 - Forks: 0
cicirello/generate-sitemap
Generate an XML sitemap for a GitHub Pages site using GitHub Actions
Language: Python - Size: 487 KB - Last synced at: 23 days ago - Pushed at: 8 months ago - Stars: 90 - Forks: 16
crwlrsoft/robots-txt
Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping
Language: PHP - Size: 32.2 KB - Last synced at: 26 days ago - Pushed at: 4 months ago - Stars: 11 - Forks: 2
jimsmart/grobotstxt
grobotstxt is a native Go port of Google's robots.txt parser and matcher library.
Language: Go - Size: 238 KB - Last synced at: about 2 months ago - Pushed at: about 3 years ago - Stars: 110 - Forks: 7
hrbrmstr/spiderbar
Lightweight R wrapper around rep-cpp for robot.txt (Robots Exclusion Protocol) parsing and path testing in R
Language: C++ - Size: 88.9 KB - Last synced at: about 1 month ago - Pushed at: 11 months ago - Stars: 9 - Forks: 2
VIPnytt/UserAgentParser
User-Agent parser for robots.txt, X-Robots-tag and Robots-meta-tag rule sets
Language: PHP - Size: 38.1 KB - Last synced at: 4 months ago - Pushed at: 4 months ago - Stars: 1 - Forks: 2
muratgozel/robotstxt-util
RFC 9309 spec compliant robots.txt builder and parser. 🦾 No dependencies, fully typed.
Language: TypeScript - Size: 136 KB - Last synced at: 28 days ago - Pushed at: 9 months ago - Stars: 3 - Forks: 1
toimik/RobotsProtocol
Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag
Language: C# - Size: 198 KB - Last synced at: 23 days ago - Pushed at: about 1 year ago - Stars: 0 - Forks: 0
drmathias/robots
Parse robots.txt and sitemaps using dotnet
Language: C# - Size: 37 MB - Last synced at: almost 2 years ago - Pushed at: almost 2 years ago - Stars: 0 - Forks: 0
bnomei/kirby-robots-writer 📦
Robots for Kirby CMS
Language: PHP - Size: 9.77 KB - Last synced at: about 1 year ago - Pushed at: over 6 years ago - Stars: 8 - Forks: 2
michielvoo/robots.txt
PowerShell module for reading robots.txt files
Language: C# - Size: 7.81 KB - Last synced at: over 2 years ago - Pushed at: about 3 years ago - Stars: 0 - Forks: 0
hgruniaux/robotstxt Fork of google/robotstxt
The repository contains Google-based robots.txt parser and matcher as a C++ library (compliant to C++17).
Language: C++ - Size: 80.1 KB - Last synced at: about 2 years ago - Pushed at: almost 5 years ago - Stars: 0 - Forks: 0
thomasleveil/pyrobots
python binding for Google robots.txt parser C++ library
Language: Python - Size: 11.7 KB - Last synced at: 2 months ago - Pushed at: almost 6 years ago - Stars: 0 - Forks: 0
