GitHub topics: robots-parser
scrapy/protego
A pure-Python robots.txt parser with support for modern conventions.
Language: DIGITAL Command Language - Size: 3.43 MB - Last synced at: 10 days ago - Pushed at: 10 days ago - Stars: 69 - Forks: 28
andreburgaud/robotspy
Alternative robots parser module for Python
Language: Python - Size: 340 KB - Last synced at: 18 days ago - Pushed at: 18 days ago - Stars: 18 - Forks: 1
jwmorley73/jwm.robotstxt
Provides python access to Googles parser for robot.txt files as used by their GoogleBot webscraper.
Language: Python - Size: 160 KB - Last synced at: 27 days ago - Pushed at: 12 months ago - Stars: 1 - Forks: 0
samclarke/robots-parser
NodeJS robots.txt parser with support for wildcard (*) matching.
Language: JavaScript - Size: 506 KB - Last synced at: 5 days ago - Pushed at: 8 months ago - Stars: 156 - Forks: 20
chrisakroyd/robots-txt-parser
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
Language: JavaScript - Size: 71.3 KB - Last synced at: 28 days ago - Pushed at: about 2 years ago - Stars: 13 - Forks: 9
nicholasbergesen/robots-parser
Parse robots.txt and traverse sitemaps.
Language: C# - Size: 6.6 MB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 2 - Forks: 2
messense/robotparser-rs
robots.txt parser for Rust.
Language: Rust - Size: 19.6 MB - Last synced at: 17 days ago - Pushed at: over 3 years ago - Stars: 18 - Forks: 9
AntoineGagne/robots
A parser for robots.txt with support for wildcards. See also RFC 9309.
Language: Erlang - Size: 30.3 KB - Last synced at: 17 days ago - Pushed at: over 1 year ago - Stars: 3 - Forks: 2
fooock/robots.txt
:robot: robots.txt as a service. Crawls robots.txt files, downloads and parses them to check rules through an API
Language: Java - Size: 1.92 MB - Last synced at: 3 months ago - Pushed at: over 4 years ago - Stars: 16 - Forks: 2
ptsochantaris/can-proceed
A small, tested, no-frills parser of robots.txt files in Swift.
Language: Swift - Size: 26.4 KB - Last synced at: about 13 hours ago - Pushed at: 9 months ago - Stars: 1 - Forks: 0
muratgozel/robotstxt-util
RFC 9309 spec compliant robots.txt builder and parser. 🦾 No dependencies, fully typed.
Language: TypeScript - Size: 136 KB - Last synced at: 24 days ago - Pushed at: 10 months ago - Stars: 3 - Forks: 1
VIPnytt/RobotsTxtParser
An extensible robots.txt parser and client library, with full support for every directive and specification.
Language: PHP - Size: 526 KB - Last synced at: 2 months ago - Pushed at: about 4 years ago - Stars: 26 - Forks: 6
ravern/gollum
Robots.txt parser and fetcher for Elixir
Language: Elixir - Size: 29.3 KB - Last synced at: 4 days ago - Pushed at: over 2 years ago - Stars: 14 - Forks: 11
samclarke/robotstxt
Go robots.txt parser
Language: Go - Size: 17.6 KB - Last synced at: about 1 year ago - Pushed at: over 7 years ago - Stars: 16 - Forks: 7
toimik/RobotsProtocol
Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag
Language: C# - Size: 198 KB - Last synced at: 15 days ago - Pushed at: about 1 year ago - Stars: 0 - Forks: 0
drmathias/robots
Parse robots.txt and sitemaps using dotnet
Language: C# - Size: 37 MB - Last synced at: almost 2 years ago - Pushed at: almost 2 years ago - Stars: 0 - Forks: 0
rimiti/robotizer
Robots.txt parser / generator
Language: TypeScript - Size: 242 KB - Last synced at: about 2 months ago - Pushed at: almost 7 years ago - Stars: 1 - Forks: 0
eliasdabbas/robotstxt_app
Visual App for Testing URLs and User-agents blocked by robots.txt Files
Language: Python - Size: 24.4 KB - Last synced at: 2 months ago - Pushed at: over 2 years ago - Stars: 4 - Forks: 0
larevanchedessites/google-robotstxt-ruby
🤖 Ruby gem wrapper around Google Robotstxt Parser C++ library
Language: Ruby - Size: 18.6 KB - Last synced at: 30 days ago - Pushed at: over 5 years ago - Stars: 4 - Forks: 3
0xIbra/robots-txt-component
Fully native robots.txt parsing component without any dependencies.
Language: JavaScript - Size: 19.5 KB - Last synced at: 4 months ago - Pushed at: over 2 years ago - Stars: 0 - Forks: 0
b4dnewz/robots-parse
A lightweight and simple robots.txt parser in node
Language: TypeScript - Size: 647 KB - Last synced at: 5 days ago - Pushed at: over 2 years ago - Stars: 7 - Forks: 0
hgruniaux/robotstxt Fork of google/robotstxt
The repository contains Google-based robots.txt parser and matcher as a C++ library (compliant to C++17).
Language: C++ - Size: 80.1 KB - Last synced at: about 2 years ago - Pushed at: almost 5 years ago - Stars: 0 - Forks: 0
rimiti/robotstxt
Robots.txt parser and generator - Work in progress
Language: Go - Size: 24.4 KB - Last synced at: about 2 months ago - Pushed at: about 7 years ago - Stars: 3 - Forks: 0
thomasleveil/pyrobots
python binding for Google robots.txt parser C++ library
Language: Python - Size: 11.7 KB - Last synced at: 3 months ago - Pushed at: almost 6 years ago - Stars: 0 - Forks: 0
mounicmadiraju/robot.txt-changes
💧 Test your robots.txt with this testing tool. Check if a URL is blocked, which statement is blocking it and for which user agent. You can also check if the resources for the page (CSS and JavaScript) are disallowed!. Robots.txt files help you guide how search engines crawl your site, and can be an integral part of your SEO strategy.
Language: Python - Size: 10.7 KB - Last synced at: over 2 years ago - Pushed at: over 7 years ago - Stars: 2 - Forks: 0
