Ecosyste.ms: Repos

An open API service providing repository metadata for many open source software ecosystems.

GitHub topics: robots-exclusion-standard

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

Language: JavaScript - Size: 71.3 KB - Last synced: 10 days ago - Pushed: 11 months ago - Stars: 11 - Forks: 7

Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag

Language: C# - Size: 198 KB - Last synced: 30 days ago - Pushed: about 1 month ago - Stars: 0 - Forks: 0

NodeJS robots.txt parser with support for wildcard (*) matching.

Language: JavaScript - Size: 600 KB - Last synced: 24 days ago - Pushed: 5 months ago - Stars: 141 - Forks: 18

Alternative robots parser module for Python

Language: Python - Size: 265 KB - Last synced: about 1 month ago - Pushed: about 1 month ago - Stars: 14 - Forks: 0

Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.

Size: 135 KB - Last synced: 3 months ago - Pushed: 3 months ago - Stars: 80 - Forks: 39

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

Language: PHP - Size: 28.3 KB - Last synced: 1 day ago - Pushed: 7 months ago - Stars: 8 - Forks: 2

Fully native robots.txt parsing component without any dependencies.

Language: JavaScript - Size: 19.5 KB - Last synced: about 2 months ago - Pushed: over 1 year ago - Stars: 0 - Forks: 0

Go robots.txt parser

Language: Go - Size: 17.6 KB - Last synced: 10 months ago - Pushed: over 6 years ago - Stars: 16 - Forks: 7

The repository contains Google-based robots.txt parser and matcher as a C++ library (compliant to C++17).

Language: C++ - Size: 80.1 KB - Last synced: about 1 year ago - Pushed: almost 4 years ago - Stars: 0 - Forks: 0

Related Keywords