Ecosyste.ms: Repos

An open API service providing repository metadata for many open source software ecosystems.

GitHub topics: robots-meta-tag

VIPnytt/UserAgentParser

User-Agent parser for robots.txt, X-Robots-tag and Robots-meta-tag rule sets

Language: PHP - Size: 33.2 KB - Last synced: 27 days ago - Pushed: about 3 years ago - Stars: 1 - Forks: 2

Related Keywords
crawler 1 rep 1 robot 1 robots-exclusion-protocol 1 robots-meta-tag 1 robots-tag 1 robots-txt 1 spider 1 user-agent 1 user-agent-parser 1 x-robots-tag 1