GitHub topics: robots-txt
php-middleware/block-robots
Middleware to avoid search engine indexing with PSR-7 using robots.txt and X-Robots-Tag
Language: PHP - Size: 13.7 KB - Last synced at: 2 months ago - Pushed at: about 4 years ago - Stars: 5 - Forks: 0
useflyyer/robots Fork of ArcaneDigital/parse-robots
Super lightweight plain TypeScript parser for robots.txt with 0 dependencies.
Language: TypeScript - Size: 320 KB - Last synced at: 29 days ago - Pushed at: over 3 years ago - Stars: 2 - Forks: 0
slemarchand/no-robots
🚫🤖 Override /robots.txt to disallow all web crawlers, regardless settings stored in the database. Compatible with Liferay 7.0, 7.1, 7.2, 7.3 and 7.4.
Language: Java - Size: 55.7 KB - Last synced at: over 2 years ago - Pushed at: almost 4 years ago - Stars: 0 - Forks: 0
tekord/robots-txt-provider
Provides various framework-agnostic ways to generate the contents of the robots.txt file
Language: PHP - Size: 11.7 KB - Last synced at: 3 months ago - Pushed at: about 4 years ago - Stars: 0 - Forks: 0
josecarneiro/mr-roboto
🤖 Handle and parse a site's robots.txt file and extract actionable information
Language: JavaScript - Size: 36.1 KB - Last synced at: 16 days ago - Pushed at: over 7 years ago - Stars: 3 - Forks: 0
benjaminestes/robots
Package robots implements robots.txt file parsing and matching based on Google's specification.
Language: Go - Size: 38.1 KB - Last synced at: about 1 year ago - Pushed at: over 5 years ago - Stars: 3 - Forks: 2
VorticonCmdr/robotstxt
Chrome extension which blocks urls based on robots.txt (compatible to Chrome 41)
Language: JavaScript - Size: 200 KB - Last synced at: about 2 years ago - Pushed at: over 6 years ago - Stars: 2 - Forks: 0
hgruniaux/robotstxt Fork of google/robotstxt
The repository contains Google-based robots.txt parser and matcher as a C++ library (compliant to C++17).
Language: C++ - Size: 80.1 KB - Last synced at: about 2 years ago - Pushed at: almost 5 years ago - Stars: 0 - Forks: 0
rimiti/robotstxt
Robots.txt parser and generator - Work in progress
Language: Go - Size: 24.4 KB - Last synced at: about 1 month ago - Pushed at: about 7 years ago - Stars: 3 - Forks: 0
infinityloop-dev/robots
:wrench: Robots.txt generator component for Nette framework.
Language: PHP - Size: 19.5 KB - Last synced at: 3 months ago - Pushed at: about 5 years ago - Stars: 2 - Forks: 1
amandeepmittal/robotize
Generates a robots.txt
Language: JavaScript - Size: 13.7 KB - Last synced at: 2 months ago - Pushed at: over 5 years ago - Stars: 4 - Forks: 0
honzahommer/request-robots
An express.js middleware for handling noisy robots.txt
Language: JavaScript - Size: 1.76 MB - Last synced at: about 1 month ago - Pushed at: over 2 years ago - Stars: 1 - Forks: 0
pierre-pvln/robots_creator
Scripts to create a robots.txt file from building blocks
Language: Batchfile - Size: 40 KB - Last synced at: about 2 years ago - Pushed at: almost 6 years ago - Stars: 0 - Forks: 0
thomasleveil/pyrobots
python binding for Google robots.txt parser C++ library
Language: Python - Size: 11.7 KB - Last synced at: 3 months ago - Pushed at: almost 6 years ago - Stars: 0 - Forks: 0
eghuro/crawlcheck
Extensible web crawler
Language: Python - Size: 3.01 MB - Last synced at: 7 days ago - Pushed at: 7 days ago - Stars: 0 - Forks: 0
hrbrmstr/robotify
🤖 Browser extension to check for and preview a site's robots.txt in a new tab (if it exists)
Language: JavaScript - Size: 14.6 KB - Last synced at: 4 months ago - Pushed at: over 6 years ago - Stars: 3 - Forks: 0
GeorgeA93/crawley
nodejs web crawler
Language: JavaScript - Size: 70.3 KB - Last synced at: 2 months ago - Pushed at: over 8 years ago - Stars: 3 - Forks: 0
mounicmadiraju/robot.txt-changes
💧 Test your robots.txt with this testing tool. Check if a URL is blocked, which statement is blocking it and for which user agent. You can also check if the resources for the page (CSS and JavaScript) are disallowed!. Robots.txt files help you guide how search engines crawl your site, and can be an integral part of your SEO strategy.
Language: Python - Size: 10.7 KB - Last synced at: over 2 years ago - Pushed at: over 7 years ago - Stars: 2 - Forks: 0
barryceelen/wp-robositemap
Manage robots.txt and sitemap.xml via the WordPress admin
Language: PHP - Size: 5.86 KB - Last synced at: 6 days ago - Pushed at: almost 7 years ago - Stars: 0 - Forks: 0
aankur/robot-webpack-plugin
A Webpack 3 plugin for generating robots.txt file
Language: JavaScript - Size: 23.4 KB - Last synced at: 25 days ago - Pushed at: almost 7 years ago - Stars: 0 - Forks: 0
josecarneiro/robotic
A robots.txt generating Express Middleware
Language: JavaScript - Size: 29.3 KB - Last synced at: 18 days ago - Pushed at: over 7 years ago - Stars: 0 - Forks: 0
james-see/random-robots-txt
Generates a random robots.txt deny list to throw script kiddies off the scent.
Language: Python - Size: 9.77 KB - Last synced at: 8 months ago - Pushed at: over 7 years ago - Stars: 1 - Forks: 0
FlorianWendelborn/robogen
🤖 Robots.txt generator done right.
Language: JavaScript - Size: 3.91 KB - Last synced at: 4 months ago - Pushed at: over 8 years ago - Stars: 3 - Forks: 0
mets634/robots-scanner
A program to scan website fro hidden files using the robots.txt file.
Language: Python - Size: 4.88 KB - Last synced at: over 2 years ago - Pushed at: about 8 years ago - Stars: 0 - Forks: 0
