peaceiris/docker-images

A collection of Docker images

Language: Makefile - Size: 36.1 KB - Last synced: 7 days ago - Pushed: 7 days ago - Stars: 0 - Forks: 0

prod3v3loper/php-honeypot-robots-bait

🍯 HoneyPot Robots Bait

Language: PHP - Size: 36.1 KB - Last synced: 12 days ago - Pushed: 12 days ago - Stars: 0 - Forks: 0

aafeher/go-sitemap-parser

Go language library for parsing Sitemaps

Language: Go - Size: 35.2 KB - Last synced: 18 days ago - Pushed: 19 days ago - Stars: 2 - Forks: 0

toimik/RobotsProtocol

Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag

Language: C# - Size: 198 KB - Last synced: 24 days ago - Pushed: 24 days ago - Stars: 0 - Forks: 0

Folyd/robotstxt

A native Rust port of Google's robots.txt parser and matcher C++ library.

Language: Rust - Size: 207 KB - Last synced: 25 days ago - Pushed: over 3 years ago - Stars: 83 - Forks: 11

alextim/astro-lib

Makes it easy to add robots.txt, sitemap and web app manifest during build to your Astro app.

Language: TypeScript - Size: 1.34 MB - Last synced: 28 days ago - Pushed: 5 months ago - Stars: 97 - Forks: 4

ropensci/robotstxt

robots.txt file parsing and checking for R

Language: R - Size: 2.41 MB - Last synced: 14 days ago - Pushed: over 1 year ago - Stars: 69 - Forks: 9

jasonmajors/robotstxt

This is a small package to make parsing robots.txt rules easier.

Language: PHP - Size: 22.5 KB - Last synced: about 2 months ago - Pushed: about 7 years ago - Stars: 3 - Forks: 0

andreburgaud/robotspy

Alternative robots parser module for Python

Language: Python - Size: 265 KB - Last synced: 27 days ago - Pushed: 27 days ago - Stars: 14 - Forks: 0

dmi3kno/polite

Be nice on the web

Language: R - Size: 1.8 MB - Last synced: 2 months ago - Pushed: 9 months ago - Stars: 318 - Forks: 13

TahaT80/Robots_Scanner

Robots Scanner

Language: Python - Size: 33.2 KB - Last synced: 3 months ago - Pushed: about 3 years ago - Stars: 1 - Forks: 0

ajayjsr/bloggerrobotstxtgenerator

The Blogger Robots.txt Generatois a tool designed to simplify the process of creating a robots.txt file for websites hosted on the Blogger platform. A robots.txt file is crucial for controlling how search engines index your site. This generator allows users to customize and generate a robots.txt file tailored to their specific needs.

Language: HTML - Size: 151 KB - Last synced: 3 months ago - Pushed: 3 months ago - Stars: 0 - Forks: 0

MaximeGuinard/Robots.txt-Viewer

🌐 Displays the contents of robots.txt and sitemap.xml files of a website google extension

Language: JavaScript - Size: 3.91 KB - Last synced: 5 months ago - Pushed: 5 months ago - Stars: 1 - Forks: 0

herrbischoff/robots.txt

A sane, minimal robots.txt file (for the western world)

Size: 1000 Bytes - Last synced: 25 days ago - Pushed: almost 4 years ago - Stars: 5 - Forks: 1

itgalaxy/robotstxt-webpack-plugin

A webpack plugin to generate a robots.txt file

Language: JavaScript - Size: 784 KB - Last synced: about 1 month ago - Pushed: about 1 year ago - Stars: 35 - Forks: 7

pdehaan/robots-parser-test

Language: JavaScript - Size: 12.7 KB - Last synced: 7 months ago - Pushed: 7 months ago - Stars: 0 - Forks: 0

Zer0Divis0r/rebuild-cache-sitemap

GitHub action to rebuild CDN cache according to sitemaps

Language: Shell - Size: 9.77 KB - Last synced: 3 months ago - Pushed: 3 months ago - Stars: 1 - Forks: 0

datatxtorg/datatxt-spec

data.txt: Specification

Size: 7.81 KB - Last synced: 7 months ago - Pushed: almost 4 years ago - Stars: 6 - Forks: 2

itgalaxy/generate-robotstxt

Generator robots.txt for node js

Language: JavaScript - Size: 2.86 MB - Last synced: about 1 month ago - Pushed: over 1 year ago - Stars: 64 - Forks: 8

Keyvanhardani/Robots.txt-finder

Robots.txt Finder V1 by Keyvan Hardani

Language: Python - Size: 5.86 KB - Last synced: 10 months ago - Pushed: almost 2 years ago - Stars: 1 - Forks: 0

rimiti/robotizer

Robots.txt parser / generator

Language: TypeScript - Size: 242 KB - Last synced: 25 days ago - Pushed: over 5 years ago - Stars: 1 - Forks: 0

ChrisWinters/robotstxt-manager

A Simple Robots.txt Manager Plugin For Wordpress.

Language: PHP - Size: 4.12 MB - Last synced: 2 months ago - Pushed: about 1 year ago - Stars: 1 - Forks: 0

littlebizzy/virtual-robotstxt

Replaces the default virtual robots.txt generated by WordPress with an editable one, and deletes any physical robots.txt file that may already exist.

Size: 68.4 KB - Last synced: about 1 year ago - Pushed: about 5 years ago - Stars: 1 - Forks: 1

MarcoDaniels/elm-robots-humans

Create robots.txt and humans.txt in Elm

Language: Elm - Size: 17.6 KB - Last synced: 10 months ago - Pushed: over 1 year ago - Stars: 1 - Forks: 0

spcanelon/2022-ccd-sips

Workshop materials for creating the Center City Sips District 2022 interactive map

Language: HTML - Size: 10.9 MB - Last synced: about 1 year ago - Pushed: over 1 year ago - Stars: 3 - Forks: 0

onuratakan/Site_Seo_Scanner

The site seo scanner.

Language: Python - Size: 16.6 KB - Last synced: 6 days ago - Pushed: almost 3 years ago - Stars: 1 - Forks: 0

vedranvinko/robots

robots.txt

Language: Rust - Size: 14.6 KB - Last synced: about 1 year ago - Pushed: over 2 years ago - Stars: 0 - Forks: 0

ossama131/Bias-to-Search-Engines-from-Robots.txt

Determining bias to search engines from Robots.txt

Language: Jupyter Notebook - Size: 224 KB - Last synced: about 2 months ago - Pushed: over 2 years ago - Stars: 0 - Forks: 0

nicholasbergesen/RobotsParser

Parse robots.txt and traverse sitemaps.

Language: C# - Size: 6.59 MB - Last synced: 6 months ago - Pushed: 6 months ago - Stars: 1 - Forks: 1

samclarke/robotstxt

Go robots.txt parser

Language: Go - Size: 17.6 KB - Last synced: 10 months ago - Pushed: over 6 years ago - Stars: 16 - Forks: 7

dpb587/go-sitemap

Stream decoders for sitemap.xml data and link feeds.

Language: Go - Size: 21.5 KB - Last synced: about 1 month ago - Pushed: over 2 years ago - Stars: 0 - Forks: 0

datatxtorg/datatxt-site

data.txt: Data for the Web

Language: HTML - Size: 22.5 KB - Last synced: 10 months ago - Pushed: over 4 years ago - Stars: 2 - Forks: 0

itechbear/robotstxt

A java clone of Google's robotst.txt parser: https://github.com/google/robotstxt

Language: Java - Size: 43 KB - Last synced: 8 months ago - Pushed: over 3 years ago - Stars: 0 - Forks: 0

hgruniaux/robotstxt Fork of google/robotstxt

The repository contains Google-based robots.txt parser and matcher as a C++ library (compliant to C++17).

Language: C++ - Size: 80.1 KB - Last synced: about 1 year ago - Pushed: almost 4 years ago - Stars: 0 - Forks: 0

thomasleveil/pyrobots

python binding for Google robots.txt parser C++ library

Language: Python - Size: 11.7 KB - Last synced: about 1 year ago - Pushed: almost 5 years ago - Stars: 0 - Forks: 0