website-crawler | Topic | Ecosyste.ms: Repos

Topic: "website-crawler"

X-SLAYER/Website-Cloner

It allows you to download a website from the Internet to a local directory, building recursively all directories, getting HTML, images, and other files from the server to your computer.

Language: Visual Basic .NET - Size: 1.11 MB - Last synced at: 4 months ago - Pushed at: almost 2 years ago - Stars: 283 - Forks: 85

MLArtist/WebScraper

Python-based web crawling script with randomized intervals, user-agent rotation, and proxy server IP rotation to outsmart website bots and prevent blocking.

Language: Python - Size: 43.9 KB - Last synced at: 9 months ago - Pushed at: 9 months ago - Stars: 61 - Forks: 14

Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis

Language: Python - Size: 19.7 MB - Last synced at: 6 days ago - Pushed at: over 1 year ago - Stars: 37 - Forks: 0

vlmaier/marvel-snap-scrapr

Scraper for https://marvelsnapzone.com to retrieve metadata of Marvel SNAP cards.

Language: Python - Size: 31.3 KB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 21 - Forks: 5

sammwyy/SpearCopy

A universal and local phishing toolkit for audit purposes

Language: Python - Size: 6.84 KB - Last synced at: 24 days ago - Pushed at: 5 months ago - Stars: 17 - Forks: 1

chandrasekharan98/Multisite-Python-Crawler

An almost generic web crawler built using Scrapy and Python 3.7 to recursively crawl entire websites.

Language: Python - Size: 15.6 KB - Last synced at: 6 months ago - Pushed at: about 3 years ago - Stars: 16 - Forks: 5

yogsec/endpoints-extractor

A powerful Bash script for extracting URLs and API endpoints from HTML, JavaScript, and JSON content of web pages. Designed for security researchers, bug bounty hunters, and developers to streamline endpoint discovery. Simple to use, supports single or multiple URLs, and offers file-saving capabilities.

Language: Shell - Size: 81.1 KB - Last synced at: 24 days ago - Pushed at: 24 days ago - Stars: 12 - Forks: 1

oxylabs/web-scraping-php

A tutorial and code samples of web scraping with PHP

Language: PHP - Size: 26.4 KB - Last synced at: 1 day ago - Pushed at: 2 months ago - Stars: 9 - Forks: 3

zebbern/ReconX

🕷️ | ReconX is a Live-Website Crawler made to gather critical information with an option to take a picture of each site crawled!

Language: Python - Size: 57.6 KB - Last synced at: 2 days ago - Pushed at: 2 months ago - Stars: 3 - Forks: 0

Mediashare/crawler

:dizzy: Crawl urls from a webpage and provide a DomCrawler with Scraper Library

Language: PHP - Size: 40 KB - Last synced at: 2 months ago - Pushed at: 5 months ago - Stars: 3 - Forks: 1

1970Mr/link-crawler

Web Link Crawler: A Python script to crawl websites and collect links based on a regex pattern. Efficient and customizable.

Language: Python - Size: 32.2 KB - Last synced at: about 1 month ago - Pushed at: 11 months ago - Stars: 2 - Forks: 1

foomo/walker

Crawls website and collect SEO relevant data

Language: Go - Size: 188 KB - Last synced at: about 2 months ago - Pushed at: over 2 years ago - Stars: 2 - Forks: 0

Deependra-Patel/websiteCrawler

Crawls a website to generate insights

Language: Go - Size: 11.7 KB - Last synced at: about 1 month ago - Pushed at: about 6 years ago - Stars: 2 - Forks: 0

vlOd2/LightshotScraper

The most advanced Lightshot (or prnt.sc) scraper ever!

Language: Java - Size: 3.35 MB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 1 - Forks: 3

ZKAW/website-crawler

Recursive website crawler

Language: Python - Size: 2.93 KB - Last synced at: almost 2 years ago - Pushed at: about 3 years ago - Stars: 1 - Forks: 0

Dyzio18/java-web-bot-library

Java website crawler - library for analyze and testing websites

Language: Java - Size: 885 KB - Last synced at: 2 months ago - Pushed at: over 3 years ago - Stars: 1 - Forks: 0

reineimi/va2crawl

Website crawler, validator and SEO optimizer

Language: Shell - Size: 16.6 KB - Last synced at: about 2 months ago - Pushed at: 6 months ago - Stars: 0 - Forks: 0

ursnj/seo-master

SEO Master is a powerful all-in-one tool developed to boost your website's visibility and rankings. With features like automatic sitemap generation, customizable robots.txt creation, SEO-optimized metadata, Image assets generation and seamless integration with major search engines.

Language: TypeScript - Size: 162 KB - Last synced at: 13 days ago - Pushed at: 6 months ago - Stars: 0 - Forks: 0