EVANONAAN/Web-Spider-Linux-shell-script

🕷️ Crawl websites efficiently with this Bash script, producing a clean list of URLs while respecting `robots.txt` and staying within specified domains.

Language: Shell - Size: 1.34 MB - Last synced at: 1 day ago - Pushed at: 1 day ago - Stars: 0 - Forks: 0

VASETO131/download-photos-from-instagram

Size: 1.14 MB - Last synced at: 4 days ago - Pushed at: 4 days ago - Stars: 0 - Forks: 0

EduardozinYT/ai-instagram-organizer

📸 Organize your Instagram posts effortlessly with AI, generating smart captions, optimized hashtags, and automatic photo arrangement for maximum engagement.

Language: Python - Size: 2.63 MB - Last synced at: 6 days ago - Pushed at: 6 days ago - Stars: 0 - Forks: 1

xl400v/scrape-json

This shows how to use github actions to do periodic data scraping

Language: JavaScript - Size: 131 MB - Last synced at: 6 days ago - Pushed at: 6 days ago - Stars: 0 - Forks: 0

omkarcloud/botasaurus

The All in One Framework to Build Undefeatable Scrapers

Language: Python - Size: 86 MB - Last synced at: 10 days ago - Pushed at: 11 days ago - Stars: 3,143 - Forks: 263

seaavey/scapers

The Scapers is a collection of tools for scraping data from the web.

Language: TypeScript - Size: 172 KB - Last synced at: 4 days ago - Pushed at: 6 months ago - Stars: 6 - Forks: 0

lorien/awesome-web-scraping

List of libraries, tools and APIs for web scraping and data processing.

Language: Makefile - Size: 427 KB - Last synced at: 13 days ago - Pushed at: 21 days ago - Stars: 7,384 - Forks: 825

Pryodon/Web-Spider-Linux-shell-script

Generate a list of file links you can feed to wget for easy downloading! Mainly used for spidering web folders with lots of files. Can even generate a sitemap text or XML file your your website!

Language: Shell - Size: 42 KB - Last synced at: 27 days ago - Pushed at: 27 days ago - Stars: 0 - Forks: 0

OpenByteDev/SourceScraper 📦

Simple library which helps you to retrieve the source of various video streaming sites.

Language: TypeScript - Size: 4.62 MB - Last synced at: 10 days ago - Pushed at: about 6 years ago - Stars: 70 - Forks: 19

pavlovtech/WebReaper

Web scraper, crawler and parser in C#. Designed as simple, declarative and scalable web scraping solution.

Language: C# - Size: 37.3 MB - Last synced at: 29 days ago - Pushed at: about 1 year ago - Stars: 128 - Forks: 32

luminati-io/Awesome-Web-Scraping

A list of libraries, tools, and APIs for web scraping and data processing. Find everything you need for extracting, managing, and processing data from the web, from HTTP libraries to browser automation tools and proxy services.

Size: 104 KB - Last synced at: 3 days ago - Pushed at: 7 months ago - Stars: 7 - Forks: 2

omkarcloud/botasaurus-starter

🚀 OFFICIAL STARTER TEMPLATE FOR BOTASAURUS SCRAPING FRAMEWORK 🤖

Language: TypeScript - Size: 402 KB - Last synced at: 30 days ago - Pushed at: 4 months ago - Stars: 27 - Forks: 9

colonelpanic8/lastfm-edit

Edit last.fm scrobbles programmatically

Language: HTML - Size: 33.3 MB - Last synced at: 18 days ago - Pushed at: about 1 month ago - Stars: 4 - Forks: 0

pim97/scrappey-wrapper-python

An API wrapper for Scrappey.com written in Python (cloudflare, datadome bypass & solver)

Language: Python - Size: 107 KB - Last synced at: about 1 month ago - Pushed at: 4 months ago - Stars: 22 - Forks: 0

dizaraj/ali-grabber

A lightweight and convenient Chrome extension that adds a sidebar to your browser, allowing you to easily download all product images, videos, and description images from any AliExpress product page with a single click. Visit: https://dizaraj.github.io/ali-grabber/

Language: CSS - Size: 482 KB - Last synced at: 2 months ago - Pushed at: 2 months ago - Stars: 0 - Forks: 0

omkarcloud/web-scraping-template

🚀 THIS WEB SCRAPING TEMPLATE PROVIDES YOU WITH A GREAT STARTING POINT WHEN CREATING WEB SCRAPING BOTS. 🤖

Language: Python - Size: 104 KB - Last synced at: 11 days ago - Pushed at: over 2 years ago - Stars: 8 - Forks: 4

yubunus/Ebay-View-Bot

A professional Python tool for generating organic views on eBay listings using rotating proxies and user agents to simulate legitimate traffic patterns.

Language: Python - Size: 13.7 KB - Last synced at: 4 months ago - Pushed at: 4 months ago - Stars: 0 - Forks: 0

ispras/web-scraper-chrome-extension

Web data extraction tool implemented as chrome extension

Language: JavaScript - Size: 4.72 MB - Last synced at: 4 months ago - Pushed at: 4 months ago - Stars: 259 - Forks: 72

lspahija/torchestrator

Spin up Tor containers and then proxy HTTP requests via these Tor instances

Language: Kotlin - Size: 77.1 KB - Last synced at: 3 months ago - Pushed at: over 4 years ago - Stars: 44 - Forks: 8

pim97/scrappey.js

Scrappey.js: A versatile JavaScript wrapper for Scrappey API for solving Cloudflare, datadome, enabling seamless web scraping of anti-bot protected websites. Simplify data extraction with robust functionality and reliable results. Unlock valuable insights effortlessly. Get started with Scrappey

Language: JavaScript - Size: 137 KB - Last synced at: 4 months ago - Pushed at: 4 months ago - Stars: 10 - Forks: 4

DZ-ABDLHAKIM/idealista-scraper-api

Extracts structured property data (price, features, contact info, images) from any Idealista.com/.pt/.it listing URL. Outputs clean JSON for real estate analysis, market research, and automation workflows. Handles anti-bot protections automatically.

Size: 116 KB - Last synced at: 4 months ago - Pushed at: 4 months ago - Stars: 1 - Forks: 0

Marcel0024/CocoCrawler

An declarative and easy to use web crawler and scraper in C#

Language: C# - Size: 83 KB - Last synced at: about 1 month ago - Pushed at: about 1 year ago - Stars: 28 - Forks: 4

Joabutt/ScrapeDogg

GPT4 Assisted Web Scraping Library

Language: JavaScript - Size: 8.79 KB - Last synced at: 28 days ago - Pushed at: almost 2 years ago - Stars: 16 - Forks: 0

LukeRenton/wsctools

Collection of helper functions designed to facilitate efficient web scraping in python

Language: Python - Size: 14.6 KB - Last synced at: 29 days ago - Pushed at: 9 months ago - Stars: 2 - Forks: 0

Ekans122/Aliexpress-scraper-without-api-free

Aliexpress products scraping.

Language: Python - Size: 75.2 KB - Last synced at: 5 months ago - Pushed at: 6 months ago - Stars: 6 - Forks: 3

franckferman/Scraping-Deputes-France

Script pour scraper les député·e·s français (Nom, Région, Email, Groupe, Circonscription) depuis le site de l'Assemblée nationale.

Language: Python - Size: 3.71 MB - Last synced at: 6 months ago - Pushed at: 6 months ago - Stars: 2 - Forks: 0

fernandod1/ProductHunt-scraper

Producthunt.com famous website scraper script. Scrap all offers and save in spreadsheet excel file.

Language: Python - Size: 9.77 KB - Last synced at: 5 months ago - Pushed at: over 3 years ago - Stars: 26 - Forks: 9

ScrapingAnt/zoominfo_scraper

Zoominfo scraper with using of rotating proxies and headless Chrome from ScrapingAnt

Language: Python - Size: 7.81 KB - Last synced at: 5 months ago - Pushed at: over 4 years ago - Stars: 33 - Forks: 9

fernandod1/Instagram-downloader

Instagram user's photos and videos downloader. Download all media files from any username. Working 2022!

Language: Python - Size: 14.6 KB - Last synced at: 6 months ago - Pushed at: over 3 years ago - Stars: 72 - Forks: 16

ScrapingAnt/alibaba_scraper

Alibaba scraper with using of rotating proxies and headless Chrome from ScrapingAnt

Language: Python - Size: 152 KB - Last synced at: 3 months ago - Pushed at: over 1 year ago - Stars: 17 - Forks: 3

solve-cloudflare/cloudflare-bypass

Size: 2.93 KB - Last synced at: 7 months ago - Pushed at: about 1 year ago - Stars: 2 - Forks: 1

JonusNattapong/Crewzombitx64

This project was inspired by the unclecode/crawl4ai repository. It provided valuable insights and ideas that helped shape the development of Crewzombitx64.

Language: Python - Size: 682 KB - Last synced at: 8 months ago - Pushed at: 8 months ago - Stars: 16 - Forks: 0

harismuneer/Android-Apps-Downloader

📱 A utility for downloading Android apps from the Google Play Store and Xiaomi App Store (the Chinese App Store).

Language: Python - Size: 2.82 MB - Last synced at: 6 months ago - Pushed at: 9 months ago - Stars: 33 - Forks: 19

lordpaoloo/Poder

Poder is A powerful tool to collect publicly available data from social media platforms such as Facebook, YouTube, and Instagram. It organizes the extracted information, including names, phone numbers, emails, and links, into an easy-to-manage Excel sheet. Features include AI chat integration, multi-platform data collection, and real-time insights.

Language: Python - Size: 368 KB - Last synced at: 9 months ago - Pushed at: 9 months ago - Stars: 1 - Forks: 0

luminati-io/aiohttp-web-scraping

Asynchronously scrape websites in Python using AIOHTTP with step-by-step guides and advanced techniques.

Size: 176 KB - Last synced at: 4 months ago - Pushed at: 9 months ago - Stars: 0 - Forks: 0

andytyler/gethtml

Utility for web scraping and fetching the html from a url, using various strategies in a 'waterfall' approach.

Language: TypeScript - Size: 11.3 MB - Last synced at: about 2 months ago - Pushed at: 10 months ago - Stars: 8 - Forks: 0

ace-cooper/gmaps-scraper

A POC for scraping business data from Google Maps

Language: Rust - Size: 7.81 KB - Last synced at: 4 months ago - Pushed at: about 1 year ago - Stars: 0 - Forks: 0

MostafaHima/Speed-Test-Twitter-Bot

A project that uses Selenium to test internet speed and automatically posts the results on Twitter.

Language: Python - Size: 5.86 KB - Last synced at: 7 months ago - Pushed at: 11 months ago - Stars: 1 - Forks: 0

danielzlatanov/youtube-watch-later-scraper

🕸️ scrape your youtube watch later playlist in seconds

Language: JavaScript - Size: 47.9 KB - Last synced at: 25 days ago - Pushed at: 11 months ago - Stars: 0 - Forks: 0

rija/ghost-ssg

A Docker-based pipeline to publish the content of a local Ghost 4 server as static pages.

Language: Shell - Size: 57.6 KB - Last synced at: about 2 months ago - Pushed at: over 2 years ago - Stars: 7 - Forks: 0

genius-codes/telegram-scraper-tool

FREE TOOL FOR EVERYONE

Language: Python - Size: 38.1 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 21 - Forks: 10

Assadzy/Selenium_walmart

Download and save Walmart product reviews to excel sheet

Language: Python - Size: 5.86 KB - Last synced at: over 1 year ago - Pushed at: almost 5 years ago - Stars: 0 - Forks: 0

scrape-do/python-sample

Best Rotating Proxy & Scraping API Alternative. Python Example.

Language: Python - Size: 120 KB - Last synced at: 9 months ago - Pushed at: about 2 years ago - Stars: 1 - Forks: 0

ayushsoni1010/portfoliogram

⚡️Elevate your portfolio analysis with our cutting-edge web scraping tool. Uncover valuable insights about individuals, their skills, and social profiles effortlessly.

Language: JavaScript - Size: 785 KB - Last synced at: over 1 year ago - Pushed at: about 2 years ago - Stars: 3 - Forks: 0

scrape-do/dotnet-example

Best Rotating Proxy & Scraping API Alternative. C# Example.

Language: C# - Size: 1000 Bytes - Last synced at: about 2 years ago - Pushed at: about 2 years ago - Stars: 0 - Forks: 0

kawsarlog/AmerisourceBergen

🛠️ Python 🐍 script automates the extraction of product pricing details from the AmerisourceBergen 🌐 website https://abcorder.amerisourcebergen.com By inputting your username, password, and National Drug Code (NDC) codes and the 📜 script navigates the website and retrieves the 💰 Average Wholesale Price (AWP) and Acquisition Cost (AC) 📊 data

Language: Python - Size: 20.5 KB - Last synced at: almost 2 years ago - Pushed at: about 2 years ago - Stars: 0 - Forks: 0

mannasoumya/instapy_pubprofiles

Download Instagram Photos and Videos given Link to Posts

Language: Python - Size: 21.5 KB - Last synced at: 8 months ago - Pushed at: over 2 years ago - Stars: 3 - Forks: 0

patgdut/GoogleMapsScraper

By scraping leads from Google Maps, you can build a database of potential customers who have shown interest in products or services related to your business. This data can be used for targeted marketing campaigns, email outreach, or sales prospecting.

Size: 4.88 KB - Last synced at: about 2 years ago - Pushed at: about 2 years ago - Stars: 0 - Forks: 0

DemonMartin/scrappey-wrapper

An API wrapper for Scrappey.com written in Node.js (cloudflare bypass & solver)

Language: JavaScript - Size: 61.5 KB - Last synced at: about 2 years ago - Pushed at: about 2 years ago - Stars: 6 - Forks: 0

MustakAbsarKhan/DSE_COMPANY_SCRAPER_Python

The DSE Company Scraper is a Python program that extracts data from the Dhaka Stock Exchange website and saves it to an Excel file for analysis.

Language: Python - Size: 54.6 MB - Last synced at: over 2 years ago - Pushed at: over 2 years ago - Stars: 1 - Forks: 0

MaxValue/IsJavascriptWorking 📦

test if your damn browser has JS enabled

Language: HTML - Size: 1000 Bytes - Last synced at: over 1 year ago - Pushed at: over 2 years ago - Stars: 1 - Forks: 1

rifki/web-scraping-job-postings

Web Scraping wirh Node.js - Puppeteer https://blog.rifkilabs.net/web-scraping-dengan-node-js.html

Language: JavaScript - Size: 6.84 KB - Last synced at: over 2 years ago - Pushed at: over 6 years ago - Stars: 2 - Forks: 0

Kareem-Emad/youtube_metadata_scraper

An expansion over the Youtube-8m Dataset to get more data about the videos such likes/views and channel info through scrapping youtube

Language: Python - Size: 9.77 KB - Last synced at: 5 months ago - Pushed at: over 2 years ago - Stars: 1 - Forks: 0

jaeyk/digital_data_collection_workshop

Digital Data Collection Workshop

Language: HTML - Size: 4.86 MB - Last synced at: over 2 years ago - Pushed at: about 5 years ago - Stars: 6 - Forks: 0

floscodes/sitescraper

Scraping Websites in Go!

Language: Go - Size: 26.4 KB - Last synced at: 24 days ago - Pushed at: about 3 years ago - Stars: 0 - Forks: 0

mapmeld/aoc_reply_dataset

Building a dataset of Twitter replies for unsupervised learning / bot-blocking

Language: Python - Size: 29.8 MB - Last synced at: about 2 months ago - Pushed at: over 5 years ago - Stars: 13 - Forks: 3

skvrahul/chegg_dl

Python script to automate the download of textbooks from Chegg

Language: Python - Size: 2.93 KB - Last synced at: over 2 years ago - Pushed at: almost 7 years ago - Stars: 8 - Forks: 5