lorien/awesome-web-scraping

List of libraries, tools and APIs for web scraping and data processing.

Language: Makefile - Size: 473 KB - Last synced at: 11 days ago - Pushed at: 5 months ago - Stars: 7,016 - Forks: 809

omkarcloud/botasaurus

The All in One Framework to Build Undefeatable Scrapers

Language: Python - Size: 62.9 MB - Last synced at: 9 days ago - Pushed at: 16 days ago - Stars: 1,946 - Forks: 165

zhuyingda/webster

a reliable high-level web crawling & scraping framework for Node.js.

Language: JavaScript - Size: 181 KB - Last synced at: 3 days ago - Pushed at: 3 months ago - Stars: 527 - Forks: 57

kennethreitz/requests-html Fork of psf/requests-html

Pythonic HTML Parsing for Humans™

Language: Python - Size: 2.7 MB - Last synced at: 23 days ago - Pushed at: 12 months ago - Stars: 320 - Forks: 42

alephdata/memorious

Lightweight web scraping toolkit for documents and structured data.

Language: Python - Size: 1.39 MB - Last synced at: 16 days ago - Pushed at: over 1 year ago - Stars: 310 - Forks: 62

Ashis678/ProxyCrawl

Size: 23.4 KB - Last synced at: 11 months ago - Pushed at: 11 months ago - Stars: 44 - Forks: 0

flulemon/sneakpeek

Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis

Language: Python - Size: 19.7 MB - Last synced at: 15 days ago - Pushed at: almost 2 years ago - Stars: 37 - Forks: 0

peterbencze/serritor

Serritor is an open source web crawler framework built upon Selenium and written in Java. It can be used to crawl dynamic web pages that require JavaScript to render data.

Language: Java - Size: 969 KB - Last synced at: about 2 months ago - Pushed at: almost 3 years ago - Stars: 32 - Forks: 15

omkarcloud/botasaurus-starter

🚀 OFFICIAL STARTER TEMPLATE FOR BOTASAURUS SCRAPING FRAMEWORK 🤖

Language: TypeScript - Size: 397 KB - Last synced at: 6 days ago - Pushed at: about 1 month ago - Stars: 25 - Forks: 8

crawlbase/proxycrawl-php 📦

ProxyCrawl PHP library for scraping and crawling websites

Language: PHP - Size: 34.2 KB - Last synced at: 1 day ago - Pushed at: almost 2 years ago - Stars: 21 - Forks: 5

pim97/scrappey-wrapper-python

An API wrapper for Scrappey.com written in Python (cloudflare, datadome bypass & solver)

Language: Python - Size: 98.6 KB - Last synced at: 1 day ago - Pushed at: 12 days ago - Stars: 20 - Forks: 0

omkarcloud/omkar-temp-mail

🚀 OMKAR TEMP MAIL HELPS YOU USE TEMPORARY EMAILS. 🤖

Language: Python - Size: 15.6 KB - Last synced at: 6 days ago - Pushed at: over 1 year ago - Stars: 13 - Forks: 4

rajanrx/php-scrape

A simple, easy to use, scalable scraping framework written in PHP

Language: PHP - Size: 8.8 MB - Last synced at: 6 months ago - Pushed at: over 4 years ago - Stars: 11 - Forks: 4

umihico/minigun-requests

Web scraping API to outsource tons of GET & xpath to cloud computing

Language: Python - Size: 392 KB - Last synced at: 6 days ago - Pushed at: about 1 year ago - Stars: 7 - Forks: 0

omkarcloud/web-scraping-template

🚀 THIS WEB SCRAPING TEMPLATE PROVIDES YOU WITH A GREAT STARTING POINT WHEN CREATING WEB SCRAPING BOTS. 🤖

Language: Python - Size: 104 KB - Last synced at: 6 days ago - Pushed at: almost 2 years ago - Stars: 7 - Forks: 3

DemonMartin/scrappey-wrapper

An API wrapper for Scrappey.com written in Node.js (cloudflare bypass & solver)

Language: JavaScript - Size: 61.5 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 6 - Forks: 0

omkarcloud/selenium-2captcha-recaptcha-solver-demo

🚀 FINAL CODE FOR TUTORIAL ON HOW TO SOLVE CAPTCHA IN SELENIUM USING 2CAPTCHA 🤖

Language: Python - Size: 5.86 KB - Last synced at: 6 days ago - Pushed at: almost 2 years ago - Stars: 6 - Forks: 2

JimmyLaurent/node-crawling-framework

✨ NodeJs crawling & scraping framework heavily inspired by Scrapy

Language: JavaScript - Size: 227 KB - Last synced at: 11 days ago - Pushed at: almost 7 years ago - Stars: 4 - Forks: 1

lauramatilda/newsScraping

M.A. Thesis work, news scraping framework/pipeline using python, beautifulsoup, newspaper3k, flask and mongodb with a custom api.

Language: Python - Size: 34.2 KB - Last synced at: 6 days ago - Pushed at: almost 7 years ago - Stars: 3 - Forks: 0

omkarcloud/dentalkart-scraper

🚀 SCRAPE 1000'S OF PRODUCTS FROM DENTALKART 🤖

Language: Python - Size: 908 KB - Last synced at: 6 days ago - Pushed at: over 1 year ago - Stars: 2 - Forks: 2

LearnCodingEasy/Web_Scraping

Web Scraping

Language: Vue - Size: 165 KB - Last synced at: about 2 months ago - Pushed at: about 2 months ago - Stars: 1 - Forks: 0

datagrind-io/amazon-products

Scrape Amazon products

Language: Python - Size: 3.91 KB - Last synced at: 9 months ago - Pushed at: 9 months ago - Stars: 1 - Forks: 0

mr-mudgal/Amazon-Scrapper

This Python-based Amazon Scraper is designed to efficiently extract detailed product data from Amazon's product pages. The tool leverages powerful libraries like BeautifulSoup4 and csv, along with the Scrapingant API to simulate browser behavior and bypass Amazon’s anti-scraping algorithms.

Language: Python - Size: 17.6 KB - Last synced at: 2 months ago - Pushed at: 5 months ago - Stars: 0 - Forks: 0

tor4z/crabs

Scraping framework for Python

Language: Python - Size: 77.1 KB - Last synced at: over 2 years ago - Pushed at: over 2 years ago - Stars: 0 - Forks: 0