GitHub topics: web-crawling

Repositories

sinanazem/web-crawling-jobinja

This project is a web scraper designed to extract information from the [Jobinja](https://jobinja.ir) website, a popular job listing website similar to Glassdoor and Indeed.

Language: Jupyter Notebook - Size: 27.3 KB - Last synced at: about 6 hours ago - Pushed at: about 7 hours ago - Stars: 1 - Forks: 0

omkarcloud/botasaurus

The All in One Framework to Build Undefeatable Scrapers

Language: Python - Size: 62.5 MB - Last synced at: about 6 hours ago - Pushed at: about 22 hours ago - Stars: 1,877 - Forks: 164

crawler-scripts are a collection of lightweight scripts designed to automate web data extraction. These scripts support various websites and allow users to gather information efficiently without manual effort.

Language: Python - Size: 1.76 MB - Last synced at: about 21 hours ago - Pushed at: about 22 hours ago - Stars: 0 - Forks: 0

pinkpixel-dev/deep-research-mcp

The Deep Research MCP Server is a Model Context Protocol (MCP) compliant server designed for comprehensive web research. It uses Tavily's Search and Crawl APIs to gather detailed information on a given topic, then structures this data in a format perfect for LLMs to create high-quality markdown documents.

Language: JavaScript - Size: 0 Bytes - Last synced at: 2 days ago - Pushed at: 2 days ago - Stars: 1 - Forks: 0

serpapi/clauneck

A tool for scraping emails, social media accounts, and much more information from websites using Google Search Results.

Language: Ruby - Size: 34.2 KB - Last synced at: about 23 hours ago - Pushed at: about 1 year ago - Stars: 178 - Forks: 11

apify/crawlee-python

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

Language: Python - Size: 27.4 MB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 5,616 - Forks: 379

luminati-io/brightdata-mcp

A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.

Language: JavaScript - Size: 63.8 MB - Last synced at: 6 days ago - Pushed at: 6 days ago - Stars: 40 - Forks: 4

apify/crawlee

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Language: TypeScript - Size: 140 MB - Last synced at: 7 days ago - Pushed at: 7 days ago - Stars: 17,594 - Forks: 810

omkarcloud/botasaurus-starter

🚀 OFFICIAL STARTER TEMPLATE FOR BOTASAURUS SCRAPING FRAMEWORK 🤖

Language: TypeScript - Size: 397 KB - Last synced at: about 6 hours ago - Pushed at: 8 days ago - Stars: 25 - Forks: 8

18520339/web-scraping-with-scrapy

Python web scraping with Scrapy

Language: Python - Size: 479 KB - Last synced at: 3 days ago - Pushed at: over 4 years ago - Stars: 3 - Forks: 0

frederik-uni/docker-cloudflare-bypasser

A simple api that runs within a container that returns user-agent and cookies

Language: Python - Size: 5.86 KB - Last synced at: 14 days ago - Pushed at: 14 days ago - Stars: 0 - Forks: 0

alanindra/news-enricher

Extracts news article metadata (title, content, date, journalist, entities) from provided URLs.

Language: Python - Size: 31.3 KB - Last synced at: 14 days ago - Pushed at: 14 days ago - Stars: 0 - Forks: 0

scrapeway/best-web-scraping-api-benchmarks

what is the best web scraping API service? Research through benchmarks

Size: 134 KB - Last synced at: 15 days ago - Pushed at: 16 days ago - Stars: 1 - Forks: 0

crwlrsoft/crawler

Library for Rapid (Web) Crawler and Scraper Development

Language: PHP - Size: 1.17 MB - Last synced at: 19 days ago - Pushed at: 19 days ago - Stars: 360 - Forks: 13

ilarionkuleshov/fastcrawl

Fast and asynchronous web crawling and scraping library for Python.

Language: Python - Size: 162 KB - Last synced at: 10 days ago - Pushed at: 13 days ago - Stars: 0 - Forks: 0

jgujerry/python-frameworks

Another curated list of Python frameworks

Language: Python - Size: 13 MB - Last synced at: 24 days ago - Pushed at: 25 days ago - Stars: 61 - Forks: 4

spyboy-productions/omnisci3nt

Omnisci3nt – See What They’ve Tried to Hide Extract deep intelligence from any domain. From subdomains to SSL certs, archived secrets to exposed ports — Omnisci3nt gives you the full picture in seconds.

Language: Python - Size: 8.21 MB - Last synced at: 27 days ago - Pushed at: 27 days ago - Stars: 235 - Forks: 38

omkarcloud/selenium-2captcha-recaptcha-solver-demo

🚀 FINAL CODE FOR TUTORIAL ON HOW TO SOLVE CAPTCHA IN SELENIUM USING 2CAPTCHA 🤖

Language: Python - Size: 5.86 KB - Last synced at: about 6 hours ago - Pushed at: almost 2 years ago - Stars: 6 - Forks: 2

maxmindlin/scout-lang

A web crawling programming language

Language: Rust - Size: 54.4 MB - Last synced at: 6 days ago - Pushed at: 9 months ago - Stars: 112 - Forks: 6

MaxValue/Terpene-Profile-Parser-for-Cannabis-Strains

Parser and database to index the terpene profile of different strains of Cannabis from online databases

Language: Python - Size: 21.4 MB - Last synced at: 20 days ago - Pushed at: about 2 years ago - Stars: 118 - Forks: 18

N4rr34n6/MetadataHarvester

MetadataHarvester is an advanced file metadata extraction tool designed for cybersecurity professionals, researchers, and analysts.

Language: Python - Size: 21.5 KB - Last synced at: 28 days ago - Pushed at: 28 days ago - Stars: 0 - Forks: 1

spyboy-productions/PhantomCrawler

Boost website hits by generating requests from multiple proxy IPs.

Language: Python - Size: 1.48 MB - Last synced at: 22 days ago - Pushed at: about 1 year ago - Stars: 64 - Forks: 9

godkingjay/selenium-twitter-scraper

This is a Twitter Scraper which uses Selenium for scraping tweets. It is capable of scraping tweets from home, user profile, hashtag, query or search, and advanced searches.

Language: Jupyter Notebook - Size: 160 KB - Last synced at: 30 days ago - Pushed at: about 1 month ago - Stars: 246 - Forks: 63

TurnerSoftware/InfinityCrawler

A simple but powerful web crawler library for .NET

Language: C# - Size: 326 KB - Last synced at: 12 days ago - Pushed at: over 1 year ago - Stars: 251 - Forks: 37

cxcscmu/Craw4LLM

Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"

Language: Python - Size: 79.1 KB - Last synced at: 30 days ago - Pushed at: 3 months ago - Stars: 608 - Forks: 56

thesp0nge/nightcrawler-mitm

A python program that crawls a website and tries to stress it, polluting forms with bogus data

Language: Python - Size: 247 KB - Last synced at: 12 days ago - Pushed at: about 1 month ago - Stars: 13 - Forks: 1

HRN-Projects/amazon-captcha-solver

A TensorFlow (Deep Learning - CNN) based solution for tackling captcha when collecting data from Amazon.

Language: Python - Size: 35.9 MB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 29 - Forks: 13

Fern-Aerell/Web-Crawling-To-TXT

A simple web crawling application that can browse URLs, extract text content, and save the results in TXT format.

Language: Python - Size: 315 KB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 4 - Forks: 0

ayakashi-io/ayakashi

:zap: Ayakashi.io - The next generation web scraping framework

Language: TypeScript - Size: 1.24 MB - Last synced at: about 2 hours ago - Pushed at: almost 2 years ago - Stars: 213 - Forks: 8

jrbadiabo/Bet-on-Sibyl

Machine Learning Model for Sport Predictions (Football, Basketball, Baseball, Hockey, Soccer & Tennis)

Language: Jupyter Notebook - Size: 17 MB - Last synced at: 29 days ago - Pushed at: about 8 years ago - Stars: 263 - Forks: 94

scrapinghub/scrapy-training

Scrapy Training companion code

Language: Python - Size: 103 KB - Last synced at: 17 days ago - Pushed at: over 6 years ago - Stars: 174 - Forks: 45

ScrapingAnt/amazon_scraper

Amazon products scraper with using of rotating proxies and headless Chrome from ScrapingAnt

Language: JavaScript - Size: 52.7 KB - Last synced at: 4 days ago - Pushed at: about 1 year ago - Stars: 81 - Forks: 19

supergillis/crawler-ts

Crawler written in TypeScript using ES6 generators.

Language: TypeScript - Size: 60.5 KB - Last synced at: 9 days ago - Pushed at: about 4 years ago - Stars: 12 - Forks: 1

omkarcloud/omkar-temp-mail

🚀 OMKAR TEMP MAIL HELPS YOU USE TEMPORARY EMAILS. 🤖

Language: Python - Size: 15.6 KB - Last synced at: about 6 hours ago - Pushed at: about 1 year ago - Stars: 13 - Forks: 4

lekhmanrus/real-shot-pdf

RealShotPDF is a Chrome extension designed to simplify the process of creating PDF documents from web content. The extension allows users to navigate through selected webpages, parse and display links in a tree view, and generate PDFs for the chosen pages. It operates locally without sending any data to external servers.

Language: TypeScript - Size: 406 KB - Last synced at: about 1 month ago - Pushed at: about 1 year ago - Stars: 6 - Forks: 1

Solrikk/DataDigger

DataDigger is a powerful and intuitive web application designed to extract and analyze data from web pages.

Language: Go - Size: 38.1 KB - Last synced at: 27 days ago - Pushed at: 2 months ago - Stars: 4 - Forks: 0

joe-stifler/crawler

Crawler is a Python package that crawls web pages and converts their content into Markdown format, making it easy to create documentation, notes, or other text-based representations. It features domain restrictions, flexible output options, and graph visualization.

Language: Python - Size: 283 KB - Last synced at: 23 days ago - Pushed at: 11 months ago - Stars: 7 - Forks: 1

Mirtia/Inappropriate-YouTube 📦

This repository contains the scripts used to obtain channel YouTube features and analyze potential disturbing channels for the publication "YouTubers Not madeForKids: Detecting Channels Sharing Inappropriate Videos Targeting Children".

Language: Python - Size: 9.04 MB - Last synced at: 6 days ago - Pushed at: about 2 years ago - Stars: 3 - Forks: 0

jonasjacek/robots.txt

Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.

Size: 135 KB - Last synced at: 3 months ago - Pushed at: 3 months ago - Stars: 87 - Forks: 38

oxylabs/pricing-data-collection-from-ecommerce-stores

Appache Airflow DAGs for e-commerce pricing collection.

Language: Python - Size: 16.6 KB - Last synced at: 2 months ago - Pushed at: 3 months ago - Stars: 1 - Forks: 0

osamikoyo/Geass

web crawler for you, with some api function and configuration

Language: Go - Size: 32.8 MB - Last synced at: about 2 months ago - Pushed at: 3 months ago - Stars: 2 - Forks: 0

JeffBla/rent-house-crawler

This is a distributed web crawler using Scrapy, Redis, and Selenium. It is designed to handle various types of websites, including static, AJAX, and dynamic pages.

Language: Python - Size: 307 KB - Last synced at: about 1 month ago - Pushed at: 8 months ago - Stars: 0 - Forks: 0

KadekM/scrawler

Scala web crawling and scraping using fs2 streams

Language: HTML - Size: 92.8 KB - Last synced at: 6 days ago - Pushed at: over 7 years ago - Stars: 16 - Forks: 3

MohamedHmini/tweetsOLAPing

implementing an end-to-end tweets ETL/Analysis pipeline.

Language: Python - Size: 5.99 MB - Last synced at: 23 days ago - Pushed at: over 2 years ago - Stars: 57 - Forks: 6

crwlrsoft/robots-txt

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

Language: PHP - Size: 32.2 KB - Last synced at: about 1 month ago - Pushed at: 3 months ago - Stars: 10 - Forks: 2

Adityasinghvats/web-crawler

This is a project which mimics the web crawlers used in large browser engines like Chromium and Gecko.

Language: JavaScript - Size: 48.8 KB - Last synced at: about 1 month ago - Pushed at: 4 months ago - Stars: 0 - Forks: 0

lewisakura/spiderboi

A web crawling library written in TypeScript.

Language: TypeScript - Size: 376 KB - Last synced at: 30 days ago - Pushed at: over 2 years ago - Stars: 7 - Forks: 1

IAmFarrokhnejad/Murkmaw

A web crawler using Rust.

Language: Rust - Size: 67.7 MB - Last synced at: about 1 month ago - Pushed at: 6 months ago - Stars: 1 - Forks: 0

MartinCastroAlvarez/unlam-android-app

Calendar Android native app.

Language: Java - Size: 3.75 MB - Last synced at: about 1 month ago - Pushed at: about 6 years ago - Stars: 0 - Forks: 0

dstark5/gnews-scraper

GNewsScraper is a TypeScript package that scrapes article data from Google News based on a keyword or phrase. It returns the results as an array of JSON objects, making it convenient to access and use the scraped information

Language: TypeScript - Size: 153 KB - Last synced at: 22 days ago - Pushed at: over 1 year ago - Stars: 10 - Forks: 3

AmirEspahbodi/google-maps-scraper

google map scraper. extract title, phone, address, latitude and longitude, category, website URL, rating, reviews number, email, active_hours, reviews and first picture of listing

Language: Python - Size: 243 KB - Last synced at: 28 days ago - Pushed at: 5 months ago - Stars: 0 - Forks: 0

shokoofa-ghods/Web-Crawling_Text-Proccessing

simple text processing program which crawls imdb and extracts keywords with TextRank algorithm and crawls Digikala special offers and extracts some feature and shows them on web using Django framework

Language: Python - Size: 1.42 MB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 1 - Forks: 0

crwlrsoft/laravel-crawler

Laravel adapter for the crwlr/crawler package.

Language: PHP - Size: 8.79 KB - Last synced at: about 1 month ago - Pushed at: over 1 year ago - Stars: 4 - Forks: 0

mike-gee/webtranspose

Web scraping API for building AI applications.

Language: Python - Size: 1.43 MB - Last synced at: 13 days ago - Pushed at: over 1 year ago - Stars: 41 - Forks: 2

miroshnikov/scrapyteer

Web crawling & scraping framework for Node.js on top of headless Chrome browser

Language: TypeScript - Size: 384 KB - Last synced at: about 1 month ago - Pushed at: about 1 year ago - Stars: 19 - Forks: 0

ruiiary/commercelaw-words

24-2 콘텐츠프로그래밍기초 기말 프로젝트 개인 과제

Language: Jupyter Notebook - Size: 1010 KB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 0 - Forks: 0

msoleimani96/minecraft-skin-scraper

Scraping Minecraft skins from two sources.

Language: Python - Size: 43 KB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 1 - Forks: 0

ahmed-alnassif/net-spider

Net-Spider is a web scraping tool designed to retrieve the source code for a web page, including front-end elements such as JavaScript, CSS, images, and fonts. It allows you to crawl and download the source code from a target website.

Language: Python - Size: 2.65 MB - Last synced at: 28 days ago - Pushed at: 11 months ago - Stars: 5 - Forks: 1

9dl/RobotsSniffer

Tool to analyze and parse website robots.txt for crawler rules.

Language: C# - Size: 48.8 KB - Last synced at: about 1 month ago - Pushed at: 6 months ago - Stars: 0 - Forks: 0

islamhafez0/web-crawler

Language: Python - Size: 1.11 MB - Last synced at: 30 days ago - Pushed at: 6 months ago - Stars: 1 - Forks: 0

simonpierreboucher/Crawler

A robust, modular web crawler built in Python for extracting and saving content from websites. This crawler is specifically designed to extract text content from both HTML and PDF files, saving them in a structured format with metadata.

Language: Python - Size: 87.9 KB - Last synced at: about 1 month ago - Pushed at: 6 months ago - Stars: 0 - Forks: 0

GoTrained/Scrapy-Craigslist

Web Scraping Craigslist's Engineering Jobs in NY with Scrapy

Language: Python - Size: 195 KB - Last synced at: 4 months ago - Pushed at: almost 8 years ago - Stars: 66 - Forks: 37

kunalPisolkar24/IR_Lab

Collection of practical codes for Savitribai Phule Pune University's Information Retrieval Lab (410247) .

Language: Jupyter Notebook - Size: 125 KB - Last synced at: 2 months ago - Pushed at: 7 months ago - Stars: 0 - Forks: 0

zytedata/spidyquotes

Example site for web scraping tutorials

Language: Julia - Size: 229 KB - Last synced at: 2 months ago - Pushed at: 7 months ago - Stars: 31 - Forks: 18

SuperBruceJia/dynamic-web-crawlering-python

This repo is mainly for dynamic web (Ajax Tech) crawling using Python, taking China's NSTL websites as an example.

Language: Python - Size: 12.8 MB - Last synced at: 21 days ago - Pushed at: almost 2 years ago - Stars: 16 - Forks: 3

ScrapingAnt/zoominfo_scraper

Zoominfo scraper with using of rotating proxies and headless Chrome from ScrapingAnt

Language: Python - Size: 7.81 KB - Last synced at: about 1 month ago - Pushed at: about 4 years ago - Stars: 32 - Forks: 9

SpeedyShot/capture

An easy-to-use library for the SpeedyShot Capture service.

Language: TypeScript - Size: 548 KB - Last synced at: 10 days ago - Pushed at: 5 months ago - Stars: 1 - Forks: 0

jonathandunn/common_crawl_corpus

Scripts for building a geo-located web corpus using Common Crawl data

Language: Python - Size: 323 KB - Last synced at: 22 days ago - Pushed at: 22 days ago - Stars: 11 - Forks: 0

breck7/crawlers

Crawlers for extracting measurements from the web for Scroll datasets

Language: JavaScript - Size: 140 KB - Last synced at: 8 months ago - Pushed at: 8 months ago - Stars: 4 - Forks: 0

chihiroanihr/COMP479-P4_F2022

Experiments with web crawling, scraping, and indexing a collection of web documents. Clustering the indexed data with k-means algorithm. Each resulting cluster is assigned a sentiment score using AFINN - a sentiment analysis script.

Language: Python - Size: 17.6 MB - Last synced at: 6 months ago - Pushed at: 8 months ago - Stars: 0 - Forks: 0

my8100/scrapyd-cluster-on-heroku

Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO :point_right:

Language: Python - Size: 236 KB - Last synced at: about 1 month ago - Pushed at: about 5 years ago - Stars: 122 - Forks: 88

Joeri-Abbo/python-credly-scraper

This project is a set of Python scripts designed to crawl and extract data from the Credly platform, focusing on skills, organizations, and badges. The scripts allow users to perform searches using command-line arguments, predefined search terms, or skills listed in a JSON file. The collected data is then saved to JSON files for further analysis an

Language: Python - Size: 63.7 MB - Last synced at: 5 months ago - Pushed at: 6 months ago - Stars: 1 - Forks: 2

kluhan/kraken

Kraken is a generic, mid-scale web crawler specifically built to crawl vertical data-sources, like Youtube or the Google Play Store.

Language: Python - Size: 92.8 KB - Last synced at: about 1 month ago - Pushed at: about 2 years ago - Stars: 2 - Forks: 0

0MeMo07/Web-Crawler

Web Crawler with Python

Language: Python - Size: 8.79 KB - Last synced at: 18 days ago - Pushed at: over 1 year ago - Stars: 7 - Forks: 0

LikithMeruvu/Framework-Docs-AI

Framework Docs AI is a powerful SaaS solution for managing framework documentation. It automatically scrapes documentation, builds a comprehensive knowledge base, and uses advanced language models to provide accurate responses to user queries. Enhance productivity and streamline your documentation process with Framework Docs AI.

Language: Python - Size: 62 MB - Last synced at: 9 months ago - Pushed at: 9 months ago - Stars: 1 - Forks: 0