lucky521/pyspider

My Web Spider

Language: Python - Size: 263 KB - Last synced: about 1 month ago - Pushed: almost 7 years ago - Stars: 0 - Forks: 0

platonai/PulsarRPAPro

Professional edition; AI for auto extraction; Web UI; examples.

Language: Kotlin - Size: 14.4 MB - Last synced: about 1 month ago - Pushed: about 1 month ago - Stars: 76 - Forks: 17

lesterrry/campfire

Shock-drop watching utility

Language: Python - Size: 146 KB - Last synced: about 2 months ago - Pushed: 8 months ago - Stars: 0 - Forks: 0

mawippel/drizly-crawler

Beers' characteristics crawler for drizly.com. Written in Python.

Language: Python - Size: 26.4 KB - Last synced: about 2 months ago - Pushed: almost 4 years ago - Stars: 1 - Forks: 1

let4be/crusty-core

A small library for building fast and highly customizable web crawlers

Language: Rust - Size: 436 KB - Last synced: 21 days ago - Pushed: over 1 year ago - Stars: 15 - Forks: 1

dotandimet/Mojo-UserAgent-Role-Queued

A role for Mojo::UserAgent that processes non-blocking requests in a rate-limiting queue.

Language: Perl - Size: 65.4 KB - Last synced: about 2 months ago - Pushed: almost 5 years ago - Stars: 3 - Forks: 2

AmeyRuikar/webCrawler

A multi-threaded web crawler using crawler4j.

Language: Java - Size: 13.7 KB - Last synced: about 2 months ago - Pushed: over 7 years ago - Stars: 0 - Forks: 0

gr1d99/scripts

Random python scripts

Language: Python - Size: 684 KB - Last synced: about 2 months ago - Pushed: over 6 years ago - Stars: 0 - Forks: 0

remram44/crawler-structure 📦

Basic Twisted structure for web crawling (doesn't actually crawl right now)

Language: Python - Size: 121 KB - Last synced: about 2 months ago - Pushed: about 9 years ago - Stars: 0 - Forks: 0

guvenonur/airtime

IMDB airtime crawler and notifier for TV shows

Language: Python - Size: 13.9 MB - Last synced: about 2 months ago - Pushed: 11 months ago - Stars: 2 - Forks: 1

marcofavorito/simple-web-crawler

A very simple web crawler.

Language: Python - Size: 7.81 KB - Last synced: about 2 months ago - Pushed: almost 6 years ago - Stars: 0 - Forks: 2

apify/crawlee

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Language: TypeScript - Size: 117 MB - Last synced: about 2 months ago - Pushed: about 2 months ago - Stars: 11,973 - Forks: 501

danielzlatanov/estate-fetch

Real estate search engine

Language: HTML - Size: 1.6 MB - Last synced: 2 months ago - Pushed: 2 months ago - Stars: 0 - Forks: 1

qfcy/Python

This repository contains the python source code, containing more than 40 python projects, involving many fields.仓库用于储存python源代码, 包含40多个python项目，涉及爬虫、算法、OpenGL、tkinter、面向对象编程等多个领域。

Language: Python - Size: 26.7 MB - Last synced: 2 months ago - Pushed: 2 months ago - Stars: 3 - Forks: 4

rzo1/crawler4j Fork of yasserg/crawler4j

Open Source Web Crawler for Java - A maintained fork of yasserg/crawler4j

Language: Java - Size: 1.9 MB - Last synced: about 1 month ago - Pushed: about 2 months ago - Stars: 22 - Forks: 4

kan01234/ur-web-spider

web spider to scan UR avialbe room and output as csv

Language: Python - Size: 37 MB - Last synced: about 2 months ago - Pushed: about 2 months ago - Stars: 4 - Forks: 1

Cirice/Krawler 📦

A complete multi-threaded web-crawler in Python3

Language: Python - Size: 58.6 KB - Last synced: 3 months ago - Pushed: over 5 years ago - Stars: 12 - Forks: 5

Misterhex/WebCrawler

Just a simple web crawler which return crawled links as IObservable using reactive extension and async await.

Language: C# - Size: 7.13 MB - Last synced: about 2 months ago - Pushed: almost 5 years ago - Stars: 60 - Forks: 33

Amrita-TIFAC-Cyber-Blockchain/MeRiT

Hyperledger Challenge 2022 : Media Tracking Platform to Tackle Online Piracy

Language: PHP - Size: 9.68 MB - Last synced: about 2 months ago - Pushed: 11 months ago - Stars: 1 - Forks: 2

devopsgroup-io/siteshooter

:camera: Automate full website screenshots and PDF generation with multiple viewport support.

Language: JavaScript - Size: 496 KB - Last synced: about 2 months ago - Pushed: about 5 years ago - Stars: 67 - Forks: 13

wyu-du/StockForecast

:dart: predict the price trend of individual stocks using deep learning and natural language processing

Language: Python - Size: 29.4 MB - Last synced: about 2 months ago - Pushed: over 6 years ago - Stars: 74 - Forks: 34

idkidkidkidkidkidkidkidk/gics-sentry-bot

尋找資安女婕思初賽PaGamO哨兵機器人

Language: Python - Size: 2.62 MB - Last synced: about 2 months ago - Pushed: about 2 months ago - Stars: 5 - Forks: 0

USCDataScience/sparkler

Spark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.

Language: Java - Size: 23.1 MB - Last synced: about 2 months ago - Pushed: about 1 year ago - Stars: 409 - Forks: 142

followthefourleafedclover/WebCrawler

A simple WebCrawler that returns title, keywords, and exports data as JSON. Made from scratch

Language: Python - Size: 60.5 KB - Last synced: about 2 months ago - Pushed: almost 2 years ago - Stars: 2 - Forks: 0

rajagopal28/simple-web-crawler

Java based web-crawler program which makes use of pool based multi-threading, simple UI with Swing and jsoup to nested web crawling

Language: Java - Size: 2.02 MB - Last synced: about 2 months ago - Pushed: over 1 year ago - Stars: 5 - Forks: 2

minseok0809/robotic-process-automation

File Management, School Automation, Text Automation, Web Crawler, Web Automation, Data Preprocessing, Dataframe Editor

Language: Jupyter Notebook - Size: 14.1 MB - Last synced: 2 months ago - Pushed: 2 months ago - Stars: 1 - Forks: 1

nicksherron/proxi 📦

Proxy pool. Finds and checks proxies with rest api for querying results. Can find over 25k proxies in under 5 minutes.

Language: Go - Size: 1.09 MB - Last synced: about 2 months ago - Pushed: about 4 years ago - Stars: 32 - Forks: 4

slickcharmer/zoominfo_scraper

Zoominfo scraper with using of rotating proxies and headless Chrome from ScrapingAnt

Language: Python - Size: 5.86 KB - Last synced: 2 months ago - Pushed: 3 months ago - Stars: 3 - Forks: 0

HopefulHeart2020/zoominfo_scraper

Zoominfo scraper with using of rotating proxies and headless Chrome from ScrapingAnt

Language: Python - Size: 6.84 KB - Last synced: 2 months ago - Pushed: 9 months ago - Stars: 1 - Forks: 0

iamtonmoy0/crawler

web crawler

Language: Go - Size: 1.95 KB - Last synced: 2 months ago - Pushed: over 1 year ago - Stars: 0 - Forks: 0

YGGverse/Yo

Micro Web Crawler in PHP & Manticore

Language: PHP - Size: 152 KB - Last synced: about 1 month ago - Pushed: about 2 months ago - Stars: 3 - Forks: 0

Keep-Current/web-miner

Crawls sites, to find new content and scrap it

Language: Python - Size: 215 KB - Last synced: about 1 month ago - Pushed: about 3 years ago - Stars: 33 - Forks: 29

sarthakagarwal18/AppStore-Crawler

A Web Crawler which extracts the names of different Apps, its Developer, and its Price.

Language: Python - Size: 20.5 KB - Last synced: 2 months ago - Pushed: almost 8 years ago - Stars: 1 - Forks: 0

sarthakagarwal18/CodeForces-Stalker

A simple python script, which displays the handles and the ratings of your friends on Codeforces.

Language: Python - Size: 17.6 KB - Last synced: 2 months ago - Pushed: almost 8 years ago - Stars: 4 - Forks: 0

zahrabayatt/web-crawler-internal-linking

This project is a command line application using Node.js. The purpose of the application is to crawl a website and produce a report of the internal linking structure of the website.

Language: JavaScript - Size: 63.5 KB - Last synced: 2 months ago - Pushed: 2 months ago - Stars: 0 - Forks: 0

siddhant-vij/JS-Web-Crawler

Web crawler that generates an "internal links" report for any website on the internet.

Language: JavaScript - Size: 41 KB - Last synced: 2 months ago - Pushed: 2 months ago - Stars: 0 - Forks: 0

ssssssss-team/spider-flow

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

Language: Java - Size: 3.23 MB - Last synced: 2 months ago - Pushed: 12 months ago - Stars: 8,991 - Forks: 1,718

MaxValue/Terpene-Profile-Parser-for-Cannabis-Strains

Parser and database to index the terpene profile of different strains of Cannabis from online databases

Language: Python - Size: 21.4 MB - Last synced: about 1 month ago - Pushed: about 1 year ago - Stars: 107 - Forks: 20

jacygao/spiderman

A simple web crawler that crawls a website n-links deep and calculate the number of unique rendered words found on each page and in total.

Language: C++ - Size: 18.6 KB - Last synced: 2 months ago - Pushed: over 7 years ago - Stars: 4 - Forks: 0

eneax/web-crawler

A web crawler built in Node.js

Language: JavaScript - Size: 72.3 KB - Last synced: about 1 month ago - Pushed: 2 months ago - Stars: 0 - Forks: 0

brianmadden/krawler

A web crawling framework written in Kotlin

Language: Kotlin - Size: 403 KB - Last synced: about 2 months ago - Pushed: almost 3 years ago - Stars: 130 - Forks: 16

chenfh5/a-simple-book-recommender

Language: Scala - Size: 76.2 KB - Last synced: 3 months ago - Pushed: over 1 year ago - Stars: 0 - Forks: 0

GobHash/Project-walker

web crawler in python

Language: Python - Size: 150 KB - Last synced: 3 months ago - Pushed: over 1 year ago - Stars: 1 - Forks: 1

srijitravi94/Web-Crawler

A crawler bot that crawls wikipedia pages

Language: Python - Size: 30.3 KB - Last synced: 3 months ago - Pushed: over 6 years ago - Stars: 0 - Forks: 0

srijitravi94/Search-Engine

Implementation of different Information Retrieval Systems to evaluate and compare their performance levels in terms of retrieval effectiveness

Language: HTML - Size: 10.7 MB - Last synced: 3 months ago - Pushed: over 6 years ago - Stars: 3 - Forks: 2

anlp-team/LTI_Neural_Navigator

"Enhancing LLM Factual Accuracy with RAG to Counter Hallucinations: A Case Study on Domain-Specific Queries in Private Knowledge-Bases" by Jiarui Li and Ye Yuan and Zehua Zhang

Language: HTML - Size: 32.3 MB - Last synced: 3 months ago - Pushed: 3 months ago - Stars: 7 - Forks: 2

PeterDulworth/company-insights-webapp

written for nesh. utility to pull in insights about different stocks. using a react front-end, and flask backend

Language: JavaScript - Size: 3.16 MB - Last synced: 3 months ago - Pushed: over 1 year ago - Stars: 3 - Forks: 1

yasszu/finagle-web-crawler

Web Crawler of Google Developers Blog with Finagle

Language: Scala - Size: 45.9 KB - Last synced: 3 months ago - Pushed: almost 6 years ago - Stars: 0 - Forks: 0

arminkz/Crawler

Crawler with Scrapy based on Mercator architecture

Language: Python - Size: 15.6 KB - Last synced: 3 months ago - Pushed: over 4 years ago - Stars: 0 - Forks: 0

abaykan/CrawlBox 📦

Easy way to brute-force web directory.

Language: Python - Size: 71.3 KB - Last synced: 2 months ago - Pushed: about 5 years ago - Stars: 137 - Forks: 42

AbssZy/web-crawler

This is a web crawler example with the help of BeautfulSoup4 in Python.

Language: Python - Size: 162 KB - Last synced: 3 months ago - Pushed: over 3 years ago - Stars: 2 - Forks: 0

xianhu/PSpider

简单易用的Python爬虫框架，QQ交流群：597510560

Language: Python - Size: 814 KB - Last synced: 3 months ago - Pushed: almost 2 years ago - Stars: 1,814 - Forks: 508

VIDA-NYU/ache

ACHE is a web crawler for domain-specific search.

Language: Java - Size: 66.6 MB - Last synced: 2 months ago - Pushed: 10 months ago - Stars: 433 - Forks: 133

spencerlepine/readme-crawler

A Node.js web crawler to download README files and follow contained links. Fetch repositories from a valid GitHub URL

Language: JavaScript - Size: 42 KB - Last synced: about 1 month ago - Pushed: almost 3 years ago - Stars: 4 - Forks: 0

fzaca/topic-web-crawler

Web Crawler que permite buscar contenido en una o varias páginas web, utilizando una lista de palabras clave y limitando la profundidad de búsqueda.

Language: Python - Size: 179 KB - Last synced: 3 months ago - Pushed: about 1 year ago - Stars: 2 - Forks: 0

AGLDWG/ldspider Fork of ldspider/ldspider

A crawler for the (Australian) Linked Data web

Language: Java - Size: 64.7 MB - Last synced: 3 months ago - Pushed: about 3 years ago - Stars: 0 - Forks: 0

agrawalnaman/web-crawler-using-Golang

web crawler which fetches URLs and outputs crawl results to console as the crawl proceeds

Language: Go - Size: 41 KB - Last synced: 3 months ago - Pushed: about 4 years ago - Stars: 0 - Forks: 0

sharad1126/WebScrapper

web scrapping with selenium using chrome driver

Language: Python - Size: 217 KB - Last synced: about 1 month ago - Pushed: 12 months ago - Stars: 3 - Forks: 2

siddhant-vij/Concurrent-Web-Crawler

Efficient crawling & data extraction from web pages using concurrency in multiple programming languages.

Language: Java - Size: 407 KB - Last synced: 2 months ago - Pushed: 2 months ago - Stars: 0 - Forks: 0

umangrmehta/nesh-test

A demo project for my recruitment process at Hello Nesh Inc.

Language: Java - Size: 13.7 KB - Last synced: 3 months ago - Pushed: about 5 years ago - Stars: 0 - Forks: 0

mlmhl/gcrawler

A very simple and flexible web crawler, support multiple run modes.

Language: Go - Size: 1.03 MB - Last synced: 3 months ago - Pushed: about 6 years ago - Stars: 0 - Forks: 0

wesleyolvr/shein_crawler

O Crawler Shein é um projeto de Web Scraping automatizado desenvolvido para extrair informações detalhadas de produtos no site Shein.

Language: Python - Size: 45.9 KB - Last synced: about 2 months ago - Pushed: about 2 months ago - Stars: 8 - Forks: 0

browsercat/playwright-automation-starter

Awesome boilerplate for writing browser automations using Playwright, with debugging and tests ready to go.

Language: TypeScript - Size: 29.3 KB - Last synced: 3 months ago - Pushed: 3 months ago - Stars: 0 - Forks: 0

Bazserpz/nstbrowser-automation-library

NSTBrowser is an advanced browser for web scraping and automation, offering proxy management and anti-detect features. Compatible with Puppeteer, Playwright, and Selenium, it excels in multi-accounting and bypassing web protections.

Size: 4.88 KB - Last synced: 3 months ago - Pushed: 3 months ago - Stars: 5 - Forks: 0

cy-zheng/pyCreeper

一个用来快速提取网页内容的信息采集（爬虫）框架， 实现了对网页的动态加载与控制。

Language: Python - Size: 383 KB - Last synced: 3 months ago - Pushed: about 7 years ago - Stars: 28 - Forks: 11

HaiyueZhang/Luyan-Logistic-Optimization

Research on Optimization Models and Vehicle routing problem. Mentored by Prof. Frank Quan

Language: Jupyter Notebook - Size: 78.4 MB - Last synced: about 2 months ago - Pushed: 5 months ago - Stars: 2 - Forks: 0

Bazserpz/Web-Scraping-Challenges

When you do webscraping, data scraping, data collection, you will end up with the need for solve a captcha or anti bot, this browser will help you to pass this and forget about this challenge.

Size: 2.93 KB - Last synced: 3 months ago - Pushed: 3 months ago - Stars: 0 - Forks: 0

ElektroStudios/FHM-Crawler-freehardmusic.com

Crawls download urls of albums from freehardmusic.com website

Language: Visual Basic .NET - Size: 10.5 MB - Last synced: about 2 months ago - Pushed: about 2 months ago - Stars: 2 - Forks: 0

GKN-Tech/BriseCheck_Website

Web Crawler, Visualizations and Game

Language: HTML - Size: 4.87 MB - Last synced: 3 months ago - Pushed: 3 months ago - Stars: 0 - Forks: 0

ScaleUnlimited/flink-crawler

Continuous scalable web crawler built on top of Flink and crawler-commons

Language: Java - Size: 1.38 MB - Last synced: 2 months ago - Pushed: about 5 years ago - Stars: 52 - Forks: 18

hominee/dyer

Dyer is designed for reliable, flexible and fast web crawling, providing some high-level, comprehensive features without compromising speed.

Language: Rust - Size: 75 MB - Last synced: 3 months ago - Pushed: over 1 year ago - Stars: 134 - Forks: 14

logan-bobo/web-crawler

A simple web crawler to recursively traverse a website and generate a report of all internal links.

Language: JavaScript - Size: 84 KB - Last synced: 3 months ago - Pushed: 3 months ago - Stars: 0 - Forks: 0

Integralist/go-web-crawler

A web crawler built in the Go programming language

Language: Go - Size: 402 KB - Last synced: about 2 months ago - Pushed: over 5 years ago - Stars: 5 - Forks: 1

R00tendo/GoldDigger

GolDigger web crawler

Language: Go - Size: 43 KB - Last synced: 19 days ago - Pushed: 4 months ago - Stars: 0 - Forks: 0

zaidongy/Spotify-Scraper

Tool to scrape the Spotify web interface to retrieve artist logos

Language: Python - Size: 106 KB - Last synced: 4 months ago - Pushed: 4 months ago - Stars: 1 - Forks: 4

raymonddeng99/chompy

Focused web crawler using distributed RL for online RAG in LLMs

Language: OCaml - Size: 394 KB - Last synced: 4 months ago - Pushed: 4 months ago - Stars: 0 - Forks: 0

A3onn/mapptth

A simple to use multi-threaded web-crawler written in C with libcURL and Lexbor.

Language: C - Size: 205 KB - Last synced: 4 months ago - Pushed: 4 months ago - Stars: 5 - Forks: 0

rt2013G/krawler

A web crawler built in Javascript

Language: JavaScript - Size: 37.1 KB - Last synced: 4 months ago - Pushed: 4 months ago - Stars: 0 - Forks: 0

arthur3486/born2crawl

A highly performant and versatile crawling engine, designed with scalability and extensibility in mind.

Language: Kotlin - Size: 624 KB - Last synced: 4 months ago - Pushed: 4 months ago - Stars: 8 - Forks: 0

SylvainDe/ComicBookMaker

Script to fetch webcomics and use them to create ebooks.

Language: Python - Size: 2.48 MB - Last synced: 3 months ago - Pushed: almost 3 years ago - Stars: 34 - Forks: 7

its-my-data/doubak

豆备 (Doubak) - 豆瓣 (Douban.com) 命令行版备份工具，备份完整页面镜像，Golang实现。

Language: Go - Size: 182 KB - Last synced: 4 months ago - Pushed: 6 months ago - Stars: 9 - Forks: 0

yields/ant

A web crawler for Go

Language: Go - Size: 160 KB - Last synced: about 2 months ago - Pushed: about 1 year ago - Stars: 276 - Forks: 16

Bartozzz/crawlerr

A simple and fully customizable web crawler/spider for Node.js with server-side DOM. Comes with elegant and hell-simple APIs.

Language: JavaScript - Size: 1.09 MB - Last synced: 34 minutes ago - Pushed: almost 3 years ago - Stars: 25 - Forks: 7

KitsuneSemCalda/Info-Elixir

A WebCrawler builded to recursive crawling

Language: Elixir - Size: 14.6 KB - Last synced: 4 months ago - Pushed: 4 months ago - Stars: 2 - Forks: 0

spk/maman

Rust Web Crawler saving pages on Redis

Language: Rust - Size: 203 KB - Last synced: 27 days ago - Pushed: almost 3 years ago - Stars: 43 - Forks: 6

sagarmk/elastic-python-jobs-crawler

Built a web crawler using elastic search along with beautiful soup on indeed

Language: Python - Size: 4.88 KB - Last synced: 3 months ago - Pushed: 3 months ago - Stars: 5 - Forks: 1

calebwin/frequent

A utility for crawling websites and building frequency lists of words

Language: Python - Size: 9.77 KB - Last synced: 2 months ago - Pushed: 4 months ago - Stars: 26 - Forks: 12

amalrajan/learncpp-download

An advanced web scraper tool that seamlessly fetches and combines over 350 online tutorials into a convenient offline PDF format.

Language: Python - Size: 467 KB - Last synced: 4 months ago - Pushed: 4 months ago - Stars: 45 - Forks: 18

LandWhale2/TD-Spider

Via Text Density Simple Web Crawler With Go

Language: Go - Size: 4.88 KB - Last synced: 4 months ago - Pushed: about 1 year ago - Stars: 13 - Forks: 0

MarginaliaSearch/MarginaliaSearch

Internet search engine for text-oriented websites. Indexing the small, old and weird web.

Language: HTML - Size: 10.2 MB - Last synced: 6 months ago - Pushed: 6 months ago - Stars: 603 - Forks: 13

g-otn/terminal-00-navigator

A small script that tried to find all Terminal 00 nodes

Language: JavaScript - Size: 34.2 KB - Last synced: 4 months ago - Pushed: 4 months ago - Stars: 0 - Forks: 0

iBug/douban-spider 📦

An alternative solution to « Web Info 2019 » experiment 3

Language: Python - Size: 55.7 KB - Last synced: about 2 months ago - Pushed: over 4 years ago - Stars: 1 - Forks: 0

aishmittal/Product-Info-Crawler

Product-Info-Crawler is a python web crawler developed using scrapy framework to crawl e-commerce websites for products matching search keyword.

Language: Python - Size: 388 KB - Last synced: about 2 months ago - Pushed: almost 3 years ago - Stars: 17 - Forks: 13

christophknabe/akka-crawl

Webpage Crawler as Demonstration for Usage of Akka Actors, Akka HTTP client, and Akka Streams

Language: Scala - Size: 396 KB - Last synced: 5 months ago - Pushed: about 4 years ago - Stars: 3 - Forks: 3

forhadsidhu/Wiki-Movies-Crawler

API for showing the crawled wiki movies content and list of movies

Language: Python - Size: 243 KB - Last synced: 5 months ago - Pushed: about 2 years ago - Stars: 1 - Forks: 1

jyotiradityaz/MagnumPy

MagnumPy Is An Web Scrapper And Web Crawler With Advanced Commands Making The Experience of Researching On The Web Easy

Language: Python - Size: 3.91 KB - Last synced: 5 months ago - Pushed: 5 months ago - Stars: 0 - Forks: 0

alu0101056944/gcceproject

Business Intelligence school project. Web Scraper with an Apache Hop workflow.

Language: JavaScript - Size: 2.18 MB - Last synced: 5 months ago - Pushed: 5 months ago - Stars: 0 - Forks: 0

aigeoo/brpb

BRPB (Buc Rating Professors Bot) is a Python Selenium-based tool that automates instructors evaluation by crawling the college website

Language: Python - Size: 6.11 MB - Last synced: 5 months ago - Pushed: 12 months ago - Stars: 0 - Forks: 0

FunnySaltyFish/bilibili_comments_crawl

基于 B 站评论区数据构建大语言模型训练用对话数据集

Language: Python - Size: 872 KB - Last synced: 3 months ago - Pushed: 3 months ago - Stars: 5 - Forks: 1

OrNixz/webcrawlerhttp

Project to learn HTTP course

Language: JavaScript - Size: 46.9 KB - Last synced: 5 months ago - Pushed: 5 months ago - Stars: 1 - Forks: 0