luiswirth/crawler

An asynchronous web crawler.

Language: Rust - Size: 442 KB - Last synced: about 23 hours ago - Pushed: 12 months ago - Stars: 3 - Forks: 0

zorlan/skycaiji

蓝天采集器是一款开源免费的爬虫系统，仅需点选编辑规则即可采集数据，可运行在本地、虚拟主机或云服务器中，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统

Language: PHP - Size: 24.8 MB - Last synced: about 23 hours ago - Pushed: 1 day ago - Stars: 1,867 - Forks: 571

Lucs1590/cobWeb

🌧 🐛.🌿 Web crawler to get data from weather, bugs and plant!

Language: Python - Size: 8.95 MB - Last synced: 1 day ago - Pushed: 2 days ago - Stars: 0 - Forks: 0

industrialsociety/Panoptes

A Python web crawler that efficiently manages URLs with SQLite, supports real-time progress updates via keypress, and robustly handles various parsing and request errors.

Language: Python - Size: 18.6 KB - Last synced: 2 days ago - Pushed: 27 days ago - Stars: 0 - Forks: 0

Acollie/Go-Webcrawler

Webcrawler in Go with a graph database and DynamoDB for backing

Language: Go - Size: 1.73 MB - Last synced: 3 days ago - Pushed: 4 days ago - Stars: 0 - Forks: 0

Lehoczky/apro-scrape

Helpful web scraper for hardverapro.hu

Language: TypeScript - Size: 3.96 MB - Last synced: 3 days ago - Pushed: 4 days ago - Stars: 5 - Forks: 0

deep5050/Abosar

অবসর 📚 A collection of short Bengali stories web scraped from various Bengali eMagazines and eNewspapers.

Language: Python - Size: 76.6 MB - Last synced: 5 days ago - Pushed: 5 days ago - Stars: 12 - Forks: 1

nomomon/Kamernet-Puppeteer

:house: Automatic message sender to new adverts on Kamernet using puppeteer.

Language: JavaScript - Size: 1.06 MB - Last synced: 8 days ago - Pushed: 8 days ago - Stars: 7 - Forks: 2

xilapa/SiteWatcher

A web crawler that send notifications to you!

Language: C# - Size: 4.93 MB - Last synced: 8 days ago - Pushed: 9 days ago - Stars: 6 - Forks: 0

scrapinghub/scrapyrt

HTTP API for Scrapy spiders

Language: Python - Size: 286 KB - Last synced: 5 days ago - Pushed: 3 months ago - Stars: 816 - Forks: 159

iiicebearrr/spiders-for-all

A set of useful and scalable spiders to crawl data/videos from bilibili, xiaohongshu, etc.

Language: Python - Size: 1.06 MB - Last synced: about 3 hours ago - Pushed: 3 months ago - Stars: 20 - Forks: 8

brownwolf1355/JournalList

This is a repository for code developed for JournalList.net.

Language: Python - Size: 4.17 MB - Last synced: 10 days ago - Pushed: 10 days ago - Stars: 0 - Forks: 0

NoAssosciation/NightFall

Introducing NightFall, a cutting-edge tool revolutionizing Open-Source Intelligence. Dive deeper into the vast web with NightFall, unlocking unparalleled data extraction capabilities. NightFall empowers users to explore uncharted territories of the dark web and unearth hidden gems with pinpoint accuracy, courtesy of its advanced keyword extraction.

Language: Python - Size: 581 KB - Last synced: 10 days ago - Pushed: 10 days ago - Stars: 2 - Forks: 0

Indigo-Coder-github/Korean_News_Crawler

Python Library for Crawling News Artircles in Korean Top 10 News Websites with Utilities

Language: Python - Size: 53.7 KB - Last synced: 9 days ago - Pushed: 10 days ago - Stars: 0 - Forks: 0

Jake-The-Human/PizzaCrawler

Using a webcrawler to learn python

Language: Python - Size: 11.7 KB - Last synced: 11 days ago - Pushed: about 7 years ago - Stars: 0 - Forks: 0

GuizaumNR/Trabalho-de-Redes-1

Este é um trabalho que fiz na disciplina de Redes, é uma aplicação desktop, contendo um web crawler que manda as informações coletas para um tela (programação visual).

Language: Java - Size: 356 KB - Last synced: 15 days ago - Pushed: almost 3 years ago - Stars: 0 - Forks: 0

FabianGroeger96/python-scrapy-web-crawler

A python scrapy web crawler

Language: Python - Size: 38.3 MB - Last synced: 15 days ago - Pushed: over 4 years ago - Stars: 0 - Forks: 0

bkeepers/spiderman

your friendly neighborhood web crawler

Language: Ruby - Size: 39.1 KB - Last synced: 13 days ago - Pushed: almost 2 years ago - Stars: 18 - Forks: 4

MCStreetguy/Crawler

An advanced web-crawler written in PHP.

Language: PHP - Size: 224 KB - Last synced: 16 days ago - Pushed: about 5 years ago - Stars: 3 - Forks: 0

biraj21/web-wanderer

A multi-threaded web crawler written in Python, utilizing ThreadPoolExecutor and Playwright to efficiently crawl dynamically rendered web pages and download them.

Language: Python - Size: 202 KB - Last synced: 16 days ago - Pushed: 16 days ago - Stars: 10 - Forks: 1

sidmishraw/broodmother-old-deprecated 📦

A multithreaded webcrawler

Language: Java - Size: 315 KB - Last synced: 19 days ago - Pushed: almost 7 years ago - Stars: 1 - Forks: 0

tashfiqul-islam/bd-news-crawler

Language: Python - Size: 26.3 MB - Last synced: 18 days ago - Pushed: 19 days ago - Stars: 0 - Forks: 0

Aurora0x27/bvget

a project for practising, down load video from bilibili

Language: Python - Size: 1.95 KB - Last synced: 19 days ago - Pushed: 19 days ago - Stars: 1 - Forks: 0

pavlovtech/WebReaper

Web scraper, crawler and parser in C#. Designed as simple, declarative and scalable web scraping solution.

Language: C# - Size: 37.3 MB - Last synced: 18 days ago - Pushed: 6 months ago - Stars: 93 - Forks: 19

EdmundMartin/Scrapio

Asyncio web crawling framework. Work in progress.

Language: Python - Size: 111 KB - Last synced: 22 days ago - Pushed: 23 days ago - Stars: 17 - Forks: 4

BruunAnd/search-engine-wi

Language: Python - Size: 1.26 MB - Last synced: 22 days ago - Pushed: over 5 years ago - Stars: 1 - Forks: 0

ShuangChenGust/USA-covid-19-geochart-python-D3.js

USA covid-19 geochart/美国疫情地图绘制，python,D3.js

Language: JavaScript - Size: 149 KB - Last synced: 24 days ago - Pushed: almost 4 years ago - Stars: 1 - Forks: 0

qpal147147/Bahamut-crawler

用於巴哈姆特網站討論串內之大樓搜尋功能

Language: Python - Size: 3.91 KB - Last synced: 24 days ago - Pushed: about 3 years ago - Stars: 0 - Forks: 0

bear01/Multi_API_translation_Webcrawler

通过爬取谷歌，百度，有道，词霸，腾讯，微软等翻译网站API做了一个集成网站，可以实现多个翻译API同步翻译

Language: Python - Size: 95.7 KB - Last synced: 25 days ago - Pushed: almost 4 years ago - Stars: 2 - Forks: 0

DedSecInside/gotor

This program provides efficient web scraping services for Tor and non-Tor sites. The program has both a CLI and REST API.

Language: Go - Size: 10.7 MB - Last synced: 25 days ago - Pushed: 25 days ago - Stars: 150 - Forks: 43

hengbinxu/Yahoo-Movie-Reviews

Yahoo Movie Reviews Analysis

Language: Jupyter Notebook - Size: 22.1 MB - Last synced: 26 days ago - Pushed: over 5 years ago - Stars: 0 - Forks: 0

havardnyboe/dagenidag

Gjenskapning av NRKs side 199 fra Tekst-TV

Language: TypeScript - Size: 3.22 MB - Last synced: 25 days ago - Pushed: about 2 months ago - Stars: 2 - Forks: 0

3nock/SpiderSuite

Advance web security spider/crawler

Size: 6.98 MB - Last synced: 26 days ago - Pushed: 8 months ago - Stars: 525 - Forks: 60

siddhantshar24/StockManager-Stock-Portfolio-Management-System

Portfolio Manager built using SpringBoot with GraphQL integration and scraping for data correctness using JSoup Library.

Language: Java - Size: 537 KB - Last synced: 26 days ago - Pushed: 26 days ago - Stars: 1 - Forks: 0

Falchizao/webscraping_python 📦

crawler to fetch data over the IMDb platform

Language: Python - Size: 39.1 KB - Last synced: 26 days ago - Pushed: over 1 year ago - Stars: 3 - Forks: 0

panakour/pkscraper

Extract structured data from the web

Language: PHP - Size: 66.4 KB - Last synced: 27 days ago - Pushed: 5 months ago - Stars: 1 - Forks: 1

sgowdaks/nichirin

RAG and Webcrawler in a single package

Language: Python - Size: 1.46 MB - Last synced: 26 days ago - Pushed: 27 days ago - Stars: 1 - Forks: 0

krupt3x/webCrawler

A basic web crawler I built using javascript and NodeJs it's a very basic web crawler nothing very interesting.

Language: JavaScript - Size: 32.2 KB - Last synced: 27 days ago - Pushed: 27 days ago - Stars: 0 - Forks: 0

crawlab-team/crawlab

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Language: Go - Size: 23.6 MB - Last synced: 27 days ago - Pushed: 27 days ago - Stars: 10,782 - Forks: 1,721

sunshineplan/web-crawler

collect data from web

Language: Python - Size: 106 KB - Last synced: 28 days ago - Pushed: over 2 years ago - Stars: 0 - Forks: 0

kaioduarte/webcrawler-contas-cm

Web Crawler feito para capturar dados da prefeitura de Campo Mourão - PR

Language: Python - Size: 3.23 MB - Last synced: 28 days ago - Pushed: about 7 years ago - Stars: 0 - Forks: 0

krithivasanchandran/ComputerVisionCrawler

Built to recognize Images , Image Patterns - Visual Web Page Representation and Visual Page Clustering - Topic Modelled Crawler - Recognize Paypal Competitors Buttons (Stripe, Amazon Pay), Analyze Javascript and other Graphical User Interface Recognitions. Image Comparison using OpenCV Java Built in library though the DOM structure is different - Comparison the Pixel by Pixel Values of the Screenshots.

Language: Java - Size: 1.1 MB - Last synced: 29 days ago - Pushed: about 4 years ago - Stars: 0 - Forks: 1

szsascha/website-observer

Get a Telegram notification as soon as the content of a website changes.

Language: Java - Size: 5.86 KB - Last synced: 29 days ago - Pushed: over 2 years ago - Stars: 0 - Forks: 0

defrank/LinkCrawler Fork of Readz/LinkCrawler

Crawl a web site and count the number of links of specific types

Language: Python - Size: 23.4 KB - Last synced: 29 days ago - Pushed: almost 7 years ago - Stars: 0 - Forks: 0

snoop2head/comments_tracker

Public Relations Tracker Slack Chatbot for Target Page: Flask, Selenium

Language: Python - Size: 21.5 KB - Last synced: 30 days ago - Pushed: over 1 year ago - Stars: 1 - Forks: 0

mrx-arafat/KingBOB-WebCrawler

Language: Python - Size: 2.93 KB - Last synced: 30 days ago - Pushed: about 1 month ago - Stars: 0 - Forks: 0

srijannnd/Data-Scraping

Scraping Data from trodly.com (just for learning purpose)

Language: Python - Size: 68.4 KB - Last synced: about 1 month ago - Pushed: over 6 years ago - Stars: 0 - Forks: 0

lansana/antidote

Crawl web pages, convert asset URL's into raw source code, and embed the source code back into the DOM.

Language: Go - Size: 7.81 KB - Last synced: about 1 month ago - Pushed: almost 4 years ago - Stars: 1 - Forks: 0

doomspork/maartz

A refactor of Maartz's web scrapper. Context: https://twitter.com/maartz4/status/1248133734760615937

Language: Elixir - Size: 8.79 KB - Last synced: about 1 month ago - Pushed: almost 4 years ago - Stars: 2 - Forks: 1

JaCraig/Spidey

A multi threaded web crawler library that is generic enough to allow different engines to be swapped in.

Language: C# - Size: 17 MB - Last synced: about 1 month ago - Pushed: about 1 month ago - Stars: 11 - Forks: 3

amirgamil/apollo

A Unix-style personal search engine and web crawler for your digital footprint.

Language: Go - Size: 532 KB - Last synced: about 1 month ago - Pushed: 7 months ago - Stars: 1,359 - Forks: 53

droxey/getgcode

🖨 Scrapes GCODE commands and documentation from numerous 3d printer firmware websites.

Language: Go - Size: 6.84 KB - Last synced: about 1 month ago - Pushed: about 1 year ago - Stars: 1 - Forks: 0

GeneralNewsExtractor/GeneralNewsExtractor

新闻网页正文通用抽取器 Beta 版.

Language: Python - Size: 17.4 MB - Last synced: about 1 month ago - Pushed: 9 months ago - Stars: 3,378 - Forks: 515

Conso1eCowb0y/Deepminer 📦

Deep web crawler and search engine

Language: Python - Size: 14.6 KB - Last synced: 5 days ago - Pushed: almost 4 years ago - Stars: 44 - Forks: 10

kaydotdev/steam-game-recommendations

Building an open hybrid recommender system for Steam

Language: Python - Size: 150 KB - Last synced: about 1 month ago - Pushed: 6 months ago - Stars: 1 - Forks: 0

kaydotdev/streamlit-ds-courses

A collection of web crawlers, preprocessing pipelines, notebooks, and interactive dashboards for the 'Online Data Science Courses' project

Language: Jupyter Notebook - Size: 8.36 MB - Last synced: about 1 month ago - Pushed: 6 months ago - Stars: 1 - Forks: 0

gdgd009xcd/AutoMacroBuilderForZAP

A ZAPROXY Add-on that allows testing of web application vulnerabilities by recording complex multi-step sequences. You can test applications that need to access pages in a specific order, such as shopping carts or registration of member information.

Language: Java - Size: 50.9 MB - Last synced: 3 days ago - Pushed: 29 days ago - Stars: 19 - Forks: 4

sjpau/pig

Web crawler to bulk-download href elements from websites.

Language: Go - Size: 27.3 KB - Last synced: about 1 month ago - Pushed: over 1 year ago - Stars: 0 - Forks: 0

ptrumpis/snap-lens-web-crawler

JavaScript library to crawl and download Snap Lenses from lens.snapchat.com with ease.

Language: JavaScript - Size: 24.4 KB - Last synced: 15 days ago - Pushed: 4 months ago - Stars: 5 - Forks: 1

kari/crawler

Checks blog's outbound links' health

Language: Go - Size: 6.84 KB - Last synced: about 1 month ago - Pushed: 5 months ago - Stars: 0 - Forks: 0

isaqueveras/scrape-google-results

Scrape Google Results in Golang

Language: Go - Size: 10.7 KB - Last synced: about 1 month ago - Pushed: over 2 years ago - Stars: 0 - Forks: 0

vojtechhabes/web-crawler

Node.js web crawler, který prochází webové stránky a získává o nich různá data.

Language: JavaScript - Size: 1.16 MB - Last synced: about 1 month ago - Pushed: 8 months ago - Stars: 9 - Forks: 0

Aravindha1234u/SocialScraper

Social Scraper is a python tool meant for Detection of Child Predators/Cyber Harassers on Social Media

Language: Python - Size: 739 KB - Last synced: 21 days ago - Pushed: over 3 years ago - Stars: 52 - Forks: 13

amartorelli/millipedes

A webcrawler written in Go which builds a graph of all the links found in a domain

Language: Go - Size: 4.68 MB - Last synced: about 1 month ago - Pushed: about 5 years ago - Stars: 0 - Forks: 0

luka2220/stackoverflow-scraper

A https://stackoverflow.com/ web crawler to scape question data for fast indexing on software development topics

Language: Python - Size: 57.6 KB - Last synced: about 1 month ago - Pushed: about 1 month ago - Stars: 0 - Forks: 0

mintsuku/wappu

Wappu is a comprehensive web scraping library written in Rust, designed for ease of use and performance. It integrates seamlessly HTTP client capabilities with powerful HTML parsing functionalities, allowing users to fetch and parse web content efficiently.

Language: Rust - Size: 114 KB - Last synced: about 2 months ago - Pushed: about 2 months ago - Stars: 3 - Forks: 0

alrafiabdullah/newspaper-crawler

A newspaper crawler made with Java.

Language: Java - Size: 2.93 KB - Last synced: about 2 months ago - Pushed: over 3 years ago - Stars: 0 - Forks: 0

amjpg/webcrawler

A python crawler that prints the frequency of words in a webpage and the HTTP links found on the webpage

Language: Python - Size: 1.95 KB - Last synced: about 2 months ago - Pushed: about 2 months ago - Stars: 0 - Forks: 0

ssssssss-team/spider-flow

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

Language: Java - Size: 3.23 MB - Last synced: about 2 months ago - Pushed: 11 months ago - Stars: 8,991 - Forks: 1,718

jaeksoft/opensearchserver

Open-source Enterprise Grade Search Engine Software

Language: Java - Size: 498 MB - Last synced: about 2 months ago - Pushed: over 1 year ago - Stars: 495 - Forks: 199

brianmadden/krawler

A web crawling framework written in Kotlin

Language: Kotlin - Size: 403 KB - Last synced: about 1 month ago - Pushed: almost 3 years ago - Stars: 130 - Forks: 16

shofiq0023/simple-web-crawler

Language: Java - Size: 1000 Bytes - Last synced: about 2 months ago - Pushed: over 2 years ago - Stars: 0 - Forks: 0

murarisumit/crawler

Sample web crawler in golang

Language: Go - Size: 15.6 KB - Last synced: about 2 months ago - Pushed: over 5 years ago - Stars: 0 - Forks: 0

es520-hsieh/ABBA-Covers-Geography-Visualization-Site

The project is aiming to visualize sweden artists ABBA covers with visual mapping

Language: JavaScript - Size: 2.41 MB - Last synced: about 2 months ago - Pushed: about 2 months ago - Stars: 1 - Forks: 0

shibu130/blockchain.com_scraper

scraping the btc address balance using beautiful soup

Language: Python - Size: 11.7 KB - Last synced: about 2 months ago - Pushed: about 3 years ago - Stars: 0 - Forks: 1

shibu130/bip44-bip39-bip84-wallet-gen-and-query

Language: Python - Size: 4.88 KB - Last synced: about 2 months ago - Pushed: almost 3 years ago - Stars: 3 - Forks: 3

tehcyx/ruby-webcrawl-test

Language: Ruby - Size: 1000 Bytes - Last synced: about 2 months ago - Pushed: about 8 years ago - Stars: 0 - Forks: 0

sanjaymurali/Information-Retrieval

Contains code for all the Assignments completed as part of CS6200

Language: Python - Size: 52.3 MB - Last synced: about 2 months ago - Pushed: over 6 years ago - Stars: 2 - Forks: 3

ivanarena/pyscraper-cli

A CLI tool to download a whole website in one click.

Language: Python - Size: 3.91 KB - Last synced: about 2 months ago - Pushed: about 2 months ago - Stars: 0 - Forks: 0

NelcifranMagalhaes/WebCrawlerRuby

A web crawler of Quotes in site http://quotes.toscrape.com/

Language: Ruby - Size: 7.81 KB - Last synced: about 2 months ago - Pushed: over 1 year ago - Stars: 0 - Forks: 0

marcos-inja/novel-ebook

webclawler, translator and ebook assembler (under construction)

Language: Python - Size: 329 KB - Last synced: 2 months ago - Pushed: over 1 year ago - Stars: 2 - Forks: 0

thulin82/LunchScraper

Scraping lunch menus in the Malmö area

Language: C# - Size: 860 KB - Last synced: 2 months ago - Pushed: 8 months ago - Stars: 0 - Forks: 0

writepython/web-crawler

Python Web Crawler with Selenium and PhantomJS

Language: Python - Size: 89.2 MB - Last synced: 2 months ago - Pushed: almost 7 years ago - Stars: 19 - Forks: 14

ParalaxRus/WebCrawler

Web crawler pet project

Language: C# - Size: 215 KB - Last synced: 2 months ago - Pushed: about 3 years ago - Stars: 0 - Forks: 0

RolandSchmid/core_crawler

Core code for a web crawler

Language: Python - Size: 11.7 KB - Last synced: 2 months ago - Pushed: 2 months ago - Stars: 0 - Forks: 0

vasugamdha/python-webCrawler

Web Crawler in Python

Language: Python - Size: 6.84 KB - Last synced: 2 months ago - Pushed: over 4 years ago - Stars: 0 - Forks: 0

ilies-t/behance-node

An asynchronous module for scrapping Behance using JavaScript.

Language: TypeScript - Size: 175 KB - Last synced: 6 days ago - Pushed: 2 months ago - Stars: 2 - Forks: 0

salimk/Rcrawler

An R web crawler and scraper

Language: R - Size: 597 KB - Last synced: about 2 months ago - Pushed: about 2 years ago - Stars: 344 - Forks: 99

RobMcH/mindfactory_crawling

A Python 3 Crawler for Mindfactory.de

Language: Python - Size: 38.1 KB - Last synced: 2 months ago - Pushed: over 1 year ago - Stars: 4 - Forks: 1

cataniamatt/maltapost-web-crawler

A Python script that gets all the addresses in the MaltaPost database

Language: Python - Size: 28.3 KB - Last synced: 2 months ago - Pushed: about 2 years ago - Stars: 0 - Forks: 0

z0m31en7/Uscrapper

Uscrapper Vanta: Dive deeper into the web with this powerful open-source tool. Extract valuable insights with ease and efficiency, from both surface and deep web sources. Empower your data mining and analysis with Vanta's advanced capabilities. Fast, reliable, and user-friendly, Uscrapper Vanta is the ultimate choice for researchers and analysts.

Language: Python - Size: 429 KB - Last synced: 3 months ago - Pushed: 3 months ago - Stars: 356 - Forks: 31

FehmiTahsinDemirkan/Mindsite-Case

Mindsite Interview Task : Powerful web scraping tool for e-commerce data with email notifications and flexible data export. Supports N11 and Trendyol.

Language: Python - Size: 51.1 MB - Last synced: 3 months ago - Pushed: 3 months ago - Stars: 0 - Forks: 0

ElektroStudios/FHM-Crawler-freehardmusic.com

Crawls download urls of albums from freehardmusic.com website

Language: Visual Basic .NET - Size: 10.5 MB - Last synced: about 1 month ago - Pushed: about 1 month ago - Stars: 2 - Forks: 0

bjoern-hempel/php-web-crawler

A php class that crawls a given url and collects recursively some data from it. The final representation will be a json object.

Language: PHP - Size: 223 KB - Last synced: 3 months ago - Pushed: 3 months ago - Stars: 9 - Forks: 1

zeuscybersec/Content_Bruter.py

A BruteForcing Tool to Find All Hidden Directories/Files in A WebServer

Language: Python - Size: 3.91 KB - Last synced: 3 months ago - Pushed: almost 3 years ago - Stars: 0 - Forks: 0

KaidiGuo/All_links_downloader

Download all links (files) from one webpage

Language: Jupyter Notebook - Size: 53.7 KB - Last synced: 3 months ago - Pushed: 3 months ago - Stars: 4 - Forks: 1

Th3-C0der/Web-Crawler

A simple WebCrawler for exploring and downloading content from web pages within a given domain/url.

Language: HTML - Size: 44.9 KB - Last synced: 3 months ago - Pushed: 3 months ago - Stars: 3 - Forks: 0

YanSchw/Cosmos

Cosmos is a WebCrawler + SearchEngine written in Java

Language: Java - Size: 271 KB - Last synced: 3 months ago - Pushed: 3 months ago - Stars: 0 - Forks: 0

RexKizzy22/gmane

A set of ETL scripts that extracts email data from the gmane server and does analysis on the top level domains

Language: Python - Size: 125 KB - Last synced: 16 days ago - Pushed: 3 months ago - Stars: 0 - Forks: 0

RexKizzy22/pagerank

A set of ETL scripts that crawls the web, extract urls and ranks them as a search engine would

Language: Python - Size: 1.02 MB - Last synced: 16 days ago - Pushed: 3 months ago - Stars: 0 - Forks: 0