webcrawling | Topic | Ecosyste.ms: Repos

internetarchive/heritrix3

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

Language: Java - Size: 11.2 MB - Last synced at: 10 days ago - Pushed at: 11 days ago - Stars: 2,970 - Forks: 759

DemonDamon/FinnewsHunter

从新浪财经、每经网、金融界、中国证券网、证券时报网上，爬取上市公司（个股）的历史新闻文本数据进行文本分析、提取特征集，然后利用SVM、随机森林等分类器进行训练，最后对实施抓取的新闻数据进行分类预测

Language: Python - Size: 5.46 MB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 1,033 - Forks: 273

scrapinghub/scrapyrt

HTTP API for Scrapy spiders

Language: Python - Size: 233 KB - Last synced at: 3 days ago - Pushed at: 11 months ago - Stars: 856 - Forks: 161

jaeksoft/opensearchserver

Open-source Enterprise Grade Search Engine Software

Language: Java - Size: 498 MB - Last synced at: about 6 hours ago - Pushed at: over 2 years ago - Stars: 506 - Forks: 190

Scrapyman数据接口服务。提供：淘宝、小红书、同程旅行、京东外卖、抖音（电商）、美团、抖音（视频）、快手、蒲公英、星图、拼多多、微信公众号、大众点评、哔哩哔哩、知乎、微博、贝壳、Bigo、Temu、Lazada、Shopee、SHEIN、百度指数、携程、Boss直聘、智联招聘、拉钩、今日头条、Facebook、Youtube、Instgram、Twitter。爬虫、采集、scrapy、接口、API。

Size: 3.81 MB - Last synced at: 3 days ago - Pushed at: 3 days ago - Stars: 189 - Forks: 7

mehmetozkaya/DotnetCrawler

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

Language: C# - Size: 70.3 KB - Last synced at: 20 days ago - Pushed at: over 2 years ago - Stars: 176 - Forks: 66

DedSecInside/gotor

This program provides efficient web scraping services for Tor and non-Tor sites. The program has both a CLI and REST API.

Language: Go - Size: 10.7 MB - Last synced at: 8 days ago - Pushed at: about 2 months ago - Stars: 166 - Forks: 46

feddelegrand7/ralger

ralger makes it easy to scrape a website. Built on the shoulders of titans: rvest, xml2.

Language: R - Size: 1.06 MB - Last synced at: 6 days ago - Pushed at: about 1 month ago - Stars: 156 - Forks: 14

voliveirajr/seleniumcrawler

An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site

Language: Python - Size: 20.5 KB - Last synced at: about 2 months ago - Pushed at: over 6 years ago - Stars: 127 - Forks: 45

DwarfThief/Raspagem-de-dados-para-iniciantes

Raspagem de dados para iniciante usando Scrapy e outras libs básicas

Language: Python - Size: 2.04 MB - Last synced at: over 1 year ago - Pushed at: almost 2 years ago - Stars: 122 - Forks: 20

andersonkrs/malheatmap

An extension for tracking your activities on myanimelist.net

Language: Ruby - Size: 51.7 MB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 101 - Forks: 2

Aavache/LLMWebCrawler

A Web Crawler based on LLMs implemented with Ray and Huggingface. The embeddings are saved into a vector database for fast clustering and retrieval. Use it for your RAG.

Language: Python - Size: 20.5 KB - Last synced at: about 2 months ago - Pushed at: over 1 year ago - Stars: 94 - Forks: 10

datawizard1337/ARGUS

ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9

Language: Python - Size: 14.7 MB - Last synced at: 2 months ago - Pushed at: over 3 years ago - Stars: 88 - Forks: 25

flickz/newspaperjs 📦

News extraction and scraping. Article Parsing

Language: HTML - Size: 3.99 MB - Last synced at: about 6 hours ago - Pushed at: about 2 years ago - Stars: 75 - Forks: 20

kafagy/fifa-FUT-Data

Web-scraping script that writes the data of all players from FutHead and FutBin to a CSV file or a DB

Language: Python - Size: 11.7 MB - Last synced at: about 2 years ago - Pushed at: over 5 years ago - Stars: 70 - Forks: 19

Skumarr53/Stock-Fundamental-data-scraping-and-analysis

Project on building a web crawler to collect the fundamentals of the stock and review their performance in one go

Language: Jupyter Notebook - Size: 20.1 MB - Last synced at: over 1 year ago - Pushed at: about 4 years ago - Stars: 55 - Forks: 25

crawler-commons/url-frontier

API definition, resources and reference implementation of URL Frontiers

Language: Java - Size: 1.13 MB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 48 - Forks: 12

rumca-js/crawler-buddy

Crawling framework, RSS reader and parser

Language: Python - Size: 1.57 MB - Last synced at: 3 days ago - Pushed at: 3 days ago - Stars: 47 - Forks: 2

rootVIII/proxy_web_crawler

Automates the process of repeatedly searching for a website via scraped proxy IP and search keywords

Language: Python - Size: 6.45 MB - Last synced at: 10 days ago - Pushed at: over 1 year ago - Stars: 45 - Forks: 13

spieredd/Ultimate-Guide-to-Sneaker-Bot-Creation

The Ultimate Guide to Sneaker Bot 🤖 Creation using JavaScript and NodeJS ☣️ . Learn how to get the most out of tools like the Chrome devTools, and JS Libraries like Puppeteer or Axios.

Size: 11.7 KB - Last synced at: over 1 year ago - Pushed at: about 4 years ago - Stars: 40 - Forks: 6

Galarzaa90/tibia.py

API to parse tibia.com content into python objects.

Language: Python - Size: 7.65 MB - Last synced at: 5 days ago - Pushed at: 7 months ago - Stars: 38 - Forks: 12

Marcel0024/CocoCrawler

An declarative and easy to use web crawler and scraper in C#

Language: C# - Size: 83 KB - Last synced at: about 2 months ago - Pushed at: 9 months ago - Stars: 27 - Forks: 3

lgcarmo/WebHunterScreen

This program aims to check active targets by saving screenshots in a project.

Language: Python - Size: 5.57 MB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 13 - Forks: 0

dhyeythumar/Search-Engine

Application made with Node.js and Python.

Language: HTML - Size: 2.06 MB - Last synced at: about 2 years ago - Pushed at: about 4 years ago - Stars: 13 - Forks: 12

lucaromagnoli/dataservice

Python async data gathering

Language: Python - Size: 630 KB - Last synced at: 7 months ago - Pushed at: 7 months ago - Stars: 11 - Forks: 0

colmex/frontera_example

Example frontera project

Language: Python - Size: 7.81 KB - Last synced at: over 2 years ago - Pushed at: almost 8 years ago - Stars: 11 - Forks: 2

joao2391/DotNetExpose

A package that helps you to scrap web pages. It shows you a lot of information about the page.

Language: C# - Size: 37.1 KB - Last synced at: 20 days ago - Pushed at: almost 3 years ago - Stars: 9 - Forks: 3

QueraTeam/dataanalysis_bootcamp_crawler

Web scraper implementations for a variety of websites.

Language: HTML - Size: 117 MB - Last synced at: about 1 year ago - Pushed at: over 2 years ago - Stars: 8 - Forks: 34

prkskrs/icd-10-Version

I have scraped International Statistical Classification of Diseases and Related Health Problems 10th Revision websites's data. It has all the diseases and health problems. I have also attached csv of scraped data which contains two column "Ids" and "Description".

Language: Jupyter Notebook - Size: 41 KB - Last synced at: about 1 year ago - Pushed at: over 2 years ago - Stars: 8 - Forks: 0

oussemabenhassena5/Crawl4DeepSeek

Crawl4DeepSeek = Crawl4AI + DeepSeek 🚀 Smart, efficient, and built for deep web exploration! 🌐🤖

Language: Python - Size: 23.4 KB - Last synced at: about 2 months ago - Pushed at: 4 months ago - Stars: 7 - Forks: 1

michaelradu/web-crawler

A Web Crawler developed in Python.

Language: Python - Size: 6.84 KB - Last synced at: about 2 months ago - Pushed at: almost 3 years ago - Stars: 7 - Forks: 2

sunil-sandhu/scrawly

Package wrapper around Node.js and Puppeteer for web crawling/scraping. Originally put together to accompany an article that can be found here: https://sunilsandhu.com/posts/how-to-scrape-data-from-a-website-with-javascript

Language: JavaScript - Size: 6.84 KB - Last synced at: about 2 years ago - Pushed at: almost 4 years ago - Stars: 7 - Forks: 5

gabriellst/WhatsAppBot

This is an automatic message fowarder bot within WhatsApp using Python and Selenium

Language: Python - Size: 37.1 KB - Last synced at: about 2 years ago - Pushed at: about 3 years ago - Stars: 6 - Forks: 0

mincloud1501/Python

Jupyter Notebook을 활용한 Time-series data 분석 및 crawling 기술, D3를 이용한 시각화 기술 구현 및 연구

Language: Jupyter Notebook - Size: 20 MB - Last synced at: about 2 years ago - Pushed at: over 5 years ago - Stars: 6 - Forks: 1

chouj/JPO_CloudofKeywords

a MATLAB script for generating cloud of keywords of the Journal of Physical Oceanography

Language: Matlab - Size: 727 KB - Last synced at: about 2 years ago - Pushed at: almost 7 years ago - Stars: 6 - Forks: 2

moehmeni/ezweb

Easy to use web page analyzer

Language: Python - Size: 533 KB - Last synced at: about 2 months ago - Pushed at: over 2 years ago - Stars: 5 - Forks: 0

yashrajkakkad/AUtomate

Scrapes attendance and marks related data from AURIS (Ahmedabad University Resource Information System) and notifies the user without him having to check his data repeatedly

Language: Python - Size: 16.6 KB - Last synced at: about 2 years ago - Pushed at: almost 3 years ago - Stars: 5 - Forks: 3

cjf8899/WebCrawler_exe

:ghost:Web Crawling and Convert to Executable with Pyinstaller

Language: Python - Size: 18 MB - Last synced at: about 2 years ago - Pushed at: over 3 years ago - Stars: 5 - Forks: 0

davidzwei/Streaming-Linebot

🎥🎞️🤖 A LineBot powered by Finite State Machine (FSM) that delivers updates on the latest and popular dramas, movies, and animations.

Language: Python - Size: 39.6 MB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 4 - Forks: 0

RobMcH/mindfactory_crawling

A Python 3 Crawler for Mindfactory.de

Language: Python - Size: 38.1 KB - Last synced at: 24 days ago - Pushed at: over 2 years ago - Stars: 4 - Forks: 1

PresearchOfficial/opensearch-frontier

Implementation of URLFrontier service using Opensearch

Language: Java - Size: 172 KB - Last synced at: about 1 year ago - Pushed at: almost 3 years ago - Stars: 4 - Forks: 2

mmd-lk/storm

sms bomber

Language: Python - Size: 12.7 KB - Last synced at: about 2 years ago - Pushed at: about 3 years ago - Stars: 4 - Forks: 0

TanishqChamoli/Newspaper_Mining

Newspaper mining and the analysis of the results using python. Cleaning the text using OCR.

Language: Python - Size: 16.5 MB - Last synced at: 11 months ago - Pushed at: over 4 years ago - Stars: 4 - Forks: 0

mnemocron/LSZHMovements

web API for ZRH/LSZH Zürich Airport Airport arrivals/departures Table

Language: Python - Size: 11.7 KB - Last synced at: about 2 years ago - Pushed at: almost 5 years ago - Stars: 4 - Forks: 1

n3wjack/sitecrawler

A command-line based web crawler

Language: C# - Size: 8.71 MB - Last synced at: 9 months ago - Pushed at: 10 months ago - Stars: 3 - Forks: 1

AnonCatalyst/WebDiver

WebDiver is a versatile Python script for crawling websites, extracting internal and external links, titles, and descriptions. It's useful for tasks such as web analysis, OSINT (Open Source Intelligence) gathering, and competitive analysis.

Language: Python - Size: 91.8 KB - Last synced at: 11 months ago - Pushed at: 11 months ago - Stars: 3 - Forks: 0

ElektroStudios/FHM-Crawler-freehardmusic.com

Crawls download urls of albums from freehardmusic.com website

Language: Visual Basic .NET - Size: 10.5 MB - Last synced at: about 2 months ago - Pushed at: about 1 year ago - Stars: 3 - Forks: 2

amrit94/DataScience

Pwskills

Language: Jupyter Notebook - Size: 25.2 MB - Last synced at: almost 2 years ago - Pushed at: about 2 years ago - Stars: 3 - Forks: 2

UdaraAlwis/GoogleFormsExperiment

Experimenting with Google Forms pages, extracting the page data programmatically and submitting to the rest endpoints..

Language: C# - Size: 65.4 KB - Last synced at: 10 months ago - Pushed at: over 2 years ago - Stars: 3 - Forks: 1

Rafsdutra/LicitaCrawler

Language: Python - Size: 75.9 MB - Last synced at: about 2 years ago - Pushed at: over 2 years ago - Stars: 3 - Forks: 0

mmd-lk/Ponisha-Job

This app is to make it easy for you to find work !!

Language: Python - Size: 5.86 KB - Last synced at: about 2 years ago - Pushed at: about 3 years ago - Stars: 3 - Forks: 0

dumbbellcode/WhatBot

Whatsapp bot that scrapes data from web on basis of queries.

Language: Python - Size: 18.6 KB - Last synced at: 5 months ago - Pushed at: almost 4 years ago - Stars: 3 - Forks: 0

namitkrarya/WebCrawler

WebCrawling python script!

Language: Python - Size: 7.81 KB - Last synced at: almost 2 years ago - Pushed at: over 4 years ago - Stars: 3 - Forks: 1

Mixnode/mixnode-node-sdk

Mixnode Node SDK

Language: JavaScript - Size: 198 KB - Last synced at: 5 days ago - Pushed at: about 6 years ago - Stars: 3 - Forks: 0

sushant097/Chatbot-using-Python-NLTK-

This is the Chatbot made with NLTK in python with Term Frequency-Inverse Document Frequencyn(TF-IDF) and Cosine Similarity

Language: Python - Size: 9.77 KB - Last synced at: about 2 years ago - Pushed at: over 6 years ago - Stars: 3 - Forks: 3

kenshin579/tutorials-python

Language: Python - Size: 1.66 MB - Last synced at: 1 day ago - Pushed at: 1 day ago - Stars: 2 - Forks: 0

starsbit/bardle

A Blue Archive Wordle game variant. Guess the character based on the given attributes.

Language: TypeScript - Size: 141 MB - Last synced at: 12 days ago - Pushed at: 12 days ago - Stars: 2 - Forks: 0

triposat/published-blogs

Content

Size: 244 KB - Last synced at: 19 days ago - Pushed at: 19 days ago - Stars: 2 - Forks: 0

V3RNE42/NEWS_CRAWLER

📰 NEWS_CRAWLER: Automate Your News Updates! 📰 A NodeJS web crawler that generates personalized newsletters using Resend and OpenAI APIs. Ideal for staying on top of web trends and automating your news feed.

Language: JavaScript - Size: 646 KB - Last synced at: 2 months ago - Pushed at: 2 months ago - Stars: 2 - Forks: 0

namedmichi/universalSitemapLoader

A Python package to find the sitemap of a URL, even if it does not have a standard path

Language: Python - Size: 17.6 KB - Last synced at: 4 days ago - Pushed at: over 1 year ago - Stars: 2 - Forks: 0

mukmookk/streamDAQ

real time nasdaq data pipeline

Language: Python - Size: 238 KB - Last synced at: almost 2 years ago - Pushed at: almost 2 years ago - Stars: 2 - Forks: 0

mr-rjh3/python-search-engine

Search engine that collects and indexes documents from a list of URLs so that they can be searched. Machine learning portion predicts topic of given URLs.

Language: Python - Size: 12.2 MB - Last synced at: over 1 year ago - Pushed at: about 2 years ago - Stars: 2 - Forks: 0

QuartzSoftwareLLC/scrapeR

An R web scraping framework inspired by scrapy

Language: R - Size: 1.01 MB - Last synced at: about 2 years ago - Pushed at: over 2 years ago - Stars: 2 - Forks: 0

noambassat/Gemstones_Project

Data Science final project

Language: Jupyter Notebook - Size: 41.2 MB - Last synced at: about 2 years ago - Pushed at: almost 4 years ago - Stars: 2 - Forks: 1

sheetalkalburgi/web-scraping

Web scraping algorithm for FDA and Health Canada website

Language: Python - Size: 13.7 KB - Last synced at: over 1 year ago - Pushed at: about 4 years ago - Stars: 2 - Forks: 0

dearopen/django-easy-scraper

Django apps to scrape data from web page easily

Language: Python - Size: 15.6 KB - Last synced at: 17 days ago - Pushed at: over 4 years ago - Stars: 2 - Forks: 1

alejandropuerto/web-scraping-network-analysis

Web scraping and web crawling using Scrapy to extract data from MyAnimeList.net to map a network and perform a Social Network Analysis using NetworkX.

Language: Julia - Size: 866 KB - Last synced at: 10 months ago - Pushed at: almost 5 years ago - Stars: 2 - Forks: 0

congcoi123/crawler-sheis

A small crawler for getting data from the website: https://sheis.vn

Language: Python - Size: 12.7 KB - Last synced at: about 14 hours ago - Pushed at: almost 5 years ago - Stars: 2 - Forks: 1

jkopka/price_tracker

Flask-Seite, um Preisüberblicke-/vergleiche bei eBay Kleinanzeigen und ebay.de zu bekommen.

Language: Python - Size: 48.8 KB - Last synced at: about 2 years ago - Pushed at: almost 5 years ago - Stars: 2 - Forks: 0

keshavnath/hackernews-webcrawler

A simple webcrawler that prints a list of all the Articles on the first X Pages of hackernews (https://news.ycombinator.com/news) with more than Y votes.

Language: Python - Size: 3.91 KB - Last synced at: almost 2 years ago - Pushed at: almost 5 years ago - Stars: 2 - Forks: 0

chouj/realtimeTyphoonGIF

Auto post GIF of Typhoon at the Western Pacific by crawling NOAA's Storm Floaters

Language: Matlab - Size: 4.88 KB - Last synced at: over 1 year ago - Pushed at: almost 7 years ago - Stars: 2 - Forks: 1

digitaldisarray/Arachnid

:mag: A web crawling app written in java.

Language: Java - Size: 353 KB - Last synced at: about 2 years ago - Pushed at: almost 7 years ago - Stars: 2 - Forks: 0

Esubaalew/2merkato

The web scraping project to extract bussiness directory from https://www.2merkato.com/directory

Language: Jupyter Notebook - Size: 466 KB - Last synced at: about 2 months ago - Pushed at: about 2 months ago - Stars: 1 - Forks: 0

ogtirth/SpiderBolt

SpiderBolt is a fast and efficient Python web scraping script that extracts links from websites using multi-threading and random user agents. It categorizes links into HTML and other types, groups them by paths, and saves them in an organized file. Customizable settings ensure flexibility for various scraping needs.

Language: Python - Size: 26.4 KB - Last synced at: 6 months ago - Pushed at: 6 months ago - Stars: 1 - Forks: 0

IAmFarrokhnejad/Murkmaw

A web crawler using Rust.

Language: Rust - Size: 67.7 MB - Last synced at: 2 months ago - Pushed at: 6 months ago - Stars: 1 - Forks: 0

SongArtish/TIL

2020년 7월 20일부터 공부한 것들을 차근차근 정리하고 있습니다.

Size: 29 MB - Last synced at: 6 months ago - Pushed at: 6 months ago - Stars: 1 - Forks: 0

WonITKorea/ColabByGoogle

1st attempt to webcrawl informations

Language: Jupyter Notebook - Size: 12.7 KB - Last synced at: 7 months ago - Pushed at: 7 months ago - Stars: 1 - Forks: 0

thomasgottvalles/GMBot Fork of omkarcloud/google-maps-scraper

This Python program is a bot designed to explore establishments on Google Maps, extract data from each establishment and store it for later use in a JSON and CSV file. Added value of the fork: the establishments websites are then explored with Scrapy in order to extract and store email addresses.

Language: Python - Size: 6.27 MB - Last synced at: 11 months ago - Pushed at: 11 months ago - Stars: 1 - Forks: 0

joshuaDeal/search-sasquatch

An internet search engine written mostly in python. Currently TF-IDF based.

Language: Python - Size: 5.74 MB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 1 - Forks: 0

Indigo-Coder-github/Korean_News_Crawler

Python Library for Crawling News Artircles in Korean Top 10 News Websites with Utilities

Language: Python - Size: 53.7 KB - Last synced at: 13 days ago - Pushed at: about 1 year ago - Stars: 1 - Forks: 0

Kim-src/StockScraper

🚀 주식 정보 수집 프로그램(Toy-Project)

Language: Python - Size: 47.9 KB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 1 - Forks: 0

Th3-C0der/Web-Crawler

A simple WebCrawler for exploring and downloading content from web pages within a given domain/url.

Language: HTML - Size: 47.9 KB - Last synced at: 1 day ago - Pushed at: over 1 year ago - Stars: 1 - Forks: 0

lgcarmo/Scrap_Forever

Get all links in all pages in one application

Language: Python - Size: 8.79 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 1 - Forks: 1

wlstmd/java_project

자바 당근마켓 웹크롤링 프로젝트

Language: Java - Size: 3.91 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 1 - Forks: 0

davitbars/Web-Search-and-Recomender-Systems

CS4250 Repository

Language: Python - Size: 22.5 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 1 - Forks: 0

hassanzadehmahdi/BioPersianWikiAnalyzer

Persian Wikipedia Bioinformatics Page Crawler and Text Preprocessor

Language: Jupyter Notebook - Size: 10.8 MB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 1 - Forks: 0

WaithakaKariuki/shoppers-haven

Built on Ruby on Rails API, React and TailwindCSS front-end (with useReducer & useContext state management)+ Sendgrid and Cloudinary

Language: JavaScript - Size: 13.9 MB - Last synced at: about 1 year ago - Pushed at: over 1 year ago - Stars: 1 - Forks: 0

thomasgottvalles/WebCrawler

This Python program is a bot designed to explore the pages of a website (crawling), extract the hyperlinks from each page and store them for later use. Each hyperlink on the website is tested with a request to obtain the HTTP response code (200, 404, 403, 500 etc) and to extract internal and external links from the page content (scraping).

Language: Python - Size: 17.6 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 1 - Forks: 0