Topic: "crawling-framework"
lorien/awesome-web-scraping
List of libraries, tools and APIs for web scraping and data processing.
Language: Makefile - Size: 473 KB - Last synced at: 5 days ago - Pushed at: 4 months ago - Stars: 6,970 - Forks: 805
howie6879/ruia
Async Python 3.6+ web scraping micro-framework based on asyncio
Language: Python - Size: 4.46 MB - Last synced at: 10 days ago - Pushed at: almost 2 years ago - Stars: 1,754 - Forks: 181
Symbo1/wsltools
Web Scan Lazy Tools - Python Package
Language: Python - Size: 1.82 MB - Last synced at: 5 months ago - Pushed at: almost 5 years ago - Stars: 315 - Forks: 31
RevoltSecurities/SpideyX
SpideyX a multipurpose Web Penetration Testing tool with asynchronous concurrent performance with multiple mode and configurations.
Language: Python - Size: 973 KB - Last synced at: 4 days ago - Pushed at: about 1 month ago - Stars: 159 - Forks: 28
crawlzone/crawlzone
Crawlzone is a fast asynchronous internet crawling framework for PHP.
Language: PHP - Size: 288 KB - Last synced at: 30 days ago - Pushed at: about 2 years ago - Stars: 80 - Forks: 10
behitek/social-scraper
Vietnamese text data crawler scripts for various sites (including Youtube, Facebook, 4rum, news, ...)
Language: Python - Size: 22 MB - Last synced at: about 1 year ago - Pushed at: over 2 years ago - Stars: 65 - Forks: 37
rollrat/custom-crawler
π High productivity semi-automatic crawler generator π οΈπ§°
Language: C# - Size: 13.6 MB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 55 - Forks: 2
flulemon/sneakpeek
Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. Itβs the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis
Language: Python - Size: 19.7 MB - Last synced at: 5 days ago - Pushed at: over 1 year ago - Stars: 37 - Forks: 0
peterbencze/serritor
Serritor is an open source web crawler framework built upon Selenium and written in Java. It can be used to crawl dynamic web pages that require JavaScript to render data.
Language: Java - Size: 969 KB - Last synced at: 3 days ago - Pushed at: almost 3 years ago - Stars: 32 - Forks: 15
omkarcloud/botasaurus-starter
π OFFICIAL STARTER TEMPLATE FOR BOTASAURUS SCRAPING FRAMEWORK π€
Language: TypeScript - Size: 397 KB - Last synced at: about 9 hours ago - Pushed at: about 2 months ago - Stars: 24 - Forks: 8
RuedigerVoigt/exoskeleton
A Python framework to build polite, but tenacious crawlers / scrapers with a MariaDB backend
Language: Python - Size: 706 KB - Last synced at: 10 days ago - Pushed at: over 1 year ago - Stars: 21 - Forks: 1
tokenmill/crawling-framework
Easily crawl news portals or blog sites using Storm Crawler.
Language: Java - Size: 918 KB - Last synced at: 1 day ago - Pushed at: over 2 years ago - Stars: 21 - Forks: 4
miroshnikov/scrapyteer
Web crawling & scraping framework for Node.js on top of headless Chrome browser
Language: TypeScript - Size: 384 KB - Last synced at: 22 days ago - Pushed at: about 1 year ago - Stars: 19 - Forks: 0
wind2sing/aCrawler
π A powerful web-crawling framework, based on aiohttp.
Language: Python - Size: 478 KB - Last synced at: 12 days ago - Pushed at: over 5 years ago - Stars: 15 - Forks: 3
leosocy/proksi
An intelligent proxy server. Provide durable, real-time, high-quality proxies as a middleman or datasource server.
Language: Go - Size: 262 KB - Last synced at: almost 2 years ago - Pushed at: about 2 years ago - Stars: 12 - Forks: 0
supergillis/crawler-ts
Crawler written in TypeScript using ES6 generators.
Language: TypeScript - Size: 60.5 KB - Last synced at: 1 day ago - Pushed at: almost 4 years ago - Stars: 12 - Forks: 1
quicklysnail/sprite
εΊδΊpythonεη¨ζ± γη¨ζ³η΅ζ΄»ηι«ζ§θ½η¬θ«ζ‘ζΆ
Language: Python - Size: 288 KB - Last synced at: 6 days ago - Pushed at: over 4 years ago - Stars: 9 - Forks: 2
BaseMax/StockExchangeCrawler
A crawler program to extract all of the data and the price for symbols in the global stock exchange.
Language: PHP - Size: 30.3 KB - Last synced at: 8 days ago - Pushed at: almost 6 years ago - Stars: 9 - Forks: 8
omkarcloud/web-scraping-template
π THIS WEB SCRAPING TEMPLATE PROVIDES YOU WITH A GREAT STARTING POINT WHEN CREATING WEB SCRAPING BOTS. π€
Language: Python - Size: 104 KB - Last synced at: 8 days ago - Pushed at: almost 2 years ago - Stars: 7 - Forks: 3
BaseMax/NetPHP
Useful functions for connecting to the network in the PHP based applications.
Language: PHP - Size: 23.4 KB - Last synced at: 8 days ago - Pushed at: almost 5 years ago - Stars: 7 - Forks: 1
Bubblbu/crawling-framework
A framework incorporating ropensci modules and several API's to crawl bibliographic data
Language: Python - Size: 259 KB - Last synced at: about 1 year ago - Pushed at: over 6 years ago - Stars: 6 - Forks: 1
crwlrsoft/laravel-crawler
Laravel adapter for the crwlr/crawler package.
Language: PHP - Size: 8.79 KB - Last synced at: 21 days ago - Pushed at: over 1 year ago - Stars: 4 - Forks: 0
Liadrinz/jsonflow
A Crawling Framework Based on Data Flow and Decorators
Language: Python - Size: 32.2 KB - Last synced at: about 2 years ago - Pushed at: over 2 years ago - Stars: 4 - Forks: 1
nasa-jpl-memex/sce-domain-discovery
Domain Discovery for the Sparkler Crawl Environment
Language: Java - Size: 11 MB - Last synced at: about 2 years ago - Pushed at: over 2 years ago - Stars: 4 - Forks: 8
JimmyLaurent/node-crawling-framework
β¨ NodeJs crawling & scraping framework heavily inspired by Scrapy
Language: JavaScript - Size: 227 KB - Last synced at: 23 days ago - Pushed at: almost 7 years ago - Stars: 4 - Forks: 1
tokenmill/crawling-framework-example
Demonstration on how to use the Crawling Framework to setup a simple science news crawler and store results in ElasticSearch. Use this configuration to set up your own crawler.
Language: Java - Size: 179 KB - Last synced at: about 2 months ago - Pushed at: over 5 years ago - Stars: 3 - Forks: 0
omkarcloud/dentalkart-scraper
π SCRAPE 1000'S OF PRODUCTS FROM DENTALKART π€
Language: Python - Size: 908 KB - Last synced at: 8 days ago - Pushed at: about 1 year ago - Stars: 2 - Forks: 2
its-my-data/android-crawler-engine
An Android app crawling framework, making automatic crawling mobile apps super easy! (if possible, iOS will be supported after Android version)
Size: 5.86 KB - Last synced at: over 1 year ago - Pushed at: over 7 years ago - Stars: 2 - Forks: 0
EbraLim/dss_project_Crawling
Crawling hotel data on 3 hotel reservation platforms in realtime, enabling users to compare them and reserve a room with the best price.
Language: Jupyter Notebook - Size: 6.3 MB - Last synced at: almost 2 years ago - Pushed at: about 4 years ago - Stars: 1 - Forks: 0
hwaking/awesome-python-crawl
python crawl projects, including selenium crawl with requests and crawl with scrapy and all esay for redesign
Language: Python - Size: 7.33 MB - Last synced at: 10 days ago - Pushed at: over 6 years ago - Stars: 1 - Forks: 0
surister/scrupy
Python library to create web Crawlers which aims to be powerful yet simple.
Language: Python - Size: 271 KB - Last synced at: about 1 month ago - Pushed at: about 1 year ago - Stars: 0 - Forks: 0
davidpasch1/crawlframej
Simple crawl framework for a focused web-crawler in Java.
Language: Java - Size: 16.4 MB - Last synced at: about 2 months ago - Pushed at: over 2 years ago - Stars: 0 - Forks: 0
vivekg13186/lucas
A web crawler
Language: Java - Size: 780 KB - Last synced at: 26 days ago - Pushed at: over 2 years ago - Stars: 0 - Forks: 0
matheusfaustino/Phrawl
Phrawl: A web crawling framework in PHP (or it seems so)
Language: PHP - Size: 104 KB - Last synced at: 2 months ago - Pushed at: about 6 years ago - Stars: 0 - Forks: 0
