scrape | Topic | Ecosyste.ms: Repos

twintproject/twint 📦

An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.

Language: Python - Size: 4.37 MB - Last synced at: about 19 hours ago - Pushed at: over 2 years ago - Stars: 16,131 - Forks: 2,763

alirezamika/autoscraper

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Language: Python - Size: 135 KB - Last synced at: 18 days ago - Pushed at: 18 days ago - Stars: 6,784 - Forks: 705

Anorov/cloudflare-scrape

A Python module to bypass Cloudflare's anti-bot page.

Language: Python - Size: 455 KB - Last synced at: about 1 month ago - Pushed at: over 1 year ago - Stars: 3,459 - Forks: 455

d60/twikit

Language: Python - Size: 3.38 MB - Last synced at: 10 days ago - Pushed at: 2 months ago - Stars: 2,721 - Forks: 324

microlinkhq/metascraper

Get unified metadata from websites using Open Graph, Microdata, RDFa, Twitter Cards, JSON-LD, HTML, and more.

Language: HTML - Size: 28.5 MB - Last synced at: 1 day ago - Pushed at: 3 days ago - Stars: 2,506 - Forks: 183

trevorhobenshield/twitter-api-client

Implementation of X/Twitter v1, v2, and GraphQL APIs

Language: Python - Size: 31 MB - Last synced at: 4 days ago - Pushed at: about 1 year ago - Stars: 1,812 - Forks: 242

Altimis/Scweet

A simple and unlimited twitter scraper : scrape tweets, likes, retweets, following, followers, user info, images...

Language: Python - Size: 27.7 MB - Last synced at: 4 days ago - Pushed at: 2 months ago - Stars: 1,184 - Forks: 238

glebarez/cero

Scrape domain names from SSL certificates of arbitrary hosts

Language: Go - Size: 40 KB - Last synced at: about 1 month ago - Pushed at: about 1 year ago - Stars: 647 - Forks: 83

markowanga/stweet

Advanced python library to scrap Twitter (tweets, users) from unofficial API

Language: Python - Size: 494 KB - Last synced at: 16 days ago - Pushed at: almost 2 years ago - Stars: 605 - Forks: 69

austinoboyle/scrape-linkedin-selenium

`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.

Language: HTML - Size: 269 KB - Last synced at: about 1 month ago - Pushed at: over 2 years ago - Stars: 490 - Forks: 166

ScriptSmith/instamancer

Scrape Instagram's API with Puppeteer

Language: TypeScript - Size: 5.4 MB - Last synced at: 1 day ago - Pushed at: over 2 years ago - Stars: 405 - Forks: 61

unixfox/pupflare

A webpage proxy that request through Chromium (puppeteer) - can be used to bypass Cloudflare anti bot / anti ddos on any application (like curl)

Language: JavaScript - Size: 36.1 KB - Last synced at: 18 days ago - Pushed at: 18 days ago - Stars: 396 - Forks: 81

Home Assistant custom component for scraping (html, xml or json) multiple values (from a single HTTP request) with a separate sensor/attribute for each value. Support for (login) form-submit functionality.

Language: Python - Size: 880 KB - Last synced at: 2 days ago - Pushed at: 2 days ago - Stars: 358 - Forks: 16

Anonyfox/elixir-scrape

Scrape any website, article or RSS/Atom Feed with ease!

Language: Elixir - Size: 926 KB - Last synced at: about 1 month ago - Pushed at: almost 5 years ago - Stars: 331 - Forks: 42

yaroslaff/nudecrawler

Crawl telegra.ph searching for nudes!

Language: Python - Size: 212 KB - Last synced at: about 18 hours ago - Pushed at: 11 months ago - Stars: 324 - Forks: 27

drudge/n8n-nodes-puppeteer

n8n node for browser automation using Puppeteer

Language: TypeScript - Size: 2.08 MB - Last synced at: 2 days ago - Pushed at: 4 days ago - Stars: 312 - Forks: 48

ultralytics/google-images-download Fork of hardikvasa/google-images-download

Google/Bing Images Web Downloader

Language: Python - Size: 362 KB - Last synced at: about 2 months ago - Pushed at: about 2 months ago - Stars: 308 - Forks: 91

andrewstuart/goq

A declarative struct-tag-based HTML unmarshaling or scraping package for Go built on top of the goquery library

Language: Go - Size: 98.6 KB - Last synced at: 2 months ago - Pushed at: almost 4 years ago - Stars: 268 - Forks: 21

JaredLGillespie/proxyscrape

Python library for retrieving free proxies (HTTP, HTTPS, SOCKS4, SOCKS5).

Language: Python - Size: 121 KB - Last synced at: 12 days ago - Pushed at: over 1 year ago - Stars: 258 - Forks: 54

evyatarmeged/Humanoid

Node.js package to bypass CloudFlare's anti-bot JavaScript challenges

Language: JavaScript - Size: 78.1 KB - Last synced at: 13 days ago - Pushed at: about 2 years ago - Stars: 227 - Forks: 26

essamamdani/search-result-scraper-markdown

This project provides a powerful web scraping tool that fetches search results and converts them into Markdown format using FastAPI, SearXNG, and Browserless. It includes the capability to use proxies for web scraping and handles HTML content conversion to Markdown efficiently.

Language: Python - Size: 86.9 KB - Last synced at: about 1 month ago - Pushed at: 6 months ago - Stars: 218 - Forks: 11

JMousqueton/ransomware.live

🏴‍☠️💰 Another Ransomware gang tracker

Language: Python - Size: 2.96 GB - Last synced at: about 2 months ago - Pushed at: about 2 months ago - Stars: 193 - Forks: 37

rocketlaunchr/google-search

scrape google search results

Language: Go - Size: 82 KB - Last synced at: 1 day ago - Pushed at: over 1 year ago - Stars: 177 - Forks: 33

tegridydev/auto-md

Convert Files / Folders / GitHub Repos Into AI / LLM-ready Files

Language: Python - Size: 88.9 KB - Last synced at: 6 days ago - Pushed at: 5 months ago - Stars: 159 - Forks: 25

oxylabs/scrape-google-python

In this tutorial, we showcase how to scrape public Google data with Python and Oxylabs API.

Size: 36.1 KB - Last synced at: about 7 hours ago - Pushed at: about 8 hours ago - Stars: 155 - Forks: 1

meetyan/raise

A simple (and unofficial) GitHub Trending client that lives in your menubar.

Language: JavaScript - Size: 13.6 MB - Last synced at: over 1 year ago - Pushed at: almost 3 years ago - Stars: 154 - Forks: 1

Jimut123/jimutmap

API to get enormous amount of high resolution satellite images from satellites.pro quickly through multi-threading! create map your own map dataset. Bringing data to Humans.

Language: Python - Size: 3.89 MB - Last synced at: 11 days ago - Pushed at: about 2 months ago - Stars: 153 - Forks: 17

luengwaiban/instagram-python-scraper

A instagram scraper wrote in python. Similar to instagram-php-scraper.Usages are in example.py. Enjoy it!

Language: Python - Size: 57.6 KB - Last synced at: about 1 month ago - Pushed at: over 2 years ago - Stars: 132 - Forks: 14

html2rss/html2rss

📰 Build RSS 2.0 feeds from websites (and JSON APIs) automatically or with a few CSS selectors.

Language: Ruby - Size: 1.21 MB - Last synced at: 12 days ago - Pushed at: 12 days ago - Stars: 127 - Forks: 10

DrKain/scrape-youtube

A lightning fast package to scrape YouTube search results

Language: JavaScript - Size: 178 KB - Last synced at: about 1 month ago - Pushed at: about 1 year ago - Stars: 121 - Forks: 30

fefit/visdom

A library use jQuery like API for html parsing & node selecting & node mutation, suitable for web scraping and html confusion.

Language: Rust - Size: 611 KB - Last synced at: 30 days ago - Pushed at: 3 months ago - Stars: 112 - Forks: 6

badoux/goscraper

Golang pkg to quickly return a preview of a webpage (title/description/images)

Language: Go - Size: 7.81 KB - Last synced at: about 1 year ago - Pushed at: over 2 years ago - Stars: 104 - Forks: 39

warifp/Shopee-Scrape

Shopee Scrape is a tool that functions to collect data - the data needed, such as finding data from photos, prices, names, store locations and others.

Language: PHP - Size: 559 KB - Last synced at: 4 days ago - Pushed at: almost 4 years ago - Stars: 93 - Forks: 27

serp-spider/core

:spider: The PHP SERP Spider - A search engine scraper

Language: PHP - Size: 514 KB - Last synced at: about 1 month ago - Pushed at: almost 3 years ago - Stars: 92 - Forks: 44

Pringleman83/SportsBook

A sports data scraping and analysis tool

Language: Python - Size: 2.08 MB - Last synced at: over 1 year ago - Pushed at: over 2 years ago - Stars: 74 - Forks: 37

Issung/GChan

Scrape boards & threads from 4chan. Download images, videos and HTML if desired.

Language: C# - Size: 2.16 MB - Last synced at: about 1 month ago - Pushed at: 6 months ago - Stars: 69 - Forks: 10

sanjit-sinha/Bhadoo-Index-Scraper

script to scrape direct download links (ddls) from google drive index.

Language: Python - Size: 15.6 KB - Last synced at: about 2 years ago - Pushed at: about 2 years ago - Stars: 69 - Forks: 70

SilentDemonSD/FZBypassBot

A Elegant Fast Multi Threaded Bypass Bot for Bigger Deeds. Try Now !!

Language: Python - Size: 281 KB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 68 - Forks: 153

lapwat/reCatchable 📦

Turn a site into a book. Download a whole website and upload it to your reMarkable.

Language: JavaScript - Size: 201 KB - Last synced at: 3 months ago - Pushed at: almost 4 years ago - Stars: 67 - Forks: 1

lvxhnat/pyetfdb-scraper

Scrape ETF data for free - the unofficial python wrapper for ETFDB

Language: Python - Size: 174 KB - Last synced at: 16 days ago - Pushed at: about 1 year ago - Stars: 60 - Forks: 7

any4ai/AnyCrawl

AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.

Language: TypeScript - Size: 831 KB - Last synced at: 3 days ago - Pushed at: 3 days ago - Stars: 58 - Forks: 9

vanyasem/VK-Scraper

Scrape VK media

Language: Python - Size: 76.2 KB - Last synced at: 15 days ago - Pushed at: over 1 year ago - Stars: 57 - Forks: 16

Swader/diffbot-php-client

[Deprecated - Maintenance mode - use APIs directly please!] The official Diffbot client library

Language: PHP - Size: 353 KB - Last synced at: 3 months ago - Pushed at: almost 7 years ago - Stars: 53 - Forks: 20

brenns10/tswift

MetroLyrics API for Python

Language: Python - Size: 38.1 KB - Last synced at: 7 days ago - Pushed at: about 1 year ago - Stars: 52 - Forks: 14

NightMachinery/readability-cli Fork of brynedwards/readability-wrapper

A CLI for Mozilla Readability. Get clean, uncluttered, ready-to-read HTML from any webpage!

Language: JavaScript - Size: 70.3 KB - Last synced at: 17 days ago - Pushed at: about 2 years ago - Stars: 51 - Forks: 2

montoyamoraga/scrapers

scrapers for building your own image databases

Language: Python - Size: 16.6 KB - Last synced at: 3 months ago - Pushed at: about 3 years ago - Stars: 51 - Forks: 7

SanjaySunil/email-scraper

Generate thousands of temporary emails within seconds!

Language: Python - Size: 341 KB - Last synced at: 2 months ago - Pushed at: 8 months ago - Stars: 50 - Forks: 8

fritzh321/logo-scrape

🕷🚀 Scrapes/Crawls the logo from a provided url(s)/website for your Node.js applications.

Language: TypeScript - Size: 529 KB - Last synced at: 18 days ago - Pushed at: over 2 years ago - Stars: 48 - Forks: 10

vincetran96/Scrape-Finance-Data-v2 📦

A standalone package to scrape financial data from listed Vietnamese companies via Vietstock

Language: Python - Size: 21.2 MB - Last synced at: almost 2 years ago - Pushed at: over 3 years ago - Stars: 44 - Forks: 27

Obscurely/falion

An open source, programmed in rust, privacy focused tool and crate for interacting with programming resources (like stackoverflow) fast, efficiently and asynchronously/parallel using the CLI or GUI.

Language: Rust - Size: 2.77 MB - Last synced at: 6 days ago - Pushed at: about 2 months ago - Stars: 43 - Forks: 0

schoenbergerb/noscrape 📦

This repository is deprecated

Language: TypeScript - Size: 2.01 MB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 41 - Forks: 8

alias-rahil/lyrics-searcher

A Simple Lyrics Finder That Just Works

Language: TypeScript - Size: 621 KB - Last synced at: 10 days ago - Pushed at: over 2 years ago - Stars: 40 - Forks: 8

ScalaWilliam/amazon-wishlist-api

Scrape Amazon wishlist and provide an API. Play 2.5, JSoup, React.

Language: Scala - Size: 525 KB - Last synced at: 2 days ago - Pushed at: about 9 years ago - Stars: 40 - Forks: 7

projectwallace/extract-css-core

Extract all CSS from a given url, both server side and client side rendered.

Language: JavaScript - Size: 498 KB - Last synced at: 5 days ago - Pushed at: over 2 years ago - Stars: 39 - Forks: 5

typicalninja/google-sr

Scrape google search results without an API key with javascript/typescript.

Language: TypeScript - Size: 1.47 MB - Last synced at: 23 days ago - Pushed at: 23 days ago - Stars: 38 - Forks: 7

swimmingkiim/n8n-nodes-crawl-and-scrape

n8n custom node to crawl and scrape website with crawlee

Language: TypeScript - Size: 160 KB - Last synced at: 8 days ago - Pushed at: about 2 months ago - Stars: 38 - Forks: 12

saadmanrafat/imgur-scraper

Retrieve years of imgur.com's data without any authentication.

Language: Python - Size: 84 KB - Last synced at: 3 days ago - Pushed at: about 2 months ago - Stars: 38 - Forks: 6

stamen/the-ultimate-tile-stitcher

stitch & scrape tiles from slippy map services

Language: Python - Size: 9.77 KB - Last synced at: 3 days ago - Pushed at: 12 months ago - Stars: 36 - Forks: 9

MeoMunDep/MeoMunDep-airdrop-data-base.

Contribute stars if you want me to make scripts fast :)

Language: JavaScript - Size: 77.1 KB - Last synced at: 6 days ago - Pushed at: 7 days ago - Stars: 35 - Forks: 1

searchformyusername/fastProxy

MultiThreaded Application to Scrape Working Web Proxies

Language: Python - Size: 119 KB - Last synced at: 11 days ago - Pushed at: 7 months ago - Stars: 34 - Forks: 10

officialpm/scrape-amazon

🤩 Python Package for Scraping Amazon Product Reviews ✨

Language: Python - Size: 134 KB - Last synced at: 11 months ago - Pushed at: over 2 years ago - Stars: 34 - Forks: 11

pH-7/Crawling-Emails

Very simple bash script to crawl email addresses from a specific website.

Language: Shell - Size: 1.95 KB - Last synced at: 3 months ago - Pushed at: almost 8 years ago - Stars: 34 - Forks: 16

oxylabs/oxylabs-sdk-go

Go SDK for the Oxylabs Scraper APIs.

Language: Go - Size: 171 KB - Last synced at: about 7 hours ago - Pushed at: about 8 hours ago - Stars: 30 - Forks: 5

veltzer/pyscrapers

project to produce various useful scrapers

Language: Python - Size: 2.03 MB - Last synced at: 3 days ago - Pushed at: 3 days ago - Stars: 30 - Forks: 7

squirm-framework/squirm

This was the night of the crawling terror!

Language: Crystal - Size: 21.5 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 30 - Forks: 2

ndgigliotti/shopify-spy

Extract structured data from Shopify websites.

Language: Python - Size: 76.2 KB - Last synced at: over 2 years ago - Pushed at: about 3 years ago - Stars: 30 - Forks: 11

pedro-stanaka/prom-scrape-analyzer

Get insights from your scrape endpoint (even it speaks Protobuf)

Language: Go - Size: 1.08 MB - Last synced at: 2 months ago - Pushed at: 2 months ago - Stars: 29 - Forks: 2

ScriptSmith/insta-scrape

Scrape Instagram

Size: 8.79 KB - Last synced at: 3 months ago - Pushed at: over 6 years ago - Stars: 29 - Forks: 13

defrindr/brainly-scraper

package to scrape www.brainly.co.id

Language: JavaScript - Size: 82 KB - Last synced at: 11 months ago - Pushed at: over 2 years ago - Stars: 28 - Forks: 14

ScoLib/cfscrape

A simple PHP module to bypass Cloudflare's anti-bot page (also known as "I'm Under Attack Mode", or IUAM)

Language: PHP - Size: 23.4 KB - Last synced at: 26 days ago - Pushed at: almost 5 years ago - Stars: 28 - Forks: 15

toolworks-dev/auto-md

Convert Files / Folders / GitHub Repos Into AI / LLM-ready Files

Language: Python - Size: 63.5 KB - Last synced at: 2 months ago - Pushed at: 7 months ago - Stars: 27 - Forks: 1

fernandod1/ProductHunt-scraper

Producthunt.com famous website scraper script. Scrap all offers and save in spreadsheet excel file.

Language: Python - Size: 9.77 KB - Last synced at: 10 days ago - Pushed at: about 3 years ago - Stars: 26 - Forks: 9

shashwatah/gitwiz

A handy portal to query public repos on multiple version control platforms.

Language: TypeScript - Size: 1.59 MB - Last synced at: 3 months ago - Pushed at: about 1 year ago - Stars: 24 - Forks: 1

ohmybahgosh/FONTS_DOT_COM_RIPPER

Script to extract entire font families from Fonts.com, rips them as woff2 and final output includes woff2 and ttf files

Language: Shell - Size: 18.6 KB - Last synced at: 7 days ago - Pushed at: over 3 years ago - Stars: 24 - Forks: 1

ThatSINEWAVE/Spy.pet-Info

This repository serves as an index for all info the community has gathered on the Spy.pet situation and as well as my own tables and tools written for these investigations. Spy.pet was taken down by Discord on 11.08.2024, this is just an archive of what bots where in each server.

Language: JavaScript - Size: 3.5 MB - Last synced at: 22 days ago - Pushed at: 4 months ago - Stars: 23 - Forks: 3

zyrouge/node-youtube-ext

A simple YouTube scraper.

Language: TypeScript - Size: 1.39 MB - Last synced at: 25 days ago - Pushed at: about 1 year ago - Stars: 23 - Forks: 5

pH-7/Emails-Scraper

:ram: Simple PHP Email Grabber to get emails from a txt file containing the list of urls (add one url per line).

Language: PHP - Size: 1.95 KB - Last synced at: 3 months ago - Pushed at: almost 8 years ago - Stars: 23 - Forks: 14

Tomas2D/puppeteer-table-parser

Scrape and parse HTML tables with the Puppeteer table parser.

Language: TypeScript - Size: 1.95 MB - Last synced at: about 11 hours ago - Pushed at: 4 months ago - Stars: 22 - Forks: 3

vitaminarts/Telegram-scraper-api

An easy-to-use PHP api for Getting contents of Telegram's public channels and groups Messages without needing to join them.

Language: PHP - Size: 14.6 KB - Last synced at: about 2 years ago - Pushed at: about 4 years ago - Stars: 22 - Forks: 11

mguideng/gdscrapeR

:package: R package to easily web scrape Glassdoor company reviews. Write up of demo:

Language: R - Size: 327 KB - Last synced at: over 1 year ago - Pushed at: almost 6 years ago - Stars: 22 - Forks: 7

Smartproxy/eCommerce-Scraping-API

eCommerce Scraping API code examples for Python, PHP and Node.js

Language: PHP - Size: 151 KB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 21 - Forks: 4

its0x08/duckduckgo

A simple DuckDuckGo URL scraper.

Language: Python - Size: 43.9 KB - Last synced at: about 1 year ago - Pushed at: over 1 year ago - Stars: 21 - Forks: 3

miroshnikov/scrapyteer

Web crawling & scraping framework for Node.js on top of headless Chrome browser

Language: TypeScript - Size: 384 KB - Last synced at: 3 months ago - Pushed at: over 1 year ago - Stars: 19 - Forks: 0

sertchan/Discord-Scraper

Discord guild members pfp, username and id scraper.

Language: JavaScript - Size: 160 KB - Last synced at: over 2 years ago - Pushed at: almost 3 years ago - Stars: 19 - Forks: 3

etix/goscrape

Go library to scrape torrents on any UDP tracker

Language: Go - Size: 6.84 KB - Last synced at: about 1 year ago - Pushed at: almost 7 years ago - Stars: 19 - Forks: 3

danieldotnl/hass-multiscrape 📦

Home Assistant custom component for scraping multiple values (from a single HTTP request) with a separate sensor for each value.

Language: Python - Size: 39.1 KB - Last synced at: over 2 years ago - Pushed at: about 4 years ago - Stars: 18 - Forks: 4

Topic: "scrape"