GitHub topics: website-scraper

Repositories

Kooboo/Kooboo

CMS, WebSite, Application and Ecommerce Development Tool Using JavaScript

Language: C# - Size: 69.9 MB - Last synced at: 3 days ago - Pushed at: 3 days ago - Stars: 326 - Forks: 101

website-scraper/website-scraper-puppeteer

Plugin for website-scraper which returns html for dynamic websites using puppeteer

Language: JavaScript - Size: 96.7 KB - Last synced at: 1 day ago - Pushed at: 10 days ago - Stars: 332 - Forks: 80

website-scraper/node-website-scraper

Download website to local directory (including all css, images, js, etc.)

Language: JavaScript - Size: 781 KB - Last synced at: 23 minutes ago - Pushed at: 24 days ago - Stars: 1,603 - Forks: 284

goclone-dev/goclone

Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.

Language: Go - Size: 123 MB - Last synced at: 9 days ago - Pushed at: 10 days ago - Stars: 1,503 - Forks: 310

shurco/goClone

🌱 goClone - clone websites in seconds

Language: Go - Size: 7.1 MB - Last synced at: 10 days ago - Pushed at: 10 days ago - Stars: 69 - Forks: 4

CRAKZOR/linkedin-post-automator

Automatically curates and posts content to LinkedIn. It can optionally use web scraping to gather data, which is then fed to ChatGPT to craft engaging LinkedIn posts.

Language: Python - Size: 34.2 KB - Last synced at: 11 days ago - Pushed at: 11 days ago - Stars: 74 - Forks: 21

OSINT-TECHNOLOGIES/dpulse

DPULSE - Tool for complex approach to domain OSINT

Language: Python - Size: 1.32 MB - Last synced at: 10 days ago - Pushed at: 10 days ago - Stars: 110 - Forks: 6

josephlimtech/linkedin-profile-scraper-api

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON.

Language: TypeScript - Size: 10.8 MB - Last synced at: 5 days ago - Pushed at: about 1 year ago - Stars: 625 - Forks: 162

html2rss/html2rss-web

🕸 Generates RSS feeds of any website & serves to the web! Automatic scraping. Ready to use configs. Write your own. Rolling Docker releases for speedy updates.

Language: Ruby - Size: 994 KB - Last synced at: 12 days ago - Pushed at: 12 days ago - Stars: 102 - Forks: 13

MikkoVasankari/7TV_scrape

Scraping https://7tv.app/ webiste for emote links

Language: Go - Size: 1.82 MB - Last synced at: 13 days ago - Pushed at: 13 days ago - Stars: 0 - Forks: 0

Uscrapper Vanta: Dive deeper into the web with this powerful open-source tool. Extract valuable insights with ease and efficiency, from both surface and deep web sources. Empower your data mining and analysis with Vanta's advanced capabilities. Fast, reliable, and user-friendly, Uscrapper Vanta is the ultimate choice for researchers and analysts.

Language: Python - Size: 438 KB - Last synced at: 17 days ago - Pushed at: 5 months ago - Stars: 583 - Forks: 62

website-scraper/website-scraper-existing-directory

Plugin for website-scraper which allows to save resources to existing directory

Language: JavaScript - Size: 58.6 KB - Last synced at: 2 days ago - Pushed at: 5 months ago - Stars: 13 - Forks: 5

Pythoript/email-scraper

Scrape emails from a website using recursive crawling, the best anti-obfuscation techniques, and validate all addresses before saving to a file.

Language: Go - Size: 22.5 KB - Last synced at: 25 days ago - Pushed at: 25 days ago - Stars: 2 - Forks: 1

erlange/wbm-dl

Wayback Machine Downloader. 🔥 Download your entire archived websites from the Internet Archive Wayback Machine.

Language: C# - Size: 295 KB - Last synced at: 19 days ago - Pushed at: over 2 years ago - Stars: 96 - Forks: 16

kajoty/NDR-Webseiten-Scraper

NDR Radio Playlist Scraper. Ein einfacher Webscraper, der die Playlists von ndr.de herunterlädt und in einer Datenbank speichert. Nutzer können anschließend verschiedene Auswertungen auf die Daten durchführen.

Language: Python - Size: 9.69 MB - Last synced at: 27 days ago - Pushed at: 28 days ago - Stars: 0 - Forks: 0

LexiestLeszek/scrapeGPT

ScrapeGPT is a RAG-based Telegram bot designed to scrape and analyze websites, then answer questions based on the scraped content. The bot utilizes Retrieval Augmented Generation and webscraping to return natural language answers to the user's queries.

Language: Python - Size: 62.5 KB - Last synced at: 14 days ago - Pushed at: about 1 year ago - Stars: 82 - Forks: 12

waived/4chan-ripper 📦

Download all resources from 4chan thread | multiple-passes | monitor thread for new content

Language: Python - Size: 75.2 KB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 0 - Forks: 0

TnYtCoder/Url_Checker

Python Script To Verify Url Existence And Provide Basic Information

Language: Python - Size: 21.5 KB - Last synced at: 11 days ago - Pushed at: over 1 year ago - Stars: 4 - Forks: 0

zebbern/ReconX

🕷️ | ReconX is a Live-Website Crawler made to gather critical information with an option to take a picture of each site crawled!

Language: Python - Size: 57.6 KB - Last synced at: 2 days ago - Pushed at: 2 months ago - Stars: 3 - Forks: 0

David0z/Wayback-Machine-Downloader

Terminal UI tool for collective download of resources from Archive.org Wayback Machine

Language: Go - Size: 43.6 MB - Last synced at: 2 months ago - Pushed at: 2 months ago - Stars: 0 - Forks: 0

xarantolus/Collect

A server to collect & archive websites that also supports video downloads

Language: TypeScript - Size: 2.07 MB - Last synced at: about 16 hours ago - Pushed at: about 2 years ago - Stars: 86 - Forks: 13

PinsaraPerera/Selenium_Scraping_bot

This is my custom build scraping bot which is running on google cloud vm. It is capable of scraping the web without getting blocked since its using IP pool to change the IP addresses.

Language: Jupyter Notebook - Size: 146 KB - Last synced at: 8 days ago - Pushed at: 10 months ago - Stars: 1 - Forks: 0

HXCKYR/Osint-Eye

OSINT Tool for Email id (breached) | Website Scrapper | Instagram id | Linkedin Profile or .... ; )

Language: Python - Size: 237 KB - Last synced at: 3 months ago - Pushed at: 3 months ago - Stars: 3 - Forks: 0

dsc8x/node-scraper

Scraping websites made easy! A minimalistic yet powerful tool for collecting data from websites.

Language: JavaScript - Size: 143 KB - Last synced at: 1 day ago - Pushed at: over 6 years ago - Stars: 10 - Forks: 0

methylDragon/fb_embedded_comment_scraper

A scraper for gathering data from Facebook's embedded comment widgets for all pages on any number of URLs! It bypasses the Facebook graph API (you don't need an access token) so there's little risk of throttling.

Language: Python - Size: 21.8 MB - Last synced at: about 1 month ago - Pushed at: about 7 years ago - Stars: 1 - Forks: 1

Chorozon666/Chad

Chad - Dorking / Website Vulnerability Tool

Language: Python - Size: 13.7 KB - Last synced at: 12 days ago - Pushed at: 5 months ago - Stars: 2 - Forks: 0

ioreshnikov/tamizdat

Yet another telegram bot for flibusta library.

Language: HTML - Size: 119 KB - Last synced at: 1 day ago - Pushed at: 3 months ago - Stars: 2 - Forks: 0

developer-sumit/web-groper-python

WebGroper is a Python class designed to recursively scrape and download media files (images, PDFs, etc.) from a specified website directory, such as the /wp-content/uploads directory of a WordPress site.

Language: Python - Size: 4.88 KB - Last synced at: 10 days ago - Pushed at: 6 months ago - Stars: 0 - Forks: 1

1970Mr/link-crawler

Web Link Crawler: A Python script to crawl websites and collect links based on a regex pattern. Efficient and customizable.

Language: Python - Size: 32.2 KB - Last synced at: about 1 month ago - Pushed at: 11 months ago - Stars: 2 - Forks: 1

ipe9/BruteDir

Language: Python - Size: 4.88 KB - Last synced at: 6 months ago - Pushed at: 6 months ago - Stars: 0 - Forks: 0

maheshpaulj/Manga-Scraper

Manga Scraping Tool made in python, It fetches the manga page from the website and downloads it in JPG format and saves it locally. This is basically web Scraping

Language: Python - Size: 4.88 KB - Last synced at: 2 months ago - Pushed at: over 1 year ago - Stars: 2 - Forks: 1

codassassin/website-url-scraper

This is a website url scraper built using python.

Language: Python - Size: 24.4 KB - Last synced at: 5 months ago - Pushed at: over 3 years ago - Stars: 8 - Forks: 1

pwaaron/jobscrapers

Scrapes job profiles of popular job portals in Singapore.

Language: Jupyter Notebook - Size: 4.88 KB - Last synced at: 7 days ago - Pushed at: over 4 years ago - Stars: 1 - Forks: 2

MehediHasannOvi/Jamuna-TV-app--Flutter-Web-Scraper

Jamuna TV App is a Flutter-based web scraping application designed to retrieve post image URLs from the Jamuna TV website. This app provides a seamless experience for users on multiple platforms, including Android, Windows, Linux, and iOS.

Language: C++ - Size: 810 KB - Last synced at: 8 months ago - Pushed at: 8 months ago - Stars: 0 - Forks: 0

cometolearnofficial/WebHawk

Website Penetration Testing Tool With Dos Attack Feature

Language: Python - Size: 8.79 KB - Last synced at: 3 months ago - Pushed at: over 4 years ago - Stars: 16 - Forks: 1

methylDragon/news-anaCrawler

Article Dataset Generator for Internet News Sites. Crawls news sites, analyses them with NLP (sentiment analysis), and pushes to a database.

Language: Jupyter Notebook - Size: 299 KB - Last synced at: 15 days ago - Pushed at: over 7 years ago - Stars: 8 - Forks: 1

psynt/DataShuffle 📦

A data visualisation tool we wrote for a uni project. It scrapes data off websites and helps the user sort through it at a glance

Language: HTML - Size: 9.58 MB - Last synced at: 9 months ago - Pushed at: almost 8 years ago - Stars: 1 - Forks: 0

MLArtist/WebScraper

Python-based web crawling script with randomized intervals, user-agent rotation, and proxy server IP rotation to outsmart website bots and prevent blocking.

Language: Python - Size: 43.9 KB - Last synced at: 9 months ago - Pushed at: 9 months ago - Stars: 61 - Forks: 14

govardhanvembadi/WebScraping

Web scraping using Scrapy framework is often the most efficient way to gather structured data from websites, and this project showcases its power and flexibility in action.

Language: Python - Size: 72.3 KB - Last synced at: 10 months ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0

rehanjavedofficial/PSL-T20-2017-Match-Details

All match details of Pakistan Super League teams. All data is scraped from www.psl-t20.com website using python.

Language: Python - Size: 15.6 KB - Last synced at: 10 months ago - Pushed at: over 7 years ago - Stars: 1 - Forks: 0

dtflare/GPTparser

Use GPTparser with your OpenAI API to scrape & parse files into structured JSON files.

Language: Python - Size: 173 KB - Last synced at: 9 months ago - Pushed at: about 1 year ago - Stars: 9 - Forks: 0

Hoccyy/WebGrabb

A simple and efficient tool written in Go for fetching and saving websites.

Language: Go - Size: 244 KB - Last synced at: about 1 month ago - Pushed at: 10 months ago - Stars: 0 - Forks: 0

website-scraper/node-website-scraper-phantom 📦

Plugin for website-scraper which returns html for dynamic websites using PhantomJS.

Language: JavaScript - Size: 17.6 KB - Last synced at: 5 days ago - Pushed at: over 3 years ago - Stars: 59 - Forks: 14

cyterat/mil-products-aggregator

Data scraping and aggregation telegram bot. Retrieves military gear from various Ukrainian online stores.

Language: Python - Size: 6.69 MB - Last synced at: 11 months ago - Pushed at: 11 months ago - Stars: 0 - Forks: 0

jeanrauwers/followers-scraper-serverless

Now you can keep track of your followers from YouTube, Instagram and Twitter accounts - Followers scraper API on AWS serverless

Language: TypeScript - Size: 2.51 MB - Last synced at: 14 days ago - Pushed at: about 2 years ago - Stars: 18 - Forks: 2

yuis-ice/jseval

Evaluate JavaScript on a URL through headless Chrome browser.

Language: JavaScript - Size: 2.93 KB - Last synced at: 13 days ago - Pushed at: almost 4 years ago - Stars: 25 - Forks: 1

nigeld3v/Tumblr_Image_scrape

Download ALL the images (JPEG/GIF/PNG) from any Tumblr website! This project employs Python3 and BeautifulSoup4 to scrape a Tumblr site (with the url provided by the user) to download, page by page, all the images from the Tumblr site's posts. Ideal for archiving other peoples' Tumblrs <3

Language: Python - Size: 43 KB - Last synced at: 21 days ago - Pushed at: about 7 years ago - Stars: 9 - Forks: 4

filesite-io/machete_hero

为Ta荐（TaJian.tv）工作的基于Hero的Node.js爬虫程序，可抓取B站、抖音、快手、西瓜视频播放页、直播页的标题和封面图

Language: JavaScript - Size: 57.6 KB - Last synced at: 11 months ago - Pushed at: 11 months ago - Stars: 2 - Forks: 0

EthoKikon/Stock-Prediction

Sentiment-driven stock market prediction

Language: Jupyter Notebook - Size: 939 KB - Last synced at: 12 months ago - Pushed at: 12 months ago - Stars: 1 - Forks: 0

faheel/file-extensions

JSON collection of scraped file extensions, along with their description and type, from FileInfo.com

Language: Python - Size: 223 KB - Last synced at: 15 days ago - Pushed at: over 2 years ago - Stars: 18 - Forks: 6

vlmaier/marvel-snap-scrapr

Scraper for https://marvelsnapzone.com to retrieve metadata of Marvel SNAP cards.

Language: Python - Size: 31.3 KB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 21 - Forks: 5

sansagara/ipvu_trends_dashboard

Python application that scraps diverse sources for covid-19 papers, applies NLP transformations and stores them in a dataset for visualizing on a Flask web application.

Language: Jupyter Notebook - Size: 25.7 MB - Last synced at: 6 days ago - Pushed at: over 2 years ago - Stars: 3 - Forks: 2

Nidhal-Bouzara/usthb-news

📰 (Live 🚀 - Check Link 🔗) University News Aggregation Website 👉 Built with AdonisJS, InertiaJS ( SSR React with Typescript, routing done server side ), MYSQL (Lucid ORM), and puppeteer.

Language: JavaScript - Size: 1.73 MB - Last synced at: about 1 year ago - Pushed at: over 3 years ago - Stars: 0 - Forks: 0

theSTremblay/GPT-Image-Scraper

A robust Image Scraper that leverages OpenAI's GPT Chat Completions to determine the relevant HTML used to Scrape Images from websites.

Language: Python - Size: 9.99 MB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 1 - Forks: 0

Mar-Issah/site_assistant_ai

Transform your website into a dynamic and interactive platform with SiteAssistant AI. Built with Python, Streamlit, LangChain, Openai - GPT 3.5

Language: Python - Size: 274 KB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 0 - Forks: 0

PickySalamander/website-alerter

Tool for alerting when a website changes

Language: TypeScript - Size: 1.05 MB - Last synced at: 14 days ago - Pushed at: 14 days ago - Stars: 0 - Forks: 0

orangmuda/SECTOOL

sᴇᴀʀᴄʜ ᴇɴɢɪɴᴇ sᴄʀᴀᴘᴇʀ ᴛᴏᴏʟ (ʙᴀsʜ)

Language: Shell - Size: 14.6 KB - Last synced at: 5 days ago - Pushed at: over 3 years ago - Stars: 12 - Forks: 4

martapanc/aoc-data-api

API to retrieve stars obtained by year from the personal account on adventofcode.com

Language: Kotlin - Size: 230 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0

antiquarybedna/scam-Drainer

scam Drainer

Size: 1000 Bytes - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0

enrycol/Sushiswap-Drainer

Sushiswap Drainer

Size: 1000 Bytes - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0

rilyndodg/website-Drainer

website Drainer

Size: 1000 Bytes - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0

hainecle/website-Drainer

website Drainer

Size: 1000 Bytes - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0

TheEmilyAves/bnao-diet-scraper

Uses Scrapy to scrape diet data from Birds of North America Online

Language: Python - Size: 9.77 KB - Last synced at: over 1 year ago - Pushed at: almost 5 years ago - Stars: 0 - Forks: 0

linusrachlis/fringr2-scraper

Scraper for show info and performance times on the Toronto Fringe website. Used for linusrachlis/fringr2-fe

Language: PHP - Size: 8.79 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 1 - Forks: 0

SamuraiPolix/openbible-verse-scraper

This script scrapes the verses and references from an openbible.info page into a JSON file - if needed, we use bible-api.com to translate to another bible version.

Language: Python - Size: 177 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0

rtk-rnjn/cricbuzz_scraper 📦

A simple async-website-scraper for cricket score.

Language: Python - Size: 11.8 MB - Last synced at: 11 months ago - Pushed at: about 3 years ago - Stars: 2 - Forks: 0

hudson-newey/Website-Text-Extractor

This is a project to systematically extract all readable text out of a web page (only works on very primitive pages at the moment)

Language: Ruby - Size: 5.86 KB - Last synced at: 2 days ago - Pushed at: over 2 years ago - Stars: 2 - Forks: 0

AnirudhaPatil-1/Github-Scrapper

Scrap the github for the treding topics. Dive in them. Find top n repos. Dive in and scrap issues. Gather issues and put them in pdf named after repo within a folder named after Topic.

Language: JavaScript - Size: 44.9 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 1 - Forks: 0

Ashwin-op/Email-Extractor

A spider to crawl webpages

Language: Python - Size: 4.88 KB - Last synced at: over 1 year ago - Pushed at: about 5 years ago - Stars: 18 - Forks: 3

hamna-moieez/Gogo-Anime-Downloader

Download all anime episodes from gogoanime simultaneously. This is provided as a Python package.

Language: Python - Size: 2.93 KB - Last synced at: almost 2 years ago - Pushed at: about 2 years ago - Stars: 0 - Forks: 0

SySyAli/mosqueswebscraping

This Python script scrapes Salatomatic for US masjid data, including names, locations, and phone numbers. It uses requests, BeautifulSoup, and csv modules for web scraping and CSV handling.

Language: Python - Size: 86.9 KB - Last synced at: almost 2 years ago - Pushed at: almost 2 years ago - Stars: 1 - Forks: 0