data-scraping | Topic | Ecosyste.ms: Repos

je-suis-tm/web-scraping

Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist

Language: Python - Size: 1.88 MB - Last synced at: 22 days ago - Pushed at: over 3 years ago - Stars: 787 - Forks: 177

ScriptSmith/reaper

Social media scraping / data collection tool for the Facebook, Twitter, Reddit, YouTube, Pinterest, and Tumblr APIs

Language: Python - Size: 7.33 MB - Last synced at: 26 days ago - Pushed at: over 6 years ago - Stars: 387 - Forks: 67

Voldrix/onlyfans-dl-2 Fork of k0rnh0li0/onlyfans-dl

OnlyFans content downloader v2

Language: Python - Size: 2.46 MB - Last synced at: 4 months ago - Pushed at: 4 months ago - Stars: 197 - Forks: 26

drshahizan/special-topic-data-engineering

This course presents to the students recent research and industrial issues pertaining to data engineering, database systems and technologies. Various topics of interests that are directly or indirectly affecting or are being influenced by data engineering, database systems and technologies are explored and discussed.

Language: Python - Size: 428 MB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 113 - Forks: 74

dddat1017/Scraping-Youtube-Comments

Scrape comments from any Youtube video

Language: Python - Size: 14.6 KB - Last synced at: 6 months ago - Pushed at: about 4 years ago - Stars: 68 - Forks: 24

zohaibbashir/Google-Maps-Scrapper

This code is used to perform web scraping and data extraction from Google Maps. It is particularly designed for obtaining information about businesses, including their name, address, website, phone number, reviews, and more.

Language: Python - Size: 15.6 KB - Last synced at: 3 months ago - Pushed at: 3 months ago - Stars: 64 - Forks: 25

kb22/GitHub-User-Insights-using-API

The project involves using the GitHub API using user authentication to fetch information such as commits and repositories for that specific user and store them as CSV files for data collection and analysis.

Language: Jupyter Notebook - Size: 192 KB - Last synced at: 2 months ago - Pushed at: over 2 years ago - Stars: 45 - Forks: 31

LakshyaKhatri/Bookshelf-Reader-API

A browsable REST API for recognizing book spines in an image.

Language: Python - Size: 6.01 MB - Last synced at: about 2 months ago - Pushed at: over 4 years ago - Stars: 45 - Forks: 18

serpapi/google-search-results-java

Google Search Results JAVA API via SerpApi

Language: Java - Size: 260 KB - Last synced at: 2 months ago - Pushed at: over 1 year ago - Stars: 43 - Forks: 24

lspahija/torchestrator

Spin up Tor containers and then proxy HTTP requests via these Tor instances

Language: Kotlin - Size: 77.1 KB - Last synced at: about 2 months ago - Pushed at: about 4 years ago - Stars: 43 - Forks: 8

deedy/cbse_schools_data

Cleaned, scraped data of all 20,367 CBSE schools, primarily in India, in 2018. Data scraped from: cbseaff.nic.in/cbse_aff/schdir_Report/userview.aspx

Language: Python - Size: 29.5 MB - Last synced at: over 1 year ago - Pushed at: about 7 years ago - Stars: 43 - Forks: 20

Atharva-Phatak/Analysing-Glassdoor-Jobs

Data Analysis of Job Postings on Glassdoor.

Language: Jupyter Notebook - Size: 4.29 MB - Last synced at: about 1 year ago - Pushed at: almost 3 years ago - Stars: 41 - Forks: 11

MuhammadAmir5670/psx-data-reader

A scraper for downloading Pakistan stock exchange's data into Python Pandas DataFrame.

Language: Python - Size: 365 KB - Last synced at: about 1 month ago - Pushed at: over 2 years ago - Stars: 39 - Forks: 21

SwatiModi/e-commerce-web-scraper

Scraping details of products from ecommerce websites using python

Language: Python - Size: 3.49 MB - Last synced at: over 2 years ago - Pushed at: almost 6 years ago - Stars: 39 - Forks: 11

ohsusannamarie/Instant-Data-Scraper-Chrome-Extension-v0.1.7

Instant Data Scraper packed Chrome extension v0.1.7 (WITH LinkedIn scraping functionality)

Language: JavaScript - Size: 337 KB - Last synced at: about 2 years ago - Pushed at: over 6 years ago - Stars: 37 - Forks: 37

simonw/disaster-data

Data scraped by https://github.com/simonw/disaster-scrapers

Size: 191 MB - Last synced at: about 2 months ago - Pushed at: over 2 years ago - Stars: 36 - Forks: 10

sushil-rgb/AmazonMe

Introducing the AmazonMe webscraper - a powerful tool for extracting data from Amazon.com using the Requests and Beautifulsoup library in Python. This scraper allows users to easily navigate and extract information from Amazon's website.

Language: Python - Size: 894 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 32 - Forks: 13

furkankose/bilkent-scheduler

Bilkent Scheduler is an open-source tool designed to assist students at Bilkent University in planning their course schedules.

Language: JavaScript - Size: 4.95 MB - Last synced at: 3 days ago - Pushed at: 3 days ago - Stars: 31 - Forks: 3

Vyary/exile-profit

This script engages with APIs to collect data and conduct fundamental computations for identifying the optimal approach to generate profits within the game. It also comes with a preconfigured and continuously updating Google Sheet for seamless utilization.

Language: Python - Size: 106 MB - Last synced at: about 9 hours ago - Pushed at: about 11 hours ago - Stars: 25 - Forks: 1

yuis-ice/jseval

Evaluate JavaScript on a URL through headless Chrome browser.

Language: JavaScript - Size: 2.93 KB - Last synced at: 2 months ago - Pushed at: almost 4 years ago - Stars: 25 - Forks: 1

Decodo/Web-Scraping-API

Web Scraping API code examples for Python, PHP and Node.js

Language: JavaScript - Size: 77.1 KB - Last synced at: 28 days ago - Pushed at: about 2 months ago - Stars: 24 - Forks: 9

lidimayra/raspagem-de-dados-fatec

:notebook: Minicurso de raspagem de dados web com Python ministrado na Semana de Tecnologia da FATEC Jundiaí

Language: Jupyter Notebook - Size: 50.8 KB - Last synced at: 3 months ago - Pushed at: 3 months ago - Stars: 24 - Forks: 22

rileynwong/spotify-analysis

Data analysis on my monthly playlists

Language: Jupyter Notebook - Size: 5.09 MB - Last synced at: 2 months ago - Pushed at: over 7 years ago - Stars: 24 - Forks: 3

somdeep/Statball

Statball - Football soccer stats analyser from top 5 european leagues with data obtained by web scraping from Fbref and Statsbomb

Language: C# - Size: 4.69 MB - Last synced at: about 2 months ago - Pushed at: about 2 months ago - Stars: 23 - Forks: 2

pinouche/investment_analysis

automatically fetch value stocks and stocks with unusual option activity (pareto front of iv skew and vrp)

Language: Jupyter Notebook - Size: 4.47 MB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 23 - Forks: 4

ccubc/ChargeUp

optimizing locations of electric vehicle charging stations in the city of Toronto

Language: Jupyter Notebook - Size: 27.6 MB - Last synced at: over 1 year ago - Pushed at: over 2 years ago - Stars: 23 - Forks: 7

chaitanyarahalkar/Financial-Info-Extractor

Extract financial information in CSV format for companies compliant to the NSE

Language: Python - Size: 36.1 KB - Last synced at: 2 months ago - Pushed at: almost 7 years ago - Stars: 22 - Forks: 7

Smartproxy/eCommerce-Scraping-API

eCommerce Scraping API code examples for Python, PHP and Node.js

Language: PHP - Size: 151 KB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 21 - Forks: 4

zero-to-mastery/Complete-Python-Developer-Manual

Class notes for Andrei Neagoie's Complete Python Developer course through Zero to Mastery.

Language: Jupyter Notebook - Size: 60.5 KB - Last synced at: over 2 years ago - Pushed at: over 2 years ago - Stars: 21 - Forks: 21

bilalahhmedd/ebay-products-scraper

This is ebay products scrapper to scrap images and metadata details of products from ebay.com

Language: Python - Size: 88.9 KB - Last synced at: 4 months ago - Pushed at: 4 months ago - Stars: 19 - Forks: 8

LakshyaKhatri/Bookshelf-Reader

An android application that provides information for books kept in a bookshelf by recognizing their spines from images.

Language: Java - Size: 7.86 MB - Last synced at: about 2 months ago - Pushed at: over 3 years ago - Stars: 18 - Forks: 9

wingkwong/hk-atm-locator

:atm: 香港自動櫃員機定位器 :atm: Centralising Automated Teller Machine (ATM) Data in Hong Kong in a well-defined yet standardised format and display in a web portal for public use

Language: JavaScript - Size: 11.1 MB - Last synced at: 12 days ago - Pushed at: over 1 year ago - Stars: 17 - Forks: 7

easonlai/facebook_post_scraping_and_text_analytics

This is demo repo to demostrate how to scrape post data from Facebook by Python with library facebook_scraper. And then use Azure Text Analytics to perform sentiment analysis for post text content.

Language: Jupyter Notebook - Size: 1.98 MB - Last synced at: 20 days ago - Pushed at: over 3 years ago - Stars: 17 - Forks: 14

yashrane/DiscordScraper

Collects and Analyzes textual data scraped from a Discord chat server

Language: Python - Size: 68.9 MB - Last synced at: over 2 years ago - Pushed at: about 7 years ago - Stars: 17 - Forks: 2

slavaspirin/Toronto-housing-price-prediction

Building Toronto Housing dataset from scratch to predict real estate prices

Language: Jupyter Notebook - Size: 23.4 MB - Last synced at: over 2 years ago - Pushed at: over 5 years ago - Stars: 16 - Forks: 8

Smartproxy/SERP-Scraping-API

SERP Scraping API code examples for Python, PHP and Node.js

Language: PHP - Size: 345 KB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 15 - Forks: 4

CharlyWargnier/S4_wiki_topic_grapher

Leverage the power of the Google Natural Language API NLP to retrieve entity relationships from Wikipedia URLs or topics! Get interactive networkx graphs of connected entities!

Language: Python - Size: 103 KB - Last synced at: 2 months ago - Pushed at: almost 4 years ago - Stars: 15 - Forks: 7

saoun/dataviz

Tips, Cheat Sheets, Resources for Data Visualization

Size: 534 KB - Last synced at: about 1 year ago - Pushed at: over 7 years ago - Stars: 13 - Forks: 2

mzubairtahir/Google-Maps-Scraper

Scraper that scrapes data from search results of google maps

Language: Python - Size: 515 KB - Last synced at: 2 months ago - Pushed at: over 1 year ago - Stars: 12 - Forks: 2

dstark5/gnews-scraper

GNewsScraper is a TypeScript package that scrapes article data from Google News based on a keyword or phrase. It returns the results as an array of JSON objects, making it convenient to access and use the scraped information

Language: TypeScript - Size: 153 KB - Last synced at: 21 days ago - Pushed at: almost 2 years ago - Stars: 12 - Forks: 3

asoul-sig/acao

🦙 阿草，数据爬取工具人 / acao, the tool man for data scraping of https://asoul.video/.

Language: Go - Size: 230 KB - Last synced at: 2 months ago - Pushed at: almost 2 years ago - Stars: 12 - Forks: 2

LoveNui/Tableau-Scraping

Language: Python - Size: 142 KB - Last synced at: 2 months ago - Pushed at: almost 2 years ago - Stars: 12 - Forks: 0

bongohead/econforecasting-r

This is the data scraping & modeling code used for models shown in https://econforecasting.com.

Language: R - Size: 687 MB - Last synced at: 7 days ago - Pushed at: over 1 year ago - Stars: 11 - Forks: 1

kuleafenu/customizable-web-crawler

This web crawler can be customized to scrape almost all types of websites.

Language: Python - Size: 231 KB - Last synced at: about 2 years ago - Pushed at: over 3 years ago - Stars: 11 - Forks: 4

easonlai/playstore_reviews_scraping_and_text_analytics

This is demo repo to demostrate how to scrape apps review data from Google Play Store by Python with library Google-Play-Scraper. And then use Azure Text Analytics to perform sentiment analysis for reviews content (aka comments).

Language: Jupyter Notebook - Size: 105 KB - Last synced at: 19 days ago - Pushed at: almost 4 years ago - Stars: 11 - Forks: 3

BBottoml/Letterboxd-friend-ranker

Program that computes, ranks a given user and their friends based on Letterboxd ratings

Language: Python - Size: 55.7 KB - Last synced at: over 2 years ago - Pushed at: over 2 years ago - Stars: 10 - Forks: 1

fdd4s/matterport-downloader

Download Skybox panoramic photos of Matterport houses

Language: PHP - Size: 37.1 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 9 - Forks: 0

sbmagar13/raidfscrape

Code to Scraping some portion of Data from forum(RaidForums[seized by FBI]) with Python SCRAPY spiders bypassing recaptcha and storing to PostgreSQL database.(used scrapper-API as captcha, proxy bypass) SQLAlchemy as ORM for PostgreSQL - Python.

Size: 482 KB - Last synced at: 4 days ago - Pushed at: over 4 years ago - Stars: 9 - Forks: 3

Yuanchao-Xu/gfer

Green Finance and Environmental Risk

Language: R - Size: 563 KB - Last synced at: 7 days ago - Pushed at: 3 months ago - Stars: 8 - Forks: 2

GithubRealFan/QT-Desktop-Application-For-Music-Collection

Data Scraping Project using QT

Language: C++ - Size: 169 KB - Last synced at: about 1 year ago - Pushed at: about 2 years ago - Stars: 8 - Forks: 0

sallamy2580/python-web-scrapping

Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist

Language: Python - Size: 1.56 MB - Last synced at: over 2 years ago - Pushed at: almost 3 years ago - Stars: 8 - Forks: 0

kaustubhgupta/blogathon-analysis

Analytics Vidhya Blogathon Data Analysis: Python Data Extraction with PowerBI dashboard

Language: Jupyter Notebook - Size: 1.91 MB - Last synced at: 2 months ago - Pushed at: over 3 years ago - Stars: 8 - Forks: 2

changwookjun/DataScrapingCrawling

Data Scraping 정리 자료

Language: HTML - Size: 81.6 MB - Last synced at: 2 months ago - Pushed at: over 6 years ago - Stars: 8 - Forks: 6

BrianWeiHaoMa/MISOReports

A comprehensive Python library for downloading Midcontinent Independent System Operator (MISO) public reports into pandas dataframes.

Language: Python - Size: 448 KB - Last synced at: 16 days ago - Pushed at: 3 months ago - Stars: 7 - Forks: 0

leabs/misc-python-apps

Miscellaneous Python applications or scripts I've made to help me learn Python and use as productivity tools.

Language: Python - Size: 56.6 KB - Last synced at: 5 days ago - Pushed at: 6 months ago - Stars: 7 - Forks: 0

duttashi/scrapers

This repo is for web data scraping. Feel free to show your :heart: by giving a star :star:

Language: Python - Size: 15.8 MB - Last synced at: 8 months ago - Pushed at: 8 months ago - Stars: 7 - Forks: 3

abtsousa/clippy

A simple file downloader / scraper for NOVA School of Science and Technology's own e-learning platform, CLIP.

Language: Python - Size: 4.54 MB - Last synced at: 9 months ago - Pushed at: 9 months ago - Stars: 7 - Forks: 1

nqtri/SSENSE-Product-Recommendation-System

A passion project on my favorite e-commerce site that scrapes product data and builds a recommendation engine

Language: Jupyter Notebook - Size: 38.6 MB - Last synced at: 2 months ago - Pushed at: about 2 years ago - Stars: 7 - Forks: 1

stalinsantillan/laravel-soccer-compare

Soccer player management PWA

Language: Blade - Size: 17.7 MB - Last synced at: 2 months ago - Pushed at: about 2 years ago - Stars: 7 - Forks: 2

sondosaabed/Palestinian-Movies-JSON-Dataset

Github allows me to call a JSON file like a server. So this is my raw data scrapped using apify (imdb-scrapper) 🇵🇸

Size: 44.9 KB - Last synced at: 4 months ago - Pushed at: over 1 year ago - Stars: 6 - Forks: 0

ryanrudes/wikimedia

A dataset comprised of over 40 million images sourced from Wikimedia Commons

Language: Python - Size: 14.6 KB - Last synced at: 7 days ago - Pushed at: over 1 year ago - Stars: 6 - Forks: 0

TravisGibbs/SportsCardToolLib

A python library for grabbing and tracking checklists for sports cards.

Language: Python - Size: 2.17 MB - Last synced at: 10 months ago - Pushed at: almost 2 years ago - Stars: 6 - Forks: 2

SiddeshSambasivam/NTUOSS-DataScraping-And-DataCleaning-Workshop

This repository contains the reference scripts and the content presented in the NTU OSS Data scraping and Data cleaning workshop.

Language: Jupyter Notebook - Size: 2.47 MB - Last synced at: 14 days ago - Pushed at: over 4 years ago - Stars: 6 - Forks: 1

erkurn/locanesia

Laravel Package to populate and searching indonesia locations. From Provinces To Villages. Data Scraped from http://www.nomor.net/

Language: PHP - Size: 2.66 MB - Last synced at: 6 months ago - Pushed at: about 7 years ago - Stars: 6 - Forks: 5

ad1m/derivative_code

A repository of code on my derivative blog

Language: Python - Size: 818 KB - Last synced at: over 2 years ago - Pushed at: over 8 years ago - Stars: 6 - Forks: 2

bitArtisan1/netDigger

A .NET 8.0 C# WPF desktop application for web scraping data into structured databases with a modern UI, comprehensive logging and optimized high performance.

Language: C# - Size: 375 KB - Last synced at: 2 months ago - Pushed at: 10 months ago - Stars: 5 - Forks: 1

opencodeiiita/News_Scraping 📦

Language: Jupyter Notebook - Size: 139 KB - Last synced at: 3 months ago - Pushed at: over 1 year ago - Stars: 5 - Forks: 22

KalyanM45/End-to-End-Image-Scraping

Image Scraper Web App is a simple Flask-based web application that enables users to search for images on Google and download them. The app scrapes the search results page for images and saves the downloaded images to a specified directory. It provides a user-friendly interface to input search queries and view the downloaded images.

Language: Jupyter Notebook - Size: 414 KB - Last synced at: about 1 year ago - Pushed at: almost 2 years ago - Stars: 5 - Forks: 0

mzubairtahir/Latest-Twitter-Scraper

This python scraper is for latest twitter website structure , that scrapes tweets of an twitter account

Language: Python - Size: 28.3 KB - Last synced at: 2 months ago - Pushed at: about 2 years ago - Stars: 5 - Forks: 0

temcavanagh/Crypto-prediction

A predictive model for forecasting the hourly Bitcoin price movements using sentiment analysis drawn from extracted from Twitter data and tree based algorithms. The resulting predictive model utilises an XGBoost gradient boosted random forest model which demonstrates a 63.16% accuracy in predicting the hourly price movements of Bitcoin.

Language: Jupyter Notebook - Size: 1.32 MB - Last synced at: almost 2 years ago - Pushed at: over 3 years ago - Stars: 5 - Forks: 1

ds4stats/r-tutorials

Language: HTML - Size: 16.9 MB - Last synced at: over 1 year ago - Pushed at: almost 4 years ago - Stars: 5 - Forks: 18

copev313/Extract-Stock-Sentiment-From-News-Headlines

We scrape news headlines for FB and TSLA then apply sentiment analysis to generate investment insight.

Language: HTML - Size: 395 KB - Last synced at: about 2 years ago - Pushed at: over 4 years ago - Stars: 5 - Forks: 7

FoamoftheSea/tda_scraper

A web-scraper which uses selenium webdriver to obtain in-depth data from TD Ameritrade website for use with python.

Language: Jupyter Notebook - Size: 190 KB - Last synced at: about 1 year ago - Pushed at: about 5 years ago - Stars: 5 - Forks: 3

TudorGruian/alerta-seism-python3

O librarie care foloseste date XML de la INFP pentru a afla magnitudinea la secunda.

Language: Python - Size: 27.3 KB - Last synced at: about 1 month ago - Pushed at: over 6 years ago - Stars: 5 - Forks: 2

youmeat6678/Instagram-Hashtag-Scraper

A Python project for scraping Instagram posts based on specific hashtags. This tool uses Selenium for browser automation and BeautifulSoup for scraping content from Instagram. It's designed to fetch posts by hashtag, with customizable options to automate login, scrape content, and interact with Instagram posts.

Language: Python - Size: 35.2 KB - Last synced at: about 5 hours ago - Pushed at: about 6 hours ago - Stars: 4 - Forks: 0

GuechtouliAnis/Football-Data-Scraping

A Jupyter Notebook project to scrape detailed football data from FBref, covering various leagues. The repository includes both the data collection process and a final cleaned dataset for immediate use in analysis. Ideal for football data science projects, including xG, passing metrics, and tactical insights.

Language: Jupyter Notebook - Size: 7.36 MB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 4 - Forks: 0