GitHub topics: scrapy-crawler

Repositories

NguyenDa18/Portland-Jail-Data-Crawler

Scraper used for recording changes to Portland jail database

Language: Jupyter Notebook - Size: 40.4 MB - Last synced at: about 6 hours ago - Pushed at: about 7 hours ago - Stars: 5 - Forks: 0

Huolix/web-scraping-beautifulsoup

A Python project that scrapes product data using BeautifulSoup

Language: Jupyter Notebook - Size: 9.77 KB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 0 - Forks: 0

heitornolla/WebScraping-MercadoLivre

This project performs Web Scraping on Mercado Livre's website, scraping data related to 5 string basses. It then cleans this data and turns it into a SQL database to be loaded on Streamlit. This can be easily adapted for other items.

Language: Python - Size: 213 KB - Last synced at: 9 days ago - Pushed at: 9 days ago - Stars: 1 - Forks: 0

Saptha-Harsh/LilHomie

A Machine Learning Project implemented from scratch which involves web scraping, data engineering, exploratory data analysis and machine learning to predict housing prices in New York Tri-State Area.

Language: Jupyter Notebook - Size: 2.22 MB - Last synced at: 9 days ago - Pushed at: 9 days ago - Stars: 0 - Forks: 0

rukshar69/scraping-projects

Scraping job lists from careerjet using Scrapy and Cohere LLM AI

Language: Python - Size: 556 KB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 0 - Forks: 0

wkunzhi/Python3-Spider

Python爬虫实战 - 模拟登陆各大网站包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

Language: Python - Size: 41.6 MB - Last synced at: 28 days ago - Pushed at: over 1 year ago - Stars: 3,201 - Forks: 1,028

lkuffo/web-scraping

Más de 50 ejemplos de web scraping utilizando: Requests | Scrapy | Selenium | LXML | BeautifulSoup

Language: Python - Size: 60.6 MB - Last synced at: about 1 month ago - Pushed at: 9 months ago - Stars: 361 - Forks: 208

pritpalxyz/souq

Scrapy Crawler for souq Website

Language: Python - Size: 12.7 KB - Last synced at: 7 days ago - Pushed at: over 8 years ago - Stars: 6 - Forks: 3

ZoranPandovski/ProdirectScraper

:necktie: Web scraper for http://www.prodirectselect.com/ :shoe:

Language: Python - Size: 116 KB - Last synced at: 12 days ago - Pushed at: 12 days ago - Stars: 17 - Forks: 9

666zyb/small-exercise-projects

网络爬虫项目，非常适合爬虫入门和新手练习的项目，收集了多个不同场景下的爬虫示例，包括电影信息抓取、图片爬取等。

Language: JavaScript - Size: 999 KB - Last synced at: about 2 months ago - Pushed at: about 2 months ago - Stars: 3 - Forks: 0

armanjscript/Web-Scraping

Web Scraping projects with Python

Language: Python - Size: 1.98 MB - Last synced at: 22 days ago - Pushed at: about 1 year ago - Stars: 2 - Forks: 0

waahah/async-spider

知乎盐选会员文章下载pdf

Language: Python - Size: 4.48 MB - Last synced at: 6 days ago - Pushed at: almost 2 years ago - Stars: 53 - Forks: 20

aydinnyunus/ScrapIn

LinkedIn Scraper

Language: Python - Size: 3.91 KB - Last synced at: about 2 months ago - Pushed at: over 5 years ago - Stars: 18 - Forks: 4

desiquant/news_scraper

Scrapes indian market news data

Language: Python - Size: 194 KB - Last synced at: 3 months ago - Pushed at: 3 months ago - Stars: 5 - Forks: 3

zhangslob/docs

《数据采集从入门到放弃》源码。内容简介：爬虫介绍、就业情况、爬虫工程师面试题；HTTP协议介绍； Requests使用；解析器Xpath介绍； MongoDB与MySQL；多线程爬虫； Scrapy介绍；Scrapy-redis介绍；使用docker部署；使用nomad管理docker集群；使用EFK查询docker日志

Size: 1.16 MB - Last synced at: 12 days ago - Pushed at: about 6 years ago - Stars: 138 - Forks: 30

kezhenxu94/house-renting 📦

Possibly the best practice of Scrapy 🕷 and renting a house 🏡

Language: Python - Size: 1.43 MB - Last synced at: 2 days ago - Pushed at: over 3 years ago - Stars: 795 - Forks: 143

Viveckh/LilHomie

A Machine Learning Project implemented from scratch which involves web scraping, data engineering, exploratory data analysis and machine learning to predict housing prices in New York Tri-State Area.

Language: Jupyter Notebook - Size: 10.5 MB - Last synced at: 3 months ago - Pushed at: over 2 years ago - Stars: 92 - Forks: 19

farukalamai/yelp-scraper-scrapy-python

Yelp Restaurant data scraping using python, scrapy spider

Language: Python - Size: 23.4 KB - Last synced at: 3 months ago - Pushed at: almost 2 years ago - Stars: 4 - Forks: 2

ahmadhuss/scrapy_universal

A universal, plug-and-play Scrapy boilerplate for scalable web scraping projects.

Language: Python - Size: 3.91 KB - Last synced at: 2 days ago - Pushed at: 3 months ago - Stars: 0 - Forks: 0

eminbustun/UFCStatsSpider

UFCStatsSpider is a web scraper that scrapes UFC fighter and event information.

Language: Python - Size: 458 KB - Last synced at: 3 months ago - Pushed at: 3 months ago - Stars: 0 - Forks: 0

lumyjuwon/KoreaNewsCrawler

A korean news crawler built to ingest large amounts of news data.

Language: Python - Size: 1.38 MB - Last synced at: 2 months ago - Pushed at: about 1 year ago - Stars: 222 - Forks: 105

mehmetozkaya/DotnetCrawler

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

Language: C# - Size: 70.3 KB - Last synced at: about 2 months ago - Pushed at: over 2 years ago - Stars: 176 - Forks: 66

kadnan/OlxScraper

OLX Scraper in Python Scrapy

Language: Python - Size: 15.6 KB - Last synced at: about 1 month ago - Pushed at: almost 4 years ago - Stars: 97 - Forks: 50

Nayemjaman/bdnews-scraper

Scrapy-Django PostgreSQL integration. Scrapes English news data from Bangladesh (4k+ pages)

Language: Python - Size: 35.2 KB - Last synced at: 3 months ago - Pushed at: 3 months ago - Stars: 5 - Forks: 1

JonusNattapong/Crewzombitx64

This project was inspired by the unclecode/crawl4ai repository. It provided valuable insights and ideas that helped shape the development of Crewzombitx64.

Language: Python - Size: 682 KB - Last synced at: 4 months ago - Pushed at: 4 months ago - Stars: 16 - Forks: 0

mahmudahsan/webscraping

Lets learn web scraping and apply them in real application.

Language: HTML - Size: 6.53 MB - Last synced at: 2 months ago - Pushed at: over 2 years ago - Stars: 32 - Forks: 18

ehsanmojab/scrapy-lms

Scrapy spider to collect student IDs with fullname from LMS website of University of Sistan and Baluchestan.

Language: Python - Size: 215 KB - Last synced at: 4 months ago - Pushed at: 4 months ago - Stars: 3 - Forks: 0

applenob/intern

使用python爬取水木清华和北大未名的实习信息，在微信客户端进行展示。

Language: JavaScript - Size: 4.08 MB - Last synced at: 3 months ago - Pushed at: over 8 years ago - Stars: 32 - Forks: 11

sam5-hub/2amz_track

amz track 1.best seller crawl daily 2.keyword track

Language: Python - Size: 54.7 KB - Last synced at: 19 days ago - Pushed at: almost 8 years ago - Stars: 2 - Forks: 1

bilalahhmedd/ebay-products-scraper

This is ebay products scrapper to scrap images and metadata details of products from ebay.com

Language: Python - Size: 88.9 KB - Last synced at: 4 months ago - Pushed at: 4 months ago - Stars: 19 - Forks: 8

oxylabs/scrapy-web-scraping

Web Scraping with Scrapy: Python Tutorial

Language: Python - Size: 812 KB - Last synced at: 4 months ago - Pushed at: 5 months ago - Stars: 1 - Forks: 0

ChiefsBestPal/Books_Database_and_Scraper

Advanced Web Scrapping and DB generation for anything related to books,series,authors,reviewers

Language: Python - Size: 18.9 MB - Last synced at: 29 days ago - Pushed at: 5 months ago - Stars: 0 - Forks: 0

SciFrozen-Git/website-scraper

A powerful and easy-to-use tool built with Scrapy and Node.js that allows you to scrape and download the entire source code and assets of any website. Perfect for developers, researchers, and web enthusiasts who need offline access to websites or want to analyze their structure.

Language: Python - Size: 14.6 KB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 0 - Forks: 0

dojutsu-user/IMDB-Scraper

Scrapy project for scraping data from IMDB with Movie Dataset including 58,623 movies' data.

Language: Python - Size: 15.4 MB - Last synced at: 3 months ago - Pushed at: over 5 years ago - Stars: 57 - Forks: 14

apehex/gdpyr

Monitor the legal commitments of data-processors :mag:

Language: HTML - Size: 4.43 MB - Last synced at: 4 months ago - Pushed at: 5 months ago - Stars: 0 - Forks: 0

NomanSiddiqui0000/Rozee.pk-jobs-Scrapper

This scraper, built in Node.js using Puppeteer and Cheerio, is designed to extract job listings from the Rozee.pk website. It can scrape multiple pages and gather detailed information, including job titles, company names, skills, and more. The output is saved in structured CSV files, with sample datasets for cities like Lahore, Karachi, etc.

Language: JavaScript - Size: 6.75 MB - Last synced at: 2 months ago - Pushed at: 6 months ago - Stars: 3 - Forks: 0

apayziev/Upwork-Successfull-Projects

The "Upwork Successful Projects Repository" is a curated collection of successful projects completed on the Upwork platform. This repository serves as a showcase of my accomplishments, skills, and expertise as a freelancer, helping you to build credibility and trust.

Language: Python - Size: 3.9 MB - Last synced at: 6 months ago - Pushed at: 6 months ago - Stars: 1 - Forks: 0

nirajmalviya/product-review-scraper

A Python-based web scraper for extracting product reviews from sites using curl_cffi and openpyxl. Includes support for sorting reviews, handling pagination, and saving results to Excel." This description highlights the key aspects of the project and tools used, making it easy for others to understand what your repository is about.

Language: Python - Size: 12.7 KB - Last synced at: 5 months ago - Pushed at: 8 months ago - Stars: 2 - Forks: 0

Tamoziit/JobListings-App

A Job Portal App (Internship Task)

Language: Python - Size: 3.42 MB - Last synced at: about 1 month ago - Pushed at: 6 months ago - Stars: 1 - Forks: 0

Jeffallan/Ebook-API

The fast and easy way to fulfill all your ebook needs

Language: Python - Size: 42 KB - Last synced at: 9 days ago - Pushed at: over 2 years ago - Stars: 2 - Forks: 1

miroshnikov/scrapyteer

Web crawling & scraping framework for Node.js on top of headless Chrome browser

Language: TypeScript - Size: 384 KB - Last synced at: 3 months ago - Pushed at: over 1 year ago - Stars: 19 - Forks: 0

franck-gaspoz/MovieDbAssistant

Generates Movie documents (HTML, movie database,...) using Web Crawlers ( IMDb, ... )

Language: C# - Size: 454 MB - Last synced at: 3 months ago - Pushed at: 7 months ago - Stars: 2 - Forks: 0

sbmagar13/raidfscrape

Code to Scraping some portion of Data from forum(RaidForums[seized by FBI]) with Python SCRAPY spiders bypassing recaptcha and storing to PostgreSQL database.(used scrapper-API as captcha, proxy bypass) SQLAlchemy as ORM for PostgreSQL - Python.

Size: 482 KB - Last synced at: 6 days ago - Pushed at: over 4 years ago - Stars: 9 - Forks: 3

notsidney/nesappscraper

Web scraper for NESA HSC past papers links

Language: Python - Size: 3.11 MB - Last synced at: 7 months ago - Pushed at: 7 months ago - Stars: 2 - Forks: 1

marcellin-d/Shopclue-Spider

Shopclue Spider is a Scrapy-based web scraper designed to extract product data from Shopclues categories. This project effectively bypasses AJAX requests to collect detailed information about products, sellers, and their ratings.

Language: Python - Size: 0 Bytes - Last synced at: 7 months ago - Pushed at: 7 months ago - Stars: 1 - Forks: 0

marcellin-d/Rightmove-Property-Scraper

A web scraping project developed with Scrapy to extract property data from the Rightmove website. This project collects detailed information on properties for sale in London, including addresses, property types, bedrooms, transactions, and geographical coordinates.

Language: Python - Size: 179 KB - Last synced at: 7 months ago - Pushed at: 7 months ago - Stars: 1 - Forks: 0

Timotej979/Luxonis-challenge

A simple webscraper for sreality.cz and a webapge display of available flats on sale :houses:

Language: Python - Size: 1.31 MB - Last synced at: 4 months ago - Pushed at: about 1 year ago - Stars: 0 - Forks: 0

nstapelbroek/estate-crawler

Scraping the real estate agencies for up-to-date house listings as soon as they arrive!

Language: Python - Size: 1.12 MB - Last synced at: 8 months ago - Pushed at: 8 months ago - Stars: 25 - Forks: 6

lemonhu/finance-qa-spider

金融问答平台文本数据采集/爬取，数据源涉及上交所，深交所，全景网及新浪股吧

Language: Python - Size: 10.7 KB - Last synced at: 2 months ago - Pushed at: almost 8 years ago - Stars: 39 - Forks: 10

GoTrained/Scrapy-Craigslist

Web Scraping Craigslist's Engineering Jobs in NY with Scrapy

Language: Python - Size: 195 KB - Last synced at: 6 months ago - Pushed at: almost 8 years ago - Stars: 66 - Forks: 37

pnguyen215/instagram-crawler

Instagram Crawler is a Python script to download posts from a specified Instagram account.

Language: Python - Size: 21.5 KB - Last synced at: 4 months ago - Pushed at: 10 months ago - Stars: 1 - Forks: 0

faisal-fida/Scraping-UEL-University

This project is a Scrapy-based web scraper designed to extract course information from the University of East London's website.

Language: Python - Size: 69.3 KB - Last synced at: 4 months ago - Pushed at: 9 months ago - Stars: 0 - Forks: 0

crawlbase/scrapy-proxycrawl-middleware 📦

Scrapy middleware interface to scrape using ProxyCrawl proxy service

Language: Python - Size: 34.2 KB - Last synced at: 25 days ago - Pushed at: almost 2 years ago - Stars: 11 - Forks: 3

hhxx2015/dialog_wechat

chat robot based on dialogue data from zhihu

Language: Python - Size: 4.42 MB - Last synced at: 6 months ago - Pushed at: over 6 years ago - Stars: 0 - Forks: 0

the-akira/Python-Web-Scraping

Explorando Técnicas de Web Scraping com Python

Language: Jupyter Notebook - Size: 729 KB - Last synced at: about 2 months ago - Pushed at: over 2 years ago - Stars: 13 - Forks: 4

ArtaXerxess/bookscraper

Language: Python - Size: 1.52 MB - Last synced at: 9 months ago - Pushed at: 9 months ago - Stars: 0 - Forks: 0

intel00000/web_scraper_Hasmo

A python web scraper designed to collect data from competitor websites, news articles, and market research reports.

Language: Jupyter Notebook - Size: 10.4 MB - Last synced at: 3 months ago - Pushed at: 10 months ago - Stars: 0 - Forks: 0

AaronAS2016/gamer-scrapy-api

API rest para el proyecto Gamerscrapy

Language: Python - Size: 85.9 KB - Last synced at: 11 months ago - Pushed at: over 4 years ago - Stars: 0 - Forks: 0

VaishnavSherla/Google-Scraper

A Simple Python Based Google Search Results Scraper.

Language: Python - Size: 16.6 KB - Last synced at: 10 months ago - Pushed at: over 2 years ago - Stars: 22 - Forks: 9

andersli53/Course-Work

My Work

Language: C++ - Size: 1.12 MB - Last synced at: 11 months ago - Pushed at: over 5 years ago - Stars: 0 - Forks: 0

openwebindex/indexbot

🤖 Primary web crawler of the Open Web Index

Language: Python - Size: 46.9 KB - Last synced at: 11 months ago - Pushed at: 11 months ago - Stars: 0 - Forks: 0

vaibhavyadav-dev/codeforces-problemset-scrapper

Web Scrapper that scrap the whole problemset of Codeforces into csv or json file.

Language: Python - Size: 805 KB - Last synced at: 4 months ago - Pushed at: 11 months ago - Stars: 0 - Forks: 0

jagmoreira/comics-scraper

Scraper of comic books information. Built using Scrapy.

Language: Python - Size: 28.3 KB - Last synced at: 11 months ago - Pushed at: 11 months ago - Stars: 1 - Forks: 0

spideynolove/scrapy-projects

🕷️📁🕸️🛠️📚🔍🤖📦🧠🔍🔍🌐🧮🔍🌐💻🔍🔍📦📊📚🔍

Language: Python - Size: 8.77 MB - Last synced at: 12 months ago - Pushed at: 12 months ago - Stars: 0 - Forks: 0

chandrasekharan98/Multisite-Python-Crawler

An almost generic web crawler built using Scrapy and Python 3.7 to recursively crawl entire websites.

Language: Python - Size: 15.6 KB - Last synced at: 8 months ago - Pushed at: over 3 years ago - Stars: 16 - Forks: 5

JacksonWuxs/easyThreading

A user-friendly threading management library for humans

Language: Python - Size: 3.91 KB - Last synced at: 21 days ago - Pushed at: over 5 years ago - Stars: 2 - Forks: 0

gkzz/deephotel 📦

scraping TripAdvisor, Booking.com with Scrapy

Language: Python - Size: 453 KB - Last synced at: 2 days ago - Pushed at: over 4 years ago - Stars: 17 - Forks: 10

seokyim8/Steam_data_pipeline

A data pipeline that provides with web-scraped information from Steam

Language: Python - Size: 177 KB - Last synced at: 12 months ago - Pushed at: 12 months ago - Stars: 1 - Forks: 0

Varadharavi/scrap-news

This project is a Scrapy-based web scraper designed to extract the latest news from moneycontrol.com and livemint.com. The scraped data is then stored in a MongoDB database.

Language: Python - Size: 37.1 KB - Last synced at: 12 months ago - Pushed at: 12 months ago - Stars: 0 - Forks: 0

NatLee/MyAnimeList-comment-crawler

Crawl all reviews and infomation of Anime works on MyAnimeList. ;)

Language: Python - Size: 50.8 KB - Last synced at: 23 days ago - Pushed at: about 2 years ago - Stars: 4 - Forks: 0

pratikwatwani/Sentiment-Analysis-on-service-reviews

Language: Jupyter Notebook - Size: 16.3 MB - Last synced at: about 1 year ago - Pushed at: about 5 years ago - Stars: 0 - Forks: 0

Shubhamdutta2000/Web-Scrapping-from-google-with-Scrapy

Python Scrapy spider that searches Google for a particular keyword and extracts all data from the SERP results. The spider will iterate through all pages returned by the keyword query.

Language: Python - Size: 208 KB - Last synced at: 3 months ago - Pushed at: over 2 years ago - Stars: 15 - Forks: 3

pritpalxyz/amazon-crawler

Language: Python - Size: 4.88 KB - Last synced at: 7 days ago - Pushed at: almost 8 years ago - Stars: 2 - Forks: 1

chen0040/scrapy-projects

Projects using selenium, requests, bs4, and scrapy for web scraping on google images, google trends and others

Language: Python - Size: 11.7 MB - Last synced at: 3 months ago - Pushed at: about 7 years ago - Stars: 10 - Forks: 2

mikefrank-ca/dealer-com-photo-crawler 📦

A Scrapy spider written to pull photos from a Dealer.com client web site.

Language: Python - Size: 4.88 KB - Last synced at: 7 months ago - Pushed at: about 7 years ago - Stars: 4 - Forks: 1

sejal-rajput/Amazon-web-scraping-using-Scrapy-Framework

A web scraping script using scrapy framework of python to extract data from amazon.com website.

Language: Python - Size: 43.9 KB - Last synced at: 4 months ago - Pushed at: over 3 years ago - Stars: 2 - Forks: 1

machinia/scraper-factory

Scraping library to retrieve data from useful pages, such as Amazon wishlists

Language: Python - Size: 65.4 KB - Last synced at: 6 days ago - Pushed at: over 2 years ago - Stars: 1 - Forks: 0

ali-bin-kashif/books-website-scraping-python

Developed a web crawler using python library scrapy. It scrapes the books data in a book store website and store it in the form of CSV file, we can store the data in any place like SQL Database or No-SQL database like Mongo-DB.

Language: Python - Size: 8.79 KB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 0 - Forks: 0