EasyPi/docker-scrapyd

🕷️ Scrapyd is an application for deploying and running Scrapy spiders.

Language: Dockerfile - Size: 57.6 KB - Last synced at: 12 days ago - Pushed at: 12 days ago - Stars: 83 - Forks: 22

q-m/scrapyd-k8s

Scrapyd on container infrastructure

Language: Python - Size: 101 KB - Last synced at: 12 days ago - Pushed at: 12 days ago - Stars: 14 - Forks: 8

ConlinH/aio-scrapy

Implement scrapy with asyncio

Language: Python - Size: 496 KB - Last synced at: 9 days ago - Pushed at: 7 months ago - Stars: 63 - Forks: 10

my8100/logparser

A tool for parsing Scrapy log files periodically and incrementally, extending the HTTP JSON API of Scrapyd.

Language: Python - Size: 172 KB - Last synced at: 11 days ago - Pushed at: 4 months ago - Stars: 91 - Forks: 25

DormyMo/SpiderKeeper

admin ui for scrapy/open source scrapinghub

Language: Python - Size: 3.62 MB - Last synced at: 10 days ago - Pushed at: almost 2 years ago - Stars: 2,763 - Forks: 505

my8100/scrapydweb

Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. Docs 文档 :point_right:

Language: Python - Size: 3.05 MB - Last synced at: 15 days ago - Pushed at: 2 months ago - Stars: 3,260 - Forks: 578

mouday/spider-admin-pro

spider-admin-pro 一个集爬虫Scrapy+Scrapyd爬虫项目查看 和 爬虫任务定时调度的可视化管理工具，SpiderAdmin的升级版

Language: Python - Size: 2.82 MB - Last synced at: 14 days ago - Pushed at: 5 months ago - Stars: 587 - Forks: 84

kezhenxu94/house-renting 📦

Possibly the best practice of Scrapy 🕷 and renting a house 🏡

Language: Python - Size: 1.43 MB - Last synced at: 4 days ago - Pushed at: over 3 years ago - Stars: 795 - Forks: 144

my8100/files

Docs and files for ScrapydWeb, Scrapyd, Scrapy, and other projects

Size: 16.7 MB - Last synced at: 17 days ago - Pushed at: about 2 months ago - Stars: 420 - Forks: 71

bitmakerla/estela

estela, an elastic web scraping cluster 🕸

Language: TypeScript - Size: 4.48 MB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 179 - Forks: 15

crawlab-team/crawlab-lite

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

Language: Vue - Size: 2.36 MB - Last synced at: about 1 month ago - Pushed at: about 2 years ago - Stars: 224 - Forks: 75

ScrapeOps/scrapeops-scrapy-sdk

Scrapy extension that gives you all the scraping monitoring, alerting, scheduling, and data validation you will need straight out of the box.

Language: Python - Size: 89.8 KB - Last synced at: about 1 month ago - Pushed at: 9 months ago - Stars: 36 - Forks: 10

jxltom/scrapymon

Simple Web UI for Scrapy spider management via Scrapyd

Language: Python - Size: 2.61 MB - Last synced at: 4 days ago - Pushed at: almost 7 years ago - Stars: 51 - Forks: 11

datawizard1337/ARGUS

ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9

Language: Python - Size: 14.7 MB - Last synced at: about 1 month ago - Pushed at: over 3 years ago - Stars: 88 - Forks: 25

Gerapy/Gerapy

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

Language: Python - Size: 36.6 MB - Last synced at: 6 months ago - Pushed at: 6 months ago - Stars: 3,350 - Forks: 644

jxltom/scrapyd-heroku

Wrapper for running Scrapyd in Heroku or locally as a service

Language: Python - Size: 44.9 KB - Last synced at: 4 days ago - Pushed at: over 4 years ago - Stars: 19 - Forks: 36

my8100/scrapyd-cluster-on-heroku

Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO :point_right:

Language: Python - Size: 236 KB - Last synced at: 15 days ago - Pushed at: about 5 years ago - Stars: 122 - Forks: 88

abebus/spider-info-webservice

Language: Python - Size: 64.5 KB - Last synced at: 11 days ago - Pushed at: 8 months ago - Stars: 1 - Forks: 0

zubedev/scrapydoo

Scrapy dappy doo crawler for proxy sites

Language: Python - Size: 379 KB - Last synced at: 9 months ago - Pushed at: 9 months ago - Stars: 1 - Forks: 0

Tiago-Lira/scrapyd-mongodb

Library designed to replace the SQLite backend by a MongoDB backend on Scrapy queue management

Language: Python - Size: 12.7 KB - Last synced at: about 22 hours ago - Pushed at: over 7 years ago - Stars: 17 - Forks: 9

Dainius-P/scrapyd-dash

Scrapyd Dashboard

Language: CSS - Size: 2.79 MB - Last synced at: about 1 month ago - Pushed at: about 3 years ago - Stars: 8 - Forks: 1

liangWenPeng/scrapy-admin

A django admin site for scrapy

Language: Python - Size: 779 KB - Last synced at: 4 months ago - Pushed at: over 7 years ago - Stars: 45 - Forks: 12

slymit/scrapyq

Scrapyd queue management using Redis.

Language: Python - Size: 7.81 KB - Last synced at: about 1 month ago - Pushed at: over 1 year ago - Stars: 2 - Forks: 0

Dog-Egg/scrapy-ui

A web service that manages and schedules Scrapyd. 🕷️

Language: TypeScript - Size: 32.5 MB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 0 - Forks: 0

harrisbaird/dockerfiles

Various Dockerfiles.

Language: Shell - Size: 21.5 KB - Last synced at: about 1 year ago - Pushed at: almost 7 years ago - Stars: 9 - Forks: 0

koneb71/SpiderManager Fork of DormyMo/SpiderKeeper

admin ui for scrapy/open source scrapinghub

Language: Python - Size: 3.64 MB - Last synced at: 9 months ago - Pushed at: almost 4 years ago - Stars: 2 - Forks: 1

casual-silva/NewsCrawl

狠心开源企业级舆情新闻爬虫项目：支持任意数量爬虫一键运行、爬虫定时任务、爬虫批量删除；爬虫一键部署；爬虫监控可视化; 配置集群爬虫分配策略；👉 现成的docker一键部署文档已为大家踩坑

Language: Python - Size: 15.2 MB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 402 - Forks: 119

slymit/scrapyduler

Scrapyd module that schedules scrapy spiders by time.

Language: Python - Size: 7.81 KB - Last synced at: 4 months ago - Pushed at: over 1 year ago - Stars: 1 - Forks: 0

speakol-ads/scrapyd-redis Fork of Tiago-Lira/scrapyd-mongodb

Library designed to replace the SQLite backend by a redis backend on Scrapy queue management

Language: Python - Size: 16.6 KB - Last synced at: 4 months ago - Pushed at: over 5 years ago - Stars: 2 - Forks: 1

WangPei0316/scrapy-zhihu-user

知乎用户爬虫，使用scrapy_redis，scrapyd，gerapy等

Language: Python - Size: 24.4 KB - Last synced at: about 1 year ago - Pushed at: almost 7 years ago - Stars: 10 - Forks: 5

baabaaox/ScrapyDouban

豆瓣电影/豆瓣读书 Scarpy 爬虫

Language: Python - Size: 31.3 KB - Last synced at: over 1 year ago - Pushed at: about 3 years ago - Stars: 636 - Forks: 195

aaldaber/Distributed-Multi-User-Scrapy-System-with-a-Web-UI

Django based application that allows creating, deploying and running Scrapy spiders in a distributed manner

Language: Python - Size: 4.97 MB - Last synced at: over 1 year ago - Pushed at: almost 7 years ago - Stars: 108 - Forks: 46

AzizNadirov/scrapy-monit

scrapy-monit: web app for monitoring, scheduling and managing scrapyd instances.

Language: Python - Size: 246 KB - Last synced at: 11 months ago - Pushed at: 11 months ago - Stars: 1 - Forks: 0

ermissa/scrapyd-django-mongodb-setup

Setup project to run Scrapy + Django and save parsed data to MongoDB.

Language: Python - Size: 12.7 KB - Last synced at: 12 months ago - Pushed at: over 4 years ago - Stars: 1 - Forks: 1

inTheRye/docker-scrapyd

docker platform sample for scrapyd

Size: 1000 Bytes - Last synced at: over 1 year ago - Pushed at: over 7 years ago - Stars: 2 - Forks: 0

ammirsm/data-grabber-cnn-twitter

Basic setup to get data from twitter and CNN with a keyword.

Language: JavaScript - Size: 689 KB - Last synced at: 25 days ago - Pushed at: over 2 years ago - Stars: 1 - Forks: 1

f213/scrapyc 📦

CLI and client library for scrapyd. Done right.

Language: Python - Size: 39.1 KB - Last synced at: about 22 hours ago - Pushed at: over 5 years ago - Stars: 3 - Forks: 0

dharrisbaird/dailyteedeals_scrapers 📦

A collection of spiders for extracting designs from daily tee websites, including: ShirtPunch, Teefury, Yetee, Qwertee and 40+ other sites.

Language: Python - Size: 48.8 KB - Last synced at: about 2 years ago - Pushed at: almost 6 years ago - Stars: 0 - Forks: 0

INNOVINATI/scrapretty

A pretty and serverless dashboard for your Scrapyd instances

Language: Vue - Size: 559 KB - Last synced at: about 2 years ago - Pushed at: over 4 years ago - Stars: 5 - Forks: 1

fliot/ScrapyKeeper Fork of DormyMo/SpiderKeeper

admin ui for scrapy/open source scrapinghub

Language: Python - Size: 4.18 MB - Last synced at: about 2 years ago - Pushed at: almost 4 years ago - Stars: 56 - Forks: 20

mpszumowski/djangocrawler

start a new (random) life | World Bank data scraper on django rest app

Language: Python - Size: 161 KB - Last synced at: almost 2 years ago - Pushed at: about 7 years ago - Stars: 4 - Forks: 3

heidudu/tophub

后端基于Python的Flask和Scrapy,前端基于React,redux，采用docker部署的资讯收集站

Language: Python - Size: 1.25 MB - Last synced at: about 2 years ago - Pushed at: over 2 years ago - Stars: 17 - Forks: 6

fuji44/iocage-plugin-scrapyd

It is an iocage-plugin made to easily use Scrapyd with FreeBSD, TrueNAS, FreeNAS.

Language: Shell - Size: 3.91 KB - Last synced at: about 2 years ago - Pushed at: over 2 years ago - Stars: 0 - Forks: 0

Benknightdark/ScrapydDashboard

Scrapyd 爬蟲儀表板

Language: TypeScript - Size: 594 KB - Last synced at: about 1 year ago - Pushed at: over 1 year ago - Stars: 3 - Forks: 0

napoler/scrapyd_docker

scrapyd_docker

Language: Dockerfile - Size: 101 KB - Last synced at: about 2 years ago - Pushed at: over 2 years ago - Stars: 1 - Forks: 0

my8100/scrapyd-cluster-on-heroku-scrapyd-app

How to set up Scrapyd cluster on Heroku

Language: Python - Size: 26.4 KB - Last synced at: 11 days ago - Pushed at: about 3 years ago - Stars: 9 - Forks: 29

xxl4tomxu98/scrapy_web_crawler

Integration of django and scrapy package for web scraping

Language: Python - Size: 21.5 KB - Last synced at: about 2 years ago - Pushed at: about 2 years ago - Stars: 1 - Forks: 0

ToonoW/SpiderManager

爬虫管理平台

Language: Python - Size: 62.5 KB - Last synced at: about 2 years ago - Pushed at: over 2 years ago - Stars: 27 - Forks: 10

civilcoder55/imdb-scrapy-app

simple imdb movie scraper, built with python scrapy

Language: Python - Size: 160 KB - Last synced at: almost 2 years ago - Pushed at: about 3 years ago - Stars: 1 - Forks: 0

rnovec/scrapy-template

Scrapy project template

Language: Python - Size: 3.49 MB - Last synced at: about 2 years ago - Pushed at: over 2 years ago - Stars: 2 - Forks: 0

quicksandznzn/scrapyd_docker

Language: Dockerfile - Size: 2.93 KB - Last synced at: about 2 years ago - Pushed at: over 3 years ago - Stars: 0 - Forks: 0

perrornet/SpiderMan

SpiderMan Based on Scrapy, scrapyd, scrapy-API, tornado spider distributed management framework.

Language: Python - Size: 2.51 MB - Last synced at: 3 months ago - Pushed at: over 1 year ago - Stars: 12 - Forks: 6

idrodriguez/scrapyd-auth

Scrapy daemon running in an authenticated way through nginx

Language: Dockerfile - Size: 7.81 KB - Last synced at: about 2 years ago - Pushed at: about 4 years ago - Stars: 0 - Forks: 0

rnovec/scrapyd-api

A Node.js wrapper for working with the Scrapyd API

Language: JavaScript - Size: 1.1 MB - Last synced at: 3 days ago - Pushed at: over 2 years ago - Stars: 0 - Forks: 0

crawlaio/scrapyd-heroku

在 heroku 上搭建 scrapyd 集群教程

Language: Python - Size: 10.7 KB - Last synced at: about 2 years ago - Pushed at: over 4 years ago - Stars: 0 - Forks: 3

shivamacs/scrapjango

A web crawling utility that gets you the links at one place.

Language: Python - Size: 19.7 MB - Last synced at: about 2 years ago - Pushed at: over 2 years ago - Stars: 0 - Forks: 0

netflame/docker-scrapyd

Dockerized Scrapyd based on Alpine

Language: Dockerfile - Size: 3.91 KB - Last synced at: about 2 years ago - Pushed at: about 6 years ago - Stars: 1 - Forks: 1

my8100/scrapyd-cluster-on-heroku-scrapydweb-app-git

How to set up Scrapyd cluster on Heroku

Language: Python - Size: 36.1 KB - Last synced at: 22 days ago - Pushed at: over 5 years ago - Stars: 1 - Forks: 21

silenceliang/ptt-crawler-scrapyRedis

ptt-crawler with scrapy-redis framework in python

Language: Python - Size: 5.89 MB - Last synced at: about 2 years ago - Pushed at: over 2 years ago - Stars: 0 - Forks: 0

ChristianYeah/scrapyd-egg-checksum

Extension of scrapyd to get egg's md5 checksum for distributed scrapyd

Language: Python - Size: 5.86 KB - Last synced at: 13 days ago - Pushed at: over 5 years ago - Stars: 0 - Forks: 0

my8100/scrapyd-cluster-on-heroku-scrapydweb-app

How to set up Scrapyd cluster on Heroku

Language: Python - Size: 43.9 KB - Last synced at: about 1 month ago - Pushed at: over 5 years ago - Stars: 0 - Forks: 3

BruceDone/python-scrapyd-api Fork of djm/python-scrapyd-api

A Python wrapper for working with Scrapyd's API.

Language: Python - Size: 46.9 KB - Last synced at: about 2 years ago - Pushed at: about 8 years ago - Stars: 1 - Forks: 0

usernamehcx/usernamehcx.github.io

my blog website

Size: 10.7 KB - Last synced at: almost 2 years ago - Pushed at: almost 7 years ago - Stars: 0 - Forks: 0

perrornet/async_scrapyd_api

异步scrapyd api实现

Language: Python - Size: 7.81 KB - Last synced at: 3 months ago - Pushed at: almost 7 years ago - Stars: 0 - Forks: 0

mushoffa/docker-scrapyd

Size: 3.91 KB - Last synced at: about 1 month ago - Pushed at: over 7 years ago - Stars: 0 - Forks: 0