GitHub / golang-collection / go-crawler-distributed
分布式爬虫项目，本项目支持个性化定制页面解析器二次开发，项目整体采用微服务架构，通过消息队列实现消息的异步发送，使用到的框架包括：redigo, gorm, goquery, easyjson, viper, amqp, zap, go-micro，并通过Docker实现容器化部署，中间爬虫节点支持水平拓展。
JSON API: http://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/golang-collection%2Fgo-crawler-distributed
PURL: pkg:github/golang-collection/go-crawler-distributed
Stars: 48
Forks: 7
Open issues: 2
License: mit
Language: Go
Size: 90.6 MB
Dependencies parsed at: Pending
Created at: almost 5 years ago
Updated at: 10 months ago
Pushed at: about 1 year ago
Last synced at: 7 months ago
Topics: crawler, docker, elasticsearch, go, go-micro, gocrawler, microservice, rabbitmq