Ecosyste.ms: Repos
An open API service providing repository metadata for many open source software ecosystems.
GitHub / brendonboshell / supercrawler
A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.
JSON API: https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brendonboshell%2Fsupercrawler
Stars: 368
Forks: 63
Open Issues: 23
License: apache-2.0
Language: JavaScript
Repo Size: 664 KB
Dependencies:
367
Created: almost 8 years ago
Updated: about 1 month ago
Last pushed: over 1 year ago
Last synced: 27 days ago
Topics: crawler, distributed-crawler, robot, sitemap, web-crawler
Files
Dependencies
- chai ^4.2.0 development
- mocha ^6.1.4 development
- nyc ^14.1.0 development
- proxyquire ^2.1.0 development
- sinon ^1.17.7 development
- bluebird ^3.5.4
- cheerio ^1.0.0-rc.3
- ioredis ^4.9.3
- lodash ^4.17.11
- mime-types ^2.1.24
- node-cache ^4.2.0
- request ^2.88.0
- robots-parser ^2.1.1
- sequelize ^5.8.5
- sqlite3 ^4.0.7