Ecosyste.ms: Repos

An open API service providing repository metadata for many open source software ecosystems.

GitHub / apify / crawlee

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

JSON API: https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apify%2Fcrawlee

Stars: 11,973
Forks: 501
Open Issues: 109

License: apache-2.0
Language: TypeScript
Repo Size: 117 MB
Dependencies: 3,263

Created: almost 8 years ago
Updated: about 2 months ago
Last pushed: about 2 months ago
Last synced: about 2 months ago

Commit Stats

Commits: 3079
Authors: 72
Mean commits per author: 42.76
Development Distribution Score: 0.789
More commit stats: https://commits.ecosyste.ms/hosts/GitHub/repositories/apify/crawlee

Topics: apify, automation, crawler, crawling, headless, headless-chrome, javascript, nodejs, npm, playwright, puppeteer, scraper, scraping, typescript, web-crawler, web-crawling, web-scraping

Files
    Loading...
    Readme
    Loading...
    Dependencies
    website/roa-loader/package.json npm
    test/e2e/cheerio-request-queue-v2/actor/package.json npm
    website/versions.json meteor
    package.json npm
    website/package.json npm
    .github/workflows/check-pr-title.yml actions
    .github/workflows/docs.yml actions
    .github/workflows/release.yml actions
    .github/workflows/test-ci.yml actions
    .github/workflows/test-e2e.yml actions
    packages/templates/templates/cheerio-js/Dockerfile docker
    packages/templates/templates/cheerio-ts/Dockerfile docker
    packages/templates/templates/getting-started-js/Dockerfile docker
    packages/templates/templates/getting-started-ts/Dockerfile docker
    packages/templates/templates/playwright-js/Dockerfile docker
    packages/templates/templates/playwright-ts/Dockerfile docker
    packages/templates/templates/puppeteer-js/Dockerfile docker
    packages/templates/templates/puppeteer-ts/Dockerfile docker
    test/e2e/automatic-persist-value/actor/Dockerfile docker
    test/e2e/autoscaling-max-tasks-per-minute/actor/Dockerfile docker
    test/e2e/cheerio-default/actor/Dockerfile docker
    test/e2e/cheerio-default-ts/actor/Dockerfile docker
    test/e2e/cheerio-enqueue-links/actor/Dockerfile docker
    test/e2e/cheerio-enqueue-links-base/actor/Dockerfile docker
    test/e2e/cheerio-ignore-ssl-errors/actor/Dockerfile docker
    test/e2e/cheerio-initial-cookies/actor/Dockerfile docker
    test/e2e/cheerio-max-requests/actor/Dockerfile docker
    test/e2e/cheerio-page-info/actor/Dockerfile docker
    test/e2e/cheerio-throw-on-ssl-errors/actor/Dockerfile docker
    test/e2e/input-json5/actor/Dockerfile docker
    test/e2e/jsdom-default-ts/actor/Dockerfile docker
    test/e2e/jsdom-react-ts/actor/Dockerfile docker
    test/e2e/linkedom-default-ts/actor/Dockerfile docker
    test/e2e/migration/actor/Dockerfile docker
    test/e2e/playwright-chromium-experimental-containers/actor/Dockerfile docker
    test/e2e/playwright-default/actor/Dockerfile docker
    test/e2e/playwright-enqueue-links/actor/Dockerfile docker
    test/e2e/playwright-enqueue-links-base/actor/Dockerfile docker
    test/e2e/playwright-firefox-experimental-containers/actor/Dockerfile docker
    test/e2e/playwright-initial-cookies/actor/Dockerfile docker
    test/e2e/playwright-introduction-guide/actor/Dockerfile docker
    test/e2e/playwright-multi-run/actor/Dockerfile docker
    test/e2e/proxy-rotation/actor/Dockerfile docker
    test/e2e/puppeteer-default/actor/Dockerfile docker
    test/e2e/puppeteer-enqueue-links/actor/Dockerfile docker
    test/e2e/puppeteer-ignore-ssl-errors/actor/Dockerfile docker
    test/e2e/puppeteer-initial-cookies/actor/Dockerfile docker
    test/e2e/puppeteer-page-info/actor/Dockerfile docker
    test/e2e/puppeteer-store-pagination/actor/Dockerfile docker
    test/e2e/puppeteer-store-pagination-jquery/actor/Dockerfile docker
    test/e2e/puppeteer-throw-on-ssl-errors/actor/Dockerfile docker
    test/e2e/request-queue-zero-concurrency/actor/Dockerfile docker
    test/e2e/request-skip-navigation/actor/Dockerfile docker
    test/e2e/session-rotation/actor/Dockerfile docker
    packages/basic-crawler/package.json npm
    packages/browser-crawler/package.json npm
    packages/browser-pool/package.json npm
    packages/cheerio-crawler/package.json npm
    packages/cli/package.json npm
    packages/core/package.json npm
    packages/crawlee/package.json npm
    packages/http-crawler/package.json npm
    packages/jsdom-crawler/package.json npm
    packages/linkedom-crawler/package.json npm
    packages/memory-storage/package.json npm
    packages/playwright-crawler/package.json npm
    packages/puppeteer-crawler/package.json npm
    packages/templates/package.json npm
    packages/templates/templates/cheerio-js/package.json npm
    packages/templates/templates/cheerio-ts/package.json npm
    packages/templates/templates/getting-started-js/package.json npm
    packages/templates/templates/getting-started-ts/package.json npm
    packages/templates/templates/playwright-js/package.json npm
    packages/templates/templates/playwright-ts/package.json npm
    packages/templates/templates/puppeteer-js/package.json npm
    packages/templates/templates/puppeteer-ts/package.json npm
    packages/types/package.json npm
    packages/utils/package.json npm
    test/e2e/automatic-persist-value/actor/package.json npm
    test/e2e/autoscaling-max-tasks-per-minute/actor/package.json npm
    test/e2e/cheerio-default/actor/package.json npm
    test/e2e/cheerio-default-ts/actor/package.json npm
    test/e2e/cheerio-enqueue-links/actor/package.json npm
    test/e2e/cheerio-enqueue-links-base/actor/package.json npm
    test/e2e/cheerio-ignore-ssl-errors/actor/package.json npm
    test/e2e/cheerio-initial-cookies/actor/package.json npm
    test/e2e/cheerio-max-requests/actor/package.json npm
    test/e2e/cheerio-page-info/actor/package.json npm
    test/e2e/cheerio-throw-on-ssl-errors/actor/package.json npm
    test/e2e/input-json5/actor/package.json npm
    test/e2e/jsdom-default-ts/actor/package.json npm
    test/e2e/jsdom-react-ts/actor/package.json npm
    test/e2e/linkedom-default-ts/actor/package.json npm
    test/e2e/migration/actor/package.json npm
    test/e2e/playwright-chromium-experimental-containers/actor/package.json npm
    test/e2e/playwright-default/actor/package.json npm
    test/e2e/playwright-enqueue-links/actor/package.json npm
    test/e2e/playwright-enqueue-links-base/actor/package.json npm
    test/e2e/playwright-firefox-experimental-containers/actor/package.json npm
    test/e2e/playwright-initial-cookies/actor/package.json npm
    test/e2e/playwright-introduction-guide/actor/package.json npm
    test/e2e/playwright-multi-run/actor/package.json npm
    test/e2e/proxy-rotation/actor/package.json npm
    test/e2e/puppeteer-default/actor/package.json npm
    test/e2e/puppeteer-enqueue-links/actor/package.json npm
    test/e2e/puppeteer-ignore-ssl-errors/actor/package.json npm
    test/e2e/puppeteer-initial-cookies/actor/package.json npm
    test/e2e/puppeteer-page-info/actor/package.json npm
    test/e2e/puppeteer-store-pagination/actor/package.json npm
    test/e2e/puppeteer-store-pagination-jquery/actor/package.json npm
    test/e2e/puppeteer-throw-on-ssl-errors/actor/package.json npm
    test/e2e/request-queue-zero-concurrency/actor/package.json npm
    test/e2e/request-skip-navigation/actor/package.json npm
    test/e2e/session-rotation/actor/package.json npm
    test/e2e/cheerio-request-queue-v2/actor/Dockerfile docker
    website/yarn.lock npm
    yarn.lock npm