paulpierre/markdown-crawler

A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG

Language: Python - Size: 1.14 MB - Last synced at: 17 days ago - Pushed at: 9 months ago - Stars: 375 - Forks: 43