GitHub / Morgscode / desktop-webpage-text-crawler

This desktop GUI will index, format and create .txt files from the text content from webpages you request, so long as HTML or JSON is sent as a response. You can crawl sites as single pages, crawl all internal links on a page, or crawl all links within the page's <nav> tag(s). You can also decide to extract only page titles, the main text content, or all text content from the page. The crawler has some built-in basic error logging.

JSON API: http://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Morgscode%2Fdesktop-webpage-text-crawler

Topics: content-services, web-crawler, web-scraping-python

• Files
• Readme
• Dependencies