GitHub / twardoch / split-markdown4gpt

A Python tool for splitting large Markdown files into smaller sections based on a specified token limit. This is particularly useful for processing large Markdown files with GPT models, as it allows the models to handle the data in manageable chunks.

JSON API: https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twardoch%2Fsplit-markdown4gpt

Topics: data-preprocessing, gpt, gpt-3, gpt-35-turbo, gpt-35-turbo-16k, gpt-4, markdown, markdown-processing, mistletoe, natural-language-processing, nlp, openai, openai-gpt, python, split-text, summarization, text-analysis, text-processing, text-summarization, text-tokenization

Files