GitHub / marcelovicentegc / counttokens

Yet another CLI tool for counting tokens in text datasets using tiktoken.

JSON API: http://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marcelovicentegc%2Fcounttokens
PURL: pkg:github/marcelovicentegc/counttokens

Stars: 0
Forks: 0
Open issues: 0

License: mit
Language: Python
Size: 18.6 KB
Dependencies parsed at: Pending

Created at: 4 months ago
Updated at: 4 months ago
Pushed at: 4 months ago
Last synced at: 4 months ago

Topics: llms, tiktoken, tiktoken-cli, tokens

Loading...