GitHub / zouharvi / tokenization-scorer

Simple-to-use scoring function for arbitrarily tokenized texts.

JSON API: http://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zouharvi%2Ftokenization-scorer
PURL: pkg:github/zouharvi/tokenization-scorer

Stars: 44
Forks: 5
Open issues: 0

License: mit
Language: Python
Size: 42 KB
Dependencies parsed at: Pending

Created at: over 2 years ago
Updated at: about 1 month ago
Pushed at: 7 months ago
Last synced at: about 1 month ago

Topics: bpe, segmentation, subword, tokenization

Loading...