GitHub / symanto-research / merge-tokenizers

Package to align tokens from different tokenizations.

JSON API: http://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/symanto-research%2Fmerge-tokenizers
PURL: pkg:github/symanto-research/merge-tokenizers

Stars: 11
Forks: 0
Open issues: 0

License: other
Language: Python
Size: 347 KB
Dependencies parsed at: Pending

Created at: over 1 year ago
Updated at: about 1 month ago
Pushed at: over 1 year ago
Last synced at: 11 days ago

Topics: distance, tokenizers, tokens, transformers

Loading...