Package Usage: pypi: utoken

utoken is a universal tokenizer (multilingual word segmenter) that divides text into words, punctuation and special tokens such as numbers, URLs, XML tags, email-addresses and hashtags. It comes with a companion detokenizer.
6 versions
Latest release: over 2 years ago
121 downloads last month

View more package details: https://packages.ecosyste.ms/registries/pypi.org/packages/utoken

View more repository details: https://repos.ecosyste.ms/hosts/GitHub/repositories/uhermjakob%2Futoken

Dependent Repos 1