Ecosyste.ms: Repos
An open API service providing repository metadata for many open source software ecosystems.
GitHub / huggingface / tokenizers
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
JSON API: https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Ftokenizers
Stars: 8,424
Forks: 725
Open Issues: 94
License: apache-2.0
Language: Rust
Repo Size: 9.6 MB
Dependencies:
1,045
Created: over 4 years ago
Updated: 1 day ago
Last pushed: 1 day ago
Last synced: 1 day ago
Commit Stats
Commits: 1640
Authors: 97
Mean commits per author: 16.91
Development Distribution Score: 0.43
More commit stats: https://commits.ecosyste.ms/hosts/GitHub/repositories/huggingface/tokenizers
Topics: bert, gpt, language-model, natural-language-processing, natural-language-understanding, nlp, transformers
Files
Dependencies
- copy-webpack-plugin ^11.0.0 development
- webpack ^5.75.0 development
- webpack-cli ^5.0.1 development
- webpack-dev-server ^4.10.0 development
- unstable_wasm file:../pkg
- actions-rs/toolchain v1 composite
- actions/cache v1 composite
- actions/checkout v1 composite
- actions/setup-node v1 composite
- actions/setup-python v1 composite
- actions-rs/cargo v1 composite
- actions-rs/toolchain v1 composite
- actions/cache v1 composite
- actions/checkout v1 composite
- actions/setup-node v1 composite
- actions-rs/toolchain v1 composite
- actions/checkout v2 composite
- conda-incubator/setup-miniconda v2 composite
- actions/checkout v1 composite
- actions/checkout v2 composite
- actions/setup-python v1 composite
- actions-rs/toolchain v1 composite
- actions/checkout v2 composite
- actions/checkout v1 composite
- actions/setup-python v1 composite
- actions/setup-python v4 composite
- actions-rs/cargo v1 composite
- actions-rs/toolchain v1 composite
- actions/cache v1 composite
- actions/checkout v1 composite
- actions/setup-python v2 composite
- actions-rs/toolchain v1 composite
- actions/cache v1 composite
- actions/checkout v1 composite
- actions-rs/cargo v1 composite
- actions-rs/toolchain v1 composite
- actions/checkout v1 composite
- pyo3 0.17.2 development
- tempfile 3.1 development
- env_logger 0.7.1
- itertools 0.9
- libc 0.2
- ndarray 0.13
- numpy 0.17.2
- onig 6.0
- pyo3 0.17.2
- rayon 1.3
- serde 1.0
- serde_json 1.0
- tokenizers *
- assert_approx_eq 1.1 development
- criterion 0.4 development
- tempfile 3.1 development
- aho-corasick 0.7
- cached-path 0.6
- clap 4.0
- derive_builder 0.12
- dirs 3.0
- esaxx-rs 0.1
- fancy-regex 0.10
- getrandom 0.2.6
- indicatif 0.15
- itertools 0.9
- lazy_static 1.4
- log 0.4
- macro_rules_attribute 0.1.2
- onig 6.0
- paste 1.0.6
- rand 0.8
- rayon 1.3
- rayon-cond 0.1
- regex 1.3
- regex-syntax 0.6
- reqwest 0.11
- serde 1.0
- serde_json 1.0
- spm_precompiled 0.1
- thiserror 1.0.30
- unicode-normalization-alignments 0.1
- unicode-segmentation 1.6
- unicode_categories 0.1
- wasm-bindgen-test 0.3.13 development
- console_error_panic_hook 0.1.6
- wasm-bindgen 0.2.63
- wee_alloc 0.4.5
- @types/jest ^26.0.24 development
- @typescript-eslint/eslint-plugin ^3.10.1 development
- @typescript-eslint/parser ^3.10.1 development
- eslint ^7.32.0 development
- eslint-config-prettier ^6.15.0 development
- eslint-plugin-jest ^23.20.0 development
- eslint-plugin-jsdoc ^30.7.13 development
- eslint-plugin-prettier ^3.4.1 development
- eslint-plugin-simple-import-sort ^5.0.3 development
- jest ^26.6.3 development
- neon-cli ^0.9.1 development
- prettier ^2.5.1 development
- shelljs ^0.8.3 development
- ts-jest ^26.5.6 development
- typescript ^3.9.10 development
- @types/node ^13.13.52
- node-pre-gyp ^0.14.0
- actions-rs/toolchain v1 composite
- actions/checkout v1 composite
- actions/setup-python v1 composite
- actions/upload-artifact v2 composite