Ecosyste.ms: Repos

An open API service providing repository metadata for many open source software ecosystems.

GitHub / ArthurZucker / tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

JSON API: https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ArthurZucker%2Ftokenizers

Fork of huggingface/tokenizers
Stars: 0
Forks: 0
Open Issues: 0

License: apache-2.0
Language: Rust
Repo Size: 8.87 MB
Dependencies: 1,686

Created: almost 2 years ago
Updated: almost 2 years ago
Last pushed: 7 months ago
Last synced: about 1 month ago

Files

Loading...

Readme

Loading...

Dependencies

.github/workflows/docs-check.yml actions

actions-rs/toolchain v1 composite
actions/checkout v1 composite
actions/setup-python v1 composite
actions/upload-artifact v2 composite

.github/workflows/node-release.yml actions

actions-rs/toolchain v1 composite
actions/cache v1 composite
actions/checkout v1 composite
actions/setup-node v1 composite
actions/setup-python v1 composite

.github/workflows/node.yml actions

actions-rs/cargo v1 composite
actions-rs/toolchain v1 composite
actions/cache v1 composite
actions/checkout v1 composite
actions/setup-node v1 composite

.github/workflows/python-release-conda.yml actions

actions-rs/toolchain v1 composite
actions/checkout v2 composite
conda-incubator/setup-miniconda v2 composite

.github/workflows/python-release-extra.yml actions

actions/checkout v1 composite
actions/checkout v2 composite
actions/setup-python v1 composite

.github/workflows/python-release.yml actions

actions-rs/toolchain v1 composite
actions/checkout v2 composite
actions/checkout v1 composite
actions/setup-python v1 composite

.github/workflows/python.yml actions

actions-rs/cargo v1 composite
actions-rs/toolchain v1 composite
actions/cache v1 composite
actions/checkout v1 composite
actions/setup-python v2 composite

.github/workflows/rust-release.yml actions

actions-rs/toolchain v1 composite
actions/cache v1 composite
actions/checkout v1 composite

.github/workflows/rust.yml actions

actions-rs/cargo v1 composite
actions-rs/toolchain v1 composite
actions/checkout v1 composite

tokenizers/Cargo.toml cargo

tokenizers/examples/unstable_wasm/Cargo.toml cargo

wasm-bindgen-test 0.3.13 development
console_error_panic_hook 0.1.6
wasm-bindgen 0.2.63
wee_alloc 0.4.5

bindings/node/package.json npm

@types/jest ^26.0.24 development
@typescript-eslint/eslint-plugin ^3.10.1 development
@typescript-eslint/parser ^3.10.1 development
eslint ^7.32.0 development
eslint-config-prettier ^6.15.0 development
eslint-plugin-jest ^23.20.0 development
eslint-plugin-jsdoc ^30.7.13 development
eslint-plugin-prettier ^3.4.1 development
eslint-plugin-simple-import-sort ^5.0.3 development
jest ^26.6.3 development
neon-cli ^0.9.1 development
prettier ^2.5.1 development
shelljs ^0.8.3 development
ts-jest ^26.5.6 development
typescript ^3.9.10 development
@types/node ^13.13.52
node-pre-gyp ^0.14.0

tokenizers/examples/unstable_wasm/www/package.json npm

copy-webpack-plugin ^5.0.0 development
webpack ^4.29.3 development
webpack-cli ^3.1.0 development
webpack-dev-server ^4.10.0 development
unstable_wasm file:../pkg

bindings/python/Cargo.toml cargo

pyo3 0.17.2 development
tempfile 3.1 development
env_logger 0.7.1
itertools 0.9
libc 0.2
ndarray 0.13
numpy 0.17.2
onig 6.0
pyo3 0.17.2
rayon 1.3
serde 1.0
serde_json 1.0
tokenizers *

bindings/node/native/Cargo.lock cargo

211 dependencies

bindings/python/Cargo.lock cargo

218 dependencies

tokenizers/examples/unstable_wasm/www/package-lock.json npm

525 dependencies

bindings/node/package-lock.json npm

627 dependencies