GitHub / Beomi / megatronlm_dataset_autotokenizer
Megatron-LM/GPT-NeoX compatible Text Encoder with 🤗Transformers AutoTokenizer.
JSON API: http://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Beomi%2Fmegatronlm_dataset_autotokenizer
Stars: 6
Forks: 1
Open issues: 0
License: apache-2.0
Language: Python
Size: 498 KB
Dependencies parsed at: Pending
Created at: over 1 year ago
Updated at: 11 months ago
Pushed at: over 1 year ago
Last synced at: 21 days ago
Topics: gpt-neox, megatron-lm, tokenizers, transformers
Loading...