Data

Tools

Indexes

Applications

Experiments

Repos

An open API service providing repository metadata for many open source software ecosystems.

Topic: "sequence-parallelism"

InternLM/InternEvo

InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.

Language: Python - Size: 6.77 MB - Last synced at: 7 days ago - Pushed at: 7 days ago - Stars: 382 - Forks: 64

xrsrke/pipegoose

Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*

Language: Python - Size: 1.26 MB - Last synced at: 3 days ago - Pushed at: over 1 year ago - Stars: 82 - Forks: 18

AlibabaPAI/FlashModels

Fast and easy distributed model training examples.

Language: Python - Size: 42.9 MB - Last synced at: 6 months ago - Pushed at: 6 months ago - Stars: 9 - Forks: 4

Related Topics
tensor-parallelism 3 data-parallelism 2 pipeline-parallelism 2 model-parallelism 2 pytorch 2 zero-1 1 910b 1 deepspeed-ulysses 1 flash-attention 1 gemma 1 internlm 1 internlm2 1 llama3 1 llava 1 llm-framework 1 llm-training 1 multi-modal 1 ring-attention 1 transformers-models 1 zero3 1 transformers 1 moe 1 mixture-of-experts 1 megatron-lm 1 megatron 1 large-scale-language-modeling 1 huggingface-transformers 1 distributed-optimizers 1 3d-parallelism 1 zero 1 xla 1 llm 1 fsdp 1 distributed-training 1 deep-learning 1