Data

Tools

Indexes

Applications

Experiments

Repos

An open API service providing repository metadata for many open source software ecosystems.

GitHub topics: sequence-parallelism

InternLM/InternEvo

InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.

Language: Python - Size: 6.78 MB - Last synced at: about 16 hours ago - Pushed at: about 17 hours ago - Stars: 386 - Forks: 64

xrsrke/pipegoose

Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*

Language: Python - Size: 1.26 MB - Last synced at: 12 days ago - Pushed at: over 1 year ago - Stars: 82 - Forks: 18

AlibabaPAI/FlashModels

Fast and easy distributed model training examples.

Language: Python - Size: 42.9 MB - Last synced at: 6 months ago - Pushed at: 6 months ago - Stars: 9 - Forks: 4

Related Keywords
tensor-parallelism 3 sequence-parallelism 3 data-parallelism 2 pytorch 2 model-parallelism 2 pipeline-parallelism 2 distributed-optimizers 1 huggingface-transformers 1 large-scale-language-modeling 1 megatron 1 megatron-lm 1 mixture-of-experts 1 moe 1 transformers 1 zero-1 1 deep-learning 1 distributed-training 1 fsdp 1 llm 1 xla 1 zero 1 3d-parallelism 1 zero3 1 transformers-models 1 ring-attention 1 multi-modal 1 llm-training 1 llm-framework 1 llava 1 llama3 1 internlm2 1 internlm 1 gemma 1 flash-attention 1 deepspeed-ulysses 1 910b 1