Data

Tools

Indexes

Applications

Experiments

Repos

An open API service providing repository metadata for many open source software ecosystems.

GitHub topics: deepspeed-ulysses

InternLM/InternEvo

InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.

Language: Python - Size: 6.78 MB - Last synced at: 1 day ago - Pushed at: 1 day ago - Stars: 386 - Forks: 64

feifeibear/long-context-attention

USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference

Language: Python - Size: 4.6 MB - Last synced at: 18 days ago - Pushed at: 18 days ago - Stars: 475 - Forks: 38

Related Keywords
deepspeed-ulysses 2 ring-attention 2 pytorch 2 llm-training 2 llm-inference 1 attention-is-all-you-need 1 zero3 1 transformers-models 1 tensor-parallelism 1 sequence-parallelism 1 pipeline-parallelism 1 multi-modal 1 llm-framework 1 llava 1 llama3 1 internlm2 1 internlm 1 gemma 1 flash-attention 1 910b 1