Data

Tools

Indexes

Applications

Experiments

Repos

An open API service providing repository metadata for many open source software ecosystems.

GitHub topics: deepspeed-ulysses

feifeibear/long-context-attention

USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference

Language: Python - Size: 4.61 MB - Last synced at: 2 days ago - Pushed at: about 2 months ago - Stars: 556 - Forks: 64

InternLM/InternEvo

InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.

Language: Python - Size: 6.79 MB - Last synced at: 16 days ago - Pushed at: 16 days ago - Stars: 404 - Forks: 70

Related Keywords
deepspeed-ulysses 2 llm-training 2 pytorch 2 ring-attention 2 zero3 1 transformers-models 1 tensor-parallelism 1 sequence-parallelism 1 pipeline-parallelism 1 multi-modal 1 llm-framework 1 llava 1 llama3 1 internlm2 1 internlm 1 gemma 1 flash-attention 1 910b 1 llm-inference 1 attention-is-all-you-need 1