GitHub topics: efficient-attention
thu-ml/SageAttention
Quantized Attention achieves speedup of 2-5x and 3-11x compared to FlashAttention and xformers, without lossing end-to-end metrics across language, image, and video models.
Language: Cuda - Size: 46.1 MB - Last synced at: 6 days ago - Pushed at: 6 days ago - Stars: 1,690 - Forks: 120

Ascend-Research/CascadedGaze
The official PyTorch implementation for CascadedGaze: Efficiency in Global Context Extraction for Image Restoration, TMLR'24.
Language: Python - Size: 478 KB - Last synced at: 19 days ago - Pushed at: 4 months ago - Stars: 72 - Forks: 3

lucidrains/ring-attention-pytorch
Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch
Language: Python - Size: 1.01 MB - Last synced at: 24 days ago - Pushed at: about 1 month ago - Stars: 513 - Forks: 31

lucidrains/CoLT5-attention
Implementation of the conditionally routed attention in the CoLT5 architecture, in Pytorch
Language: Python - Size: 187 KB - Last synced at: 23 days ago - Pushed at: 9 months ago - Stars: 227 - Forks: 13

gmlwns2000/sea-attention
Official Implementation of SEA: Sparse Linear Attention with Estimated Attention Mask (ICLR 2024)
Language: Python - Size: 372 MB - Last synced at: 3 months ago - Pushed at: 5 months ago - Stars: 7 - Forks: 1

robflynnyh/hydra-linear-attention
Implementation of: Hydra Attention: Efficient Attention with Many Heads (https://arxiv.org/abs/2209.07484)
Language: Python - Size: 8.79 KB - Last synced at: about 2 months ago - Pushed at: over 2 years ago - Stars: 13 - Forks: 0

HolmesShuan/Compact-Global-Descriptor
Pytorch implementation of "Compact Global Descriptor for Neural Networks" (CGD).
Language: Python - Size: 2.55 MB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 25 - Forks: 7

pszemraj/samba-pytorch
Minimal implementation of Samba by Microsoft in PyTorch
Language: Python - Size: 34.1 MB - Last synced at: 3 months ago - Pushed at: 7 months ago - Stars: 0 - Forks: 0

MAGICS-LAB/NonparametricHopfield
Nonparametric Modern Hopfield Models
Language: Jupyter Notebook - Size: 167 MB - Last synced at: about 1 year ago - Pushed at: over 1 year ago - Stars: 1 - Forks: 0

davidsvy/cosformer-pytorch
Unofficial PyTorch implementation of the paper "cosFormer: Rethinking Softmax In Attention".
Language: Jupyter Notebook - Size: 243 KB - Last synced at: over 2 years ago - Pushed at: over 3 years ago - Stars: 40 - Forks: 7
