Topic: "efficient-attention"
thu-ml/SageAttention
Quantized Attention achieves speedup of 2-3x and 3-5x compared to FlashAttention and xformers, without lossing end-to-end metrics across language, image, and video models.
Language: Cuda - Size: 46.1 MB - Last synced at: 2 days ago - Pushed at: 2 days ago - Stars: 1,357 - Forks: 94
lucidrains/ring-attention-pytorch
Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch
Language: Python - Size: 1.01 MB - Last synced at: 9 days ago - Pushed at: 6 months ago - Stars: 510 - Forks: 30
lucidrains/CoLT5-attention
Implementation of the conditionally routed attention in the CoLT5 architecture, in Pytorch
Language: Python - Size: 187 KB - Last synced at: 14 days ago - Pushed at: 8 months ago - Stars: 226 - Forks: 13
Ascend-Research/CascadedGaze
The official PyTorch implementation for CascadedGaze: Efficiency in Global Context Extraction for Image Restoration, TMLR'24.
Language: Python - Size: 474 KB - Last synced at: 2 months ago - Pushed at: 2 months ago - Stars: 49 - Forks: 3
davidsvy/cosformer-pytorch
Unofficial PyTorch implementation of the paper "cosFormer: Rethinking Softmax In Attention".
Language: Jupyter Notebook - Size: 243 KB - Last synced at: about 2 years ago - Pushed at: over 3 years ago - Stars: 40 - Forks: 7
HolmesShuan/Compact-Global-Descriptor
Pytorch implementation of "Compact Global Descriptor for Neural Networks" (CGD).
Language: Python - Size: 2.55 MB - Last synced at: 3 months ago - Pushed at: 3 months ago - Stars: 25 - Forks: 7
robflynnyh/hydra-linear-attention
Implementation of: Hydra Attention: Efficient Attention with Many Heads (https://arxiv.org/abs/2209.07484)
Language: Python - Size: 8.79 KB - Last synced at: 5 months ago - Pushed at: over 2 years ago - Stars: 11 - Forks: 0
gmlwns2000/sea-attention
Official Implementation of SEA: Sparse Linear Attention with Estimated Attention Mask (ICLR 2024)
Language: Python - Size: 372 MB - Last synced at: about 1 month ago - Pushed at: 4 months ago - Stars: 7 - Forks: 1
MAGICS-LAB/NonparametricHopfield
Nonparametric Modern Hopfield Models
Language: Jupyter Notebook - Size: 167 MB - Last synced at: about 1 year ago - Pushed at: over 1 year ago - Stars: 1 - Forks: 0
pszemraj/samba-pytorch
Minimal implementation of Samba by Microsoft in PyTorch
Language: Python - Size: 34.1 MB - Last synced at: 30 days ago - Pushed at: 5 months ago - Stars: 0 - Forks: 0
