An open API service providing repository metadata for many open source software ecosystems.

GitHub topics: efficient-attention

thu-ml/SageAttention

Quantized Attention achieves speedup of 2-5x and 3-11x compared to FlashAttention and xformers, without lossing end-to-end metrics across language, image, and video models.

Language: Cuda - Size: 46.1 MB - Last synced at: 6 days ago - Pushed at: 6 days ago - Stars: 1,690 - Forks: 120

Ascend-Research/CascadedGaze

The official PyTorch implementation for CascadedGaze: Efficiency in Global Context Extraction for Image Restoration, TMLR'24.

Language: Python - Size: 478 KB - Last synced at: 19 days ago - Pushed at: 4 months ago - Stars: 72 - Forks: 3

lucidrains/ring-attention-pytorch

Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch

Language: Python - Size: 1.01 MB - Last synced at: 24 days ago - Pushed at: about 1 month ago - Stars: 513 - Forks: 31

lucidrains/CoLT5-attention

Implementation of the conditionally routed attention in the CoLT5 architecture, in Pytorch

Language: Python - Size: 187 KB - Last synced at: 23 days ago - Pushed at: 9 months ago - Stars: 227 - Forks: 13

gmlwns2000/sea-attention

Official Implementation of SEA: Sparse Linear Attention with Estimated Attention Mask (ICLR 2024)

Language: Python - Size: 372 MB - Last synced at: 3 months ago - Pushed at: 5 months ago - Stars: 7 - Forks: 1

robflynnyh/hydra-linear-attention

Implementation of: Hydra Attention: Efficient Attention with Many Heads (https://arxiv.org/abs/2209.07484)

Language: Python - Size: 8.79 KB - Last synced at: about 2 months ago - Pushed at: over 2 years ago - Stars: 13 - Forks: 0

HolmesShuan/Compact-Global-Descriptor

Pytorch implementation of "Compact Global Descriptor for Neural Networks" (CGD).

Language: Python - Size: 2.55 MB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 25 - Forks: 7

pszemraj/samba-pytorch

Minimal implementation of Samba by Microsoft in PyTorch

Language: Python - Size: 34.1 MB - Last synced at: 3 months ago - Pushed at: 7 months ago - Stars: 0 - Forks: 0

MAGICS-LAB/NonparametricHopfield

Nonparametric Modern Hopfield Models

Language: Jupyter Notebook - Size: 167 MB - Last synced at: about 1 year ago - Pushed at: over 1 year ago - Stars: 1 - Forks: 0

davidsvy/cosformer-pytorch

Unofficial PyTorch implementation of the paper "cosFormer: Rethinking Softmax In Attention".

Language: Jupyter Notebook - Size: 243 KB - Last synced at: over 2 years ago - Pushed at: over 3 years ago - Stars: 40 - Forks: 7