GitHub topics: self-attention

Repositories

SmallDoges/flash-dmattn

Flash Dynamic Mask Attention

Language: C++ - Size: 3.16 MB - Last synced at: 1 day ago - Pushed at: 1 day ago - Stars: 271 - Forks: 34

sayakpaul/robustness-vit

Contains code for the paper "Vision Transformers are Robust Learners" (AAAI 2022).

Language: Jupyter Notebook - Size: 4.22 MB - Last synced at: 1 day ago - Pushed at: almost 3 years ago - Stars: 124 - Forks: 19

The official PyTorch implementation of the paper "SAITS: Self-Attention-based Imputation for Time Series". A fast and state-of-the-art (SOTA) deep-learning neural network model for efficient time-series imputation (impute multivariate incomplete time series containing NaN missing data/values with machine learning). https://arxiv.org/abs/2202.08516

Language: Python - Size: 603 KB - Last synced at: 1 day ago - Pushed at: 12 days ago - Stars: 455 - Forks: 62

nitinuprety/secure-attachment-blueprint

Discover the foundation of healthy relationships and self-worth, and learn practical steps to cultivate a more secure attachment style.

Language: HTML - Size: 257 KB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 0 - Forks: 0

microsoft/DeBERTa

The implementation of DeBERTa

Language: Python - Size: 237 KB - Last synced at: 5 days ago - Pushed at: almost 2 years ago - Stars: 2,140 - Forks: 235

gordicaleksa/pytorch-GAT

My implementation of the original GAT paper (Veličković et al.). I've additionally included the playground.py file for visualizing the Cora dataset, GAT embeddings, an attention mechanism, and entropy histograms. I've supported both Cora (transductive) and PPI (inductive) examples!

Language: Jupyter Notebook - Size: 25.2 MB - Last synced at: 7 days ago - Pushed at: almost 3 years ago - Stars: 2,583 - Forks: 343

github/CodeSearchNet 📦

Datasets, tools, and benchmarks for representation learning of code.

Language: Jupyter Notebook - Size: 28.6 MB - Last synced at: 9 days ago - Pushed at: over 3 years ago - Stars: 2,363 - Forks: 405

esceptico/perceiver-io

Unofficial implementation of Perceiver IO

Language: Python - Size: 16.6 KB - Last synced at: 9 days ago - Pushed at: about 3 years ago - Stars: 126 - Forks: 5

Audio-WestlakeU/FS-EEND

The official Pytorch implementation of "Frame-wise streaming end-to-end speaker diarization with non-autoregressive self-attention-based attractors". [ICASSP 2024] and "LS-EEND: long-form streaming end-to-end neural diarization with online attractor extraction". [TASLP 2025]

Language: Python - Size: 5.67 MB - Last synced at: 11 days ago - Pushed at: 12 days ago - Stars: 140 - Forks: 6

NVlabs/MambaVision

[CVPR 2025] Official PyTorch Implementation of MambaVision: A Hybrid Mamba-Transformer Vision Backbone

Language: Python - Size: 2.76 MB - Last synced at: 13 days ago - Pushed at: about 2 months ago - Stars: 1,671 - Forks: 90

miniHuiHui/awesome-high-order-neural-network

Size: 43.9 KB - Last synced at: 3 days ago - Pushed at: 12 months ago - Stars: 51 - Forks: 5

cmhungsteve/Awesome-Transformer-Attention

An ultimately comprehensive paper list of Vision Transformer/Attention, including papers, codes, and related websites

Size: 5.65 MB - Last synced at: 13 days ago - Pushed at: about 1 year ago - Stars: 4,920 - Forks: 498

NVlabs/FasterViT

[ICLR 2024] Official PyTorch implementation of FasterViT: Fast Vision Transformers with Hierarchical Attention

Language: Python - Size: 1.22 MB - Last synced at: 14 days ago - Pushed at: about 2 months ago - Stars: 870 - Forks: 66

hyuki875/Transformers

The Transformers repository provides a comprehensive implementation of the Transformer architecture, a groundbreaking model that has revolutionized both Natural Language Processing (NLP) and Computer Vision tasks. Introduced in the seminal paper "Attention is All You Need" by Vaswani et al.

Size: 1.95 KB - Last synced at: 17 days ago - Pushed at: 17 days ago - Stars: 1 - Forks: 0

GuanRunwei/Awesome-Vision-Transformer-Collection

Variants of Vision Transformer and its downstream tasks

Size: 59.6 KB - Last synced at: 7 days ago - Pushed at: about 3 years ago - Stars: 244 - Forks: 29

Separius/awesome-fast-attention 📦

list of efficient attention modules

Language: Python - Size: 156 KB - Last synced at: 1 day ago - Pushed at: about 4 years ago - Stars: 1,012 - Forks: 106

Syeda-Farhat/awesome-Transformers-For-Segmentation

Semantic segmentation is an important job in computer vision, and its applications have grown in popularity over the last decade.We grouped the publications that used various forms of segmentation in this repository. Particularly, every paper is built on a transformer.

Size: 300 KB - Last synced at: 3 days ago - Pushed at: 6 months ago - Stars: 36 - Forks: 2

alohays/awesome-visual-representation-learning-with-transformers

Awesome Transformers (self-attention) in Computer Vision

Size: 73.2 KB - Last synced at: 9 days ago - Pushed at: about 4 years ago - Stars: 270 - Forks: 38

sim1-99/ProtACon

A pipeline for the analysis of Bert self-attention mechanism applied to proteins.

Language: Python - Size: 7.17 MB - Last synced at: 29 days ago - Pushed at: 29 days ago - Stars: 1 - Forks: 1

WHU-Sigma/HyperSIGMA

The official repo for [TPAMI'25] "HyperSIGMA: Hyperspectral Intelligence Comprehension Foundation Model"

Language: Python - Size: 80.6 MB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 273 - Forks: 22

adarsh-crafts/llama-llm-from-scratch

Educational, from-scratch implementation of a LLaMA-style LLM using PyTorch to explore Transformer architecture fundamentals.

Language: Jupyter Notebook - Size: 139 KB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 3 - Forks: 0

daiquocnguyen/Graph-Transformer

Universal Graph Transformer Self-Attention Networks (TheWebConf WWW 2022) (Pytorch and Tensorflow)

Language: Python - Size: 109 MB - Last synced at: 10 days ago - Pushed at: about 3 years ago - Stars: 674 - Forks: 77

zhouhaoyi/Informer2020

The GitHub repository for the paper "Informer" accepted by AAAI 2021.

Language: Python - Size: 6.34 MB - Last synced at: about 2 months ago - Pushed at: 3 months ago - Stars: 6,056 - Forks: 1,250

kaituoxu/Speech-Transformer

A PyTorch implementation of Speech Transformer, an End-to-End ASR with Transformer network on Mandarin Chinese.

Language: Python - Size: 678 KB - Last synced at: about 2 months ago - Pushed at: over 2 years ago - Stars: 796 - Forks: 197

mahshid1378/Parallel-Tacotron2

PyTorch Implementation of Google's Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling

Language: Python - Size: 99.3 MB - Last synced at: about 1 month ago - Pushed at: 6 months ago - Stars: 1 - Forks: 0

keonlee9420/VAENAR-TTS

PyTorch Implementation of VAENAR-TTS: Variational Auto-Encoder based Non-AutoRegressive Text-to-Speech Synthesis.

Language: Python - Size: 122 MB - Last synced at: about 2 months ago - Pushed at: about 4 years ago - Stars: 72 - Forks: 14

KhushiRajurkar/Vision-Transformer-Image-Classification

A Vision Transformer (ViT) implementation for image classification using CIFAR-10 dataset, leveraging HuggingFace's Trainer API for computational efficiency

Language: Jupyter Notebook - Size: 191 KB - Last synced at: about 2 months ago - Pushed at: about 2 months ago - Stars: 0 - Forks: 0

awsaf49/gcvit-tf

Tensorflow 2.0 Implementation of GCViT: Global Context Vision Transformer

Language: Jupyter Notebook - Size: 27.6 MB - Last synced at: 2 days ago - Pushed at: over 1 year ago - Stars: 26 - Forks: 6

ubisoft/ubisoft-laforge-daft-exprt

PyTorch Implementation of Daft-Exprt: Robust Prosody Transfer Across Speakers for Expressive Speech Synthesis

Language: Python - Size: 1.44 MB - Last synced at: about 1 month ago - Pushed at: over 2 years ago - Stars: 129 - Forks: 23

kaushalshetty/Structured-Self-Attention

A Structured Self-attentive Sentence Embedding

Language: Python - Size: 492 KB - Last synced at: about 2 months ago - Pushed at: almost 6 years ago - Stars: 493 - Forks: 104

samaraxmmar/transformer-explained

A hands-on guide to understanding and building Transformer models from scratch, with detailed explanations and practical Jupyter notebooks.

Language: Jupyter Notebook - Size: 1.78 MB - Last synced at: 2 months ago - Pushed at: 2 months ago - Stars: 0 - Forks: 0

aravindsankar28/DySAT

Representation learning on dynamic graphs using self-attention networks

Language: Python - Size: 2.39 MB - Last synced at: about 2 months ago - Pushed at: over 2 years ago - Stars: 296 - Forks: 42

The-AI-Summer/self-attention-cv

Implementation of various self-attention mechanisms focused on computer vision. Ongoing repository.

Language: Python - Size: 291 KB - Last synced at: 2 months ago - Pushed at: almost 4 years ago - Stars: 1,209 - Forks: 155

neilatuniofyorkspace/ri-language-transformer-backpropagation

Size: 6.84 KB - Last synced at: 3 months ago - Pushed at: 3 months ago - Stars: 0 - Forks: 0

veb-101/keras-vision

Porting vision models to Keras 3 for easily accessibility. Contains MobileViT v1, MobileViT v2, fastvit

Language: Jupyter Notebook - Size: 4.45 MB - Last synced at: 2 months ago - Pushed at: 5 months ago - Stars: 14 - Forks: 3

jancarauma/nanoGPT

nanoGPT - A simple GPT-Style Transformer from Scratch in PyTorch

Language: Python - Size: 9.77 KB - Last synced at: 3 months ago - Pushed at: 3 months ago - Stars: 0 - Forks: 0

NVlabs/GCVit

[ICML 2023] Official PyTorch implementation of Global Context Vision Transformers

Language: Python - Size: 858 KB - Last synced at: 3 months ago - Pushed at: over 1 year ago - Stars: 437 - Forks: 51

datawhalechina/leedl-tutorial

《李宏毅深度学习教程》（李宏毅老师推荐👍，苹果书🍎），PDF下载地址：https://github.com/datawhalechina/leedl-tutorial/releases

Language: Jupyter Notebook - Size: 295 MB - Last synced at: 3 months ago - Pushed at: 3 months ago - Stars: 15,209 - Forks: 3,037

sparks-baird/CrabNet Fork of anthony-wang/CrabNet

Predict materials properties using only the composition information!

Language: HTML - Size: 393 MB - Last synced at: 22 days ago - Pushed at: 3 months ago - Stars: 17 - Forks: 5

Razvan48/Proiect-Natural-Language-Processing-NLP

Proiect Natural Language Processing (NLP) Anul 3, Semestrul 2, Facultatea de Matematica si Informatica, Universitatea din Bucuresti

Language: Jupyter Notebook - Size: 132 MB - Last synced at: 3 months ago - Pushed at: 3 months ago - Stars: 0 - Forks: 0

PetarV-/GAT

Graph Attention Networks (https://arxiv.org/abs/1710.10903)

Language: Python - Size: 4.6 MB - Last synced at: 3 months ago - Pushed at: over 3 years ago - Stars: 3,364 - Forks: 665

jayparks/transformer

A Pytorch Implementation of "Attention is All You Need" and "Weighted Transformer Network for Machine Translation"

Language: Python - Size: 55.7 KB - Last synced at: 3 months ago - Pushed at: almost 5 years ago - Stars: 557 - Forks: 122

Diego999/pyGAT

Pytorch implementation of the Graph Attention Network model by Veličković et. al (2017, https://arxiv.org/abs/1710.10903)

Language: Python - Size: 207 KB - Last synced at: 3 months ago - Pushed at: about 2 years ago - Stars: 3,035 - Forks: 699

NhanPhamThanh-IT/Fingerprint-Matching-System 📦

This project presents a fingerprint matching system utilizing deep learning. It features multiple models, including VGG-based, SENet, CBAM, Self-Attention, and Dual-Attention architectures. Pre-trained MobileNet models with Self-Attention and SENet are also included. A web application allows for easy demonstration of the model's capabilities.

Language: Python - Size: 25.1 MB - Last synced at: 4 months ago - Pushed at: 4 months ago - Stars: 1 - Forks: 0

fudan-zvg/SOFT

[NeurIPS 2021 Spotlight] & [IJCV 2024] SOFT: Softmax-free Transformer with Linear Complexity

Language: Python - Size: 5.06 MB - Last synced at: 3 months ago - Pushed at: over 1 year ago - Stars: 310 - Forks: 25

NVlabs/FAN

Official PyTorch implementation of Fully Attentional Networks

Language: Python - Size: 8.6 MB - Last synced at: 3 months ago - Pushed at: over 2 years ago - Stars: 478 - Forks: 28

speedinghzl/CCNet

CCNet: Criss-Cross Attention for Semantic Segmentation (TPAMI 2020 & ICCV 2019).

Language: Python - Size: 3.88 MB - Last synced at: 4 months ago - Pushed at: over 4 years ago - Stars: 1,461 - Forks: 278

brightmart/bert_language_understanding

Pre-training of Deep Bidirectional Transformers for Language Understanding: pre-train TextCNN

Language: Python - Size: 16 MB - Last synced at: 4 months ago - Pushed at: over 6 years ago - Stars: 966 - Forks: 211

Dhanush-R-git/MH-Analysis

The MHRoberta is Mental Health Roberta model. The pretrained Roberta transformer based model fine-tunned on Mental Health dataset by adopting PEFT method.

Language: Jupyter Notebook - Size: 3.67 MB - Last synced at: 4 months ago - Pushed at: 4 months ago - Stars: 1 - Forks: 0

lucidrains/global-self-attention-network

A Pytorch implementation of Global Self-Attention Network, a fully-attention backbone for vision tasks

Language: Python - Size: 95.7 KB - Last synced at: 2 months ago - Pushed at: almost 5 years ago - Stars: 95 - Forks: 7

MuzzammilShah/GPT-TransformerModel-2

An end-to-end PyTorch implementation of a GPT-2 style language model (124M) released by OpenAI and inspired by Karpathy’s NanoGPT. Covers core components like tokenization, multi-head self-attention, transformer blocks, positional embeddings and various other key ML concepts.

Language: Jupyter Notebook - Size: 3.35 MB - Last synced at: 4 months ago - Pushed at: 4 months ago - Stars: 0 - Forks: 0

nicolay-r/AREnets

Tensorflow-based framework which lists attentive implementation of the conventional neural network models (CNN, RNN-based), applicable for Relation Extraction classification tasks as well as API for custom model implementation

Language: Python - Size: 1.34 MB - Last synced at: 18 days ago - Pushed at: 4 months ago - Stars: 7 - Forks: 0

codewithdark-git/Transformers

Language: Jupyter Notebook - Size: 2.09 MB - Last synced at: 29 days ago - Pushed at: 4 months ago - Stars: 0 - Forks: 0

deshwalmahesh/ML-Models-from-Scratch

Repo for ML Models built from scratch such as Self-Attention, Linear +Logistic Regression, PCA, LDA. CNN, LSTM, Neural Networks using Numpy only

Language: Jupyter Notebook - Size: 38.4 MB - Last synced at: 4 months ago - Pushed at: 7 months ago - Stars: 49 - Forks: 8

xxxnell/how-do-vits-work

(ICLR 2022 Spotlight) Official PyTorch implementation of "How Do Vision Transformers Work?"

Language: Python - Size: 18.3 MB - Last synced at: 4 months ago - Pushed at: about 3 years ago - Stars: 815 - Forks: 79

VSainteuf/pytorch-psetae

PyTorch implementation of the model presented in "Satellite Image Time Series Classification with Pixel-Set Encoders and Temporal Self-Attention"

Language: Python - Size: 1.98 MB - Last synced at: 4 months ago - Pushed at: over 3 years ago - Stars: 198 - Forks: 43

emadeldeen24/ECGTransForm

[Biomedical Signal Processing and Control] ECGTransForm: Empowering adaptive ECG arrhythmia classification framework with bidirectional transformer

Language: Python - Size: 1.11 MB - Last synced at: 4 months ago - Pushed at: 4 months ago - Stars: 40 - Forks: 8

ntat/Class-Conditional-Diffusion

Conditional Diffuser from scratch, applied on CelebA-HQ, Cifar10 and MNIST.

Size: 4.36 MB - Last synced at: 4 months ago - Pushed at: 4 months ago - Stars: 0 - Forks: 0

Namkwangwoon/Saliency-Attention-based-DETR

SA-DETR: Saliency Attention-based DETR for Salienct Object Detection

Language: Python - Size: 338 KB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 1 - Forks: 0

VSainteuf/utae-paps

PyTorch implementation of U-TAE and PaPs for satellite image time series panoptic segmentation.

Language: Jupyter Notebook - Size: 3.03 MB - Last synced at: 4 months ago - Pushed at: about 1 year ago - Stars: 161 - Forks: 58

mayankmittal29/CalcuTron-Neural-Math-Solver-with-Transformers

CalcuTron is a transformer-based sequence-to-sequence model engineered for symbolic arithmetic reasoning. Leveraging multi-head self-attention, positional encoding, and deep encoder-decoder layers, it learns to perform multi-digit addition and subtraction. It generalizes to longer sequences without explicit rules, showcasing emergent algorithmics.

Language: Jupyter Notebook - Size: 20.2 MB - Last synced at: about 2 months ago - Pushed at: 5 months ago - Stars: 0 - Forks: 0

theJingqiZhou/DS-AGC-Colab Fork of Vesan-yws/DS-AGC

A Pytorch re-implementation of paper "Semi-Supervised Dual-Stream Self-Attentive Adversarial Graph Contrastive Learning for Cross-Subject EEG-based Emotion Recognition" by Ye et al.

Language: Python - Size: 58.6 KB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 0 - Forks: 0

tiongsikng/gc2sa_net

Self-Attentive Contrastive Learning for Conditioned Periocular and Face Biometrics

Language: Jupyter Notebook - Size: 15.1 MB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 3 - Forks: 2

srinadh99/AstroFormer

Photometry Guided Cross Attention Transformers for Astronomical Image Processing

Language: Jupyter Notebook - Size: 22.2 MB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 0 - Forks: 0

asalekin-ubiquitouslab/Modality-wise-Multple-Instance-Learning

The repository contains our implementation for the work to be presented at Ubicomp 2022

Language: Jupyter Notebook - Size: 31.1 MB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 3 - Forks: 0

wenwenyu/MASTER-pytorch

Code for the paper "MASTER: Multi-Aspect Non-local Network for Scene Text Recognition" (Pattern Recognition 2021)

Language: Python - Size: 4.33 MB - Last synced at: 3 months ago - Pushed at: over 3 years ago - Stars: 280 - Forks: 51

cocoalex00/Mamba2D

Official PyTorch Implementation of Mamba2D: A Natively Multi-Dimensional State-Space Model for Vision Tasks

Language: Python - Size: 4.34 MB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 3 - Forks: 1

pashtari/deconver

Official PyTorch Implementation of "Deconver: A Deconvolutional Network for Medical Image Segmentation"

Language: Python - Size: 20.5 KB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 4 - Forks: 0

saadwazir/HistoSeg

HistoSeg is an Encoder-Decoder DCNN which utilizes the novel Quick Attention Modules and Multi Loss function to generate segmentation masks from histopathological images with greater accuracy. This repo contains the code to Test and Train the HistoSeg

Language: Python - Size: 22.8 MB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 23 - Forks: 5

zxuu/Self-Attention

Transformer的完整实现。详细构建Encoder、Decoder、Self-attention。以实际例子进行展示，有完整的输入、训练、预测过程。可用于学习理解self-attention和Transformer

Language: Python - Size: 4.79 MB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 72 - Forks: 12

babycommando/neuralgraffiti

Live-bending a foundation model’s output at neural network level.

Language: Jupyter Notebook - Size: 31.3 KB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 212 - Forks: 16

cbaziotis/neat-vision

Neat (Neural Attention) Vision, is a visualization tool for the attention mechanisms of deep-learning models for Natural Language Processing (NLP) tasks. (framework-agnostic)

Language: Vue - Size: 25.4 MB - Last synced at: 5 months ago - Pushed at: over 7 years ago - Stars: 250 - Forks: 24

francomano/PSA-GAN

PSA-GAN implementation in pytorch

Language: Jupyter Notebook - Size: 127 MB - Last synced at: 5 months ago - Pushed at: over 2 years ago - Stars: 19 - Forks: 1

anthony-wang/CrabNet

Predict materials properties using only the composition information!

Language: Python - Size: 429 MB - Last synced at: 4 months ago - Pushed at: over 2 years ago - Stars: 100 - Forks: 31

SCCSMARTCODE/attention-is-all-you-need-from-scratch

A complete implementation of the Transformer architecture from scratch, including self-attention, positional encoding, multi-head attention, and feedforward layers. This repository provides a deep understanding of Transformers and serves as a foundation for advanced NLP and deep learning models.

Language: Jupyter Notebook - Size: 25.4 KB - Last synced at: 5 months ago - Pushed at: 9 months ago - Stars: 1 - Forks: 0