GitHub topics: inference-optimization

Repositories

google/XNNPACK

High-efficiency floating-point neural network inference operators for mobile, server, and Web

Language: C - Size: 162 MB - Last synced at: about 9 hours ago - Pushed at: about 9 hours ago - Stars: 2,003 - Forks: 412

alibaba/BladeDISC

BladeDISC is an end-to-end DynamIc Shape Compiler project for machine learning workloads.

Language: C++ - Size: 21.2 MB - Last synced at: 14 days ago - Pushed at: 4 months ago - Stars: 856 - Forks: 165

imedslab/pytorch_bn_fusion

Batch normalization fusion for PyTorch

Language: Python - Size: 54.7 KB - Last synced at: 4 days ago - Pushed at: about 5 years ago - Stars: 197 - Forks: 29

vbdi/divprune

[CVPR 2025] DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models

Language: Python - Size: 11 MB - Last synced at: 21 days ago - Pushed at: 21 days ago - Stars: 7 - Forks: 0

Keshavpatel2/local-llm-workbench

🧠 A comprehensive toolkit for benchmarking, optimizing, and deploying local Large Language Models. Includes performance testing tools, optimized configurations for CPU/GPU/hybrid setups, and detailed guides to maximize LLM performance on your hardware.

Language: Shell - Size: 8.79 KB - Last synced at: 24 days ago - Pushed at: 24 days ago - Stars: 0 - Forks: 0

ZFTurbo/Keras-inference-time-optimizer

Optimize layers structure of Keras model to reduce computation time

Language: Python - Size: 77.1 KB - Last synced at: 15 days ago - Pushed at: almost 5 years ago - Stars: 157 - Forks: 18

ccs96307/fast-llm-inference

Accelerating LLM inference with techniques like speculative decoding, quantization, and kernel fusion, focusing on implementing state-of-the-art research papers.

Language: Python - Size: 142 KB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 7 - Forks: 1

keli-wen/AGI-Study

The blog, read report and code example for AGI/LLM related knowledge.

Language: Python - Size: 19.5 MB - Last synced at: 12 days ago - Pushed at: 3 months ago - Stars: 36 - Forks: 2

ksm26/Efficiently-Serving-LLMs

Learn the ins and outs of efficiently serving Large Language Models (LLMs). Dive into optimization techniques, including KV caching and Low Rank Adapters (LoRA), and gain hands-on experience with Predibase’s LoRAX framework inference server.

Language: Jupyter Notebook - Size: 2.34 MB - Last synced at: 26 days ago - Pushed at: about 1 year ago - Stars: 11 - Forks: 3

shreyansh26/Accelerating-Cross-Encoder-Inference

Leveraging torch.compile to accelerate cross-encoder inference

Language: Python - Size: 3.84 MB - Last synced at: about 1 month ago - Pushed at: about 2 months ago - Stars: 0 - Forks: 0

grazder/template.cpp

A template for getting started writing code using GGML

Language: C++ - Size: 40 KB - Last synced at: 18 days ago - Pushed at: 12 months ago - Stars: 9 - Forks: 0

EZ-Optimium/Optimium

Your AI Catalyst: inference backend to maximize your model's inference performance

Language: C++ - Size: 101 MB - Last synced at: about 17 hours ago - Pushed at: 4 months ago - Stars: 5 - Forks: 0

OneAndZero24/TRTTL

TensorRT C++ Template Library

Language: C++ - Size: 423 KB - Last synced at: 4 months ago - Pushed at: 4 months ago - Stars: 0 - Forks: 0

Harly-1506/Faster-Inference-yolov8

Faster inference YOLOv8: Optimize and export YOLOv8 models for faster inference using OpenVINO and Numpy 🔢

Language: Python - Size: 49.8 MB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 8 - Forks: 1

amazon-science/mlp-rank-pruning

MLP-Rank: A graph theoretical approach to structured pruning of deep neural networks based on weighted Page Rank centrality as introduced by the related thesis.

Language: Python - Size: 60.5 KB - Last synced at: 16 days ago - Pushed at: about 1 year ago - Stars: 3 - Forks: 1

piotrostr/infer-trt

Interface for TensorRT engines inference along with an example of YOLOv4 engine being used.

Language: Python - Size: 17.6 KB - Last synced at: 2 months ago - Pushed at: almost 3 years ago - Stars: 2 - Forks: 0

jiazhihao/TASO

The Tensor Algebra SuperOptimizer for Deep Learning

Language: C++ - Size: 1.21 MB - Last synced at: 6 months ago - Pushed at: about 2 years ago - Stars: 690 - Forks: 91

yester31/TensorRT_Examples

All useful sample codes of tensorrt models using onnx

Language: Python - Size: 240 KB - Last synced at: 6 months ago - Pushed at: 6 months ago - Stars: 1 - Forks: 1

matteo-stat/transformers-nlp-multi-label-classification

This repo provides scripts for fine-tuning HuggingFace Transformers, setting up pipelines and optimizing multi-label classification models for inference. They are based on my experience developing a custom chatbot, I’m sharing these in the hope they will help others to quickly fine-tune and use models in their projects! 😊

Language: Python - Size: 31.3 KB - Last synced at: about 2 months ago - Pushed at: 8 months ago - Stars: 0 - Forks: 0

matteo-stat/transformers-nlp-ner-token-classification

This repo provides scripts for fine-tuning HuggingFace Transformers, setting up pipelines and optimizing token classification models for inference. They are based on my experience developing a custom chatbot, I’m sharing these in the hope they will help others to quickly fine-tune and use models in their projects! 😊

Language: Python - Size: 22.5 KB - Last synced at: about 2 months ago - Pushed at: 8 months ago - Stars: 0 - Forks: 0

manickavela29/EmoTwitter

OnnxRT based Inference Optimization of Roberta model trained for Sentiment Analysis On Twitter Dataset

Language: Jupyter Notebook - Size: 12.7 KB - Last synced at: 11 months ago - Pushed at: 11 months ago - Stars: 0 - Forks: 0

Bisonai/ncnn Fork of Tencent/ncnn

Modified inference engine for quantized convolution using product quantization

Language: C++ - Size: 7.96 MB - Last synced at: 12 months ago - Pushed at: almost 3 years ago - Stars: 4 - Forks: 0

cedrickchee/pytorch-mobile-android Fork of pytorch/android-demo-app

PyTorch Mobile: Android examples of usage in applications

Size: 53 MB - Last synced at: about 1 year ago - Pushed at: over 5 years ago - Stars: 0 - Forks: 1

cedrickchee/pytorch-mobile-ios Fork of pytorch/ios-demo-app

PyTorch Mobile: iOS examples

Size: 47.3 MB - Last synced at: about 1 year ago - Pushed at: over 5 years ago - Stars: 1 - Forks: 0

ieee820/ncnn Fork of Tencent/ncnn

ncnn is a high-performance neural network inference framework optimized for the mobile platform

Language: C++ - Size: 6.81 MB - Last synced at: almost 2 years ago - Pushed at: almost 6 years ago - Stars: 0 - Forks: 0

Related Keywords

inference-optimization 36 deep-learning 8 onnx 6 pytorch 5 machine-learning 5 neural-network 5 tensorrt 4 onnxruntime 4 object-detection 3 inference 3 inference-engine 3 quantization 3 openvino-toolkit 2 ultralytics 2 yolov8 2 cuda 2 huggingface-transformers 2 acceleration 2 edge-ai 2 libtorch 2 computer-vision 2 pytorch-mobile 2 amd 2 cpp 2 huggingface-pipelines 2 nlp 2 transformers 2 tensorflow 2 cpu 2 deep-neural-networks 2 huggingface 2 llm 2 fine-tuning 2 pruning 2 pandas 1 ray-tune 1 optimization-algorithms 1 darknet 1 openvino-inference-engine 1 live-streaming 1 average-precision 1 onnx-torch 1 ios-app 1 yolov3 1 acceleration-model 1 multi-label-classification 1 text-classification 1 android-app 1 product-quantization 1 mobile-deep-learning 1 inference-acceleration 1 edge-machine-learning 1 sentiment-analysis 1 roberta-model 1 named-entity-recognition 1 ner 1 token-classification 1 bert-models 1 avx512 1 avx2 1 vgg 1 tensorflow-models 1 conv1d 1 eigen 1 eigen3 1 lstm 1 cuda-kernels 1 mobilenet-v2 1 data-parallelism 1 dl-compiler 1 dl-optimization 1 tvm 1 batch-optimization 1 g2o 1 lie-groups 1 state-estimation 1 expectation-maximisation-algorithm 1 expectation-maximization 1 feasibility 1 network-inference 1 arm-neon 1 mobile-networks 1 ncnn 1 cnn-classification 1 cudf 1 cuml 1 nvidia-rapids 1 python3 1 torchscript 1 xgboost 1 cnn 1 parallelism 1 amdgpu 1 caffe 1 docker-images 1 inceptionv4 1 mivisionx 1 mivisionx-inference-analyzer 1 nnef 1 nnir 1