Topic: "dynamic-batching"
NetEase-Media/grps
Deep Learning Deployment Framework: Supports tf/torch/trt/trtllm/vllm and other NN frameworks. Support dynamic batching, and streaming modes. It is dual-language compatible with Python and C++, offering scalability, extensibility, and high performance. It helps users quickly deploy models and provide services through HTTP/RPC interfaces.
Language: C++ - Size: 67.8 MB - Last synced at: 24 days ago - Pushed at: about 2 months ago - Stars: 164 - Forks: 13
microsoft/batch-inference
Dynamic batching library for Deep Learning inference. Tutorials for LLM, GPT scenarios.
Language: Python - Size: 271 KB - Last synced at: 5 days ago - Pushed at: 10 months ago - Stars: 97 - Forks: 3
chncwang/InsNet
InsNet Runs Instance-dependent Neural Networks with Padding-free Dynamic Batching.
Language: C++ - Size: 6.52 MB - Last synced at: about 1 year ago - Pushed at: over 3 years ago - Stars: 65 - Forks: 12
theIbrahimStudio/finder.LiveBatch
A lightweight, framework-agnostic middleware that dynamically batches inference requests in real time to maximize GPU/TPU utilization.
Language: Go - Size: 19.5 KB - Last synced at: 12 days ago - Pushed at: about 1 month ago - Stars: 0 - Forks: 0
