NetEase-Media/grps_trtllm

Higher performance OpenAI LLM service than vLLM serve: A pure C++ high-performance OpenAI LLM service implemented with GPRS+TensorRT-LLM+Tokenizers.cpp, supporting chat and function call, AI agents, distributed multi-GPU inference, multimodal capabilities, and a Gradio chat interface.

Language: Python - Size: 128 MB - Last synced at: about 19 hours ago - Pushed at: about 20 hours ago - Stars: 133 - Forks: 9