GitHub / fcn94 / llm_stream_endpoint
Simple LLM Rest API using Rust, Warp and Candle. Dedicated for quantized version of either phi-2 ( default) , Mistral, or Llama. Work using CPU or CUDA
JSON API: http://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fcn94%2Fllm_stream_endpoint
Stars: 4
Forks: 1
Open issues: 0
License: apache-2.0
Language: Rust
Size: 74.2 KB
Dependencies parsed at: Pending
Created at: over 1 year ago
Updated at: 8 months ago
Pushed at: about 1 year ago
Last synced at: 10 days ago
Commit Stats
Commits: 51
Authors: 2
Mean commits per author: 25.5
Development Distribution Score: 0.02
More commit stats: https://commits.ecosyste.ms/hosts/GitHub/repositories/fcn94/llm_stream_endpoint
Topics: api, candle, cpu, cuda, cuda-kernels, endpoint, llama2, llm, llm-server, mistral-7b, phi-2, quantized, rest, rest-api, rust, streaming