LLM server for inference
A reference table of the main LLM inference engines and servers along with their GitHub and documentation resources.
Less than 1 minute(122 words)simple
Quick Navigation
Difficulty: Intermediate
Estimated Time: 5-10 minutes
Prerequisites: basic LLM knowledge, familiarity with GPUs, Python environment, Docker basics
Here's the complete table with only the first and last columns in a single table:
| Inference Engine/Server | Resources |
|---|---|
| vLLM | GitHub, Documentation |
| TensorRT-LLM | GitHub, Documentation |
| Hugging Face Text Generation Inference | GitHub, Documentation |
| RayLLM with RayServe | GitHub, Documentation |
| Triton Inference Server with TensorRT-LLM | GitHub, Documentation |
| Ollama | Website, GitHub |
| NVIDIA NIM (NeMo Inference Manager) | Website, GitHub |
| llama.cpp | GitHub |
This table provides the names of the inference engines/servers along with their respective resource links.