llm-d Distributed LLM Inference: Run the Router Without Kubernetes and Tune It
What llm-d is, how its inference-aware router works, how to run it with Docker and no Kubernetes, and how each router and vLLM parameter changes latency and throughput
Quick Navigation
Difficulty: Advanced
Estimated Time: 30-45 minutes
Prerequisites: Working knowledge of vLLM, NVIDIA GPUs with CUDA drivers, Docker and Docker Compose, Understanding of KV cache and prefill/decode phases
What You'll Learn
This tutorial covers llm-d, the CNCF Sandbox project for distributed LLM inference:
- Architecture - Router, Endpoint Picker, vLLM workers, and how a request flows
- Core Features - Inference-aware routing, prefill/decode disaggregation, KV-cache offloading
- Version History - What changed from v0.5 to v1.0
- No-Kubernetes Deployment - File discovery, Envoy, and vLLM workers in containers
- Tuning - Router scorer weights and vLLM flags, with the effect of raising or lowering each
- Monitoring and Benchmarking - Which signals to watch while you tune
- Troubleshooting - Common failures and fixes
Prerequisites
- Working knowledge of vLLM serving and its main flags
- One or more NVIDIA GPUs with current drivers and the NVIDIA Container Toolkit
- Docker and Docker Compose
- Understanding of KV cache, prefix caching, and the prefill and decode phases
- A HuggingFace token if your model is gated
Related Tutorials
- vLLM Inference - Install and serve models with vLLM
- LLM KV Cache Calculation - Size the KV cache for your model
- LLM Inference Server - Inference server fundamentals
- LLM Performance Monitoring - Metrics for serving latency and throughput
- Scaling LLMs Challenges - Why LLM serving is hard to scale
- Main Tutorials Hub - Step-by-step implementation guides
Introduction
A standard load balancer treats LLM replicas as interchangeable. They are not. Each vLLM replica holds a KV cache keyed by prompt prefix, and a request routed to a replica with a matching prefix skips most of its prefill work. Round-robin throws that advantage away and spreads long and short requests without regard to queue depth.
llm-d fixes this at the routing layer. It is a Kubernetes-native distributed inference stack built on top of vLLM, launched in May 2025 by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA. It was accepted into the CNCF Sandbox on March 24, 2026.
Kubernetes is the reference deployment, but the routing logic does not depend on it. The router can read its endpoints from a YAML file, which lets you run the same scheduler in Docker, on Slurm, on Ray, or on bare metal. This tutorial focuses on that path.
How llm-d Works
llm-d is a layer, not an engine. vLLM still performs batching, PagedAttention, and prefix caching inside each replica. llm-d adds three capabilities around it.
Inference-Aware Routing
The Endpoint Picker (EPP) discovers endpoints, filters them, scores them, and picks one. Typical scorers include:
- Prefix-cache scorer - favors replicas that already hold the request's prompt prefix
- Queue-depth scorer - favors replicas with shorter request queues
- KV-cache-utilization scorer - favors replicas with free cache space
- No-hit LRU scorer - for cold requests, favors replicas that have never handled one, then the least recently used
Prefill/Decode Disaggregation
Prefill processes the whole prompt in one forward pass and is compute-bound. Decode generates tokens one at a time and is memory-bandwidth-bound. llm-d runs them on separate replicas. The decode replica pulls KV blocks from the prefill replica over RDMA (InfiniBand, RoCE, or EFA) using NIXL.
This helps most for long-context workloads, around 10:1 input-to-output tokens, on medium-to-large models.
KV-Cache Offloading
Through vLLM's KVConnector API, caches can move to host memory, remote storage, or systems like LMCache, extending capacity beyond GPU memory.
Version History
| Version | Highlights |
|---|---|
| v0.5 (Feb 2026) | Reproducible benchmark workflows, hierarchical KV offloading, cache-aware LoRA routing, active-active HA, scale-to-zero autoscaling |
| v0.7 (May 2026) | Core component renamed the llm-d Router; standalone mode by default; Kustomize replaces Helm for guides; experimental Batch Gateway |
| v1.0 | Described by the project as multi-tenant quality of service, zero-downtime upgrades, and agent-aware serving on a stable v1 API |
Operational notes from the renames:
llm-d-inference-schedulerbecamellm-d-router- The old
llm-d-infraandllm-d-modelserviceHelm charts are deprecated - CUDA images moved to CUDA 13.0.2 in v0.7, which requires NVIDIA driver 580 or later
Pin image versions and read the release notes before upgrading.
Running Without Kubernetes
What You Run
The no-Kubernetes path runs three kinds of processes or containers:
- vLLM workers - one per GPU or GPU group
- The EPP - image
ghcr.io/llm-d/llm-d-router-endpoint-picker - Envoy - the proxy, configured with ext_proc pointed at the EPP
Traffic flows as follows:
client -> Envoy :8081
-> ext_proc gRPC :9002 (EPP picks a worker and sets a header)
-> vLLM worker chosen by the EPP
The Endpoints File
The EPP gets its worker list from a YAML file through the file-discovery plugin instead of watching a Kubernetes InferencePool.
- Each entry needs a unique
nameand a literaladdress:port - Hostnames are not resolved, so use IP addresses
- Optional
labelsare passed to scheduler plugins, for examplellm-d.ai/role: prefillordecode - With
watchFile: true, the file hot-reloads on atomic rewrite, so adding or removing a worker needs no EPP restart
One line in the EPP config switches discovery to the file: dataLayer.discovery.pluginRef: file-discovery. The upstream repo ships a ready config at guides/no-kubernetes-deployment/router/epp/config.yaml with the Optimized Baseline plugin mix already wired in. Start from that file.
Start the vLLM Workers
This Compose file runs two workers, each pinned to its own GPU:
services:
vllm-a:
image: vllm/vllm-openai:latest
ipc: host
environment:
- HF_TOKEN=${HF_TOKEN}
command: >
--model Qwen/Qwen2.5-7B-Instruct
--enable-prefix-caching
--gpu-memory-utilization 0.9
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids: ["0"]
capabilities: [gpu]
networks:
llmd:
ipv4_address: 172.28.0.11
vllm-b:
image: vllm/vllm-openai:latest
ipc: host
environment:
- HF_TOKEN=${HF_TOKEN}
command: >
--model Qwen/Qwen2.5-7B-Instruct
--enable-prefix-caching
--gpu-memory-utilization 0.9
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids: ["1"]
capabilities: [gpu]
networks:
llmd:
ipv4_address: 172.28.0.12
networks:
llmd:
ipam:
config:
- subnet: 172.28.0.0/24
The fixed IP addresses matter. Since the endpoints file takes literal address:port values, the workers need stable addresses.
Add the epp and envoy services using the commands and Envoy config from the upstream no-Kubernetes guide, and list 172.28.0.11:8000 and 172.28.0.12:8000 in the endpoints file.
Send a Test Request
Requests go to Envoy, not to a worker:
curl http://localhost:8081/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-7B-Instruct",
"messages": [{"role": "user", "content": "Summarize what a KV cache is."}],
"max_tokens": 128
}'
Send the same long system prompt several times and watch the prefix-cache metrics on the workers. Repeat requests should land on the same replica.
Limits Without Kubernetes
Out of scope:
InferenceObjective-driven flow control- The
InferenceModelRewritemodel-name rewriter PodMonitor-based Prometheus discovery
Still working:
- Scoring and prefix-cache affinity
- Saturation-based admission
- Prometheus metrics on
--metrics-port
Lifecycle also changes. On Kubernetes a dead pod leaves the pool automatically. Here, your own script or health checker must remove a failed worker from the endpoints file.
Tuning Parameters
Two layers interact: scorer weights in the EPP and flags on each vLLM worker. Change one at a time and measure.
Router Scorer Weights
| Parameter | Increase | Decrease |
|---|---|---|
| Prefix-cache scorer weight | More cache hits, lower TTFT on shared prompts; traffic concentrates on fewer replicas | More even spread; less cache reuse |
| Queue-depth / load scorer weight | Better balance and tail latency | Higher risk of overloading one replica |
| KV-cache-utilization scorer weight | Avoids nearly full replicas, fewer evictions | Risk of cache pressure and preemption |
In llm-d's own test with heavy prefix reuse, a prefix-only scorer overloaded replicas and succeeded on about 55% of requests, while prefix plus load kept 100% success. With little prefix sharing, both performed alike. Keep both weights non-zero and raise the prefix weight only when prompts share long prefixes, such as system prompts, RAG context, or chat history.
vLLM Flags
| Flag | Increase | Decrease |
|---|---|---|
--gpu-memory-utilization | More KV-cache space and concurrency; higher out-of-memory risk | More headroom; smaller cache |
--max-num-seqs | Higher throughput; higher per-request latency | Snappier single requests; underused GPU |
--max-num-batched-tokens | Better prefill throughput and TTFT on long prompts; worse inter-token latency | Smoother inter-token latency; slower prefill |
--tensor-parallel-size | Fits larger models, lower per-token latency; communication overhead, fewer replicas to route between | More replicas and routing choice |
--max-model-len | Accepts longer requests; less memory for cache | Frees memory; long requests rejected |
Prefill/Decode Ratio
llm-d's reference example uses 4 prefill replicas at TP=1 and 1 decode replica at TP=5. Add prefill replicas when TTFT is high. Add decode capacity when tokens stream slowly. Skip disaggregation for short prompts on small models, where KV transfer cost can outweigh the gain.
Monitoring and Benchmarking
Watch these signals while you tune:
- TTFT - time to first token
- Inter-token latency - gap between streamed tokens
- Queue depth per replica
- KV-cache utilization per replica, also exported as an average across the pool
- Prefix-cache hit rate
Use the llm-d-benchmark tooling for reproducible runs, and test with traffic shaped like your own. A benchmark with unique prompts will understate prefix-aware routing, and one with a single shared prompt will overstate it.
Troubleshooting
Issue: Requests Never Reach a Worker
Solution: Confirm the endpoints file uses literal IP addresses, not hostnames, and that each entry has a unique name.
Issue: A Dead Worker Still Receives Traffic
Solution: Without Kubernetes, remove failed workers from the file yourself. Add a small health check that rewrites it atomically.
Issue: CUDA Out of Memory on a Worker
Solution: Lower --gpu-memory-utilization or --max-model-len.
Issue: One Replica Is Overloaded
Solution: Raise the queue-depth or load scorer weight, or lower the prefix-cache weight.
Issue: Images Fail on an Older Host
Solution: CUDA images from v0.7 onward need NVIDIA driver 580 or later. Upgrade the driver or pin an earlier image.
Production Considerations
Stability
- llm-d is a CNCF Sandbox project, so expect API changes between releases
- Pin image versions and test upgrades on a staging pool
Operations
- Own worker lifecycle: detect failures and rewrite the endpoints file
- Scrape metrics from the EPP and every worker
- Prefer Kubernetes for autoscaling and rolling upgrades at scale
Performance Claims
- Published benchmark figures are the project's own and depend on workload shape
- Reproduce them on your traffic before sizing a GPU purchase
Conclusion
llm-d moves LLM request placement from random to informed. The router scores each replica on cache locality, queue depth, and cache utilization, and the discovery plugin decides only where endpoints come from, which is why the same scheduler runs in Docker, on Slurm, or on Kubernetes.
Treat its gains as workload-dependent. Heavy prefix reuse and long contexts reward it most, while short unique prompts on a single GPU may not need it.
Key Takeaways
- Prefix-Aware Routing - Routing to the replica that already holds the prefix cuts redundant prefill
- Load Still Matters - Prefix-only scoring overloaded replicas in llm-d's own test, so combine it with load signals
- Kubernetes Is Optional - File discovery runs the same router with no cluster
- You Own the Lifecycle Off-Cluster - Failed workers must be removed from the endpoints file
- Measure Your Own Traffic - Published numbers are a starting hypothesis
Tags: #llmd #CNCF #vLLM #LLMInference #KVCache #PrefixCaching #MLOps #AIInfrastructure #Docker