llm-d Distributed LLM Inference: Run the Router Without Kubernetes and Tune It

What llm-d is, how its inference-aware router works, how to run it with Docker and no Kubernetes, and how each router and vLLM parameter changes latency and throughput

8-14 minutes(1906 words)complex

Quick Navigation

Difficulty: Advanced
Estimated Time: 30-45 minutes
Prerequisites: Working knowledge of vLLM, NVIDIA GPUs with CUDA drivers, Docker and Docker Compose, Understanding of KV cache and prefill/decode phases

What You'll Learn

This tutorial covers llm-d, the CNCF Sandbox project for distributed LLM inference:

  • Architecture - Router, Endpoint Picker, vLLM workers, and how a request flows
  • Core Features - Inference-aware routing, prefill/decode disaggregation, KV-cache offloading
  • Version History - What changed from v0.5 to v1.0
  • No-Kubernetes Deployment - File discovery, Envoy, and vLLM workers in containers
  • Tuning - Router scorer weights and vLLM flags, with the effect of raising or lowering each
  • Monitoring and Benchmarking - Which signals to watch while you tune
  • Troubleshooting - Common failures and fixes

Prerequisites

  • Working knowledge of vLLM serving and its main flags
  • One or more NVIDIA GPUs with current drivers and the NVIDIA Container Toolkit
  • Docker and Docker Compose
  • Understanding of KV cache, prefix caching, and the prefill and decode phases
  • A HuggingFace token if your model is gated

Introduction

A standard load balancer treats LLM replicas as interchangeable. They are not. Each vLLM replica holds a KV cache keyed by prompt prefix, and a request routed to a replica with a matching prefix skips most of its prefill work. Round-robin throws that advantage away and spreads long and short requests without regard to queue depth.

llm-d fixes this at the routing layer. It is a Kubernetes-native distributed inference stack built on top of vLLM, launched in May 2025 by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA. It was accepted into the CNCF Sandbox on March 24, 2026.

Kubernetes is the reference deployment, but the routing logic does not depend on it. The router can read its endpoints from a YAML file, which lets you run the same scheduler in Docker, on Slurm, on Ray, or on bare metal. This tutorial focuses on that path.

How llm-d Works

llm-d is a layer, not an engine. vLLM still performs batching, PagedAttention, and prefix caching inside each replica. llm-d adds three capabilities around it.

Inference-Aware Routing

The Endpoint Picker (EPP) discovers endpoints, filters them, scores them, and picks one. Typical scorers include:

  • Prefix-cache scorer - favors replicas that already hold the request's prompt prefix
  • Queue-depth scorer - favors replicas with shorter request queues
  • KV-cache-utilization scorer - favors replicas with free cache space
  • No-hit LRU scorer - for cold requests, favors replicas that have never handled one, then the least recently used

Prefill/Decode Disaggregation

Prefill processes the whole prompt in one forward pass and is compute-bound. Decode generates tokens one at a time and is memory-bandwidth-bound. llm-d runs them on separate replicas. The decode replica pulls KV blocks from the prefill replica over RDMA (InfiniBand, RoCE, or EFA) using NIXL.

This helps most for long-context workloads, around 10:1 input-to-output tokens, on medium-to-large models.

KV-Cache Offloading

Through vLLM's KVConnector API, caches can move to host memory, remote storage, or systems like LMCache, extending capacity beyond GPU memory.

Version History

VersionHighlights
v0.5 (Feb 2026)Reproducible benchmark workflows, hierarchical KV offloading, cache-aware LoRA routing, active-active HA, scale-to-zero autoscaling
v0.7 (May 2026)Core component renamed the llm-d Router; standalone mode by default; Kustomize replaces Helm for guides; experimental Batch Gateway
v1.0Described by the project as multi-tenant quality of service, zero-downtime upgrades, and agent-aware serving on a stable v1 API

Operational notes from the renames:

  • llm-d-inference-scheduler became llm-d-router
  • The old llm-d-infra and llm-d-modelservice Helm charts are deprecated
  • CUDA images moved to CUDA 13.0.2 in v0.7, which requires NVIDIA driver 580 or later

Pin image versions and read the release notes before upgrading.

Running Without Kubernetes

What You Run

The no-Kubernetes path runs three kinds of processes or containers:

  1. vLLM workers - one per GPU or GPU group
  2. The EPP - image ghcr.io/llm-d/llm-d-router-endpoint-picker
  3. Envoy - the proxy, configured with ext_proc pointed at the EPP

Traffic flows as follows:

client -> Envoy :8081
       -> ext_proc gRPC :9002 (EPP picks a worker and sets a header)
       -> vLLM worker chosen by the EPP

The Endpoints File

The EPP gets its worker list from a YAML file through the file-discovery plugin instead of watching a Kubernetes InferencePool.

  • Each entry needs a unique name and a literal address:port
  • Hostnames are not resolved, so use IP addresses
  • Optional labels are passed to scheduler plugins, for example llm-d.ai/role: prefill or decode
  • With watchFile: true, the file hot-reloads on atomic rewrite, so adding or removing a worker needs no EPP restart

One line in the EPP config switches discovery to the file: dataLayer.discovery.pluginRef: file-discovery. The upstream repo ships a ready config at guides/no-kubernetes-deployment/router/epp/config.yaml with the Optimized Baseline plugin mix already wired in. Start from that file.

Start the vLLM Workers

This Compose file runs two workers, each pinned to its own GPU:

services:
  vllm-a:
    image: vllm/vllm-openai:latest
    ipc: host
    environment:
      - HF_TOKEN=${HF_TOKEN}
    command: >
      --model Qwen/Qwen2.5-7B-Instruct
      --enable-prefix-caching
      --gpu-memory-utilization 0.9
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["0"]
              capabilities: [gpu]
    networks:
      llmd:
        ipv4_address: 172.28.0.11

  vllm-b:
    image: vllm/vllm-openai:latest
    ipc: host
    environment:
      - HF_TOKEN=${HF_TOKEN}
    command: >
      --model Qwen/Qwen2.5-7B-Instruct
      --enable-prefix-caching
      --gpu-memory-utilization 0.9
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["1"]
              capabilities: [gpu]
    networks:
      llmd:
        ipv4_address: 172.28.0.12

networks:
  llmd:
    ipam:
      config:
        - subnet: 172.28.0.0/24

The fixed IP addresses matter. Since the endpoints file takes literal address:port values, the workers need stable addresses.

Add the epp and envoy services using the commands and Envoy config from the upstream no-Kubernetes guide, and list 172.28.0.11:8000 and 172.28.0.12:8000 in the endpoints file.

Send a Test Request

Requests go to Envoy, not to a worker:

curl http://localhost:8081/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "messages": [{"role": "user", "content": "Summarize what a KV cache is."}],
    "max_tokens": 128
  }'

Send the same long system prompt several times and watch the prefix-cache metrics on the workers. Repeat requests should land on the same replica.

Limits Without Kubernetes

Out of scope:

  • InferenceObjective-driven flow control
  • The InferenceModelRewrite model-name rewriter
  • PodMonitor-based Prometheus discovery

Still working:

  • Scoring and prefix-cache affinity
  • Saturation-based admission
  • Prometheus metrics on --metrics-port

Lifecycle also changes. On Kubernetes a dead pod leaves the pool automatically. Here, your own script or health checker must remove a failed worker from the endpoints file.

Tuning Parameters

Two layers interact: scorer weights in the EPP and flags on each vLLM worker. Change one at a time and measure.

Router Scorer Weights

ParameterIncreaseDecrease
Prefix-cache scorer weightMore cache hits, lower TTFT on shared prompts; traffic concentrates on fewer replicasMore even spread; less cache reuse
Queue-depth / load scorer weightBetter balance and tail latencyHigher risk of overloading one replica
KV-cache-utilization scorer weightAvoids nearly full replicas, fewer evictionsRisk of cache pressure and preemption

In llm-d's own test with heavy prefix reuse, a prefix-only scorer overloaded replicas and succeeded on about 55% of requests, while prefix plus load kept 100% success. With little prefix sharing, both performed alike. Keep both weights non-zero and raise the prefix weight only when prompts share long prefixes, such as system prompts, RAG context, or chat history.

vLLM Flags

FlagIncreaseDecrease
--gpu-memory-utilizationMore KV-cache space and concurrency; higher out-of-memory riskMore headroom; smaller cache
--max-num-seqsHigher throughput; higher per-request latencySnappier single requests; underused GPU
--max-num-batched-tokensBetter prefill throughput and TTFT on long prompts; worse inter-token latencySmoother inter-token latency; slower prefill
--tensor-parallel-sizeFits larger models, lower per-token latency; communication overhead, fewer replicas to route betweenMore replicas and routing choice
--max-model-lenAccepts longer requests; less memory for cacheFrees memory; long requests rejected

Prefill/Decode Ratio

llm-d's reference example uses 4 prefill replicas at TP=1 and 1 decode replica at TP=5. Add prefill replicas when TTFT is high. Add decode capacity when tokens stream slowly. Skip disaggregation for short prompts on small models, where KV transfer cost can outweigh the gain.

Monitoring and Benchmarking

Watch these signals while you tune:

  • TTFT - time to first token
  • Inter-token latency - gap between streamed tokens
  • Queue depth per replica
  • KV-cache utilization per replica, also exported as an average across the pool
  • Prefix-cache hit rate

Use the llm-d-benchmark tooling for reproducible runs, and test with traffic shaped like your own. A benchmark with unique prompts will understate prefix-aware routing, and one with a single shared prompt will overstate it.

Troubleshooting

Issue: Requests Never Reach a Worker

Solution: Confirm the endpoints file uses literal IP addresses, not hostnames, and that each entry has a unique name.

Issue: A Dead Worker Still Receives Traffic

Solution: Without Kubernetes, remove failed workers from the file yourself. Add a small health check that rewrites it atomically.

Issue: CUDA Out of Memory on a Worker

Solution: Lower --gpu-memory-utilization or --max-model-len.

Issue: One Replica Is Overloaded

Solution: Raise the queue-depth or load scorer weight, or lower the prefix-cache weight.

Issue: Images Fail on an Older Host

Solution: CUDA images from v0.7 onward need NVIDIA driver 580 or later. Upgrade the driver or pin an earlier image.

Production Considerations

Stability

  • llm-d is a CNCF Sandbox project, so expect API changes between releases
  • Pin image versions and test upgrades on a staging pool

Operations

  • Own worker lifecycle: detect failures and rewrite the endpoints file
  • Scrape metrics from the EPP and every worker
  • Prefer Kubernetes for autoscaling and rolling upgrades at scale

Performance Claims

  • Published benchmark figures are the project's own and depend on workload shape
  • Reproduce them on your traffic before sizing a GPU purchase

Conclusion

llm-d moves LLM request placement from random to informed. The router scores each replica on cache locality, queue depth, and cache utilization, and the discovery plugin decides only where endpoints come from, which is why the same scheduler runs in Docker, on Slurm, or on Kubernetes.

Treat its gains as workload-dependent. Heavy prefix reuse and long contexts reward it most, while short unique prompts on a single GPU may not need it.

Key Takeaways

  • Prefix-Aware Routing - Routing to the replica that already holds the prefix cuts redundant prefill
  • Load Still Matters - Prefix-only scoring overloaded replicas in llm-d's own test, so combine it with load signals
  • Kubernetes Is Optional - File discovery runs the same router with no cluster
  • You Own the Lifecycle Off-Cluster - Failed workers must be removed from the endpoints file
  • Measure Your Own Traffic - Published numbers are a starting hypothesis

Tags: #llmd #CNCF #vLLM #LLMInference #KVCache #PrefixCaching #MLOps #AIInfrastructure #Docker