LLM Caching with LiteLLM and vLLM

Quick Navigation

Difficulty: Intermediate
Estimated Time: 25-35 minutes
Prerequisites: Basic LLM concepts, Familiarity with Redis, Python or YAML configuration, Understanding of API latency

Caching an ordinary API saves milliseconds. Caching an LLM saves money, and on self-hosted infrastructure it saves GPU time, which is usually the scarcer resource. The techniques differ enough that applying classic cache thinking to an LLM gateway gets you the wrong answer.

Cache-Aside: The Baseline Pattern

The most common pattern for any API: check Redis first, hit the database only on a miss.

1. Build a readable key: user:42, products:page=2&sort=price
2. GET the key. If found, return it (cache hit)
3. Otherwise query the database, SET with a TTL, then return
4. Invalidate with DEL on every write that touches that data

import { createClient } from 'redis';
const redis = await createClient({ url: process.env.REDIS_URL }).connect();

app.get('/products/:id', async (req, res) => {
  const key = `product:${req.params.id}`;
  const cached = await redis.get(key);
  if (cached) return res.json(JSON.parse(cached));

  const product = await db.findProduct(req.params.id);
  await redis.set(key, JSON.stringify(product), { EX: 300 }); // 5 min TTL
  res.json(product);
});

app.put('/products/:id', async (req, res) => {
  const product = await db.updateProduct(req.params.id, req.body);
  await redis.del(`product:${req.params.id}`);
  res.json(product);
});

Four things that are easy to get wrong:

• Always set a TTL, even a long one. It is your safety net against stale data you forgot to invalidate.
• Never block on Redis. Wrap calls in try/catch and fall through to the database if Redis is down.
• Never cache per-user responses without the user ID in the key. This is the classic data leak.
• Serialize to JSON. Redis stores strings and bytes, nothing else.

Why Cache At All

• Latency — Redis answers from memory in under 1 ms. A SQL query with joins, or an external API call, runs 50 to 500 ms.
• Database load — often the real win. The database is the hardest resource to scale. If 90 percent of reads are served by Redis, you postpone adding replicas for a long time.
• Cost and traffic spikes — absorb a spike without multiplying servers, and protect against the one popular endpoint that flattens the database.

When It Is Not Worth It

Data that changes on every read, low traffic, or a query that already runs in 5 ms with a good index. Caching adds real complexity: invalidation, stale data, and one more component that can fail.

Before caching, check your indexes. A large share of "performance problems" are just a badly indexed query.

Useful rule: cache what is read often and written rarely. Product catalogues, configuration, public profiles, aggregations.

Caching for LLMs

The economics change. A request costs money, not just milliseconds.

Three Levels

1. Exact cache — the key is a hash of the full prompt, plus model, temperature and system prompt. Simple and reliable, but the hit rate is low because humans rephrase. Only meaningful at temperature=0.

2. Semantic cache — embed the question, find the nearest stored prompt, and reuse the answer if similarity exceeds a threshold (around 0.95). Much better hit rates on a support chatbot. The risk is real: too permissive a threshold returns the right answer to the wrong question. "How do I enable X" and "how do I disable X" are semantically very close.

3. Provider prefix caching — Anthropic and OpenAI cache the prefix of the prompt: a large system prompt, RAG documents, few-shot examples. It does not skip the call, it makes the call cheaper. Usually the best effort-to-reward ratio, and there is nothing to invalidate.

Also Worth Caching

• Embeddings — deterministic, immutable, and expensive to recompute. Use a very long TTL.
• Vector search results for a RAG system over a stable corpus.

LLM-Specific Traps

• Per-user isolation. If the prompt contains personal data, the user ID must be part of the key.
• Streaming. Cache the complete response, then re-stream it artificially so the user experience stays consistent.
• Short TTL on factual answers. A response from three weeks ago may simply be wrong now.

Recommended order: provider prompt caching and embedding caching first, because the gains are immediate and the risk is zero. Semantic caching last, once you have volume and genuinely repetitive questions.

Configuring Caching in LiteLLM

LiteLLM handles all of this as configuration rather than code.

Available backends: local in-memory, Redis, redis-semantic, valkey-semantic, S3, GCS, disk, Qdrant, and Azure Blob.

Python SDK

from litellm.caching.caching import Cache
import litellm

litellm.cache = Cache(type="redis", host=..., port=..., password=..., ttl=600)

# or semantic:
litellm.cache = Cache(
    type="redis-semantic",
    similarity_threshold=0.8,
    redis_semantic_cache_embedding_model="text-embedding-ada-002",
)

Proxy Configuration

litellm_settings:
  cache: true
  cache_params:
    type: redis
    ttl: 600
    supported_call_types: ["acompletion", "aembedding"]
    mode: default_off   # opt-in per request

Implementation Details Worth Knowing

• DualCache — an L1 layer in local process memory in front of a shared L2 Redis. Redis hits are promoted into local memory, avoiding a network round trip on repeated requests.
• Circuit breaker on Redis — if Redis fails, calls are short-circuited rather than adding latency, with a HALF_OPEN state that lets a probe request through.
• Per-request controls, modelled on HTTP: no-cache forces a fresh call, no-store skips writing, ttl overrides the default, and namespace isolates the key. These are passed in a cache dict in the request body.
• namespace is the per-user isolation mechanism. Use it.
• supported_call_types selects which endpoints are cached. Not everything is supported.

Do not confuse this with provider prompt caching, which is a separate feature: cache_control_injection_points automatically injects cache_control directives onto the system prompt or tool definitions. Note that a cache write costs more than a normal call, so it only pays off if the prefix is genuinely reused.

Two useful endpoints: /cache/ping to check the cache responds, and /cache/delete to purge keys.

LiteLLM in Front of Self-Hosted vLLM

Self-hosting changes the economics. There is no per-token cost any more, but the GPU is the scarce resource. Caching no longer saves dollars per call; it lets you serve more concurrent requests on the same hardware. That is often the bigger win.

Two layers that are easy to conflate.

Layer 1: vLLM Prefix Caching (APC)

The most valuable layer, and almost free. vLLM hashes KV-cache blocks by content and reuses already-computed blocks when a request shares the same prefix. It is enabled by default on the V1 engine (--no-enable-prefix-caching turns it off).

A 2000-token system prompt, few-shot examples, RAG documents at the head of the prompt: the prefill is computed once. This is the equivalent of Anthropic or OpenAI prompt caching, but free and automatic.

That also means cache_control_injection_points in LiteLLM does nothing useful here. Skip it.

How to exploit it:

• Order prompts from most stable to most variable: system, then fixed context, then history, then the question. A prefix that changes at the first token destroys the entire benefit. The classic mistake is putting a timestamp or user ID at the start of the system prompt.
• Use cache_salt per request in multi-tenant setups. The value is mixed into the hash of the first block, so only requests sharing a salt reuse KV blocks. This protects against timing attacks.
• With multiple vLLM replicas, prefix-aware routing is mandatory. Requests with similar prefixes must reach the same replica, or round-robin will destroy the cache.

Layer 2: LiteLLM Response Cache

Still useful, for a different reason: a hit here consumes no GPU time at all, whereas APC only speeds up the prefill. On an endpoint with repetitive questions, that is throughput recovered directly.

litellm_settings:
  cache: true
  cache_params:
    type: redis
    ttl: 3600
    supported_call_types: ["acompletion", "aembedding"]

Implementation Order

1. Confirm APC is running. Check gpu_prefix_cache_hit_rate in vLLM's Prometheus metrics. If the rate is low, restructure the prompt ordering before doing anything else.
2. Cache embeddings in Redis via LiteLLM.
3. Add the exact response cache only once logs show real repetition.
4. Semantic caching last. It is the only layer that can return a wrong answer.

The Self-Hosted Trap

Check that --gpu-memory-utilization leaves enough room for the KV cache. Effective prefix caching needs free VRAM. If it is saturated by an oversized batch, cached blocks are evicted continuously and the benefit disappears entirely.

Tags: #LLM #Caching #LiteLLM #vLLM #Redis #Performance