LLM: How to Calculate KV Cache

The KV cache — the per-token memory transformers must retain at inference — has quietly become the dominant memory cost for any deployment that touches long context.

9-15 minutes(2019 words)complex

Quick Navigation

Difficulty: Advanced
Estimated Time: 15-25 minutes
Prerequisites: transformer architecture fundamentals, attention mechanisms (MHA/GQA), GPU memory and precision formats, LLM inference basics

A single Llama 3.1 405B user at 128k tokens burns 66 GB of GPU memory before the model even responds. Here's the math that explains why long context is the most expensive feature in modern AI.

Most engineers fine-tune their LLM cost model around two numbers: parameter count and tokens per second. Both miss the real bottleneck. The KV cache — the per-token memory transformers must retain at inference — has quietly become the dominant memory cost for any deployment that touches long context.

The numbers are stark. A Llama 3.1 405B model at 128k tokens needs 66 GB of KV cache for a single user in FP16. Scale that to a modest batch of 16 concurrent users and you're at over a terabyte of GPU memory — for cache alone, before counting model weights or activations. This isn't an edge case. It's what happens the moment you offer 128k context to more than a handful of people simultaneously.

The good news: the formula is simple, the variables are public, and the optimization levers are well-understood. The bad news: most LLM deployment guides skip past it with a hand-wave about "memory considerations." This article fixes that. We'll work through the exact math, run the numbers for models from 7B to 800B parameters, and map out the three optimization axes — architecture, precision, and system — that actually matter in production.

The Formula That Governs Every Long-Context Deployment

The complete KV cache calculation looks like this:

KV Cache = 2 × L × H_kv × d_h × S × B × P

Seven variables, one multiplication. The factor of 2 comes from storing both Keys and Values. L is the number of transformer layers. H_kv is the number of key/value heads — and this is where the first major optimization lives. d_h is the dimension of each head. S is the sequence length in tokens. B is the batch size. P is bytes per parameter: 2 for FP16/BF16, 1 for FP8 or INT8, 0.5 for INT4.

For a single user, batch drops out and you get the practical formula every inference engineer should have memorized:

Key insight: KV = 2 × L × H_kv × d_h × S × P. This single line determines whether your 128k context deployment fits on one GPU or requires twelve.

The variables are not hidden. They live in every model's config.json on Hugging Face. The mistake most engineers make is using num_attention_heads when they should use num_kv_heads. In a model with Grouped Query Attention, those two numbers differ by a factor of 4 to 8. Get this wrong and your memory estimate is off by an order of magnitude.

Worked Example: Llama 3 70B at 128k Tokens

Llama 3 70B exposes these architectural parameters: 80 layers, 8 KV heads (GQA), 128 dimensions per head. Run them through the formula at 128k tokens, batch of 1, FP16:

2 × 80 × 8 × 128 × 128,000 × 2 = 41,943,040,000 bytes ≈ 42 GB

Step by step: 2 × 80 = 160. Times 8 heads = 1,280. Times 128 = 163,840. Times 128,000 tokens = 20,971,520,000. Times 2 bytes for FP16 = 41.9 GB.

Reframed per-token, that's 327,680 bytes — about 320 KB of KV cache for every single token the user sends or the model generates. At 1,000 tokens you've spent 320 MB. At 32k tokens, 10 GB. At 128k, 42 GB. Linear growth, no tricks.

"The KV cache scales linearly with context length, but the constants are large enough that 'linear' starts to look exponential in dollar terms."

What Happens at 405B, 671B, and Beyond

Llama 3.1 405B has 126 layers, 8 KV heads, 128 head dimension. Per token, that's 504 KB. At 128k, 66 GB of KV cache — for one user.

The pattern holds across the frontier:

  • Llama 3 8B / Mistral 7B: 32 layers, ~16.8 GB at 128k FP16
  • Llama 3 70B / Qwen2 72B: 80 layers, ~42 GB
  • Command R+ (104B): 64 layers, ~33.6 GB
  • Mixtral 8x22B (141B MoE): 56 layers, ~29.4 GB
  • Llama 3.1 405B: 126 layers, ~66 GB
  • Hypothetical 700B GQA model: ~160 layers, ~84 GB
  • Hypothetical 800B GQA model: ~170 layers, ~89 GB
  • GPT-3 175B (vintage MHA, no GQA): 96 layers × 96 heads, ~604 GB

That last number is not a typo. GPT-3 predates Grouped Query Attention. Its KV cache at 128k tokens would consume 604 GB — roughly fifteen times what a modern Llama 70B needs, despite GPT-3 having a similar parameter count. The architecture matters more than the model size.

The most interesting outlier is DeepSeek-V3 at 671B parameters, which uses Multi-head Latent Attention (MLA) instead of GQA. MLA compresses the KV cache into a latent dimension of roughly 512 per layer. The result: ~70 KB per token, or about 9 GB at 128k tokens — for a model nearly twice the size of Llama 3.1 405B. This is why DeepSeek can offer long context at prices that look impossible if you assume conventional architecture.

Note: Multi-head Latent Attention is the single most important architectural innovation of 2024 for long-context economics. Expect every major lab to adopt some variant within the next 12–18 months.

When Users Multiply, So Does the Cache

The single-user math is sobering enough. The batched math is brutal. For Llama 3.1 405B in FP16 at 128k tokens:

  • 1 user: 66 GB
  • 4 users: 264 GB
  • 8 users: 528 GB
  • 16 users: 1.05 TB
  • 32 users: 2.11 TB

There is no production deployment of Llama 3.1 405B serving 32 simultaneous 128k-token users on a single node. The cache alone would require 27 H100s before counting weights. This is why every serious inference stack — vLLM, TensorRT-LLM, SGLang — treats KV cache management as its central engineering problem.

"The era of treating model weights as the dominant memory cost is over. For any deployment touching long context, the KV cache is the budget."

The total memory picture for a single Llama 3.1 405B user at 128k tokens in FP16 looks like this:

  • Model weights (405B × 2 bytes): 810 GB
  • KV cache: 66 GB
  • Activations (~5% of weights): ~40 GB
  • CUDA / framework overhead: ~10 GB
  • Total: ~926 GB

That's twelve H100s minimum, for one user. Drop to INT4 weights plus FP8 KV cache and the same workload fits in roughly 280 GB — four H100s. The quantization choice doesn't change the model; it changes the entire economics of who can deploy it.

The Architecture Tax: MHA vs GQA vs MQA vs MLA

Modern attention variants represent escalating compression of the KV cache. At a fixed 70B model size and 128k context, the ratios are dramatic:

  • MHA (Multi-Head Attention) — GPT-3, Llama 1, Llama 2 7B: ~2.6 MB per token. The baseline.
  • GQA (Grouped Query Attention) — Llama 3, Mistral, Qwen2: ~328 KB per token. 8× compression over MHA.
  • MQA (Multi-Query Attention) — Falcon, PaLM: ~41 KB per token. 64× compression, but at measurable quality cost.
  • MLA (Multi-head Latent Attention) — DeepSeek-V2/V3: ~70 KB per token. 37× compression without the quality penalty of MQA.
  • SWA (Sliding Window Attention) — Mistral 7B v0.1: constant memory regardless of context, but loses long-range information.

GQA has become the default for good reason. It's the inflection point where compression doesn't hurt quality. MLA is what comes next — and the fact that DeepSeek-V3's KV footprint at 128k is smaller than Llama 3 70B's despite the model being nearly 10× larger should be a flashing red signal for anyone planning multi-year infrastructure.

"MLA isn't an incremental improvement. It's the move from compressing the cache to redefining what the cache is."

The 1M and 10M Token Problem

The linear scaling of KV cache becomes a wall at extreme context lengths. For Llama 3.1 405B in FP16:

  • 32k tokens: 16.5 GB
  • 128k tokens: 66 GB
  • 256k tokens: 132 GB
  • 1M tokens: 516 GB
  • 10M tokens (Gemini's marketed maximum): 5.16 TB

At 10 million tokens, a single user's KV cache would require 65 H100s just to hold the cache in memory. This is mathematically impossible without aggressive compression. Whatever Google is running under the Gemini 1.5 Pro hood at 10M context, it involves some combination of MLA-style latent compression, hierarchical attention, KV offloading to CPU and SSD, and dynamic pruning of low-importance tokens. The naive transformer formula breaks down — the engineering doesn't.

This matters for anyone building on top of long-context APIs. The pricing of long-context inference is not a markup on compute. It's a markup on the rare engineering capability to make long-context tractable at all.

The Three Optimization Levers Worth Knowing

Production inference engineering converges on three optimization axes. Knowing them is the difference between a deployment that works and one that bankrupts you.

Precision is the cheapest win. Moving the KV cache from FP16 to FP8 halves memory with negligible quality loss. INT4 cuts it by 4× with a small but measurable degradation. Tools like KIVI and KVQuant push this further with KV-specific quantization schemes. For Llama 3.1 405B at 128k, FP8 alone takes the cache from 66 GB to 33 GB. There is no reason not to use it.

System optimizations come from the inference stack. PagedAttention, pioneered by vLLM, treats the KV cache like virtual memory — allocating in blocks, reducing fragmentation by 60–80%, and enabling prefix sharing across users. Prefix caching alone saves 30–90% in production workloads where users share system prompts. Continuous batching, sliding-window attention, and CPU/SSD offloading round out the toolkit.

Architecture is the deepest lever. GQA is now table stakes; MLA is the frontier. Sliding-window attention and StreamingLLM-style anchor tokens trade off quality for bounded memory. H2O and Scissorhands dynamically prune low-importance tokens. The choice of base model — and specifically its attention architecture — locks in a 5–40× memory difference that no amount of system optimization can fully recover.

A Rule of Thumb You'll Use Weekly

For any modern GQA model with H_kv = 8 and d_h = 128, this approximation gets you within 5% in seconds:

KV (in GB) ≈ (L × S) / 250,000 [FP16, 1 user]

Llama 3 70B at 128k: 80 × 128,000 / 250,000 = 40.9 GB. Actual: 42 GB.

Llama 3.1 405B at 128k: 126 × 128,000 / 250,000 = 64.5 GB. Actual: 66 GB.

Memorize it. You'll reach for it more often than you expect.

Key insight: If the model uses GQA and you know the layer count and context length, you can estimate KV memory to within a few percent without opening a calculator. That's the level of fluency engineers deploying long-context LLMs need.

Conclusion: The Cache Is the Bottleneck Now

For most of the deep learning era, model size dominated cost conversations. That era is ending. Modern GQA models compress weights so aggressively — through quantization, pruning, and architectural efficiency — that the KV cache for long contexts now rivals or exceeds the cost of holding the model itself.

This shifts what matters when evaluating an LLM for production. Parameter count tells you something about capability. Attention architecture tells you something about cost. A 671B MLA model and a 405B GQA model can have wildly different deployment economics at the same context length — and the difference is governed entirely by how the KV cache is structured.

The next wave of inference improvements won't come from making models smaller. Models are already small enough. The wave will come from rethinking what attention even means at scale — compressed caches, hierarchical retention, learned eviction policies, and architectures that treat memory as a first-class design variable rather than an emergent cost.

If you're betting on long context as a product surface, the question isn't whether your model can handle 128k tokens. The question is whether your inference stack can afford to.