KV-Cache Offloading: Tiered Memory for LLM Inference

Q: What is KV-cache offloading and why does it matter?

KV-cache offloading is storing the transformer attention cache (the key-value state that lets an LLM generate each token without recomputing the whole prompt) across a memory hierarchy instead of only in GPU HBM, so that cache which would otherwise be discarded when GPU memory fills is instead demoted to cheaper tiers and fetched back on demand. It matters because the alternative is recompute, and recompute is pure economic loss. When a serving stack runs out of HBM, the default behavior is to evict cache and, on the next request that needs it, re-run the prefill phase from scratch — re-executing the attention math for a system prompt, a retrieved context, or an agent scaffold that was already computed, possibly thousands of times a day. Prefill is compute-bound and expensive (GPU-seconds); fetching the same cache back from DRAM or even NVMe is a memory transfer that is faster and vastly cheaper than the matrix multiplies that would regenerate it. The consequence at fleet scale, and the headline of NVIDIA's 2026 GTC inference push, is serving several times more concurrent users on the same GPU: a single 80 GB H100 that holds only a few dozen long-context sessions in HBM can keep hundreds warm across DRAM and NVMe, because a returning user's context becomes a fetch rather than a re-prefill. The teams who discover this usually arrive convinced they need more GPUs, when what they actually have is a fleet paying twice for context it already computed once.

Q: How does the three-tier memory hierarchy for KV cache work?

It mirrors the CPU cache hierarchy you already understand, applied to attention state, with each tier chosen for a different cache temperature and bandwidth. Tier one is GPU HBM at roughly 3.3 TB/s — where compute happens, holding hot cache for the sequences actively being decoded right now; it is the fastest and the scarcest, so you keep only what you are generating against. Tier two is CPU DRAM, reached over PCIe 5.0 at roughly 60 GB/s — an order of magnitude slower than HBM but still sub-millisecond to low-millisecond per block, holding warm cache: finished conversation turns and contexts likely to be reused soon, demoted out of HBM under pressure but not thrown away. Tier three is NVMe SSD at roughly 7 GB/s — another order of magnitude down, holding cold cache: prefix caches for cross-request reuse, historical context, and shared system prompts that many future requests will hit; a single-digit-to-tens-of-milliseconds fetch, still dramatically cheaper than recomputing. An emerging fourth option, CXL memory over PCIe Gen5, sits between DRAM and NVMe and lets you expand warm capacity without stranding DRAM. The scheduler's job is temperature-aware placement: keep active decode on HBM, demote the coldest HBM blocks to DRAM and the coldest DRAM blocks to NVMe as pressure rises, and promote a block back toward HBM when a request needs it. The design payoff is that concurrent-user capacity decouples from HBM size — the GPU serves as many users as the whole hierarchy can keep warm, not as many as fit in 80 GB.

Q: What is LMCache and how does it reduce inference latency?

LMCache is an open-source KV-cache engine that gives inference servers a real memory hierarchy: it plugs into vLLM, SGLang, and NVIDIA Dynamo, serializes KV blocks to configurable storage backends (local NVMe disk, Redis, remote object stores), and manages placement across GPU, CPU DRAM, and disk so that cache survives eviction from HBM instead of being discarded. Its headline benefit is turning cross-request prefix reuse into a first-class, shared layer: when many requests share a prefix — a common system prompt, a scaffold, a retrieved document — LMCache lets the prefill for that prefix be computed once and reused across all of them, spanning GPU, CPU, disk, and network rather than being trapped in one GPU's memory. The reported results are why it became ubiquitous in 2026: paired with vLLM, LMCache has shown 3x to 10x latency reductions on the workloads where context recurs — multi-turn question answering, retrieval-augmented generation, and document-level reasoning — which describes the majority of production LLM applications. The latency win comes from two mechanisms working together: avoided recompute (a cache hit skips the entire prefill phase for the cached portion, going straight to generating the unique suffix) and tiered retention (a block that would have been evicted and recomputed is instead fetched from DRAM or NVMe far faster than regeneration). It is, in effect, the self-hosted systems-layer version of the prompt-caching discount that model vendors sell through their APIs — the same economics, built into your own serving stack where you control it.

Q: When does KV-cache offloading help and when does it not?

It helps most when contexts are long and reused, and least when every request is short and unique — because the entire value is capturing cache that would otherwise be recomputed, and there is nothing to capture if nothing recurs. The strong-fit workloads: multi-turn conversations (each turn reuses the whole prior conversation's cache), retrieval-augmented generation (the retrieved context and system prompt recur across turns and often across users), agent pipelines (a long scaffold or tool-definition prefix is rebuilt on every step), and any application with a large shared system prompt hitting every request — there, offloading converts repeated multi-thousand-token prefills into single computes plus cheap fetches, and the concurrent-user capacity of a GPU rises several-fold. The weak-fit workloads: short, unique, single-shot requests with no shared prefix and no follow-up (classification of distinct inputs, one-off completions) — the cross-request hit rate approaches zero, and offloading only adds fetch latency and complexity for cache no one reuses. The genuine cost even in the strong case is latency: a cold NVMe fetch adds milliseconds that a hot HBM hit does not, so a strict interactive p99 budget bounds how deep you can demote — you trade tail latency for capacity, and that trade must be sized against the SLA. And offloading moves the bottleneck rather than eliminating all of them: once prefill recompute stops dominating, decode-phase memory bandwidth and batching efficiency become the next ceiling. The decision rule: measure your cross-request prefix overlap and your prefill-versus-decode split first; high overlap with prefill-heavy load is the green light, and low overlap means the lever is elsewhere.

Q: How is KV-cache offloading different from prompt caching that vendors sell?

They solve the same economics — avoid recomputing a repeated prefix — at different control points, and which one you use depends on whether you run the GPUs. Vendor prompt caching (Anthropic's prompt caching, Amazon Bedrock's prompt caching, and similar) is an API-level feature: you mark a portion of your prompt as cacheable, the provider retains its KV state on their infrastructure for a short window, and subsequent requests reusing that prefix are billed at a steep discount and served faster. You get the benefit without operating anything, but you also do not control the cache policy, the retention window, or the tiers, and it only applies to the model you are renting. KV-cache offloading is the self-hosted systems-layer version: on your own GPUs, an engine like LMCache manages the KV cache across HBM, DRAM, and NVMe, giving you the same avoided-recompute economics for models you serve yourself, with full control over what gets cached, how long it lives, which tier holds it, and how it is shared across requests and sessions. The relationship is complementary rather than competitive: if you call a hosted model, use the vendor's prompt caching for the discount; if you serve your own model, offloading is how you build that discount into your stack and own the policy. Teams running open-weight models at scale reach for offloading precisely because the vendor option does not exist for them — the cache economics are theirs to capture, and offloading is the mechanism that captures them.

Q: What hardware do I need for KV-cache offloading — DRAM, NVMe, or CXL?

You need whatever tiers your reuse pattern actually exercises, sized from measured working-set data rather than round numbers, and the good news is that the first big win requires no new hardware at all. Start with CPU DRAM offloading: every inference host already has host memory, and demoting warm KV blocks from HBM to DRAM (over PCIe 5.0 at roughly 60 GB/s) is usually a configuration change in LMCache or the framework — it captures the warm reuse that eviction was throwing away and typically raises concurrent-user capacity immediately, paying for itself in the first billing cycle. Add NVMe as the cold tier when your reuse spans longer time horizons or more sessions than DRAM can hold: shared system prompts and prefix caches that many future requests will hit belong on fast local NVMe (roughly 7 GB/s), which is cheap capacity per gigabyte and still beats recompute handily. Consider CXL memory — an emerging option over PCIe Gen5 — when you want to expand warm capacity beyond what the host's DRAM slots allow without stranding memory; it sits between DRAM and NVMe in both bandwidth and latency. The sizing discipline that matters: measure your reuse distance (how long between the first and subsequent uses of a cache block) and your working-set size (how much cache is live across concurrent sessions), then size DRAM to hold the warm working set and NVMe to hold the cold reuse horizon. Sizing by vibes strands capacity in the wrong tier — too little DRAM and warm blocks spill to slow NVMe, too much NVMe for a workload with short reuse distance and the disk sits idle while HBM thrashes.

Q: Does KV-cache offloading hurt latency?

It adds latency on cold fetches and removes far more on cache hits, so the net effect is strongly positive for the right workload — but the trade is real and must be sized against your latency budget. The cost side: a block served from NVMe arrives milliseconds later than one already resident in HBM, and a block promoted from DRAM adds sub-millisecond to low-millisecond overhead; under a strict interactive p99 target, this bounds how aggressively you can demote cache to the slowest tier, because a deep cold fetch on a latency-sensitive request risks a tail-latency violation. The benefit side, which usually dominates: a cache hit skips the entire prefill phase for the cached portion, and prefill on a long prompt is far more expensive than any fetch — so where reuse is high, offloading dramatically lowers time-to-first-token by turning a multi-thousand-token recompute into a fetch plus a jump straight to generating the unique suffix. The reported 3x-to-10x latency reductions on multi-turn and RAG workloads are exactly this effect. The way to manage the trade rather than fear it: treat tier depth as a latency-budget decision, keeping the hottest and most latency-critical cache on HBM and DRAM while relegating only genuinely cold, reuse-oriented blocks to NVMe; instrument fetch latency per tier so you can see when demotion is approaching your p99 ceiling; and remember that the comparison for a cold block is not HBM-versus-NVMe but NVMe-fetch-versus-full-recompute, where the fetch wins decisively. Offloading is a latency optimization for reuse-heavy workloads and a latency cost only for workloads that had no reuse to capture in the first place.

Q: Will KV-cache offloading reduce my inference costs?

For reuse-heavy workloads, materially — because the cost it removes is one of the largest and most silently avoidable line items in an LLM inference bill: recomputed prefill. Every time a serving stack discards cache under memory pressure and re-prefills a shared prefix, it spends GPU-seconds regenerating attention state it already had, and at fleet scale a system prompt or RAG context re-prefilled on nearly every request represents an enormous quantity of compute poured into duplicate work. Offloading converts that recompute into a cheap fetch from DRAM or NVMe, which frees GPU cycles for actual generation and, crucially, raises the number of concurrent users a single GPU can serve — often several-fold — which directly lowers cost-per-user because the expensive asset (the GPU) is amortized across more served requests. The savings show up two ways: fewer GPUs needed for the same traffic (the "we need more H100s" that turns out to be "we are recomputing cache we already had"), and lower cost per request on the traffic you have. The magnitude depends entirely on cross-request prefix overlap: high-overlap workloads (shared prompts, multi-turn, RAG, agents) see large reductions, while low-overlap workloads see little because there was little duplicate prefill to eliminate. The way to size the opportunity before committing: measure your prefill-versus-decode compute split and your cross-request prefix overlap — high prefill share plus high overlap is the signal that a large fraction of your GPU spend is recompute, which is precisely what offloading recovers, usually starting to pay back in the first billing cycle since the initial DRAM tier is a configuration change on hardware you already own.

Satyam Kumar

返回博客

ai-architecture

KV-Cache Offloading: Serving 10x More Users by Not Recomputing

By Satyam KumarJuly 13, 20269 min read

kv cache offloading llm inference optimization lmcache tiered memory nvme kv cache cxl memory vllm nvidia dynamo prefix caching gpu memory inference economics prefill recompute multi-turn inference rag serving ai infrastructure llm cost optimization 2026

KV-Cache Offloading: Serving 10x More Users by Not Recomputing

Frequently Asked Questions

分享这篇文章

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

人工智能和云架构师。帮助团队构建可扩展到数百万的系统。

LinkedIn GitHub

KV-Cache Offloading: Serving 10x More Users by Not Recomputing

Frequently Asked Questions

分享这篇文章

Comments

Leave a comment