KV-Cache Side Channels: Cross-Tenant Prompt Leakage

Q: How can a prefix cache leak one tenant's prompt to another tenant?

A prefix cache identifies each block of tokens by a hash of its content chained with the hashes of the blocks before it, and reuses the stored key/value tensors whenever a new request's leading blocks match entries already resident. Because the key is derived only from content, the cache has no notion of which tenant created an entry. An attacker on the same deployment sends a candidate prefix and measures time-to-first-token: if the leading blocks were resident, prefill is skipped and the response is measurably faster, revealing that someone else recently sent exactly that prefix. Since hashes chain block by block, the attacker can extend a confirmed prefix one block at a time by testing candidates and keeping the one that hits, reconstructing another tenant's system prompt, few-shot examples or in-context customer data. This is the mechanism demonstrated by the PromptPeek paper at NDSS 2025 against vLLM and SGLang.

Q: Is this only a self-hosted vLLM or SGLang problem, or does it affect hosted APIs too?

It affects any shared prefix cache whose key does not include a tenant boundary, hosted or self-hosted. A 2025 Stanford audit of prompt caching in commercial LLM APIs used a purely statistical test — send a prompt from one account, then measure latency for the same prompt and variants from a second account — and detected global cache sharing across users at seven providers, including at one point OpenAI. The major vendors now document their isolation: Anthropic states caches are isolated per workspace and never shared across organisations, OpenAI states caches are not shared across organisations or regional processing boundaries, and Gemini uses explicit per-project cache objects. That closes the channel between vendor customers, but a platform that puts thousands of its own end customers behind one vendor organisation inherits only per-organisation isolation, so the cross-customer channel is the platform's to close.

Q: Should we just disable prefix caching to be safe?

Usually not. Disabling caching closes the cross-tenant channel but forfeits one of the largest cost and latency levers in production serving — a warm hit on a long system prompt or a growing agent transcript can cut prefill from hundreds of milliseconds to tens and reduce input cost by up to 90 percent. The same protection is available at a fraction of the cost by scoping the cache key: mixing a tenant identifier (and, where hit rate allows, a user identifier) into the block hash so that identical content from two tenants produces two different keys and can never collide. Because most real reuse is intra-session — the same agent or chat re-sending its own history — a per-tenant or per-user key preserves most of the benefit. Turning caching off tends to be an over-correction that is quietly reverted the next time the GPU bill is reviewed.

Q: What exactly should go into the cache key in a multi-tenant deployment?

At minimum the tenant or organisation identifier, so that no two tenants can share entries. Where the hit rate tolerates it, add the end-user identifier, which removes the residual tenant-internal channel in which one user of an organisation probes another user's prompts. Include the LoRA adapter identifier and the system-prompt version so that different applications in one tenant do not cross-hit. vLLM already mixes extra keys such as LoRA ID and multimodal hashes into its block hash, and a tenant ID is one more extra key; in SGLang the equivalent is rooting each tenant in its own radix subtree. It is reasonable to keep a separate cross-tenant tier for your own platform system prompt, which is not secret from your tenants, and a scoped tier for everything after it. Whatever key you choose must be carried unchanged into any CPU or disk offload tier.

Q: How large is the timing signal, and can we hide it with random delay?

The difference between a cache hit and a miss on a long prefix is typically tens to hundreds of milliseconds of time-to-first-token, because a hit skips the prefill computation for the cached blocks entirely. Network jitter is smaller than that once an attacker averages a handful of samples, so the signal is easy to recover. Adding random delay is a weak control: to hide a 100 ms gap you need jitter of roughly the same magnitude, which destroys the latency benefit you kept the cache for, and statistical averaging over a few dozen probes defeats modest jitter anyway. The strong control is scoping the key so that a hit against another tenant's entry is impossible; the supporting controls are not returning cache-hit metadata such as cached-token counts to callers below the organisation boundary, per-tenant rate limits on requests, and anomaly detection on prompt-prefix churn.

Q: Does KV-cache offloading or prefill/decode disaggregation change the risk?

Yes, it widens it. Offloading moves KV blocks from one GPU's HBM into CPU memory and disk tiers shared across a cluster, and disaggregated prefill ships KV over the network between prefill and decode nodes. Every one of those tiers is now a copy of tenants' prompt state, and the disk tier in particular persists across restarts and may be restorable. The scoped cache key must be carried through every tier — a tenant-salted key in HBM combined with a content-only key in an LMCache-style disk tier means the leak has moved rather than closed. Offloaded KV should be encrypted at rest, given a TTL, and included in your data-deletion and retention story, because it is derived confidential data even though it does not look like a document.

Q: What does scoping the cache key not protect against?

It closes the cross-tenant channel but not the timing side channel itself. Within a tenant, a malicious employee or compromised sub-account can still probe the tenant's own cache unless the key is per-user. Prompt length leaks through time-to-first-token regardless of caching, and response length leaks through streaming shape; those are separate channels. Speculative decoding, batching order and mixture-of-experts routing introduce their own data-dependent timing. And if the leak is in the model rather than the cache — a shared fine-tune that memorised one customer's data — no cache keying helps; that is a model-extraction and memorisation problem. For workloads where even tenant-internal residual leakage is unacceptable, the honest answer is dedicated capacity or confidential inference inside a trusted execution environment, not a cleverer hash.

Q: How do we verify our deployment is not leaking across tenants?

Run the same audit the researchers ran, from a second account you control. Send a distinctive long prompt from account A, then from account B send the identical prompt and a set of variants that differ only in the final blocks, measuring time-to-first-token for each over a few dozen samples. If account B's latency distribution for the identical prompt is significantly lower than for the variants, entries written by A are being hit by B and the cross-tenant channel is open. After scoping the key the shift must disappear. Automate this as a CI check against every serving-stack upgrade, because a new engine version can silently change what goes into the block hash, and publish your cache-isolation model to customers the way the API vendors now do — the customers who ask during procurement are the ones you most want to keep.