Enterprise takeaway: Prefix caching improves language-model serving economics by reusing computation for repeated prompt prefixes. In a shared inference service, however, the cache is also a security boundary. Buyers should ask how the serving stack separates tenant cache entries, prevents timing inference and controls offloaded KV data—not merely whether the provider encrypts the model’s API traffic.
The issue is documented in production-oriented engineering guidance. vLLM’s security documentation describes a prefix-cache timing side channel, identified as CVE-2025-46570, and an opt-in cache_salt mitigation. NVIDIA Dynamo’s standalone indexer documentation describes separate indexing trees for each model and tenant. These facts illustrate different layers of isolation; neither is blanket proof that every hosted deployment is tenant-safe.
What the KV cache actually stores
During transformer inference, key-value (KV) tensors retain intermediate attention state so the model does not recompute earlier tokens every time it generates another token. Prefix caching can reuse those blocks across requests with matching prefixes. A system serving repeated instructions or long shared context may save prefill work, lower time to first token and improve throughput. But the performance benefit depends on how the runtime identifies cache entries and which requests are allowed to reuse them.
The relevant question is not whether the application logs or stores the original prompt in plaintext. Cached intermediate state can still be associated with sensitive input. The threat model includes both direct access to cache artifacts and indirect signals created by cache hits. These are distinct from the retrieval-permission issue discussed in our RAG access-revocation analysis.
Timing is an information channel
vLLM’s security guide explains that when different tenants share a prefix cache, a caller may infer whether another user’s guessed prompt prefix is already cached by measuring differences in time to first token. The guide cites research finding a strong distinguishing signal under experimental conditions. That finding should not be generalized to every cloud service, but it is enough to make shared-prefix behavior a testable requirement.
For example, an attacker may submit carefully chosen prefixes and compare latency distributions. A cache hit could reveal that a particular phrase or structure has appeared in another request, even if the attacker never receives that user’s full response. Noise, rate limits, model scheduling and cache policy affect feasibility. The correct procurement response is to test the actual deployment, not assume that a theoretical timing channel always yields useful secrets.
Isolation must survive the routing layer
Modern inference stacks distribute work across routers, prefill workers, decode workers and external cache tiers. A tenant label enforced at the API gateway is insufficient if downstream cache lookup omits it. NVIDIA’s documented Dynamo standalone indexer maintains a separate radix tree for each (model_name, tenant_id) pair; that is a concrete design choice for segregating lookup metadata. It does not independently establish end-to-end protection for storage, credentials or every execution path.
The isolation review should therefore trace tenant identity from authenticated request to router, block hash, index lookup, remote cache and eviction path. A missing label, an unsafe default tenant or an incorrectly shared salt can undermine an otherwise sound perimeter.
An original three-layer cache isolation model
Authenticated tenant ID; trusted routing metadata; admission policy
Per-tenant salt; model/tenant cache key; no cross-tenant hits
GPU/CPU/disk lifecycle; deletion; access controls; audit
Proposed review model. A pass at one layer does not substitute for evidence at the others.
What cache salting can and cannot do
vLLM documents a request-level cache_salt that participates in the first block’s hash, so requests using different salts cannot share cached prefix blocks. The security guide says this mitigation is opt-in, and recommends a per-tenant secret in multi-tenant deployments. A deployment that never sets the salt should not claim to be protected by it. Conversely, setting a salt does not replace resource authorization, host isolation, secret rotation or protection of offloaded data.
The same vLLM guidance warns about user-provided UUIDs for multimodal media and recommends cryptographically random, unique identifiers across callers. That is a separate identity-collision issue, illustrating why a cache design needs more than one guardrail.
Offloaded caches create a second procurement question
GPU memory is not the only possible residence for KV blocks. NVIDIA Dynamo’s KV Block Manager documentation describes movement across GPU memory, host memory, remote access and SSD/object storage. This is a capability description, not evidence that a given supplier has enabled every tier. When cache state moves off GPU, enterprises should ask about storage access, encryption where applicable, retention, data deletion, operational debugging access and incident response.
Those questions connect directly to the workload-placement trade-offs in our agent fleet density-versus-isolation analysis and the self-hosted versus managed model break-even framework. An apparently cheaper shared endpoint may require additional controls or contractual assurance.
A vendor acceptance test, not a marketing checklist
Request the exact serving engine, version, cache mode and deployment topology. Verify tenant identity propagation at every routing hop. Test repeated known prefixes within one tenant to establish expected cache reuse, then repeat across tenants to check that reuse is not shared. Measure latency distributions with appropriate controls rather than drawing conclusions from a single response. Ask how cache blocks are purged when a tenant leaves, whether external tiers are shared and who can inspect diagnostic data.
Document what the provider can actually demonstrate. A vendor may offer physically isolated endpoints, a logical tenant boundary or a managed service whose internal cache policy is not exposed. Those are different assurance levels. Procurement should describe the residual uncertainty instead of labeling them all ‘private inference.’
The decision criterion
Cache reuse is a valuable performance technique, not a defect to disable reflexively. The enterprise decision is whether its economic benefit is compatible with the workload’s confidentiality and isolation requirements. Evaluate the tenant boundary and the cache lifecycle with the same rigor used for network access and database permissions. A faster first token is not evidence of a safer service.
References and limitations
- vLLM security guide — prefix-cache timing side channel, cache salting and multimodal UUID guidance; version-specific.
- NVIDIA Dynamo standalone KV indexer — model/tenant indexing design.
- NVIDIA Dynamo KV Block Manager — heterogeneous cache storage tiers; feature availability depends on deployment.
Leave A Comment