vLLM Prefix Cache Not Hitting on Multi-Turn Agent Requests

The line comes from vllm/v1/metrics/loggers.py, formatted as Prefix cache hit rate: %.1f%% and fed from prefix_caching_metrics.hit_rate. It is token-weighted, so a hundred short requests that hit perfectly and one 180k-token turn that misses do not average out the way intuition says.

It is also windowed. CachingMetrics takes max_recent_requests: int = 1000, hardcoded; an issue asking for it to be configurable, opened 20 July 2026, was still open on 13 August. On a busy server 1000 requests can be under a minute, so the line describes recent mixed traffic rather than your agent fleet.

For alerting, use the counters: vllm:prefix_cache_queries and vllm:prefix_cache_hits, exported in tokens with _total appended by prometheus_client, and vllm:external_prefix_cache_queries / vllm:external_prefix_cache_hits for connectors, both cumulative since process start.

The per-request field is the one most teams are missing. enable_prompt_tokens_details defaults to False in vllm/entrypoints/openai/cli_args.py, and _make_prompt_tokens_details returns None when it is off, so cached_tokens reads as a zero hit rate rather than a disabled feature.

vllm bench serve does not report hit rate at all. The request for it, opened 28 June 2026, was still open on 13 August.

vLLM's prefix cache is hash-based over fixed-size blocks, not a prefix tree, which is why the failure is all-or-nothing past the divergence point. The tree-based alternative belongs to a different engine (SGLang vs vLLM).

A block hash is hash(tuple[components]) over the parent block's hash value, the tokens in this block, and extra keys such as LoRA IDs, multimodal input hashes and cache salts. The parent hash is the whole story: change one token in block 2 and block 3's parent changes, and so on to the end of the prompt. One edited token at position 40 costs every block after position 32.

Only full blocks are cached. The design doc's worked example uses block size 4: a request whose first 10 tokens match hits only the first 2 blocks, 8 tokens, because the third block matches 2 of 4. DEFAULT_BLOCK_SIZE is 16 in vllm/config/cache.py, so a real hit truncates down to the last full 16-token boundary.

Some extra keys apply to every block. In vllm/v1/core/kv_cache_utils.py, _gen_lora_extra_hash_keys returns the adapter name for every block, so switching LoRA adapter mid-session voids the entire conversation. cache_salt enters only block 0 via generate_block_hash_extra_keys, and the chain propagates it anyway.

This is the cause almost nobody checks, and on agent workloads it is usually the dominant one.

Open Qwen/Qwen3-8B's tokenizer_config.json and read chat_template. It computes ns.last_query_index by scanning in reverse for the last user message whose content is not wrapped in <tool_response>. Assistant messages with loop.index0 > ns.last_query_index keep their reasoning block:

{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}

Assistant messages at or before that index render as '<|im_start|>' + message.role + '\n' + content, with the <think> block dropped.

Read that as an agent loop. At turn N the last user message sits at index 4 and the assistant message at index 5 carries its reasoning. The user sends turn N plus 1, last_query_index jumps to index 6, and index 5 now renders without it. The turn N plus 1 prompt is shorter in the middle than the turn N prompt, diverging at the first assistant message, a few dozen tokens in. Everything after that recomputes on every turn.

gpt-oss has the same bug in a different place. Its chat_template.jinja sets future_final_message by scanning loop_messages[loop.index:] for a later assistant message without tool calls, and its own comment says it drops the tool call analysis message when one exists. Once a final answer arrives, every analysis message from that turn vanishes from the render.

The same template has a blunter problem. Inside build_system_message():

{{- "Knowledge cutoff: 2024-06\n" }}
{{- "Current date: " + strftime_now("%Y-%m-%d") + "\n\n" }}

strftime_now is a Jinja global transformers installs as datetime.now().strftime(format). It sits in the first block of every prompt, so at local midnight the hash of block 0 changes and the whole prefix cache goes cold in one step, with no config change and no log line. Llama 3.x does the same job safely: unsloth/Llama-3.3-70B-Instruct sets date_string to a frozen "26 Jul 2024" default.

The fourth variant is tool arguments. Qwen3 passes tool_call.arguments through tojson when it is not already a string, and transformers overrides Jinja's tojson with a wrapper around json.dumps, so separators fall back to ', ' and ': '. If your framework parsed the model's arguments into a dict and hands them back structurally, {"city":"Boston"} replays as {"city": "Boston"}: two space characters, and a miss to the end of the prompt on every later turn.

Do not reason about your template. Render both turns and diff the token IDs.

POST /v1/chat/completions/render takes a normal chat completion request and returns the GenerateRequest the engine would execute, token IDs included, generating nothing.

# what vLLM will feed the model, no generation
curl -s -X POST http://localhost:8000/v1/chat/completions/render \
  -H 'Content-Type: application/json' \
  -d @turn_n.json > render_n.json

# flush so you are not measuring a warm server
curl -s -X POST 'http://localhost:8000/reset_prefix_cache?reset_running_requests=true'
# {"success":true}

/reset_prefix_cache can legitimately return false while blocks are still held by running requests or in-flight async KV offload transfers, so read the body, not the status code.

return_token_ids gives the same information through the normal path: token IDs alongside the generated text, with prompt_token_ids in the first chunk when streaming and token_ids carrying the deltas.

import json, urllib.request

BASE = "http://localhost:8000"

def find(obj, key):
    if isinstance(obj, dict):
        if obj.get(key) is not None:
            return obj[key]
        for value in obj.values():
            hit = find(value, key)
            if hit is not None:
                return hit
    if isinstance(obj, list):
        for value in obj:
            hit = find(value, key)
            if hit is not None:
                return hit
    return None

def prompt_tokens(messages, tools=None):
    body = {
        "model": "your-served-model-name",
        "messages": messages,
        "max_tokens": 1,
        "temperature": 0,
        "return_token_ids": True,
    }
    if tools:
        body["tools"] = tools
    request = urllib.request.Request(
        f"{BASE}/v1/chat/completions",
        data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json"},
    )
    with urllib.request.urlopen(request) as response:
        return find(json.load(response), "prompt_token_ids")

turn_n = prompt_tokens(history)
turn_n_plus_1 = prompt_tokens(history + new_turn)

BLOCK_SIZE = 16  # DEFAULT_BLOCK_SIZE in vllm/config/cache.py
for i, (a, b) in enumerate(zip(turn_n, turn_n_plus_1)):
    if a != b:
        print(f"first divergence at token {i} of {len(turn_n)}")
        print(f"reusable: {(i // BLOCK_SIZE) * BLOCK_SIZE} tokens, full blocks only")
        break
else:
    print(f"clean prefix extension, all {len(turn_n)} tokens reusable")

The divergence index names the cause. Near zero is the system message: a live date, a rotating salt, a tool schema your framework reorders. At the first assistant message it is history rewriting; at a tool call, argument re-serialization; past the last cached turn, a clean render and a capacity problem.

For load rather than a single diff, vLLM ships benchmarks/multi_turn/benchmark_serving_multi_turn.py, run in its README as python benchmark_serving_multi_turn.py --model $MODEL_PATH --served-model-name Llama --input-file generate_multi_turn.json --num-clients 2 --max-active-conversations 6. Its prompt_input block takes num_turns, prefix_num_tokens and num_tokens, each sampled from a named distribution.

The first two rows deserve ten minutes each: both produce a clean 0 and both are invisible in a normal log. _set_default_chunked_prefill_and_prefix_caching_args in vllm/engine/arg_utils.py resolves the serve default as the model supporting prefix caching and not being hybrid, with a comment that hybrids keep it opt-in while the feature matures. is_hybrid is the registry IsHybrid protocol: models like Jamba with both attention and mamba blocks.

Two behaviours cap reuse before any of this applies. Sliding-window layers need only the last sliding_window_size tokens resident for a hit, and for models mixing sliding-window and full attention vLLM intersects the longest hit from each group; why those layers cannot be checkpointed is in prompt reprocessing on SWA and hybrid models. And every compaction event is a deliberate total flush, since rewriting history changes the render from the first message: the cost side of context compaction for long-running agents.

Fix the template and you meet the second problem. vLLM's retention RFC, opened 13 March 2026 and still open on 13 August, states it plainly: over 90 percent of tokens in a typical agent turn are prefixes reused verbatim from the previous turn, 40 to 60 percent of session wall time is spent paused on tool calls, and during those pauses the agent's blocks are unreferenced. Other agents evict them by LRU, and resuming recomputes a context that reaches 70,000 to 200,000 tokens by session end.

The free-queue ordering makes it worse. Freed blocks go to the tail in reverse order, because the last block of a request hashes more tokens and is less likely to be reused by anyone else. Eviction pops the head. For a shared prefix that is correct; for one agent's session it is backwards, and the newest, most context-rich blocks die first.

vLLM ships the histograms that prove this, off by default. kv_cache_metrics is False in vllm/config/observability.py with kv_cache_metrics_sample at 0.01. --kv-cache-metrics exposes vllm:kv_block_lifetime_seconds, vllm:kv_block_idle_before_evict_seconds and vllm:kv_block_reuse_gap_seconds, bucketed out to 1800 seconds. Mass in the middle metric at your tool-call latency means the cache dies while the tool runs.

There is no retention, pinning or TTL argument in v0.27.1; that capability is the open RFC, which argues hard pinning does not scale while LRU sees only recency. What exists is offload: --kv-offloading-size in GiB (summed across TP ranks when TP is above 1) with --kv-offloading-backend set to native or lmcache. No published measurement says whether it rescues a session across a 60-second pause under concurrency.

The RFC leans on four results worth reading first: 10 percent of KV blocks accounting for 77 percent of reuses, with workload-aware eviction cutting mean response time up to 41.9 percent against LRU (arXiv:2506.02634); 1.12x to 3.66x delay reduction from time-to-live based multi-turn scheduling (arXiv:2511.02230); up to 2.19x for multi-agent workflows (arXiv:2507.07400); and up to 34.4x higher token hit rates on hybrid models (arXiv:2411.19379). Those are the RFC's figures, not our replication.

If the cache cannot hold one session, none of this helps and you have a sizing problem, worked through in vLLM KV cache OOM.

A hosted API hides the router and the eviction policy, the trade in debugging a prompt cache TTL regression. On your own cluster nothing is hidden, and nothing is handled for you.

The default data-parallel router scatters sessions. In vllm/v1/engine/core_client.py, DPLBAsyncMPClient.get_core_engine_for_request scores each engine as max(self.client_count * inflight, waiting + running), adds waiting * 6.0 * max(0.0, kv_cache_usage - 0.5) when requests are waiting, picks the minimum, and rotates the scan start so ties spread. Nothing in it knows which engine holds your conversation, so -dp 4 can send a four-turn conversation to four cold engines.

What prefix awareness is worth was measured in a September 2025 benchmark of a vLLM-based scheduler: 8 pods of 2 NVIDIA H100 each, Qwen-32B, 307,328 tokens of KV cache per pod, 150 simulated customer groups with a 6,000-token shared prefix each, and Poisson load ramping 3 to 60 QPS.

That is roughly 57x better P90 TTFT than the approximate variant and over 170x better than random, with 8,730.0 output tokens per second against 6,944.4. The gap is the argument for exact token IDs: approximate mode has no tokenizer and estimates from character-to-token ratios.

Exact mode reads the render endpoint plus real-time KV cache events, which on the vLLM side is --kv-events-config: enable_kv_cache_events (default False), a publisher that becomes zmq when events are on, endpoint at tcp://*:5557, plus replay_endpoint, buffer_steps, hwm, max_queue_size and topic. Without a scheduler, --data-parallel-external-lb or --data-parallel-hybrid-lb moves routing to a front end where you add session affinity yourself. A miss here blows the TTFT budget outright.

A shared prefix cache is a cross-tenant timing side channel. vLLM documents cache_salt as the control: the salt enters the hash of the first block so only requests carrying it reuse those blocks, preventing timing attacks where an adversary infers cached content from latency differences. The guidance is a random value, protected from third parties and long enough to be unpredictable, for example 43 base64 characters for 256 bits. One stable salt per tenant, held for the whole session.

Your hash algorithm is a reproducibility decision. prefix_caching_hash_algo defaults to sha256, and the Literal is exactly ["sha256", "sha256_cbor", "xxhash", "xxhash_cbor"].

Both xxHash variants carry an explicit warning that a hashing algorithm not considered cryptographically secure theoretically increases collision risk, which can leak private information in multi-tenant environments. On a mixed-version or air-gapped fleet rebuilt from a local mirror, sha256_cbor is what makes a hash computed on one node valid on another; mirror mechanics in air-gapped vLLM deployment.

The one thing to do this week is step 3: take your two most recent agent turns, send both to /v1/chat/completions/render, and print the index where the token IDs first differ. If that number is smaller than your system prompt plus your first assistant message, no amount of GPU memory will fix your hit rate.