Full Prompt Re-Processing: llama.cpp Cache Fixes That Work

The tell is a flat prefill curve across repeated turns. An independent reproducer in issue #22746 makes it unmistakable: five simulated agents at fixed prompt sizes, three rounds, identical prompts each round. If prefix reuse worked, rounds two and three would be near-instant.

Rounds two and three land where round one did. The script labelled every one "LIKELY COLD (cache miss)". That run was on a Qwen3.6-35B-A3B build, on different hardware from the issue author's.

This hides well. The answer is correct, so no eval flags it, and a cold turn just looks like a longer prompt. It reads as a defect only once you plot time-to-first-token against turn number and see a flat line instead of a cliff after turn one. Our guide on fixing slow LLM latency in production apps covers the wider measurement order.

The server does tell you. The searchable string, hyphen included, is:

forcing full prompt re-processing due to lack of cache data

The rendered line appends "(likely due to SWA or hybrid/recurrent memory, see ...)" plus a link to a 2025 pull request comment. Do not follow it: that comment is a two-sentence design note about supporting both the old and new SWA cache behind a flag, and says nothing about checkpoints.

Here is the part that costs people days. On current master the message is emitted through SLT_TRC, which resolves to LOG_LEVEL_TRACE = 4 in common/log.h, while common/common.h sets the default verbosity to 3 (info). On a default llama-server run the line is never printed. Older builds surfaced neighbouring checkpoint lines at warning level, which is why "I checked my logs and it is not there" is worth nothing as evidence.

Start the server at trace verbosity and filter:

llama-server \
  --model /models/your-model.gguf \
  --host 127.0.0.1 --port 8080 \
  --ctx-size 65536 \
  -lv 4 \
  2>&1 | grep -E "forcing full prompt|context checkpoint|Checking checkpoint|n_past ="

The lines that matter, all from tools/server/server-context.cpp:

A context checkpoint is not a copy of the KV cache. That single misunderstanding explains most of the confused advice here.

In server-context.cpp, checkpoint creation and restore both pass LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY, and the comment on that flag in include/llama.h is explicit: it works "only with partial states, such as SWA KV cache or recurrent cache (e.g. Mamba)". A checkpoint stores only the parts of memory that cannot be reconstructed by rolling back. The full-attention KV for the prefix is not in there, because it is still sitting in the slot and can simply be truncated.

Truncation is the primitive everything hangs off. llama_memory_seq_rm removes tokens in [p0, p1) and, per its doc comment, "returns false if a partial sequence cannot be removed." llama.cpp probes this at startup and classifies the context:

The source states the recurrent case plainly: "models like Mamba or RWKV can't have a state partially erased at the end of the sequence because their state isn't preserved for previous tokens." A recurrent layer carries one running state overwritten in place at every step, so there is no per-token history to truncate back to. llama.cpp does implement a bounded rewind via a ring of per-token snapshots, succeeding only when cell.pos - (p0 - 1) falls between 1 and n_rs_seq: a small fixed lookback window, the SEQ_RM_TYPE_RS row above.

For sliding window attention, the SWA cache pull request (PR #13194, merged 2025-05-20) is the cleanest statement available: "advanced cache operations such as removing tokens or shifting their positions are not possible when using SWA cache, because token information becomes lost when the window slides."

Checkpoints exist to paper over exactly these two cases, and creation is gated on it: the context must be FULL, be RS, or the model must have n_swa > 0.

llama_pos pos_next = slot.prompt.tokens.pos_next(n_past);
const bool has_new_tokens = (n_past < slot.task->n_tokens());
const auto pos_min_thold = std::max(0, pos_next - n_swa - (has_new_tokens ? 0 : 1));

The maintainers' position on the open issues is that most reports are the client mutating the prefix: "You simply cannot cache something that changes." So your repro has to take the harness out of the picture entirely.

One caution about the repro in issue #21831, which sends "My name is Tester. Remember it." then "What is my name?" as two curl calls. The second logged prompt_tokens: 18, so the body carried only the new message, and /v1/chat/completions is stateless. The model answering "I don't know your name" proves nothing. The log line and the restart at position zero are the evidence. The answer text is not. Send the full history every turn.

#!/usr/bin/env bash
set -euo pipefail
HOST=${HOST:-http://127.0.0.1:8080}
HIST='[{"role":"user","content":"Summarise the attached spec."},
       {"role":"assistant","content":"Sure. Paste it."}]'

# Grow the conversation by one fixed turn per iteration and time TTFT.
for i in $(seq 1 8); do
  BODY=$(printf '{"model":"local","stream":true,"max_tokens":16,"messages":%s}' \
         "$(printf '%s' "$HIST" | sed 's/\]$/,{"role":"user","content":"Turn '"$i"'. Continue."}]/')")
  T=$(curl -s -o /dev/null -w '%{time_starttransfer}' \
        -H 'Content-Type: application/json' -d "$BODY" \
        "$HOST/v1/chat/completions")
  printf 'turn %d  ttft=%ss\n' "$i" "$T"
  HIST=$(printf '%s' "$HIST" | sed 's/\]$/,{"role":"user","content":"Turn '"$i"'. Continue."},{"role":"assistant","content":"ok"}]/')
done

With stream: true, curl's time_starttransfer is a usable proxy for time-to-first-token. Run it against a server at -lv 4 and read both streams side by side. Healthy: a restored context checkpoint line and a roughly flat TTFT. Broken: forcing full prompt re-processing plus a TTFT scaling linearly with conversation length, forever.

If the curl repro is clean and your harness still reprocesses, the prefix is being mutated between turns. Add --log-prompts-dir /tmp/llama-prompts and diff consecutive dumps; that is the maintainer-endorsed way to prove it. Usual culprits: reasoning traces stripped from earlier turns, tool definitions regenerated in a different order, and harnesses that rewrite history in place. On reasoning, the maintainer's position is that this is "normal behavior for chat templates that do not keep the reasoning contents of previous turns," and that even -rea off still trashes context, because the assistant turn emits two <think></think> tokens that get stripped next turn.

The clearest logged event in the threads is worth walking through, because a naive reading produces the wrong conclusion. Setup: a 27B dense Qwen quant on a 16 GB RX 7800 XT, --ctx-size 131072 with --parallel 2, so every log line reads n_ctx_slot = 65536. That is the number that matters, not the 131,072 in the script.

The wrong reading is "high-similarity checkpoints failed validation." Those similarity numbers come from slot selection on the five earlier successful requests and say nothing about checkpoint validation. The real story is step 2 into step 3: an oversized request was rejected, the next got assigned by least-recently-used rather than prefix match, its reuse point collapsed to position 3, and because the model's state cannot be rewound, all 15 checkpoints became garbage at once.

Note n_swa = 0 on every erasure line: this model has no sliding window at all. The message is one generic string covering two distinct architectures, and for these Qwen builds the trigger is the hybrid recurrent side. That has a direct consequence for the advice you will find online.

First, the terminology trap. llama.cpp has three separate things that all sound like caching, and conflating them is why so much advice here is wrong:

Every default below was read from common/arg.cpp, common/common.h and tools/server/README.md on master, with arg.cpp last touched 2026-07-27 and the server README 2026-07-28.

Read the last two rows twice, because the most-copied workaround for this problem is dead. --checkpoint-every-n-tokens 1024 --ctx-checkpoints 256 fails to parse on any current build, and the replacement is not a rename: the old flag set a creation cadence, -cms sets an eviction floor. Checkpoints are now created at user message boundaries (PR #24176, merged 2026-06-23), and PR #25472 (2026-07-12) evicts any landing within min-step of an earlier one.

That combination has a practical consequence nobody documents. If your agent turns are 800 tokens apart and -cms sits at its 8192 default, most per-user-message checkpoints are created and then immediately evicted for being too close together, and the log says so: erasing context checkpoint too close to an earlier one. Lower -cms toward your median turn length to keep them, at a memory cost you can compute. Observed checkpoint sizes in these threads run 62.813 MiB, 149.626 MiB, and 188.302 to 213.707 MiB, growing with position. Thirty-two at 150 MiB is 4.7 GiB, more than half the default 8192 MiB --cache-ram, before you multiply by --parallel.

Every checkpoint above lives in system RAM. On a discrete-GPU box that is a genuine second tier: spend host memory, avoid recomputing on the accelerator. On unified memory it is not a tier at all.

The open feature request in issue #20697 states it precisely, from a 128 GB AMD Strix Halo box running an 85 GB Qwen3.5-122B-A10B quant at 220k context on Vulkan: "on UMA (Unified Memory Architecture) systems such as AMD Strix Halo, system RAM and VRAM are physically the same memory pool. This means checkpoint caching actively competes with model weights and KV cache for the same limited resource, rather than providing relief." Or bluntly: "--cache-ram doesn't offload anything."

That covers Apple Silicon, Strix Halo, and any integrated-memory accelerator you would pick for large-model private inference, which is a large share of the on-premise deployments we see. There, the default 8 GiB prompt cache is 8 GiB not spent on weights or KV. Size --cache-ram deliberately.

The proposed fix, a disk tier via --cache-disk and --cache-disk-max, does not exist: zero occurrences of either string in arg.cpp on master, no maintainer reply, no linked pull request, last activity 2026-07-17. Its performance case is spec-sheet arithmetic, an estimated 10 to 20 ms restore of a "typical checkpoint (50-100 MiB)", it was challenged in the thread, and that size premise understates the checkpoints actually observed, which reach 213.707 MiB.

The one shipping disk path is --slot-save-path with the manual /slots endpoints, and it is a different mechanism: full sequence state on demand, not partial checkpoints. A user in the thread put it correctly, that it survives a server restart and will not save unit checkpoints for later reuse. Warm-start tooling, not a checkpoint tier.

This is not a law of hybrid architectures. It is a llama.cpp implementation state, and the two production serving engines took a different route.

vLLM's hybrid KV cache manager documentation is explicit about where it stops. It rejects naive round-robin allocation for sliding windows precisely because that is incompatible with prefix caching, freeing blocks outside the window instead. But the coordinator "handles exactly two KV cache groups (must include one full-attention group plus one other efficient-attention group). Other cases are not implemented."

On the Mamba side, mind the gap between vLLM's docs and its code. The usage guide still says prefix caching "is not yet supported" for these models, while main ships --mamba-cache-mode with modes none, align and all, plus --mamba-block-size and a SupportsMambaPrefixCaching interface. Seven model files declare that interface; the Qwen3-Next and Qwen3.5 files do not, so those fall back to align, which requires chunked prefill. The runtime log calls Mamba-mode prefix caching "experimental" and asks you to report issues. Shipped and mode-dependent, not solved. Our writeup on debugging vLLM KV cache OOM covers what the block manager does with the memory you give it.

SGLang went furthest architecturally. Its recurrent checkpoint pool stores int8-compressed linear-attention states, one per radix node, and dequantizes into a fresh active slot copy-on-write on a hit. The source header describes int8 as giving roughly twice the cached-prefix capacity at fixed memory versus bf16, quantized once on store and dequantized once on a hit so it never re-enters the recurrence as a quantize-dequantize loop. That is a design claim in a code comment, not a benchmark, but the shape is what matters: it caches the state itself rather than a partial snapshot that needs a valid rollback to be useful. Some acceleration paths are mutually exclusive with radix prefix caching, so read the validation errors first.

Hosted APIs hide all of this by construction. You do not get the server log, so whatever the engine decided about your prefix is unobservable, and the only signal is whatever cache accounting the provider reports. That is its own discipline, and we wrote up a version of it in debugging a five-minute prompt cache TTL regression. Self-hosting is what buys you the trace line.

Preserving reasoning is a real flag now, not a template hack. --reasoning-preserve exists on master alongside --chat-template-kwargs, for templates advertising supports_preserve_reasoning. Do not reach for -rea off instead: the assistant turn still emits two empty thinking tokens, they still get stripped next turn, and the context still churns.

The slot-flip mitigation is underrated. In the event above, nothing was wrong with the checkpoints. An oversized request got rejected and the conversation lost its slot. If your harness can exceed the per-slot context, it will eventually evict its own warm state. Compact before you hit the wall; our context compaction guide for long-running agents covers where that check belongs.

**LLAMA_ATTN_ROT_DISABLE=1 is a debugging lever, not a fix.** A real environment variable that force-disables attention rotation, undocumented in the server README, meaningful only with a quantized KV cache type, and reported in-thread as a workaround for this symptom. Use it to isolate, not to ship.

Model choice is the highest-leverage knob and the one teams consider last. Checkpoints are created only for contexts that cannot do arbitrary partial removal, or for models with n_swa > 0. A full-attention model has no checkpoint problem at all on llama.cpp, because its prefix KV can simply be truncated. Weigh prefix reuse alongside the usual quality and memory math; our guide to running GLM 5.2 locally walks the memory side of that decision.

None of the three issues is closed. Both reprocessing reports carry bug-unconfirmed and both were still active on 2026-07-28. Several merged pull requests through 2026 improved the bookkeeping and community reports on those builds conflict, so do not assume it is fixed and do not assume it is broken. Measure your own build.

The practical version is short. Run at -lv 4. Grep for forcing full prompt re-processing due to lack of cache data. If it fires, dump prompts and find out whether your harness is rewriting the prefix, because that is the cheapest fix by a wide margin. If the prefix is stable, tune -cms and -ctxcp against a real memory budget and stop copying the flag deleted in May. If neither helps, the answer is a different model or a different engine. We run this pass early on private inference deployments at Particula Tech, because prefix reuse is usually worth more than any quantization or batching change available on the same hardware. More on tooling decisions for these stacks sits in our AI development tools pillar.