KV Cache Quantization: What FP8 and q8_0 Cost in Accuracy

FP8 attention accumulates partial products. On Hopper, once the contraction dimension of the attention GEMM reaches 100K or more, the accumulator's effective precision degrades enough that the softmax numerator and denominator drift and the model attends to the wrong span. The fix is two-level accumulation: write partial results into an actual FP32 register rather than trusting the Tensor Core accumulator across the full contraction. That recovered the score to 89%, a residual two-point gap, not full recovery.

Say the scope plainly. This was a Hopper problem with Flash Attention 3, and vLLM's own text notes that on B200 the accumulation issue is fixed and no two-level accumulation is needed. It is fixed in mainline now, so nobody should conclude FP8 KV cache is broken today.

What matters is the shape of the failure. A precision defect that only appears past 100K of contraction dimension is invisible to every test most teams run: short-context evals pass, perplexity looks fine, throughput improves. The regression lives only in the part of the input distribution your suite does not reach, which on most on-prem deployments is the part users care most about, a pattern we have looked at from the retrieval side in long-context LLM performance issues and context rot at long context lengths.

Two numbers get conflated constantly. The storage number is arithmetic: FP8 stores K and V at one byte per element against BF16's two, exactly half the bytes by element width. That is the dtype's definition, not a benchmark result. The performance number is measured, and it is not 50%. vLLM reports the inter-token-latency slope, the marginal millisecond cost of each additional token of context.

That 4.37e-05 to 2.37e-05 pair is the origin of the widely quoted "54%" figure. It is a slope ratio, not a memory figure, and the post gives no absolute byte measurements. If you see 54% cited as a memory reduction, whoever wrote it did not read the source.

Break-even matters more for capacity planning: it says where the constant overhead stops eating the gain.

Read the third row again. Before the improvements, one model's BF16 and FP8 slopes were nearly identical and break-even sat past 700,000 tokens, meaning FP8 bought that model nothing at any length anyone serves. The comparison spans v0.10.2 to v0.19.1, so if you run an older pinned build, the FP8 numbers in a 2026 blog post do not describe your deployment.

Under load, on Llama-3.1-8B with 150 requests at concurrency 8, FP8 delivered 14.9% higher output throughput, 13.0% faster total runtime, and 14.8% lower median ITL. Real gains, smaller than the headline framing implies. Our LLM latency targets for TTFT and tokens per second covers what to hold a deployment to.

This finding should change how you configure any cluster serving several model families. An independent April 2026 benchmark loaded the same BF16 GGUF three times on one machine, only KV precision changing, measuring token-by-token KL divergence between top-40 log-probability distributions across roughly 250,000 tokens in six categories.

At the identical q8_0 flag, the Gemma MoE sits at 0.377 while both Qwen models stay under 0.04, at least a tenfold spread produced by nothing but the model file. The MoE is roughly 3.5 times worse than its own dense sibling, which suggests routing sparsity interacts badly with cache precision, though nobody has isolated why.

The damage distribution differs too, which determines whether your evals would catch it. Gemma degrades fairly uniformly, its science category at KL 0.214. Qwen concentrates it: the weakest q8_0 category is long documents at 0.142, and at q4_0 that becomes 0.581 against 0.086 for tool calling. A tool-call suite would have cleared Qwen at q4_0 while long-document behavior degraded nearly seven times as much.

Two caveats belong inline. KL divergence is a distributional proxy, not a task score: a model can shift its distribution measurably and still return the same answer. And that benchmark published neither its llama.cpp version nor its context length, which matters for reasons the next section makes concrete.

The datacenter side shows the same model-dependence. On a 256k retrieval task, one 30B MoE dropped from 45.8% to 43.1% AUC on FP8 alone, a 2.7-point loss before any 4-bit format entered the picture. Across vLLM's evaluations FP8 recovery runs from about 94% to 99% of baseline depending on model and setting, with a 70B model retaining 97% to 98% at 128k, all with uncalibrated per-tensor scales that the authors call the worst case.

On 2026-04-01, mainline llama.cpp merged a change that applies a normalized Hadamard rotation to Q, K and V before they are cached. Attention runs in the rotated space and the output is rotated back. Dot products are preserved exactly; outliers get spread across coordinates, and block quantization tolerates a flat distribution far better than a spiky one. Backend-agnostic, compatible with every existing quantization type, opt-out by environment variable.

# rotation is on by default in mainline builds after 2026-04-01
# set this to reproduce pre-merge numbers, or to isolate it during debugging
LLAMA_ATTN_ROT_DISABLE=1 llama-server -m model.gguf \
  --cache-type-k q8_0 --cache-type-v q8_0 -c 32768

The reported effect is not marginal.

The 0.6B model at q5_1 went from unusable to nearly baseline. The 8B model barely moved, because it was never badly broken. Small models are far more sensitive to KV precision, so a 0.6B benchmark tells you almost nothing about a 70B deployment.

The task scores in that pull request are worth pinning to the wall. AIME25 went from 2.0% to 21.7% at q4_0 KV, 25.4% to 32.5% at q5_0, and 31.7% to 37.1% at q8_0. Even q8_0, the setting everyone calls lossless, moved 5.4 points from a change to how activations are laid out before caching. It had been leaving accuracy on the table unnoticed, because the comparison everyone runs is q8_0 against f16 on the same build, not q8_0 against what q8_0 could be.

So any llama.cpp KV benchmark is valid only for builds on one side of 2026-04-01. Record the build number with your results. The KL benchmark above published no version, so its Gemma-versus-Qwen ordering is trustworthy while its absolute values are not portable. We flagged the same variable as a debugging lever in prompt reprocessing on hybrid SWA models; here it is an accuracy control.

One naming correction, repeated widely: what mainline shipped is that Hadamard rotation, not TurboQuant. TurboQuant is a separate method accepted at ICLR 2026 out of Google Research. Its llama.cpp integration pull request closed without merging in June 2026, a code search for its cache type identifiers returns zero results there, and it survives only in unmaintained forks. It did land in vLLM as a set of turboquant_* cache dtypes, a different codebase and a different discussion.

A single-operator harness published in June 2026 ran a 7B coding model, itself already quantized to Q4_K_M weights, on a consumer 16 GB NVIDIA card under llama.cpp: greedy decoding, 256 tokens per prompt, a fixed 12-prompt suite spanning code, math and reasoning.

f16 to q8_0 saves 208 MB. The rest of the way to q4_0 saves another 112 MB and takes similarity from 81.6% to 8.3%. Not a trade, a cliff, and the reason q4_0 is not a general-purpose setting.

Now look at the speed column: 81.8, 76.4, 80.3, essentially flat, with q8_0 the slowest. A stack watching tokens per second, TTFT and p99 latency shows a green board across a change that collapsed output quality by an order of magnitude.

State the caveats, because the source does. That similarity figure is a mean string-ratio comparison against f16, not a task score: 81.6% does not mean the model is 18% worse, it means the token sequence diverged. The suite is 12 prompts, one model, one consumer GPU, one person's harness with no replication. And the same author calls 81.6% similarity "lossless," which is itself an argument for task scores.

Start with vLLM's own four conditions for staying on BF16, the most useful thing in the source literature and the part almost nobody quotes:

That fourth flag earns its own experiment: the gpt-oss-20b slope improves from 7.14e-06 to 6.34e-06 ms/token purely by skipping the sliding-window layers.

Then fix calibration, where most of the recoverable loss lives. vLLM documents three strategies: no calibration with all scales pinned at 1.0, random-token calibration during warmup, and dataset calibration through llm-compressor, the recommended path, which supports per-attention-head quantization. Every accuracy figure above came from the uncalibrated worst case, so calibrating on your own data is the cheapest accuracy you will buy all quarter.

# datacenter starting point, long context, hybrid-attention model
vllm serve <model> \
  --kv-cache-dtype fp8_e4m3 \
  --kv-cache-dtype-skip-layers sliding_window \
  --max-model-len 65536
# then re-run your eval suite; the flag alone is not the deliverable

The engine's own runtime warning for quantized cache dtypes says it may cause an accuracy drop without a proper scaling factor. Treat that as a work item, not log noise.

Perplexity and KL divergence are proxies. String similarity is a worse proxy. A June 2026 preprint, not yet peer reviewed, reports that low-bit KV quantization degrades safety alignment behavior while perplexity stays acceptable, on the theory that alignment features occupy a low-dimensional activation subspace unusually sensitive to quantization. Whether that mechanism holds, the point stands: aggregate distributional metrics do not tell you which behavior broke.

Run this loop instead, on your model, your build, your context lengths:

Two things remain open. Nobody has re-tested the model-family KL spread on a known llama.cpp build. And no published study covers tool calling or structured output under KV quantization: vLLM's May 2026 evaluation says explicitly that it did not test them, so agentic operators are extrapolating and the honest move is to run the test. Worth tracking: variance-normalized approaches pairing Hadamard rotation with dual-scaling across both axes of K and V to attack error accumulation during decoding (arXiv 2606.03458), and NVFP4 as the Blackwell path, whose headline numbers are vendor-published and not independently reproduced.

The bottom line: FP8 and q8_0 are defensible defaults above roughly 7k tokens on most models, q4_0 is not a general-purpose setting, and neither is safe unvalidated on a model family you have not tested. Fleet sizing is covered in self-hosting Kimi K3 on vLLM, engine choice in our SGLang versus vLLM comparison, and the rest in the LLM models pillar.

Particula Tech builds the acceptance suites that sit between a dtype flag and production on regulated on-premise deployments: task-level scoring at your real context lengths, tail-percentile divergence tracking, and a re-test gate on every engine upgrade.