KV Cache Duplication Factor: Divide tp_size by KV Heads

The vLLM doc defines the variable precisely: "For a model with H kv-heads, a request with T tokens in the context needs to store H * T key/value tensors in the KV cache." H is the checkpoint's kv-head count, not its attention head count, and not the number of GPUs.

The factor is max(1, tp_size // H). Integer division, floored at one, because a layout cannot duplicate less than once.

Work the ordinary case before the alarming one. Qwen3-32B's config.json reads num_attention_heads 64 and num_key_value_heads 8. Serve it on a single eight-card node with -tp 8 and the factor is max(1, 8 // 8), which is 1. Each rank holds exactly one kv-head, no rank holds a head another rank already has, and nothing is wasted. Decode context parallelism has nothing to recover here, and as the validator section shows, vLLM refuses to enable it. If that describes your layout, go back to the OOM triage order: your KV shortage, if you have one, comes from somewhere else.

That generalises. The factor only leaves 1 when the tensor-parallel size strictly exceeds H, which on an ordinary eight-card node means a checkpoint with fewer than eight kv-heads: multi-head latent attention checkpoints, where H is 1, and grouped-query checkpoints with a small head count such as 4.

The obvious audit gives the wrong answer in the worst possible direction. Reading num_key_value_heads out of config.json and dividing tp_size by it produces a confident "no duplication" for exactly the two checkpoints carrying the most of it.

Columns two through four are read from the config.json files those repositories ship. Column five is the H that governs the factor: 1 whenever use_mla is true, because ModelConfig.get_num_kv_heads returns 1 unconditionally in that case with the source explanation that when using MLA during decode it becomes MQA, and otherwise whatever get_total_num_kv_heads reads out of the config. Do the naive division on row one: 8 // 128 is 0, clamps to 1, and reports a clean bill of health for a layout that is storing the cache eight times.

vLLM decides use_mla from two conditions that must both hold. The model_type has to appear in a list described in source as a manually maintained list of model types for vLLM model implementations, which at v0.28.0 includes deepseek_v2, deepseek_v3, kimi_k2, longcat_flash and roughly twenty entries in total. And the checkpoint has to carry kv_lora_rank. Membership alone is not sufficient. Separately, use_mla returns false when the environment variable VLLM_MLA_DISABLE is truthy, which vLLM reads as bool(int(os.getenv("VLLM_MLA_DISABLE", "0"))), so a launch script you inherited that sets it to 1 sends H back to the checkpoint's own kv-head count and changes every number here. We make no claim about what else that switch does.

The second trap is the attribute name. get_total_num_kv_heads tries five keys in order, n_head_kv, num_kv_heads, num_key_value_heads, multi_query_group_num and num_attention_groups, taking the first present, and falls back to the total attention head count when none is, with the comment that for non-grouped-query attention models the number of KV heads is equal to the number of attention heads. A grep for num_key_value_heads alone silently misreads Falcon-style, ChatGLM-style and plain multi-head checkpoints.

The audit that reproduces vLLM's answer therefore has to check MLA first, then the five keys, then fall back. This script does that. Its MLA type set is deliberately abridged, and the compress_ratios branch that newer DeepSeek variants use is not modelled, so treat it as a first pass and confirm anything surprising against the model's own config.

import json, sys

# Abridged from vLLM v0.28.0: ModelArchConfigConvertorBase.is_deepseek_mla
# lists ~20 model_type values; membership plus kv_lora_rank means MLA.
MLA_MODEL_TYPES = {
    "deepseek_v2", "deepseek_v3", "deepseek_v32", "deepseek_v4",
    "kimi_k2", "kimi_linear", "longcat_flash", "glm4_moe_lite",
}
# vLLM reads the first of these that exists, else falls back to head count.
KV_HEAD_KEYS = ("n_head_kv", "num_kv_heads", "num_key_value_heads",
                "multi_query_group_num", "num_attention_groups")

cfg = json.load(open(sys.argv[1]))
cfg = cfg.get("text_config", cfg)
tp = int(sys.argv[2])

mla = cfg.get("model_type") in MLA_MODEL_TYPES and "kv_lora_rank" in cfg
h = 1 if mla else next((cfg[k] for k in KV_HEAD_KEYS if k in cfg),
                       cfg["num_attention_heads"])
factor = max(1, tp // h)
print(f"model_type={cfg.get('model_type')} mla={mla} H={h} tp={tp} "
      f"duplication_factor={factor} max_dcp={factor}")

Run against the four config.json files above, it reproduces all three cases vLLM documents in prose and returns 1 for Qwen3-32B at -tp 8. max_dcp equals the factor because that is exactly where vLLM caps decode context parallelism.

vLLM's documentation gives three worked cases, written as prose under a "Case study:" line. There is no table in that document, so the table below is our summary of those three cases plus two rows of arithmetic we did against Qwen3-32B's config and the validator source.

The first three rows are vLLM's own recommendations, and they translate into these commands. They are the only per-model DCP invocations we print, because they are the only ones the project documents.

# DeepSeek-R1: MLA, 1 kv-head, single node of eight cards.
# -tp 8 alone stores the cache eight times.
vllm serve deepseek-ai/DeepSeek-R1 -tp 8 -dcp 8

# Qwen3-235B-A22B: 4 kv-heads. -tp 8 stores it twice.
vllm serve Qwen/Qwen3-235B-A22B -tp 8 -dcp 2

# Kimi-K2: MLA, two nodes. -dcp 16 removes duplication entirely;
# -dcp 8 leaves 2x but keeps the DCP traffic inside one node.
vllm serve moonshotai/Kimi-K2-Instruct -tp 16 -dcp 16
vllm serve moonshotai/Kimi-K2-Instruct -tp 16 -dcp 8

The Kimi-K2 pair is the interesting one, and the residual is vLLM's own figure rather than our arithmetic: the doc says -dcp 16 removes the duplication completely at the cost of more communication overhead, and that -dcp 8 reduces it to 2x while keeping DCP traffic inside a single node. That is a deliberate trade of two copies of the cache against a cross-node collective on every decode step, and vLLM publishes no figure that settles it for your interconnect.

The last two rows are the case the documentation does not cover: a conventional 8-kv-head checkpoint. On one node it has no duplication and no DCP. Push it to -tp 16 across two nodes, a layout you might already run for latency or for weights rather than for cache, and the factor becomes 2 while -dcp 2 becomes legal. The interconnect side of that decision is in our comparison of RTX PRO 6000, H100 and L40S.

Raising --tensor-parallel-size when the KV cache is short is standard advice, and it is ours: our vLLM KV cache OOM post names it in the memory-budget table, in the error table, in the swap-space footgun, and in the last step of the triage order. Nothing here contradicts that. What it adds is the ceiling.

The advice holds cleanly while tp_size is at or below H. Every rank you add there takes a real slice: the heads divide further and per-rank KV bytes fall proportionally. Past H the arithmetic changes character. max(1, H // tp) has already bottomed out at one head per rank, so the next rank does not subdivide anything. It holds another copy of a head that already exists on a peer, per-rank cache stops shrinking, and what you bought is weight sharding and more collective traffic. Those may still be worth buying. They are just not KV cache room.

So the qualifier is a condition, not a correction: read H before you treat tensor parallelism as a memory lever. Above H it is a different lever that happens to share a flag.

Once you are past H, the head dimension has no more room, so vLLM offers a different dimension. The doc: "then we need to add decode context parallel to further shard the KV cache along the T dimension. This is as simple as adding -dcp <size> to the command line." T is the token dimension, and the layout is interleaved so that tokens generated later shard naturally as they arrive rather than requiring a redistribution.

The property that makes it worth considering at all is that it does not ask for hardware. The doc states it plainly: the size "does not increase the number of GPUs we need to launch, but just reduces the KV cache duplication." The field's own docstring at v0.28.0 says the same thing and one more thing besides.

decode_context_parallel_size: int = Field(default=1, ge=1)
"""Number of ranks that shard the decode KV cache. DCP does not expand
the process world size. Without PCP, DCP reuses TP ranks. With PCP, DCP
either spans the PCP axis or the full TP x PCP block."""

The last sentence is the part that gets dropped when this docstring is quoted, and it is the part with an open pull request attached. Prefill context parallel is a separate axis with the opposite property: its docstring describes it as the number of ranks that split prefill sequence computation, which "expands the process world size but does not increase the KV-cache shard count." PCP is not an alternative to DCP for duplication, because it does not shard the cache at all. Splitting prefill from decode across separate replicas is a third thing again, a serving-topology decision we cover in prefill-decode disaggregation, and it is not what -pcp does.

Two more facts before the bound. Coverage is broader than the MLA case that motivates it: the doc says decode context parallel is supported for both MLA and GQA models, and that some attention backends also support combining DCP with multi-token prediction. And DCP changes the effective block size, since kv_cache_utils computes the group block size as cache_config.block_size * dcp with the comment that attention groups are scaled by DCP. That matters when you read the startup log, because the block accounting behind the concurrency figure has moved.

The doc gives the range and the reason for the ceiling: "the dcp size should lie in the range of [1, tp_size/H]. With larger dcp size, the KV cache duplication is reduced, but the communication overhead increases." Going past the ceiling is described as theoretically possible but left out for simplicity, because "it's unclear what should we do for the remaining dcp_size - tp_size / H GPUs for non-attention layers."

Documentation is guidance. The question for anyone writing a launch script is what the code refuses, and two different files do the refusing. Reading only the first gives a wrong answer: ParallelConfig validates the geometry, and nothing in it mentions kv-heads.

tp = self.tensor_parallel_size
pcp = self.prefill_context_parallel_size
dcp = self.decode_context_parallel_size
if pcp > 1 and self.data_parallel_size > 1:
    raise ValueError("PCP does not support data parallelism yet.")
if pcp == 1:
    # DCP reuses the TP ranks when PCP is disabled.
    if tp % dcp != 0:
        raise ValueError(f"tp_size={tp} must be divisible by dcp_size={dcp}.")
elif dcp not in (1, pcp, tp * pcp):
    raise ValueError(
        "When PCP is enabled, DCP must be disabled, span the PCP "
        "axis, or span the full TP x PCP axis. "
        f"Got TP={tp}, PCP={pcp}, DCP={dcp}; valid DCP sizes are "
        f"{sorted({1, pcp, tp * pcp})}."
    )

At default settings, -pcp being 1, that is a single divisibility rule. Ask for -tp 8 -dcp 3 and it produces ValueError: tp_size=8 must be divisible by dcp_size=3 before any model-aware check runs, because ParallelConfig is validated on construction while the kv-head check happens later. That ordering is a reading of the two call sites rather than an observed run.

The kv-head bound lives in ModelConfig.verify_with_parallel_config, which VllmConfig.__post_init__ calls at startup. Three checks run there, and all three are skipped for MLA models, where H is 1 and the divisibility rule already caps dcp at tp.

decode_context_parallel_size = parallel_config.decode_context_parallel_size
if decode_context_parallel_size > 1 and not self.use_mla:
    total_num_kv_heads = self.get_total_num_kv_heads()
    if tensor_parallel_size <= total_num_kv_heads:
        raise ValueError(
            "Decode context parallelism for GQA/MQA requires "
            f"`--tensor-parallel-size` ({tensor_parallel_size}) to be "
            "greater than the model's total number of KV heads "
            f"({total_num_kv_heads}). Increase `--tensor-parallel-size` "
            "or set `--decode-context-parallel-size 1`."
        )

    max_dcp_size = tensor_parallel_size // total_num_kv_heads
    if decode_context_parallel_size > max_dcp_size:
        raise ValueError(
            "`--decode-context-parallel-size` "
            f"({decode_context_parallel_size}) exceeds the maximum "
            f"supported value ({max_dcp_size}) for "
            f"`--tensor-parallel-size` ({tensor_parallel_size}) and "
            f"{total_num_kv_heads} model KV heads."
        )

    num_q_per_kv = total_num_attention_heads // total_num_kv_heads
    if num_q_per_kv % decode_context_parallel_size != 0:
        raise ValueError(
            "The model's number of query heads per KV head "
            f"({num_q_per_kv}) must be divisible by "
            "`--decode-context-parallel-size` "
            f"({decode_context_parallel_size}) for GQA/MQA."
        )

Two of those three rejections fire on configurations from the table above. Qwen3-32B has 8 kv-heads, so -tp 8 -dcp 2 enters the guard (DCP is above 1, the model is not MLA), hits 8 <= 8 on the first check, and never starts.

ValueError: Decode context parallelism for GQA/MQA requires
`--tensor-parallel-size` (8) to be greater than the model's total number
of KV heads (8). Increase `--tensor-parallel-size` or set
`--decode-context-parallel-size 1`.

Move the same model to two nodes and ask for more than the heads allow. At -tp 16 the first check passes, max_dcp_size is 16 // 8, which is 2, and -dcp 4 trips the second. The attention-head divisibility rule that also lives in this method does not fire first, because 64 attention heads divide by 16 cleanly.

ValueError: `--decode-context-parallel-size` (4) exceeds the maximum
supported value (2) for `--tensor-parallel-size` (16) and 8 model KV heads.

Both are the messages those f-strings produce for those values, read from source rather than copied from a terminal. The consequence is the good one: on a GQA checkpoint you cannot quietly serve a bad DCP size. The residual gap is prefill context parallel, which is what pull request 55030 addresses. Open at the time of writing, it touches only vllm/config/model.py and the config tests, and its diff extends this guard rather than creating one, replacing the tensor-parallel-only bound with max(1, tp // H) multiplied by the prefill context parallel size. Finally, --decode-context-parallel-size 0 is not how you turn DCP off. The field is declared Field(default=1, ge=1), so zero is rejected by the constraint. Disabled is 1, which is already the default.

Communication first, because the doc puts it in the same sentence as the benefit: with larger dcp size the duplication falls and the overhead rises. That is the whole published characterisation. vLLM ships no measured throughput or latency comparison for DCP at v0.28.0, so a quoted speedup for the flag is measuring something else. The one quantified figure in the source is a comparison between DCP backends, not between DCP and no DCP: --dcp-comm-backend accepts ag_rs (allgather plus reducescatter, the default) or a2a, and the a2a docstring says it reduces NCCL calls from three to two per layer for MLA models. Note also the shape of the Kimi-K2 recommendation, where the project offers keeping 2x alongside the duplication-free option, on the grounds that it holds the collective inside one node. That is the cost curve stated as a choice.

Then churn. Two DCP bugfix pull requests were open in the vLLM repository in the days around this writing, 55030 on the validation path and 55168 on CPU load alignment for hybrid models under DCP. A correctness defect on the DCP path affecting dense prefill on non-owner ranks was filed and closed inside 48 hours in the same window. Fast turnaround is a good sign about the maintainers and a bad sign about running whatever tag your build pulled last night. Pin the vLLM tag in your image and record it next to your acceptance run.

The arithmetic does not change on-premise. max(1, tp_size // H) is the same on a rented cluster and on a purchased rack. What changes is that on a rack you own you cannot rent a ninth card. Your tp was chosen to make the weights fit across the cards in the chassis, your H was fixed by whichever checkpoint cleared your model-approval process, and the duplication factor is a downstream property of two procurement decisions already made and expensive to unmake. The only free HBM on a fixed rack is the HBM you are already spending on copies, which is why the audit is worth ten minutes even when the answer is 1.

Governance next. In a validated environment -dcp is not a tuning knob. It changes the serving topology and the numerical path through attention, since partial outputs are now combined across ranks, so it is a re-run of the acceptance set behind a new pinned image, with the flag recorded in the deployment manifest. The audit itself needs no network: the script above reads a config.json off the same local checkpoint mirror you already run for air-gapped vLLM deployment.

A procurement note follows from the same arithmetic. A throughput or concurrency chart labelled TP 8 on an MLA model was measured on a layout storing the KV cache eight times, unless the run notes say the benchmark set -dcp. That does not make the chart dishonest, but it does make it un-transferable in both directions, so the question to ask is what the decode context parallel size was during the benchmark. If nobody knows, the concurrency figure is not the one your rack will produce.

One caveat on recovered capacity. Removing duplication only buys concurrency when KV cache is the binding constraint, and on MLA checkpoints carrying linear or hybrid attention layers it may not be, which we work through in GPU sizing for a 1,561 GB MLA checkpoint. The measurement, not the factor, is the deliverable.

The instrument for the comparison is one line vLLM already logs at startup, emitted by update_kv_cache_capacity.

# vllm/v1/core/kv_cache_utils.py, update_kv_cache_capacity(), v0.28.0
logger.info_once(
    "GPU KV cache size: %s tokens, "
    "Maximum concurrency for %s tokens per request: %.2fx",
    f"{num_tokens:,}",
    f"{max_model_len:,}",
    max_concurrency,
)

The concurrency figure is the number of KV blocks available divided by the blocks a full-length request needs, so it moves when the duplication moves, in the unit you care about: how many maximum-length conversations fit at once.

So the week's work is small and ordered. Run the audit script against the config.json in your checkpoint mirror, for every model you serve, at the tp you actually use. Write down the factor. For every model where it is 1, close the ticket and change nothing. For the models where it is not, capture the GPU KV cache size and Maximum concurrency line from your current startup log, add -dcp at the bound the table gives you, restart on a pinned tag, and capture the same line again. If the second number is no better on the max_model_len you actually serve, you learned that for the price of one restart and you keep the simpler layout. Where the cache itself is the constraint rather than its layout, what fp8 KV quantisation costs in accuracy is the next lever to price.