Prefill-Decode Disaggregation for LLM Serving at Scale

They fight because they stress opposite parts of the hardware. Prefill saturates compute; decode saturates memory bandwidth; a single GPU running both at once is always starving one to feed the other.

Prefill is the phase where the model reads the entire input prompt in one parallel pass and builds the KV cache, the set of key and value attention tensors for every token it will later attend to. Because it processes all prompt tokens simultaneously, prefill is a dense matrix-multiply workload that pins the GPU's compute units. Decode is the autoregressive phase that generates output one token at a time, and each step re-reads the whole KV cache to produce the next token. That makes decode bound by how fast the GPU can move the cache in and out of memory, not by raw compute.

The Towards Data Science explainer put the framing in its title in April 2026: "Prefill Is Compute-Bound. Decode Is Memory-Bound. Why Your GPU Shouldn't Do Both." When both phases live on the same device, the scheduler has to interleave a compute-hungry burst with a bandwidth-hungry stream, and neither gets what it needs. Add continuous batching, where new prompts join the batch while existing requests are still generating, and you get the worst case: a long prefill arriving mid-batch stalls every request already in the decode phase.

Across the inference stacks we audit at Particula Tech, this mismatch is the most common root cause of latency SLOs that pass in a load test and fail in production: the test sends uniform request sizes, and real traffic mixes short and very long prompts on the same GPU.

P99 latency spikes because one long prefill blocks the decode steps of every other request sharing the GPU, and that blocking lands on the tail, not the average. P99 latency is the response time at the 99th percentile, the slow experience that roughly one request in a hundred actually gets.

Picture a GPU serving fifty concurrent chat sessions, all mid-generation, each expecting a token every few tens of milliseconds. A new request arrives with a 32,000-token prompt. Its prefill is a large compute burst, and while the GPU services it, the fifty in-flight decodes wait. Every one of those users sees their inter-token latency jump for the duration of that prefill. Inter-token latency (ITL, sometimes called TTIT) is the pause a user watches between words appearing on screen, and it is the metric that makes a stream feel smooth or stuttering.

The damage is invisible on an average-latency dashboard because the spikes are episodic and concentrated. Your P50 looks healthy; your P99 quietly doubles whenever a big prompt lands. This is the same tail-latency trap we unpack in our guide to fixing slow LLM latency in production apps, and it gets worse as prompts get longer, which is why teams running long-context workloads hit performance problems first. The longer the prompts and the higher the concurrency, the more often a prefill burst collides with a batch full of decodes.

Disaggregation attacks the collision directly. If prefill never runs on the same device as decode, a long prompt cannot stall anyone's token stream. That is the core promise, and it is why the primary payoff of the architecture is tail-latency smoothness, not average throughput.

Prefill-decode disaggregation works by dedicating one GPU pool to prefill and another to decode, then streaming the KV cache from the prefill pool to the decode pool once the first token is ready. Each pool is sized and tuned for its own bottleneck.

The request lifecycle looks like this. A prompt arrives at a prefill worker, which runs the compute-heavy forward pass over the full input, produces the KV cache and the first token, then transfers that cache across the network to a decode worker. The decode worker loads the cache and takes over autoregressive generation, emitting the rest of the tokens at a steady cadence. Because the prefill pool only ever does bursts of compute and the decode pool only ever does bandwidth-bound generation, each can be provisioned independently: more prefill capacity when prompts are long, more decode capacity when generations are long.

prompt --> [ Prefill pool ]  compute-bound: builds KV cache + first token
                 |
                 v   KV cache streamed over the network (RDMA)
           [ Decode pool ]   memory-bandwidth-bound: generates the rest --> stream

This is a different question from which engine to run. Comparing vLLM, Ollama, and TensorRT-LLM, as we do in our model-serving engine comparison, is about picking the software that executes the model. Disaggregation is an architecture decision that sits above the engine: it is about how many pools you run and what each one does. You can disaggregate within vLLM, within SGLang, or within an NVIDIA Dynamo stack, so the SGLang versus vLLM comparison is the right place to start on engine choice, and this post is about the topology you wrap around whichever engine you pick.

One nuance: the prefill-to-decode ratio is not one to one. Decode needs far more capacity, because generation runs many more steps than the single prefill pass, as DeepSeek's production numbers show below.

Disaggregation only works if the KV-cache transfer between pools is faster than the latency it saves; otherwise you have simply moved the bottleneck onto the network. The KV cache for a long prompt is measured in gigabytes, and it has to cross the wire before the decode pool can emit the first token, so a slow transfer lands directly in time to first token.

This is why production disaggregated serving runs on RDMA-class fabrics: InfiniBand or RoCE (RDMA over Converged Ethernet), which move data between GPU memories without round-tripping through the CPU. The vLLM project, whose disaggregated prefilling is officially experimental, ships a set of KV-transfer connectors for exactly this handoff: NixlConnector (built on NVIDIA's NIXL library), LMCacheConnector, and MooncakeConnector, among others. In their September 2025 test the PyTorch and vLLM teams used plain TCP, which functions but is the slower path and is best treated as a starting point rather than a production fabric.

The engineering rule is blunt: your KV-transfer path must cost less time than the decode-stall it eliminates. On a fast RDMA fabric with long prompts, that inequality holds comfortably, and disaggregation is a clear win. On commodity Ethernet with short prompts, the transfer overhead can swamp the benefit, which is the first sign you should not split. This is also why disaggregation is an infrastructure decision, not a config flag, and why its cost math ties into the broader self-hosting versus API break-even question: the RDMA fabric that makes it pay is part of the fixed cost you amortize.

The strongest evidence comes from three public deployments, and all three confirm the same thing: disaggregation is real, in production, at scale. DeepSeek runs it, SGLang reproduced it in the open, and Meta enabled it in its internal stack.

DeepSeek published the most detailed account. According to DeepSeek's DeepSeek-V3/R1 inference system overview (February 2025), the team "adopted prefill-decode disaggregation architecture" and gave the numbers: a prefill deployment unit of 32 H800 GPUs (4 nodes) and a decode unit of 144 H800 GPUs (18 nodes), sustaining roughly 73,700 input tokens per second per node during prefill (including cache hits) and about 14,800 output tokens per second per node during decode. Over a single profiled 24-hour window in late February 2025, the system processed 608 billion input tokens and 168 billion output tokens, at an infrastructure cost DeepSeek put at $87,072 per day (assuming $2 per GPU-hour) against a theoretical revenue of $562,027. That margin is what makes the operational complexity worth carrying.

SGLang then proved the design was not proprietary magic. According to the LMSYS SGLang team (May 2025), an open-source prefill-decode disaggregation deployment on H100 nodes reached 54,543 input tokens per second per prefill node and 22,282 output tokens per second per decode node on 2,000-token prompts, within 5.6% of DeepSeek's official prefill profile and 6.6% below on decode. They reported a cost of roughly $0.20 per million output tokens, about one-fifth the cost of the official DeepSeek Chat API, and described it as the first open-source implementation to nearly match DeepSeek's reported throughput.

Meta added the enterprise-scale data point on a different model. According to the PyTorch and vLLM engineering teams (September 2025), running Llama4 Maverick on hosts of eight H100 GPUs with TCP-based KV transfer, prefill-decode disaggregation delivered higher throughput at the same batch size and, more importantly, "better control for the overall latency due to the much smoother TTIT curve." Their setup held HBM utilization near 90% at a 40-to-50% prefix cache hit rate and used larger KV-cache block sizes (128 and 256) than vLLM's default of 16. They also flagged the honest failure mode: TTFT regresses sharply under very heavy load. Meta reported that prefill-decode disaggregation has been enabled in its internal inference stack serving large-scale traffic, with improved TTFT and inter-token latency versus its prior stack.

Note what is not on this list: a single latency-reduction percentage you can quote across all three. Each team optimized different things on different hardware, and the honest headline is qualitative: disaggregation reliably smooths the latency tail and, in full serving stacks, raises throughput. Be skeptical of any writeup that attaches a precise "40% latency cut" to these deployments; the 40-to-50% figure in Meta's report is a prefix cache hit rate, not a latency reduction.

Prefill-decode disaggregation pays off when you have consistently long prompts, high sustained concurrency, and fast KV-transfer networking, all three at once. Miss any one of them and the split is likely to cost more than it saves.

The logic follows from the mechanics. Long prompts make prefill expensive enough that isolating it protects a lot of decode steps from stalling. High concurrency means there are always decodes in flight to protect, so the isolation is constantly earning its keep. Fast networking keeps the KV-cache transfer from eating the latency you freed up. A reasonable rule of thumb is that prompts in the 8,000-plus token range, under real concurrent load, are where the tradeoff starts to favor splitting, though this is guidance rather than a benchmark: the true crossover depends on your exact hardware, network, and traffic shape.

There is a refinement worth watching for multi-turn workloads. An arXiv preprint on partial-prefill disaggregation for multi-turn serving (arXiv 2603.13358, 2026) reported that routing selected prefills to run locally on decode nodes cut time to first token by roughly 68% on subsequent conversation turns while keeping competitive generation speed. The lesson is that "prefill always goes to the prefill pool" is not the final word: for conversational traffic with heavy cache reuse, keeping some prefill next to the decode state can beat shipping the KV cache back and forth.

Do not disaggregate when your prompts are short, your concurrency is low, or your network is commodity Ethernet, and do not expect it to raise throughput on its own. The vLLM documentation states this last point directly: disaggregated prefilling does not improve throughput, its purpose is latency control, separating time-to-first-token tuning from inter-token-latency tuning.

That caveat is the one most teams miss. The most common mistake we see across the serving setups we review is reaching for disaggregation to raise throughput, then discovering the split alone does nothing for it. The throughput gains reported in the wild, DeepSeek's economics and SGLang's cost per token, come from full disaggregated-serving stacks that pair the split with expert parallelism, large batches, and prefix caching. They do not come from flipping on vanilla vLLM disaggregated prefilling. If your goal is higher tokens per second on a single replica and your tail latency is already fine, disaggregation is the wrong tool and you should look at batching and quantization instead.

For a large share of teams, the honest answer is: stay co-located. If you are serving a few hundred requests per second with short-to-medium prompts on a single well-tuned server, continuous batching plus a good engine already gives you smooth latency, and cross-node KV transfer would only add moving parts. Disaggregation earns its complexity at the scale where you are already running multiple GPU pools and your P99, not your average, is the thing breaking.

Split prefill and decode when tail latency, not average throughput, is your problem, and when you already operate at multi-pool scale with fast networking. Otherwise stay co-located and fix latency with batching, quantization, and prompt hygiene first.

Concretely: if your P99 inter-token latency spikes whenever long prompts hit a busy GPU, and you have RDMA-class networking and enough traffic to keep both pools full, disaggregation is a proven, production-grade fix that DeepSeek, SGLang, and Meta have all demonstrated. Start with the framework you already run: vLLM's experimental disaggregated prefilling is the lightest entry point for latency-only tuning inside an existing deployment (pin your versions, it is still experimental), while SGLang and NVIDIA Dynamo are the mature choices when you want the throughput gains of a full disaggregated stack. Do not switch engines just to get this feature.

If your prompts are short, your load is modest, or your network is ordinary Ethernet, do not split. You would be paying KV-transfer overhead and operational complexity for a benefit you cannot realize, and vLLM's own docs will tell you the throughput gain is not there.

This is exactly the tradeoff we size in Particula Tech's inference-serving audits: measuring whether your latency pain is a scheduling-contention problem that disaggregation solves or a throughput problem it will not, before you commit to the networking and the second GPU pool. Get that call right and prefill-decode disaggregation turns a lumpy tail into a smooth one at the scale where it counts. Get it wrong and you have built a distributed system to solve a problem a single well-tuned server already handled.