Speculative Decoding Slower Than Baseline: How to Fix It

vLLM issue #19254 is the canonical false positive. The reporter ran n-gram speculation against a Qwen3-32B-FP8 target on an 8x H20 node, measured higher inter-token latency, and filed a regression. Their own follow-up settled it: "only the ITL time is longer, but in fact both E2E and output throughput are better than before."

One verification step commits several tokens, so the gap between streamed chunks widens while the total time to produce them falls. Working speculation looks like broken speculation on a streaming dashboard. We keep the difference between TTFT, TPOT and inter-token latency in a separate post.

Adjudicate on output token throughput, both runs on the same binary, prompts and concurrency.

# Run twice: once with --speculative-config on the server, once without.
vllm bench serve \
  --model <target-model> \
  --dataset-name hf --dataset-path philschmid/mt-bench \
  --num-prompts 200 \
  --max-concurrency 32 \
  --ignore-eos \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,95

--ignore-eos earns its place: generation length drifts between runs, so the published evaluation compares token throughput rather than latency, and vLLM qualifies its losslessness guarantee to "the precision limits of hardware numerics", the same family of effect as temperature 0 nondeterminism.

No instrumentation is required. vLLM logs a line per interval with no extra flag:

SpecDecoding metrics: Mean acceptance length: %.2f, Accepted throughput: %.2f tokens/s,
Drafted throughput: %.2f tokens/s, Accepted: %d tokens, Drafted: %d tokens,
Per-position acceptance rate: %s, Avg Draft acceptance rate: %.1f%%

Mean acceptance length is 1 + (num_accepted_tokens / num_drafts), so 1.0 means every draft was rejected and you are paying the drafter for nothing. Read the per-position vector when the aggregate looks fine but the speedup does not: one that collapses after position 1 says your draft depth is too deep, which is configuration, not a bad drafter.

The same counters export to Prometheus as vllm:spec_decode_num_drafts, vllm:spec_decode_num_draft_tokens, vllm:spec_decode_num_accepted_tokens and the vector vllm:spec_decode_num_accepted_tokens_per_pos.

# draft acceptance rate
rate(vllm:spec_decode_num_accepted_tokens_total[5m])
  / rate(vllm:spec_decode_num_draft_tokens_total[5m])

# per-position acceptance rate vector
vllm:spec_decode_num_accepted_tokens_per_pos[5m]
  / vllm:spec_decode_num_drafts[5m]

llama.cpp prints the same idea per slot at info level as draft acceptance = %0.5f (%5d accepted / %5d generated), mean len = %5.2f, with a per-position vector at trace level, and echoes - n_max=%d, n_min=%d, p_min=%.2f at init. Read that init line first: it reports the window in force rather than the one you think you set. Response timings carry draft_n and draft_n_accepted.

Acceptance is workload-shaped, not a model constant, as self-hosting Kimi K3 with vLLM measured: one pair, very different accepted tokens per step on coding versus creative writing. Compare against your own workload, not that table. Per-request acceptance metrics are still landing in vLLM and are not in v0.27.1.

The 2022 paper that introduced speculative decoding gives the walltime improvement in closed form.

improvement = (1 - a^(g+1)) / ((1 - a) * (g*c + 1))

a = expected acceptance rate
g = draft tokens proposed per step
c = time for one draft forward pass / time for one target forward pass

Its corollary is the operational part: an improvement exists only when a > c, and where it does, the floor at g = 1 is (1 + a) / (1 + c). No acceptance-rate threshold appears in that paper, or in any of the primary sources behind this post.

c is the term people skip, and it is not a property of the model pair. The paper says so directly: unlike acceptance, c depends on hardware and implementation. Its own experiments had c below 0.05 because the drafter was two orders of magnitude smaller, which is why almost anything worked. On current hardware a 1B draft against a 70B target at TP4 runs about 12.5 percent, a 0.6B draft against an 8B target about 37.5 percent. Both drafters are around a billion parameters or smaller, and their break-even points are nowhere near each other, because the card decides where the compute-bound crossover sits.

llama.cpp issue #25908 is the empirical demolition of threshold advice. That Qwen3-4B and Qwen3-0.6B pair, both fully offloaded, hit 0.898 draft acceptance and still committed 164.3 tok/s against 181.3 tok/s plain. Nearly nine drafts in ten accepted, and still a loss.

Measuring your own c takes one run: time the draft alone at the batch shape your target sees during decode, and divide its per-step time by the target's.

Total arithmetic operations go up; only memory accesses go down, because the target's weights and KV cache are read once per run instead of once per committed token. Speculative decoding buys latency with FLOPs, and only while you have spare FLOPs.

The most common cause in production, and the easiest to miss: teams tune at batch size one and deploy behind continuous batching. vLLM states the mechanism: speculation verifies K tokens per sequence, so the effective batch becomes BS times K, and past a critical batch size that verification compute hurts time per output token.

All three rows are EAGLE. The 70B falls more than three times faster from batch 1 to 32 because a 70B at TP4 is already compute-bound at small batch, leaving less idle compute to borrow. The execution-time breakdown agrees: verification takes 42 to 95 percent of decode runtime across every method measured and grows with model and batch size, while sampling stays under 1.7 percent and framework overhead runs 3 to 12 percent.

Prefill is untouched, which is why the two halves of serving want different hardware profiles. Rule out the engine before blaming the drafter: how the two engines differ covers scheduling and attention choices that move throughput on their own. On reasoning workloads, long generations fill the KV cache and trigger preemption, so check preemption counters.

If cause 1 is your problem, the fix is not to disable speculation but to make K a function of batch size. vLLM exposes num_speculative_tokens_per_batch_size, a list of [start_bs, end_bs, K] triples inside --speculative-config, where K = 0 produces no draft tokens in that range.

VLLM_USE_V2_MODEL_RUNNER=0 vllm serve <target-model> \
  --speculative-config '{
    "method": "eagle3",
    "model": "<eagle3-draft-head>",
    "num_speculative_tokens": 5,
    "num_speculative_tokens_per_batch_size": [
      [1, 16, 5],
      [17, 32, 4],
      [33, 64, 3],
      [65, 128, 1],
      [129, 512, 0]
    ]
  }'

Set num_speculative_tokens to the largest K in your schedule. vLLM stores min(num_speculative_tokens, K) in every batch-size slot, so the docs' own example, which pairs this schedule with num_speculative_tokens: 3, never drafts the 5 or the 4 it appears to ask for.

Three limitations bite. It is tested with Eagle, Eagle-3 and DFlash only. Full CUDA graph requires Model Runner V2, while V1 gets piece-wise CUDA graph. And with --data-parallel-size > 1 vLLM drops the schedule and reverts to the static num_speculative_tokens, logging a warning rather than refusing to start.

SGLang's equivalent is --speculative-adaptive, which keeps independent EMA trackers per batch-size range and switches tiers only between rounds, because CUDA graphs are shape-dependent. The tuning knobs live in a JSON file passed with --speculative-adaptive-config, which is optional and does nothing on its own. Adaptive mode supports EAGLE and EAGLE3 with --speculative-eagle-topk 1 only, and falls back to static settings otherwise. The docs put it plainly: at high batch sizes the cost of each wasted draft step is multiplied across all sequences in the batch.

llama.cpp has no batch schedule, only a probability gate. --spec-draft-p-min defaults to 0.00, so the drafter never exits early and always fills the whole --spec-draft-n-max window. That default is how issue #25908 got 0.070 acceptance at --spec-draft-n-max 64 and 32.6 tok/s; raising the gate to 0.75 recovered both.

llama-server \
  -m <target>.gguf \
  --spec-draft-model <draft>.gguf \
  --spec-type draft-simple \
  --spec-draft-n-max 8 \
  --spec-draft-p-min 0.75 \
  -ngl 99 --spec-draft-ngl 99 \
  -c 8192 -fa on --parallel 1

The headroom is real: on one code-editing workload at batch size 1, an oracle proposing exactly the number of tokens that will be accepted reaches roughly 2.75x against roughly 2.1x for the best fixed length, and the gap widens with batch size.

Check what you copied from older tutorials against what the engines accept today.

When acceptance looks healthy and throughput is still down, the drafter is eating the win. Drafting share of decode runtime spans an order of magnitude: under 2 percent for n-gram, 12 to 20 percent for EAGLE and EAGLE-3 at batch size 1 falling to 3 to 7 percent at batch 512, and, also at batch size 1, 21 percent for a 1B draft against a 70B target and 47 percent for a 0.6B draft against an 8B target.

That last number explains a result that reads backwards. On GSM8K the 70B pair accepts 75 percent of drafts and the 8B pair accepts 81 percent, and the 8B pair gains far less speedup, entirely because c is three times larger. Any tuning loop that optimises acceptance without measuring draft cost walks into this.

The other version is a draft running eager while the target runs graphed. In vLLM issue #48766, a Tencent Hy3 target at TP4 across four RTX PRO 6000 Blackwell cards ran 119 tok/s plain and 94 tok/s with MTP depth 1 at 56.2 percent position-0 acceptance. Acceptance stayed in band, so the loss was pure draft-cycle cost. The asymmetry is checkable: vllm/model_executor/models/deepseek_mtp.py applies @support_torch_compile and vllm/model_executor/models/hy_v3_mtp.py does not. SGLang's equivalent knob is --enable-torch-compile.

One pairing choice costs you before you start: cross-vocabulary drafting in vLLM (use_heterogeneous_vocab) works only with method='draft_model' and constrains draft logits to the shared token intersection, costing acceptance by construction.

Turning speculation on changes the scheduler, the memory budget and, in some engines, the batching strategy.

In SGLang, NGRAM and DFLASH both disable the overlap scheduler and mixed chunked prefill, and NGRAM is CUDA-only and rejects --enable-dp-attention. The subtler trap is topk: the overlap scheduler supports --speculative-eagle-topk 1 only. Setting it higher explicitly errors out; omitting it lets auto-tuning pick a higher value for some models, incompatible with the overlap scheduler and without always erroring. Set all three tree parameters explicitly or leave them unset.

Structured output quietly cuts your effective draft count: vLLM defines per-request draft tokens as proposals minus drafts invalidated by structured-output constraints, so traffic running through a JSON schema pays the draft cost and discards part of the yield. If that is your workload, the constrained decoding setup is the lever to pull first. Pipeline parallelism is a harder wall, documented as not composable with speculative decoding as of vllm<=0.15.0.

Memory is the cause that disguises itself best. Draft weights, the draft KV pool and wider CUDA graph buffers come out of your KV cache budget: static overhead alone measured 1.8 percent for a 70B target with a 1B draft and 7.3 percent for an 8B target with a 0.6B draft, before graph buffers. Lose enough KV cache and maximum concurrency drops, requests queue earlier, and the queueing appears as a decode slowdown. Confirm the ceiling moved with the flags that fix a vLLM memory ceiling. In llama.cpp, --spec-draft-type-k and --spec-draft-type-v quantize the draft's KV separately, cheap headroom once you know what quantizing the KV cache costs.

SGLang's OOM ladder runs in a fixed order: lower --mem-fraction-static first (0.5 in the documented example), then --cuda-graph-max-bs-decode to 4 or 2, then shrink the draft tree to --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4, then cap --max-running-requests at 4.

The mechanism is identical behind a firewall. The operating point is not, and that flips the answer for most private deployments.

vLLM's own framing is that speculative decoding targets medium-to-low QPS, memory-bound workloads. A private cluster serving a few hundred internal staff on fixed GPUs is exactly that: tens of concurrent requests, not hundreds. The decay curve agrees: EAGLE still returns 1.61x at batch 32 on an 8B model, and only falls to 1.21x once you push concurrency to 128. So the on-prem advice is not "be careful, this might hurt" but measure at your real P95 concurrency, probably 20 to 60, not at the batch 512 vendor charts use or the batch 1 blog posts use.

Two operational consequences do change on your own hardware.

The draft is a second model artifact. An EAGLE head, an MTP checkpoint or a small draft model clears the same approval, provenance, scanning and internal-mirror path as the target weights. In an air-gapped estate it cannot be pulled from a hub at server start, so a serving config naming a draft repo id fails to boot on the isolated side of the wall (an air-gapped vLLM deployment covers blocking those calls). That is a procurement question as much as an engineering one.

On fixed hardware the draft's memory comes straight out of KV cache. On elastic capacity you add a node; on a fixed rack you lose concurrency headroom. Re-measure maximum concurrent requests with speculation on before you re-measure tokens per second, and if the ceiling drops far enough to push you into queueing at peak, spend that VRAM on KV cache instead and leave speculation off.

This week, do one measurement and one read. Run vllm bench serve twice at your P95 concurrency, with and without --speculative-config and --ignore-eos set both times, and compare output token throughput. Then time your draft alone at the same batch shape, divide by the target's per-step time, and compare that ratio to the acceptance rate in your server log. If acceptance is below it, the feature costs you throughput whatever the ITL panel says. The rest of the serving stack sits in our large language models pillar.