vLLM rope_scaling YaRN: What 128k Context Really Costs

Launch Qwen3-8B and ask for the window the model card advertises, and the engine refuses before it loads anything:

vllm serve Qwen/Qwen3-8B --max-model-len 131072

# ValueError: User-specified max_model_len (131072) is greater than the derived
# max_model_len (max_position_embeddings=40960 or model_max_length=None in
# model's config.json).

That refusal is useful. It names the config key it read and the number it found, so the derived ceiling for any checkpoint is one failed launch away.

The fix printed on the vendor pages is the removed one. Pull request #28006, "Remove deprecated --rope-scaling and --rope-theta", was created on 3 November 2025 and merged the next day, changing two files. Its body is blunt about why: the flags had been deprecated for a year, and clearing them was a prerequisite for supporting Transformers v5 style RoPE configuration per layer type. At tag v0.11.0, released 2 October 2025, vllm/engine/arg_utils.py registers both arguments at lines 541 to 543. At v0.11.1, released 18 November 2025, the string rope-scaling does not appear in that file at all. At v0.28.0, the string rope does not appear in it in any case.

So this block, which is what both Qwen sources currently show, no longer parses:

# Still printed by the vendor docs. Removed from vLLM by a merge on 2025-11-04.
vllm serve Qwen/Qwen3-8B \
  --rope-scaling '{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}' \
  --max-model-len 131072

vLLM's shipped documentation for this feature, added in April 2026, states the correction in its first line: the --rope-scaling parameter used in older versions of vLLM is no longer supported, and the --hf-overrides method with rope_parameters should be used instead.

The middle row is the trap for anyone pinned mid-migration: a command copied from either era fails there, one because the flag is gone and one because the key has not been renamed yet.

The corrected form on v0.28.0, with the --hf-overrides flag doing the work that how vLLM resolves a model config and where --hf-overrides is applied covers in detail:

vllm serve Qwen/Qwen3-8B \
  --hf-overrides '{"rope_parameters": {"rope_type": "yarn", "factor": 4.0, "original_max_position_embeddings": 32768, "rope_theta": 1000000}}' \
  --max-model-len 131072

The 131,072 is not a number you pick. Read the derivation in vllm/config/model.py: for rope_type yarn the engine assigns derived_max_model_len = rp["original_max_position_embeddings"], throwing away whatever the checkpoint declared, and after the loop multiplies by factor. The ceiling is literally factor times original_max_position_embeddings, which is 32,768 x 4.0 for the values above. One family is exempt: the same function skips the multiplication for Gemma3, with a source comment noting that its 128K window is already scaled by RoPE scaling.

Without an override, the derived value comes from derive_max_model_len_and_key in vllm/transformers_utils/model_arch_config_convertor.py, which checks nine keys (max_position_embeddings, n_positions, max_seq_len, seq_length, model_max_length, max_target_positions, max_sequence_length, max_seq_length, seq_len) and takes the smallest present. Qwen3-8B's config.json declares max_position_embeddings 40960, rope_scaling null and rope_theta 1000000, so the answer is 40,960. That is the number in the error above, and it is neither the 32,768 the model card calls native nor the 131,072 it advertises with YaRN.

Four fields go inside the dict; the fifth row is the flag that stays outside it. Keep --max-model-len and factor in step: change one and the other has to move, or you have either bought a window you cannot request or asked for one the engine will refuse. And raising the window multiplies the KV budget every concurrent sequence consumes, which is the other half of the sizing conversation and is covered in the KV cache arithmetic that decides your concurrency ceiling.

For batch jobs the same dict is the interface, passed to the constructor rather than the CLI. vLLM ships this example against Qwen3-0.6B; retargeted to Qwen3-8B, whose config.json declares the same rope_theta of 1,000,000, the four fields are unchanged:

from vllm import LLM

hf_overrides = {
    "rope_parameters": {
        "rope_theta": 1000000,
        "rope_type": "yarn",
        "factor": 4.0,
        "original_max_position_embeddings": 32768,
    },
}

llm = LLM(model="Qwen/Qwen3-8B", hf_overrides=hf_overrides, max_model_len=131072)

Set the serving limit through the constructor's own max_model_len argument. The shipped example also puts a max_model_len key inside the overrides dict; the nine-key list vLLM derives the ceiling from does not contain it, so leave it out.

RotaryEmbeddingBase.__init__ computes the cache and registers it as a buffer: if init_cache: cache = self._compute_cos_sin_cache(), then self.register_buffer("cos_sin_cache", cache, persistent=False). There is no per-request recomputation path anywhere in that class.

For the yarn subclass, _compute_cos_sin_cache builds positions with t = torch.arange(self.max_position_embeddings * self.scaling_factor, ...) and finishes with cos = freqs.cos() * self.mscale and sin = freqs.sin() * self.mscale. Every entry in the table, position 0 included, is multiplied by the same constant. That constant is set in the constructor as yarn_get_mscale(self.scaling_factor) * attn_factor, and yarn_get_mscale is two branches: return 1.0 when the scale is at most 1, otherwise 0.1 * math.log(scale) + 1.0.

Factor 4.0 therefore scales every cosine and sine entry about 13.9 percent above the unscaled model, on the 900-token prompt as much as on the 120,000-token one. The Qwen deployment page describes the consequence in plain language: vLLM implements static YaRN, the scaling factor remains constant regardless of input length, potentially impacting performance on shorter texts, and the configuration should be added only when processing long contexts is required. The Qwen3-8B model card generalises the same warning to every notable open source framework. The vendor is blunter still about a sibling model: the Qwen2.5-VL-32B-Instruct card, after giving the same yarn block, says the method has a significant impact on the performance of temporal and spatial localization tasks and is therefore not recommended there.

What does not change with factor is which frequencies get touched. The split comes from yarn_find_correction_range, whose inputs are beta_fast, beta_slow, the rotary dimension, the RoPE base and original_max_position_embeddings. Factor is not among them. Running it with Qwen3-8B's numbers (rotary dimension 128, base 1,000,000, original window 32,768, beta_fast 32, beta_slow 1) gives a low bound of 23 and a high bound of 40 across the 64 frequency pairs:

A larger factor compresses the interpolated band harder. It does not move where that band begins. The method itself is described in arXiv:2309.00071, which vLLM's source credits directly.

The vendor gives the sizing rule, and it points away from the model card headline. The Qwen deployment page: if the typical context length for your application is 65,536 tokens, it would be better to set factor as 2.0. And below 32,768, do not enable it at all, since the default 40,960-token allocation already reserves 32,768 tokens for outputs and 8,192 for typical prompts.

So the correct input is your own length distribution, not the maximum document anyone has ever pasted. Take the P99 of prompt plus expected output from your gateway logs, divide by 32,768, round up to the next factor you are willing to pay for. A corpus whose P99 sits near 60,000 tokens wants factor 2.0, not 4.0:

vllm serve Qwen/Qwen3-8B \
  --hf-overrides '{"rope_parameters": {"rope_type": "yarn", "factor": 2.0, "original_max_position_embeddings": 32768, "rope_theta": 1000000}}' \
  --max-model-len 65536

The second half of the rule is harder, because factor is a property of the engine and not of the request. get_rope builds one rotary embedding per unique configuration key and caches it in a module-level _ROPE_DICT, keyed on head size, rotary dimension, max position, Neox style, the rope_parameters tuple, the dual chunk attention arguments and dtype. Those parameters come from the engine's ModelConfig. One served model in one process means one YaRN configuration for everything that process accepts. Switching rope_type to dynamic changes nothing: DynamicNTKScalingRotaryEmbedding also computes its base from a fixed scaling factor inside _compute_cos_sin_cache, which the base class calls once.

Nor is there a request-level escape hatch waiting in a release. Issue #8793, "Is Qwen2.5's long context YARN handled?", ran for 21 comments from September 2024 before being closed as not planned on 1 September 2025, and a proposal to choose a factor at admission time, opened 26 August 2026, is still unmerged.

That leaves one honest option, which is two engine processes:

# short traffic: no override at all, native geometry
vllm serve Qwen/Qwen3-8B \
  --served-model-name qwen3-8b \
  --max-model-len 32768 \
  --port 8000

# long traffic: extended, and only requests that need it are routed here
vllm serve Qwen/Qwen3-8B \
  --served-model-name qwen3-8b-long \
  --hf-overrides '{"rope_parameters": {"rope_type": "yarn", "factor": 4.0, "original_max_position_embeddings": 32768, "rope_theta": 1000000}}' \
  --max-model-len 131072 \
  --port 8001

Two distinct --served-model-name values are what let a router address them, and the router's rule is a token count against a threshold below 32,768. Be clear about the price: that is two engine processes and therefore two resident copies of the weights. It is a real hardware cost, not a configuration trick, and it is the reason why most requests should never approach the window at all is worth settling before you buy the second copy.

Run the extended endpoint with a native MTP draft and something strange happens: long requests still return sensible output, but draft acceptance falls off a cliff. Issue #37435, opened 18 March 2026 and still open with seven comments, reports it as MTP working very well at shorter or ordinary contexts while the draft acceptance rate collapses to roughly zero on very large prompts, with the target model still appearing usable. The reporter's logs show average draft acceptance between 80.0 and 100.0 percent on ordinary requests, and 0.0 percent, zero accepted of 890 drafted, on large ones, from the same running server.

The mechanism is documented in vLLM's own source rather than inferred:

# vllm/config/speculative.py
#   "Dict overrides are target-specific key patches and are not applied
#    to the draft."

if not callable(target_hf_overrides):
    return SpeculativeConfig.hf_config_override

The docstring of compose_draft_hf_overrides explains the intent: callable overrides on the target are config-to-config transforms and must also reach the draft, while dict overrides are target-specific patches and are not. The catch is that --hf-overrides on a command line can only ever produce a dict. vllm/config/model.py splits the CLI value by type, sending dict-valued entries down the dict path, and rope_parameters is a dict. The callable route exists only in Python, so from vllm serve the propagating path is unreachable.

For native MTP the draft is built through that same branch: SpeculativeConfig.__post_init__ sets the draft model to the target model, so construction falls through to a ModelConfig built with the composed (empty) draft overrides. The target keeps its extension. The draft is the component running base positional parameters, which is why nothing errors and the only symptom is speculation stopping paying for itself. Reading that symptom correctly, and deciding whether speculation was worth it at your batch size, is the subject of reading and tuning speculative decoding acceptance.

There is no upstream fix to wait for. The bugfix pull request opened alongside the issue on 18 March 2026 is still open and unmerged. A second reporter hit the same non-propagation in July 2026 on a different key, where it surfaced as an AssertionError about loading a weight of shape 168 into a parameter of shape 256 inside the drafter's load_model, with nothing in the message pointing at hf_overrides; their working fix was to correct the checkpoint's config.json so no override was needed at all. Until that changes, treat long-context serving and speculative decoding as configurations you validate together, or run the extended endpoint without speculation.

The rename is present tense, not a hazard on the horizon. Pull request #28542, "Update rope_scaling to rope_parameters in preparation for Transformers v5", merged on 19 November 2025 across 104 files, and the new key first appears in a tagged release at v0.12.0 on 3 December 2025: vllm/config/model.py has zero occurrences of rope_parameters at v0.11.2 and nine at v0.12.0. Anything you write today targets rope_parameters.

Three more things worth pinning down before an upgrade.

vLLM and Transformers accept different key sets. get_rope's yarn branch forwards a whitelist of six extra keys to the rotary embedding: extrapolation_factor, attn_factor, beta_fast, beta_slow, apply_yarn_scaling and truncate. Transformers accepts a different vocabulary. Its _validate_yarn_rope_parameters declares rope_type, factor and original_max_position_embeddings as required, and rope_theta, attention_factor, beta_fast, beta_slow, mscale, mscale_all_dim and truncate as optional, and _compute_yarn_parameters reads attention_factor, mscale and mscale_all_dim. Those three are not in vLLM's list, so a config.json carrying attention_factor is honoured by Transformers and dropped by vLLM.

That gap is reported and unresolved. Three pull requests against vLLM's YaRN handling are open or closed unmerged, covering two distinct findings. One, that generic YaRN ignores mscale and mscale_all_dim, was opened 14 July 2026 and closed unmerged on 27 August 2026. The other, that an explicit attention_factor in the rope config is not honoured, was opened 21 June 2026, closed unmerged on 11 July, and resubmitted on the same day under an identical title, where it is still open. None of them have landed, so do not write configs that depend on those keys reaching vLLM.

Only one of the four fields fails loudly when you omit it. Dict overrides are applied before RoPE normalisation: vllm/transformers_utils/config.py runs config.update(hf_overrides_kw), then any callable override, then patch_rope_parameters, which calls the config's own standardize_rope_params and validate_rope. That standardisation backfills what you left out. It sets rope_theta from the top-level config attribute, and for yarn it does setdefault("original_max_position_embeddings", self.max_position_embeddings). On Qwen3-8B, omitting original_max_position_embeddings therefore does not fail. It silently adopts 40,960, which gives you a ceiling of 40,960 x factor and an interpolation geometry built around the wrong base. Only factor has no default anywhere in that path: leave it out and Transformers raises a KeyError naming it. Write all four fields every time.

The Qwen3-8B model card's first recommended path is to modify the model files, adding the rope scaling fields directly into the checkpoint's config.json, and it notes that a GGUF build has to be regenerated afterwards. In a regulated perimeter that is the wrong branch, and it is a fork that does not exist on a hosted endpoint.

Inside an air gap the checkpoint is normally a hash-pinned artifact in a content-addressed mirror, and editing one file inside it changes the digest. The mirror entry, the attestation and whatever approval record points at that digest all stop matching, and a context-window tweak becomes a re-approval of the model. Passing the same four values through --hf-overrides leaves the artifact bit-identical and moves the change into the deployment manifest, where it is diffable, reviewable and revertible by the people who already review the rest of the serving config. That argument holds anywhere and is strongest here, next to the config and tokenizer resolution steps in serving vLLM with no path to Hugging Face.

Two consequences follow. The extended endpoint is a different configuration of the model, not the same model with a larger number after it: every request it serves sees positional geometry scaled by 1.1386 that the approved configuration did not have. If the validation evidence that cleared the model was produced without the override, the extended endpoint needs its own record.

And the two-endpoint routing rule costs hardware you may not have. On a cloud you add a replica. On a fixed allocation of four cards you are choosing between two resident copies of the weights, time-slicing one card set between two configurations, or accepting the static scaling on all traffic, and that choice belongs in the sizing conversation before anyone commits to 128k in a requirements document. The workloads that make the extension genuinely worth it are the ones that made the question urgent in the first place: whole contracts, full claim files, complete case histories that cannot be chunked without destroying the cross-references the reviewer is being asked to find.

The long-context check is the one the request came with: feed a 120,000-token document, confirm the answer comes back. It will pass, and it says nothing about the requests that never needed the extension.

Build the cheaper check alongside it. Take a few hundred of your own sub-8k prompts captured from real traffic, replay them through both the unscaled endpoint on port 8000 and the extended one on port 8001, and diff the outputs with whatever grader already gates your releases. You are not looking for a benchmark number. You are looking for a class of prompt that changed answer, before the router sends production traffic at the extended engine rather than after.

Run the load half separately with vllm bench serve, vLLM's own online serving throughput subcommand, so latency and throughput are measured against a real request mix rather than a synthetic long-prompt one. And keep the quality expectations for the long endpoint honest while you are there, because a longer window is not a promise of accuracy across it: what happens to retrieval accuracy across a long window is a separate problem from whether the positions decode correctly.

This week, do three things. Launch the model with no override at all and read the derived ceiling off the ValueError, so you know what your checkpoint actually declares. Pull the P99 of prompt plus output length from your gateway logs and divide by 32,768, which gives you the factor you need rather than the one on the model card. Then, before you enable anything, check whether a speculative draft config is in the launch command, because that override is not going to reach it. More on serving decisions like this one in our LLMs and models pillar.