Qwen3-Reranker is a Qwen3ForCausalLM checkpoint that judges relevance by comparing the logits of the tokens yes and no after a fixed prompt, so vLLM needs two things to score it correctly: a yes/no score head and the qwen3_reranker.jinja template. On vLLM 0.34.0, uploaded to PyPI on 2026-11-16, four launch configurations start cleanly and return HTTP 200 with scores that do not rank. Plain --runner pooling resolves to --convert embed and scores cosine similarity, never touching the yes/no head. The documented --hf-overrides without --chat-template concatenates query and document raw; in one public report on vLLM 0.28.0 that let an off-topic sentence score 0.958 against 0.804 for the correct answer, and 0.000 against 0.975 once the template was added. A converted seq-cls checkpoint without the template fails the same way with no warning at all, and --convert classify without overrides, which raised a missing score.weight error on vLLM 0.14.1, now starts with a score head never loaded from the checkpoint. Long pairs split across prefill chunks under torch.compile scored near 0 before vLLM 0.26.0. Start by grepping your startup log for the convert-embed INFO line and the no-template WARNING, then send a three-document probe to /v1/rerank.
Your Qwen3 reranker in vLLM starts without an error, /v1/rerank returns HTTP 200, the scores sit neatly between 0 and 1, and the unrelated chunk ranks first. Nothing in the logs looks like a failure. The retrieval quality is simply worse than with no reranker at all, and every health check you have says the service is fine.
A report filed on the vLLM issue tracker in September 2026 measured what this looks like. On vLLM 0.28.0 with Qwen3-Reranker-8B and a Turkish query asking for the capital of Türkiye, the correct answer about Ankara scored 0.804 without the reranker template and 0.975 with it. An unrelated sentence about apples and pears ripening in autumn scored 0.958 without the template and 0.000 with it. Without the template, the unrelated sentence ranked first.
The cause is structural. Qwen3-Reranker needs two things vLLM will not supply on its own: a score head that reads the model's yes/no judgement, and the exact prompt the model was trained on. All five failure modes below return HTTP 200, and four of them still do on the current release. Everything in this post is checked against the source at the vLLM v0.34.0 tag, the latest final release on PyPI as of 2026-11-18 (uploaded 2026-11-16).
Why Qwen3-Reranker is not a normal cross-encoder
A classic cross-encoder such as a BGE reranker ships with a classification head: feed it a query and a document separated by a special token and it emits one relevance logit. Qwen3-Reranker does not. Its config.json still declares Qwen3ForCausalLM, a plain language model with a vocabulary of 151,669 tokens, and relevance is a question it answers in text.
The model card spells out the prompt. A system turn says: "Judge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be "yes" or "no"." The user turn is <Instruct>: {instruction}, <Query>: {query} and <Document>: {doc} on separate lines. The assistant turn opens with an empty <think> block. The score is read at the last position: take the logits for the tokens yes and no, apply a softmax over the pair, and the yes probability is the relevance score. The default instruction is "Given a web search query, retrieve relevant passages that answer the query", and the card reports that task-specific instructions typically add 1% to 5% on its downstream evaluations.
vLLM turns this into something its pooling runner can serve. With the right overrides it loads the checkpoint as Qwen3ForSequenceClassification, a one-label classifier whose weight is the yes row of the language-model head minus the no row, computed in float32. The output is logit(yes) minus logit(no), and the default sigmoid activation turns that into P(yes), which is mathematically the same number as the two-token softmax. vLLM's example file notes that this avoids computing logits for all 151,669 vocabulary tokens on every pair.
Two consequences follow. The number only means anything if the prompt matches training, because the model is answering a question it has to be asked in its own format. And the head must be the yes/no difference, because any other pooled vector carries no relevance judgement at all.
The official Hugging Face repos for the 0.6B, 4B and 8B models were updated on 2026-04-16 and now include a chat_template.jinja for Sentence Transformers. vLLM 0.34.0 does not use it: it only auto-applies a checkpoint's Sentence Transformers template when modules.json lists exactly one module and the configured task is sequence classification, and the Qwen repos list two modules with the task text-generation. The template in the repo is invisible to vLLM. Which reranker model to run in the first place is a separate question, covered in our comparison of reranker models by latency and nDCG.
Symptom table: which mistake produces which scores
Five configurations go wrong, and each returns HTTP 200 with scores that do not reflect relevance. The fifth row used to crash at load; at v0.34.0 it starts.
Resolved `--convert auto` to `--convert embed`. Pass the value explicitly to silence this message.
Serving an original Qwen3 reranker (Qwen/Qwen3-Reranker-0.6B) without a --chat-template. The model was trained with a specific prompt template; running without it may produce inaccurate relevance scores. Consider specifying `--chat-template examples/pooling/score/template/qwen3_reranker.jinja`.
ValueError: Following weights were not initialized from checkpoint: {'score.weight'}Row one: the embedder path
vllm serve Qwen/Qwen3-Reranker-0.6B --runner pooling looks reasonable, since a reranker is a pooling model. But the checkpoint's architecture is Qwen3ForCausalLM, which matches no pooling suffix, so vLLM's default convert logic picks embed and logs: From then on /v1/score and /v1/rerank run the bi-encoder path: query and document are pooled separately and compared with cosine similarity. The yes/no head is never read, and vLLM's scoring docs state that it does not apply a score template for embedding models. The endpoint works, the numbers look like scores, and they measure something the model was never trained to output. The runner and convert options are the embedder serving path, which our comparison of TEI, vLLM and Infinity for embeddings covers in depth.
Rows two and three: the missing template
With the overrides set but no template, vLLM's cross-encoder processor falls back to default tokenization. Because the overrides mark the model as an LLM-as-reranker with no separator token, the prompt is literally the query string followed directly by the document string. No system prompt, no <Instruct>, <Query> or <Document> tags, no <think> block. That is the configuration behind the September 2026 numbers in the intro, and an earlier July 2025 issue on the 4B model described the same configuration as "the output score is incorrect and the ranking is very poor, seemingly random." Since v0.30.0, released 2026-09-22, vLLM prints a WARNING in this case, added by pull request #56017: It warns; it does not load the template. And it fires only when is_original_qwen3_reranker is true, which is why row three exists: a checkpoint you converted to seq-cls format has no such flag, takes the same raw-concatenation path, and logs nothing. The same near-zero symptom wording appears in a llama.cpp issue about Qwen3-based rerankers, which closed as stale without a confirmed cause.
Row five: the crash that went quiet
An issue filed in May 2026 against vLLM 0.14.1 records what launching the original 8B checkpoint with --runner pooling --convert classify and no overrides printed at load: Without classifier_from_token or a method, vLLM's sequence-classification adapter uses the plain weight loader, and the checkpoint has no score.weight tensor. The same ValueError still exists in the v0.34.0 loader, but its missing-weight check now marks every parameter of a layer with a quantization method as loaded when the model is unquantized, and the score layer is such a layer. A unit test at the tag asserts exactly that case: an unloaded score weight on an unquantized model produces no error. So on v0.34.0 this launch is expected to start and serve scores from a head the checkpoint never filled. The row that used to stop you now returns 200 like the others.
| What you launched | What vLLM 0.34.0 does | What you see | Fix |
|---|---|---|---|
--runner pooling, no overrides | Resolves to --convert embed, scores cosine similarity of separately pooled query and document vectors | INFO line at startup, embedding-similarity scores unrelated to the yes/no judgement | Add the --hf-overrides JSON and --chat-template |
--hf-overrides, no --chat-template | Concatenates query and document strings with no system prompt, tags or <think> suffix | WARNING at startup (v0.30.0 and later), off-topic text can outrank the answer | Add --chat-template |
Converted seq-cls checkpoint, no --chat-template | Same raw concatenation | No warning at all, same bad ranking | Add --chat-template |
| Long pairs on vLLM older than 0.26.0 | Chunked prefill under torch.compile read the pooled state too early | Matching pairs collapse toward 0, varying run to run | Upgrade to 0.26.0 or later, keep a long-pair probe |
--convert classify, no overrides | Plain weight loader finds no score.weight in the checkpoint, and the missing-weight check exempts unquantized linear layers | Server starts with no error at v0.34.0; scores come from a head never loaded from the checkpoint | Use the overrides instead of --convert classify |
Serve the original checkpoint correctly
The full command for vLLM 0.34.0, taken from the online example in the repo at that tag:
vllm serve Qwen/Qwen3-Reranker-0.6B \
--runner pooling \
--hf-overrides '{"architectures": ["Qwen3ForSequenceClassification"], "classifier_from_token": ["no", "yes"], "is_original_qwen3_reranker": true}' \
--chat-template /opt/templates/qwen3_reranker.jinjaThe template path is a local copy you make of examples/pooling/score/template/qwen3_reranker.jinja from the v0.34.0 tag. Use the same flags for the 4B and 8B checkpoints by swapping the model id; vLLM's own MTEB test for this model uses identical overrides for 0.6B and 4B. For an air-gapped host, the model id can be a local path.
Each override key does one job, and the code at v0.34.0 shows what:
architectures routes loading to Qwen3ForSequenceClassification instead of the causal LM. With this set, vLLM would infer the pooling runner on its own, but keep --runner pooling explicit as the example does.classifier_from_token must be ["no", "yes"] in that order. The loader takes the first token as the false label and the second as the true label, and sets the score weight to the true row minus the false row. Reverse the list and every logit changes sign, so the best document gets the lowest score. The config check asserts the list has exactly two entries.is_original_qwen3_reranker set to true is what makes vLLM choose the from_2_way_softmax method for building that single-label head, and it is the flag the no-template warning keys on.The example file spells the flag --hf_overrides. The registered name is --hf-overrides, and vLLM's argument parser accepts both underscore and dash forms. How that JSON gets merged into the model config is the same mechanism covered in fixing Transformers does not recognize this architecture in vLLM.
vLLM's scoring docs at v0.34.0 list Qwen3ForSequenceClassification with the qwen3_reranker.jinja template in their model table, but the note that shows how to load the original checkpoint prints the vllm serve command with the overrides and without --chat-template. Copy that note and you land in row two.
What the template produces matters as much as that it exists. The v0.34.0 file reproduces the model card's system turn, then writes <Instruct>: followed by the request's instruction variable (falling back to the default web-search instruction), then the query and document on single newlines, and closes with the empty <think> block. That matches the card's Transformers reference. The card also carries an older vLLM generation snippet that uses double newlines between the tags; vLLM's template follows the Transformers version.
from pathlib import Path
from vllm import LLM
template = Path("/opt/templates/qwen3_reranker.jinja").read_text()
llm = LLM(
model="Qwen/Qwen3-Reranker-0.6B",
runner="pooling",
hf_overrides={
"architectures": ["Qwen3ForSequenceClassification"],
"classifier_from_token": ["no", "yes"],
"is_original_qwen3_reranker": True,
},
chat_template=template,
)
outs = llm.score("What is the capital of China?", [
"The capital of China is Beijing.",
"Gravity is a force that attracts two bodies towards each other.",
])
print([o.outputs.score for o in outs])Offline: put the template on the constructor
For batch scoring with the Python API, the shipped offline example passes chat_template to llm.score(). vLLM 0.34.0 also accepts it on the LLM constructor, and its docs say a template passed there applies to every score call, with a per-call template overriding it. Set it once at construction, so a forgotten argument on one call cannot fall back to raw concatenation:
Or convert once to a seq-cls checkpoint you own
The alternative to runtime overrides is to bake the yes/no head into the weights. vLLM ships a conversion script at examples/pooling/score/convert_model_to_seq_cls.py. It loads the causal LM with Transformers, writes a classification head whose weight is the yes row minus the no row, marks the config as not using a separator token, and saves model and tokenizer to a path you choose. It imports torch and transformers only, not vLLM.
python convert_model_to_seq_cls.py \ --model_name Qwen/Qwen3-Reranker-0.6B \ --classifier_from_tokens '["no", "yes"]' \ --method from_2_way_softmax \ --path ./Qwen3-Reranker-0.6B-seq-cls
The token order rule is the same as for the override: no first, yes second. The converted directory serves without overrides, and it still needs the template:
vllm serve ./Qwen3-Reranker-0.6B-seq-cls \ --runner pooling \ --chat-template /opt/templates/qwen3_reranker.jinja
This is row three of the table. Forget --chat-template here and vLLM gives you no warning, because the converted config does not carry is_original_qwen3_reranker. The head is right and the prompt is wrong, and nothing in the log says so.
Converted Qwen3 reranker checkpoints exist on the Hub under individual accounts, and vLLM's own examples point at one. Inside a regulated perimeter, run the conversion yourself against the official weights. It is one script on one machine, and it keeps third-party modified weights out of your supply chain.
The overrides route keeps you on the official artifact and gets the startup warning as a safety net. The converted route removes the JSON from your launch command but loses that warning. If you convert, the smoke test below is your only guard against a missing template.
| Original checkpoint plus overrides | Converted seq-cls checkpoint | |
|---|---|---|
| Weights | Official repo, unmodified | Your own derived artifact |
| Serve flags | --runner pooling, --hf-overrides, --chat-template | --runner pooling, --chat-template |
| Template still required | Yes | Yes |
| Warning if template is missing | Yes, since v0.30.0 | No |
| Extra build step | None | One conversion run with Transformers |
| Provenance to record | Model revision, override JSON, template hash | Model revision, conversion command, output hash, template hash |
The long-document case: chunked prefill under torch.compile
There is one wrong-score failure that the correct flags do not prevent. In July 2026 an issue reported that Qwen3-Reranker-0.6B on vLLM's V1 engine returned scores "abnormally low (close to 0)" for pairs longer than 8K tokens, while the same inputs through offline LLM.score() scored close to 1. The reporter confirmed the token ids were identical in both paths.
The pull request that fixed it, #48901, found that when a long prompt was split across prefill chunks under torch.compile, LAST pooling read the final token's hidden state before the kernel producing it had completed. Its own measurements: about 0.33 with compile and chunked prefill, about 0.83 with compile and a single chunk, and matching pairs varying run to run between roughly 0.01 and 0.36 when chunked. The fix added a device synchronization in the V1 runner's pooling step; its author called it "arguably a workaround" and suspected a buffer-lifetime issue in the compiler rather than claiming a definitive root cause. It merged on 2026-07-17 and shipped in v0.26.0, uploaded to PyPI on 2026-07-25.
At v0.34.0 the picture has moved. The release notes say "Model Runner V1 is gone. Model Runner V2 is now the only GPU runner," so the file that patch touched no longer exists. The regression test from that fix still ships: it scores a long repeated document with max_num_batched_tokens at 2048 and at 16384 and asserts the two scores agree within 0.05. That is evidence of intent, not proof that every chunking path is covered. Be on 0.26.0 or later, and keep a pair longer than your max_num_batched_tokens in your smoke test so a regression shows up as a number.
Truncation is the adjacent trap. The template puts the scoring position at the very end of the prompt, after the document. If you cap length with truncate_prompt_tokens, the cut can remove that suffix; vLLM's docs at v0.34.0 note that it does not restore the template suffix after truncation and recommend max_tokens_per_doc instead, which limits the document before the prompt is composed. A separate fix in v0.31.0 (#57666) stopped a non-binding max_tokens_per_doc from changing token ids for rerankers without a separator, so set that field on a release at or above it.
A smoke test that catches all of it before users do
Every row above should fail the same simple check: give the reranker one obviously correct document, one obviously unrelated one and one near-miss, and see whether the scores separate. Send it to /v1/rerank:
curl -s http://127.0.0.1:8000/v1/rerank \
-H 'Content-Type: application/json' \
-d '{
"model": "Qwen/Qwen3-Reranker-0.6B",
"query": "What is the capital of China?",
"documents": [
"The capital of China is Beijing.",
"Gravity is a force that attracts two bodies towards each other.",
"Shanghai is the largest city in China by population."
],
"instruction": "Given a web search query, retrieve relevant passages that answer the query",
"max_tokens_per_doc": 4096
}'The response carries results entries with an index and a relevance_score between 0 and 1. The pass condition is shape, not an exact value: document 0 clearly above documents 1 and 2, and the three scores spread apart rather than clustered. Gravity scoring close to Beijing means the yes/no judgement is not what you are reading. The model value must match the served model name.
The instruction field works per request at v0.34.0. A May 2026 change (#42412) added it to the scoring and rerank request schema; vLLM folds it into the template's instruction variable, and an explicit chat_template_kwargs key wins over it. Use it to set a domain instruction per index, written in English as the model card recommends for multilingual use.
/v1/score takes the same model in a different shape. Query and document lists are paired by position, and a single query is broadcast across all documents:
{
"model": "Qwen/Qwen3-Reranker-0.6B",
"queries": ["What is the capital of China?", "Explain gravity"],
"documents": [
"The capital of China is Beijing.",
"Gravity is a force that attracts two bodies towards each other."
]
}The response is a data list with one score per pair.
Use the /v1 paths, not /rerank or /score. vLLM 0.34.0 registers /score, /v1/score, /rerank, /v1/rerank and /v2/rerank, but the --api-key middleware only guards paths starting with /v1, /v2, /inference and /cohere. The unprefixed routes answer without a key. The full route coverage is in our vLLM inference server hardening guide. Add an Authorization: Bearer header to the probe when the server runs with a key.
Complete the test with two more checks. Send one extra pair whose document is longer than your max_num_batched_tokens, with a document that answers the query, and confirm it scores high like the short correct document, not near 0. Then search the startup log for either log line shown in the table section: the --convert embed INFO line or the no-template WARNING. Either one on a reranker deployment means the scores are not the yes/no judgement.
One last scale note. Sentence Transformers' CrossEncoder returns the raw logit difference for this model by default (the model card's example prints 7.625 and -11.375), while vLLM applies a sigmoid and returns 0 to 1. If two stacks disagree on the scale, that alone is not a bug. Set use_activation in the vLLM request if you need the raw value for comparison.
Running it inside a regulated perimeter
When a reranker inside a bank, insurer or hospital runs from a mirrored checkpoint with no Hub access, that changes where the fix lives. The template is not part of the weights vLLM reads for this model, and whether examples/ is present in your vLLM wheel or container image is not something to assume. Copy qwen3_reranker.jinja from the v0.34.0 tag into your own image or model store, next to the weights, and treat it as a versioned artifact with the same weight as the checkpoint. The general mirroring process is in our guide to air-gapped vLLM deployment; the reranker adds one file to the list.
Record four things in the model inventory: the Hugging Face revision you mirrored, the override JSON (or the conversion command and output hash), the template file hash, and the vLLM version. A change to any of them changes every score, and a missing template file is otherwise invisible to change control because the server still starts and still returns 200.
The reranker decides which passages a reviewer or a generation model sees first. A silent prompt mismatch is a quality and audit defect that no latency SLO, uptime probe or HTTP status will catch, so put the three-document probe and the long pair into the deployment gate and fail the rollout if the correct document does not come out on top. Serve clients on /v1/rerank so --api-key applies. Keep --trust-request-chat-template off, so callers cannot send their own template and change the scoring prompt; a Jinja rendering advisory (CVE-2026-115911) that let unbounded templates stall the renderer was fixed in 0.32.0, and your pin is above it.
Whether a reranker belongs in the pipeline at all is a separate decision, made on your own queries, as laid out in when RAG reranking actually improves retrieval. Once it is in, the three-document probe is a seed: grow it into a labelled set with the method in testing RAG retrieval accuracy, and the rest of the retrieval stack is mapped in our RAG systems pillar.
This week, run the three-document probe against your live /v1/rerank endpoint and search the startup log of that same server for the convert-embed line and the no-template warning. If the gravity sentence scores near Beijing, or either line is there, add the overrides and --chat-template before you tune anything else in retrieval.
FAQ
Quick answers to the questions this post tends to raise.



