HyDE's headline result was measured against an unsupervised encoder: with a 175B InstructGPT generator sampled 8 times per query, it took Contriever from 44.5 to 61.3 nDCG@10 on TREC DL19. On a fine-tuned encoder the same paper shows HyDE lowering nDCG@10 with an 11B generator (62.1 to 60.2 on DL19, 63.2 to 62.1 on DL20) and a 52B one, and helping only with the 175B model (+5.3 and +0.3). The authors call that setup not the intended usage. Query2doc, which keeps the query and appends the generated passage, lifted BM25 by 15.0 and 15.2 nDCG@10 points on DL19 and DL20 without training, but its dense gains required retraining the retriever and shrank to +0.6 to +1.9 on distilled retrievers. Both papers show the gain scaling with generator size, and neither tested a current open-weight 7B to 70B model. On a self-hosted stack the extra generation also sits in front of time to first token on the same GPUs. Start by pulling 100 labeled queries from your logs and comparing your baseline against HyDE with the query vector averaged in, at three seeds, before you route any traffic through it.
HyDE RAG starts from one headline number: with InstructGPT writing hypothetical answers, the unsupervised Contriever encoder went from 44.5 to 61.3 nDCG@10 on TREC DL19. That is a 16.8 point jump from a method that needs no training data, which is why hypothetical document embeddings look like a cheap upgrade for retrieval augmented generation.
The same paper has a second table, and it decides whether that upgrade applies to you. On a fine-tuned encoder, HyDE with an 11B Flan-T5 generator moved DL19 from 62.1 to 60.2. With a 52B generator it still lost. Only the 175B model helped, and on DL20 by 0.3 points. The authors' own description of that configuration is blunt: "HyDE with fine-tuned encoder is not the intended usage."
Whether HyDE helps depends on two things you already know about your stack: the retriever you run and the generator you can afford to put in front of it. For a team inside a regulated perimeter, with an in-domain embedder and a 7B to 70B open-weight model on shared GPUs, both of those point toward the rows where the published gains are smallest. HyDE is a measured, gated choice for that stack, not a default, and the only result that settles it is one you produce on your own labeled queries.
How the HyDE RAG Technique Works, and What the Paper Ran
HyDE comes from "Precise Zero-Shot Dense Retrieval without Relevance Labels" (arXiv 2212.10496), submitted 20 December 2022. The mechanism has four steps:
The bet is that a fake answer sits closer in embedding space to the real answer than the question does, and that the encoder's dense bottleneck discards the invented details. That second part is the paper's stated expectation, not a measured guarantee.
The setup behind the headline numbers
The details matter because they set the cost you would actually be reproducing. The generator was InstructGPT text-davinci-003, sampled at temperature 0.7. The paper text leaves the number of samples unspecified, but the reference notebook for DL19 calls the model with n=8 and max_tokens=512, then averages the 8 passage embeddings plus the original query embedding, nine vectors in total. The headline DL19 result came from eight sampled passages per query, not one. Prompts were per dataset. Web search asked for a passage that answers the question; SciFact asked for a scientific paper passage that supports or refutes the claim; FiQA asked for a financial article passage. The authors attribute HyDE's weaker FiQA and DBPedia results to "under-specification of the instruction", which is a useful warning for anyone writing one generic prompt for a corpus of policies or SOPs.
Two reasons you cannot reuse the published setup
First, the generator is gone. OpenAI shut down text-davinci-003 on 4 January 2024, so nobody can rerun the published configuration. Any HyDE result you rely on has to come from your own generator. Second, framework defaults are not the paper. LangChain's HypotheticalDocumentEmbedder in langchain-classic 1.0.8 runs the model once, embeds one generated document, and averages only the generated embeddings, leaving the query out. That is a single-sample, query-free variant. It may be the right variant for your stack, but it was not what produced the numbers in the next section.
HyDE vs a Fine-Tuned Embedding Model: What Table 4 Shows
The abstract's claim is precise: HyDE "significantly outperforms the state-of-the-art unsupervised dense retriever Contriever and shows strong performance comparable to fine-tuned retrievers." Comparable is not better. On DL20, HyDE with the 175B generator scored 57.9 against 63.2 for the fine-tuned ContrieverFT, which the authors describe as around 10% lower.
Table 4 is the experiment that answers the question a production team actually has: what happens when you add HyDE on top of an encoder that has already been trained?
Read it twice, once per encoder.
On the unsupervised encoder, every generator helped, and bigger helped more. The authors say so in Section 5.1: all models improve unsupervised Contriever, "with larger models bringing larger improvements."
On the fine-tuned encoder, two of three generators made retrieval worse on both query sets. The authors' wording is that "less powerful instruction LMs can negatively impact the overall performance of the fine-tuned retriever," and that the degradations "remain small." The 175B result, they suggest, means "there may still exist certain factors not captured by the fine-tuned encoder but only by the generative model." That is a real signal, but it came from one generator on one query set: on DL20 the same model added 0.3 points.
Why this baseline favors the fine-tuned encoder, and the gap it leaves
ContrieverFT was fine-tuned on MS MARCO, and TREC DL19 and DL20 are built from MS MARCO queries. The authors call it "in-domain supervisedly fine-tuned." That makes Table 4 a demanding test for HyDE: the encoder it competes with was trained on this exact distribution. It also leaves a gap. The fine-tuned rows exist only for DL19 and DL20. The BEIR results in Table 2 are HyDE over the unsupervised encoder, where HyDE beat ContrieverFT on SciFact (69.1 vs 67.7), ArguAna and TREC-NEWS, and lost on TREC-COVID, FiQA (27.3 vs 32.9) and DBPedia (36.8 vs 41.3). Nobody in the paper tested HyDE on top of a fine-tuned encoder out of domain. If your embedder was fine-tuned on one corpus and serves queries from another, the evidence is silent.
The authors' own routing advice
The conclusion frames HyDE as a cold-start tool. Early in a search system's life, HyDE offers performance comparable to a fine-tuned model; "as the search log grows, a supervised dense retriever can be gradually rolled out," with "only less common and emerging ones going to HyDE backend." That is a gating design, written by the people who built the method. If you already have labels, the cheaper path to better retrieval is the one they point at: fine-tuning the embedding model on your own query and passage pairs.
| Encoder | Generator | DL19 nDCG@10 | Change | DL20 nDCG@10 | Change |
|---|---|---|---|---|---|
| Contriever (unsupervised) | none | 44.5 | 42.1 | ||
| Contriever | Flan-T5 11B | 48.9 | +4.4 | 52.9 | +10.8 |
| Contriever | Cohere 52B | 53.8 | +9.3 | 53.8 | +11.7 |
| Contriever | InstructGPT 175B | 61.3 | +16.8 | 57.9 | +15.8 |
| ContrieverFT (fine-tuned) | none | 62.1 | 63.2 | ||
| ContrieverFT | Flan-T5 11B | 60.2 | -1.9 | 62.1 | -1.1 |
| ContrieverFT | Cohere 52B | 61.4 | -0.7 | 63.1 | -0.1 |
| ContrieverFT | InstructGPT 175B | 67.4 | +5.3 | 63.5 | +0.3 |
Query2doc vs HyDE: Keep the Query, Expand It
Query2doc (arXiv 2303.07678, EMNLP 2023) is a second pre-retrieval generation method with published tables, and it makes a different choice. Instead of replacing the query, it appends to it. A few-shot prompt ("Write a passage that answers the given query:" with four labeled MS MARCO examples, re-sampled per call) produces one pseudo-document of at most 128 tokens at temperature 1. For BM25, the query is repeated five times and concatenated with the pseudo-document so its terms keep their weight. For dense retrieval, the input is the query, a separator token, then the pseudo-document.
The pseudo-document alone, which is the HyDE-style replacement, scored below the original query. The combination is what works, and the authors call the two "complementary." If you run a hybrid stack, this is the leg where generated expansion has its clearest evidence; the fusion and sparse side are covered in our guide to hybrid dense and sparse search.
BM25 is where the no-training gain lives
The abstract reports BM25 gains of "3% to 15%" on MS MARCO and TREC DL without fine-tuning. Those are absolute points on different metrics, not relative percentages: MRR@10 on MS MARCO dev went from 18.4 to 21.4, and nDCG@10 on DL19 went from 51.2 to 66.2 (+15.0) and on DL20 from 47.7 to 62.9 (+15.2). Classic pseudo-relevance feedback (BM25 plus RM3) reached 52.2 and 47.4 on the same sets. The ablation in Table 4 of that paper settles the replace-or-expand question for lexical search:
Dense gains required retraining
Every dense query2doc row in Query2doc's Table 1 is marked as fine-tuned: the retrievers were retrained on expanded queries, with maximum query length raised to 144 tokens to fit the pseudo-documents, and most of the roughly 550,000 API calls went to generating pseudo-documents for training queries. This is not a drop-in change to a dense retriever you already serve. The paper never tests the dense benefit without retraining. Even with retraining, gains shrank as the retriever got stronger. DPR gained 4.0 and 3.0 nDCG@10 points on DL19 and DL20; SimLM gained 1.5 and 1.9; E5-base with knowledge distillation gained 0.6 and 1.8. The authors write that the gain "tends to diminish when using intermediate pre-training or knowledge distillation from cross-encoder re-rankers." Out of domain on BEIR, BM25 improved on all five datasets, while dense results were "mixed": SciFact fell 2.9 points for both SimLM and E5. One more number belongs next to every Query2doc delta. Across three random runs on the sparse setup, DL19 scored 64.8 plus or minus 1.14 and DL20 60.9 plus or minus 1.63, against single-run main-table figures of 66.2 and 62.9. Randomness from few-shot example selection and sampling moves results by more than some of the dense gains above.
| BM25 input | DL19 nDCG@10 | DL20 nDCG@10 |
|---|---|---|
| Query only | 51.2 | 47.7 |
| Pseudo-document only | 48.7 | 44.5 |
| Query plus pseudo-document | 66.2 | 62.9 |
Which LLM for HyDE: Generator Size Decides the Gain
Both papers ran a generator-size sweep, and both point the same way.
The Query2doc authors write that "small language models only provide marginal improvements over baselines," and that text from smaller models tends "to be shorter and contain more factual errors." In the HyDE rows, the 11B generator over Contriever reached 48.9 on DL19, below plain BM25's 50.6. On that query set, a team with an 11B generator and an unsupervised encoder would have done better skipping both.
The honest limit of this table is its age. Every generator tested dates from 2023 or earlier, and none is a current open-weight instruct model in the 7B to 70B range. A modern 8B model may write better hypothetical passages than Flan-T5 11B did; it may not. No paper read for this post measures it, and we are not going to guess. What the table does establish is direction: the gain is a property of the generator, so the generator you can host is the variable you have to test.
Self-verification did not help either. Query2doc tried GPT-4 iteratively checking its own pseudo-documents and got 68.6 and 64.8 against 69.2 and 64.5 without it, with no meaningful change. Its Table 9 shows a generated passage with a wrong fax number, the kind of confident detail a retrieval system then searches for.
| Paper and retriever | Generator | DL19 nDCG@10 | DL20 nDCG@10 |
|---|---|---|---|
| Query2doc, BM25 | none | 51.2 | 47.7 |
| Query2doc, BM25 | babbage 1.3B | 52.0 | 50.2 |
| Query2doc, BM25 | curie 6.7B | 55.1 | 50.1 |
| Query2doc, BM25 | davinci-001 175B | 63.5 | 58.2 |
| Query2doc, BM25 | davinci-003 175B | 66.2 | 62.9 |
| Query2doc, BM25 | GPT-4 | 69.2 | 64.5 |
| HyDE, Contriever | Flan-T5 11B | 48.9 | 52.9 |
| HyDE, Contriever | Cohere 52B | 53.8 | 53.8 |
| HyDE, Contriever | InstructGPT 175B | 61.3 | 57.9 |
The Cost of HyDE on a Self-Hosted Stack
With a hosted API, HyDE costs tokens. Inside your own perimeter it costs something less visible: GPU time on the same pool that serves answers, spent before the answer can start.
The structure is fixed regardless of hardware. HyDE puts a complete generation in front of retrieval, and retrieval sits in front of answer generation. Time to first token for the user cannot begin until the hypothetical passage has finished decoding, the embedding has run, and the search has returned. The paper's reference setup decodes up to eight passages of up to 512 tokens each, which on a local GPU is a batch of decode work per query that competes for the same slots your answer model needs. Query2doc caps it at one passage of 128 tokens. If you are already budgeting against time to first token and tokens per second targets, the extra generation goes in that budget, not beside it.
There is no primary measurement of HyDE latency on an open-weight generator to quote here, so this post does not print a millisecond figure. The one hardware-meaningful number in either paper is on the search side: Query2doc reports BM25 index search at 16 ms per query, rising to 177 ms once the query is expanded, because a longer query touches more posting lists. Its LLM call is listed at over 2,000 ms, but that was a 2023 remote API whose latency the authors say depends on server load, so it says nothing about your rack.
Moving generation off the request path
Query-time generation is one of two places to spend LLM compute on retrieval. The other is index time, where generated context or questions are attached to chunks once and every query benefits without waiting; we compare those options in contextual retrieval vs late chunking. A July 2026 paper on hypothetical prompt embeddings (arXiv 2607.29402) applies the HyDE idea in reverse: it precomputes hypothetical questions per chunk, so retrieval becomes question-to-question matching with no generation at query time. For a team whose GPU budget is tightest during business hours, shifting generation into a batch indexing job is a structural win worth evaluating alongside HyDE. HyDE is also one rewrite among several. Generating multiple query variations as a recall fix is a separate pattern, covered under query expansion in when you need reranking in RAG, and grade-then-rewrite loops and query decomposition belong to agentic RAG, a different design with a different cost profile.
When to Use HyDE RAG: A Decision Table
The papers give enough to set an expectation for each common stack and to name what to test. They do not give enough to predict your result.
Gating rules
- 1. Keep the original query in the signal. Average its embedding into the HyDE vector, as the reference code does, or keep it repeated in the BM25 string, as Query2doc does. The Query2doc ablation shows what replacement alone costs on lexical search.
- 2. Route by first-pass strength. Run normal retrieval first, and send a query through HyDE only when the top-1 similarity falls below a threshold you calibrate on labeled queries, or when the query is a short keyword string. Never rewrite a query that already hit. This matches the HyDE authors' own advice to send rare and emerging queries to the HyDE backend.
- 3. Write a task-specific instruction. The paper's prompts were per dataset, and the authors blame its FiQA and DBPedia shortfalls on under-specified ones. "Write a passage from an internal credit policy that answers the question" is a different prompt from a generic one.
- 4. Treat the asymmetric encoder choice as an experiment. The Qwen3-Embedding card says each query "must come with a one-sentence instruction" and that documents need none. A hypothetical passage is document-shaped text standing in for a query, and neither the paper nor the card says which side to encode it on.
The regulated-perimeter version of the answer
Inside a bank, insurer or hospital, HyDE has a second cost beyond latency. The hypothetical passage is model-written text that decides which real documents reach the answer model. In a corpus of policies, SOPs or filings with superseded versions, a confident invented detail (a wrong threshold, a wrong effective date) can pull the outdated document. Log the generated passage with the retrieval trace so an auditor can see why a document was retrieved, and keep the original query in the vector so a hallucinated detail cannot redirect the search on its own. The local generator you are allowed to run is also likely smaller than the 175B models behind the headline gains, which is the row where both papers report the least. For the wider set of patterns across this cluster, see our RAG systems guide.
| Your stack | What the papers show | Expected effect | What to test |
|---|---|---|---|
| Unsupervised or general embedder, no labels, strong generator | HyDE gains of 4.4 to 16.8 nDCG@10 over Contriever | Likely positive | HyDE with query averaged in, at your generator size |
| In-domain fine-tuned embedder, small local generator | Losses of 0.1 to 1.9 points with 11B and 52B generators | Flat to slightly negative | Gated HyDE on weak-first-pass queries only |
| BM25, or the lexical leg of a hybrid | Query2doc +15.0 and +15.2 points; pseudo-doc alone below query | Positive if the query is kept | Query repeated plus pseudo-doc on the BM25 string |
| Asymmetric, instruction-aware embedder | Not tested in either paper | Unknown | Encode the passage query-side with instruction vs document-side without |
| Regulated corpus with superseded versions | Generated passages contain false details (Query2doc Table 9) | Risk of steering toward the wrong version | Audit logging plus query kept in the vector |
How to A/B Test HyDE on Your Own Queries
None of the published numbers describe your embedder, your generator or your corpus, so the decision comes from an evaluation you run. Build the labeled set the way we describe in testing RAG retrieval accuracy: 100 to 300 real queries from your logs, each with the passages that should come back.
Run each generated arm with your production generator at its production settings. Then:
If the HyDE arms lose to a fine-tuned embedder, you have reproduced Table 4, and the next investment is the one its authors name: more labels and a stronger supervised retriever.
This week, pull 100 real queries with known answers from your logs, run your baseline against HyDE with the query embedding averaged in, using your production generator at three seeds, and look at the fine-tuned encoder rows of Table 4 before you route a single production query through it.
| Arm | What changes | Why it is in the test |
|---|---|---|
| Baseline | Production retrieval, untouched | The number to beat |
| HyDE replace | Generated passage embedding only | What one-sample framework defaults do |
| HyDE plus query average | Passage embeddings averaged with the query embedding | What the paper's code did |
| Query2doc on the BM25 leg | Query repeated plus pseudo-document, lexical side only | Where the no-training evidence is strongest |
FAQ
Quick answers to the questions this post tends to raise.
