Add reranking when retrieval returns the right documents in the wrong order, and measure the gain on your own queries before keeping it. Retrieve 20-50 candidates, not 100+. Use Cohere Rerank for managed APIs or BGE-reranker-v2-m3 for self-hosted. Skip reranking for simple factual queries, small corpora under 1K docs, or latency-sensitive apps where sub-second responses matter more.
Adding reranking to a RAG pipeline because tutorials mention it, rather than because retrieval is actually mis-ordering results, has a predictable outcome: added latency on every query, no meaningful accuracy gain, and a customer support chatbot that now feels sluggish for queries that were simple factual lookups in the first place.
RAG reranking is a standard recommendation in retrieval-augmented generation, and for good reason: on complex queries, a cross-encoder reranker can recover relevant documents that vector search ranked too low. But it is also easy to spend weeks implementing reranking for a use case that doesn't benefit from it. This post lays out the framework we use at Particula Tech to decide when reranking pays off, and when engineering time is better spent elsewhere.
What Is Reranking in RAG?
Reranking in RAG is a second-stage retrieval process that re-scores documents after your initial vector search. The two-stage pipeline works like this:
Stage 1, Bi-encoder retrieval: Your embedding model encodes queries and documents independently into vectors. You retrieve the top-K candidates by cosine similarity. This is fast but lossy. Compressing an entire document into a single vector inevitably drops nuanced relevance signals.
Stage 2, Cross-encoder reranking: A reranking model processes each query-document pair together through cross-attention layers. Unlike bi-encoders, the reranker sees the actual interaction between query terms and document content simultaneously. This captures relationships that independent embeddings miss.
Think of it like hiring. The initial vector search is your resume screen, fast but coarse. Reranking is the structured interview where you evaluate candidates against specific requirements with full context.
The reason this two-stage approach exists is computational cost. A cross-encoder needs one forward pass per query-document pair, so running it against your entire corpus at query time is impractical. By first narrowing candidates with fast vector search, then applying the expensive reranker to just 20-50 documents, you get cross-encoder quality at practical speed.
For a deeper look at how embedding quality feeds into this pipeline, see our guide on which embedding model to use for RAG and semantic search.
How Much Does Reranking Actually Improve RAG Accuracy?
Reranking discussions often hand-wave about "improved relevance" without saying where the gain comes from. Here is what the public evidence supports, and why your own number will differ.
Research Benchmarks
The BEIR benchmark evaluated retrieval models zero-shot across 18 datasets. Its authors found that re-ranking and late-interaction models achieved the best average zero-shot performance, but at high computational cost, while dense and sparse retrievers were more efficient and often underperformed. BM25 held up as a robust baseline. That is the shape of the trade-off this post is about: rerankers buy relevance with compute. But benchmark gains come from public datasets with controlled queries, and production looks different.
Why Production Gains Differ From Benchmarks
The gain you measure on your own system can be well below what benchmarks suggest, for three reasons: The reverse also holds. In financial services contract analysis, queries often combine several constraints at once: date ranges, clause types and jurisdictional requirements. The initial vector search retrieves relevant contracts, and a reranker that reads the query and document together is better placed to prioritize documents satisfying all of them. When each surfaced document has to be verified by a lawyer, better ordering translates directly into review time saved. A healthcare FAQ chatbot is the opposite case. When queries are simple enough that the embedding model already surfaces the right answer in position one, reranking has nothing left to fix and only adds latency.
- Simpler queries: If many of your queries are straightforward questions, initial retrieval already handles them well and leaves little for a reranker to fix
- Domain-tuned embeddings: If your system uses a fine-tuned or domain-specific embedding model, it may already capture relevance signals that a generic model misses, which narrows the room a reranker has to improve on
- Corpus quality: A well-curated document collection has less ambiguity than a research dataset
How Many Candidates Should You Retrieve for Reranking?
Retrieve between 20 and 50 candidates for reranking. Getting this wrong either wastes compute or leaves accuracy on the table, and the trade-off follows from how cross-encoders work: each candidate is a separate model pass, so cost grows linearly with candidate count, while accuracy only improves if the extra candidates contain relevant documents the smaller set missed.
Going past 50 only pays if your labeled queries show relevant documents sitting between positions 50 and 100 of your first-stage results. If they don't, the extra candidates are pure latency.
My recommendation: Start with 30 candidates. If your corpus is large (100K+ documents) with many semantically similar entries, push to 50. If latency is tight, drop to 20. Never go below 10, you're undermining the point of having a reranker.
One nuance: the optimal number depends on your initial retrieval quality. If your embeddings are domain-tuned, 20 candidates likely contains everything relevant. If you're using a generic embedding model on a broad corpus, you might need 50 to ensure recall. For guidance on embedding selection, our post on embedding dimensions for RAG covers how vector size affects retrieval coverage.
| Candidates | When it fits | Cost driver |
|---|---|---|
| 10 | Floor; below this the reranker has too little to reorder | Lowest; recall limited by stage 1 |
| 20 | Tight latency budget, domain-tuned embeddings with strong recall | Twice the reranker work of 10 |
| 30 | Default starting point | Three times the reranker work of 10 |
| 50 | Large corpora (100K+ documents) with many near-duplicates | Five times the reranker work of 10 |
| 100 | Rarely justified; test it before adopting it | Ten times the reranker work of 10 |
When RAG Reranking Delivers Measurable Value
Reranking has the clearest case in four scenarios.
Multi-Intent Queries
When users ask questions with multiple requirements, "Find contracts from 2023 with renewal clauses mentioning price escalation", reranking excels. The initial vector search pulls documents matching parts of the query, but a cross-encoder scoring the full query against each document can prioritize the ones satisfying all criteria simultaneously. This is the scenario where we would test a reranker first.
Large Corpora with Semantic Overlap
If your vector search returns many near-identical similarity scores, say, 50 documents all scoring 0.82-0.88, reranking breaks the tie with finer-grained semantic understanding. Technical documentation corpora are a classic case: hundreds of articles about the same API for different purposes, tutorials, reference docs, troubleshooting guides. A reranker that reads the query against each article has a better chance of matching query intent to the right document type than a similarity score that barely separates them.
Domain-Specific Relevance Beyond Embeddings
General-purpose embedding models don't understand industry-specific relevance hierarchies. In healthcare, "patient compliance" could mean medication adherence, appointment scheduling, or treatment protocols, all semantically similar but requiring domain-specific prioritization. A reranker can learn that prioritization only if it was trained on relevance judgments from the domain. If you have labeled query-passage pairs from your own clinical content, fine-tuning a reranker on them is a way to encode that weighting; a general-purpose reranker carries no such guarantee, so test it on your own queries before relying on it. The key diagnostic: if your users regularly scroll past the first few results or rephrase queries to find what they need, your retrieval has a ranking problem, not a recall problem. That's exactly what reranking fixes.
When You Can't Fine-Tune Embeddings
If you're using an off-the-shelf embedding model and don't have the resources to fine-tune it on your domain, reranking is the next best approach. It's faster to implement than embedding fine-tuning and doesn't require curated training data. Think of it as a shortcut to domain-aware relevance without the up-front data investment.
When to Skip Reranking Entirely
The cost and complexity of reranking isn't always justified. Here's when to skip it.
Simple Factual Queries
If your RAG system handles straightforward questions, "What's our return policy?", "Who manages the ACME account?", and your embedding model already surfaces the right document in position one, skip reranking. An e-commerce FAQ system in that state gains nothing in answer quality from a reranker and still pays its latency on every query.
Small, Well-Curated Collections
With fewer than 1,000 documents, especially if they're topically distinct and well-organized, initial retrieval is likely sufficient. For a small internal knowledge base, say, a 50-person startup, plain vector search is fine and reranking would be pure engineering overhead.
Latency-Critical Applications
Reranking adds a model pass over every candidate on every query. For a live customer chat where users expect instant responses, that delay degrades experience. Removing reranking from a chat-style RAG pipeline trades some accuracy for response time, a worthwhile trade when sub-second latency is the constraint.
Already Fine-Tuned Embeddings
If you've fine-tuned your embedding model on your corpus, you may have already captured much of what reranking provides. Measure reranking against the fine-tuned baseline rather than the generic one; if the remaining gain is small, it won't justify the added complexity and latency. Our guide to fine-tuning an embedding model for RAG sets up that comparison as an evaluation gate on held-out real queries: base model, fine-tuned model, and each with the reranker. The decision rule: if your initial retrieval puts the right answer in the top 3 results more than 85% of the time and latency matters, you probably don't need reranking.
Reranking Models Worth Evaluating for Production RAG (2026)
If you've decided reranking is worth it, here are the options we would evaluate first. And once you know you need a reranker, it pays to compare Cohere vs Voyage vs Jina vs BGE on your own corpus before locking one in.
My go-to stack: Cohere Rerank for managed deployments where accuracy is the priority. BGE-reranker-v2-m3 on GPU for self-hosted systems where you need cost control and data privacy. MiniLM for quick prototyping and English-only use cases where every millisecond counts.
One practical tip: BGE-reranker-v2-m3 is a 0.6B-parameter cross-encoder, and running it on CPU over 30 candidates per query is far slower than on GPU. If you don't have GPU resources for inference, a managed API can be cheaper than the engineering time spent optimizing CPU performance.
| Model | Type | Size | Languages | Fits |
|---|---|---|---|---|
| Cohere Rerank v4.0 | Managed API | Not published | Multilingual | Managed deployments, no GPU to run |
| BAAI/BGE-reranker-v2-m3 | Open weights (on bge-m3) | 0.6B params | Multilingual | Self-hosted, cost-sensitive |
| ms-marco-MiniLM-L-6-v2 | Open weights | ~22.7M params | English only | Prototyping, low latency |
| Jina Reranker v2 | Managed API | Not listed | Multilingual | Alternative to Cohere |
How to Test If Reranking Helps Your System
Don't implement reranking based on blog posts, including this one. Test it against your actual data.
Build a Representative Query Set
Collect 100-200 real user queries spanning the full complexity range your system handles. Include both simple factual lookups and complex multi-intent queries. A common failure mode: test sets that are too simple, missing the multi-intent and ambiguous queries production users actually send, making reranking look less valuable than it is.
Establish Baselines
Measure your current system's retrieval quality: Track business metrics too, task completion rate, user satisfaction, time-to-answer. A technically impressive MRR delta might not translate into meaningful user impact.
- MRR (Mean Reciprocal Rank): Where does the correct answer appear on average?
- nDCG@10: How well-ordered are the top 10 results overall?
- Precision@3: Is the answer in the top 3?
- Latency P95: What's the worst-case response time?
Run an A/B Comparison
Use an off-the-shelf reranker (Cohere or BGE) for initial testing. Route 20% of traffic through the reranking pipeline. Compare not just accuracy metrics but actual user behavior, are people finding answers faster? Rephrasing less? Completing tasks more often?
Calculate the ROI
If MRR improves from 0.78 to 0.85 but P95 latency doubles, quantify what that means for your business. For a legal research platform, a better-ordered result list cuts the time attorneys spend verifying documents, and you can estimate that saving from your own review logs. For a customer support chatbot, the latency hit might increase abandonment rates more than the accuracy gain reduces them.
Alternatives Worth Trying Before Adding Reranking
Before committing to reranking's complexity, these approaches can deliver comparable improvements with less overhead.
Fix Your Chunking First
Chunk boundaries decide what a retriever can return at all: if the answer is split across two chunks, no reranker can put it back together. Start by testing chunk size and overlap on your own query set rather than jumping to the most sophisticated strategy. A 2024 study of chunking strategies found that the computational costs of semantic chunking were not justified by consistent performance gains over simpler fixed-size chunking. For practical guidance, see our deep dive on document chunking strategies for RAG.
Use Hybrid Search
Combining BM25 keyword search with vector similarity catches exact matches and domain terms that embeddings miss. It's computationally cheaper than reranking and can deliver comparable improvements when the failures are on exact terms. Read more in our guide to hybrid dense-sparse search approaches.
Implement Query Expansion
Transform user queries using an LLM to generate multiple search variations before retrieval. This surfaces relevant documents that single-query embedding similarity misses, which makes it a recall fix rather than an ordering fix, and it needs no post-retrieval processing. The published gains are largest on sparse or unsupervised retrievers and shrink on fine-tuned embedders, so check our breakdown of when HyDE and Query2doc expansion help or hurt retrieval before adding the extra generation call.
Add Metadata Filtering
If your documents have structured metadata (date, department, document type), filter before vector search. This reduces your candidate set and improves relevance without latency cost. It's especially powerful combined with reranking, filter first, then rerank the focused set. For a comprehensive overview of RAG optimization techniques, visit our RAG Systems pillar page.
Your Reranking Decision Framework
Here's the decision sequence we use:
RAG reranking is a powerful technique, when matched to the right problem. Complex queries, large corpora with semantic overlap, and domain-specific relevance hierarchies are where it has room to help. Simple queries, small collections, and latency-sensitive applications usually don't benefit enough to pay for it.
Getting reranking right means testing it rigorously, measuring business impact, and making deliberate trade-offs between accuracy, latency, and cost. Don't add it because every tutorial mentions it. Add it because your data proves it solves a real problem in your pipeline.
FAQ
Quick answers to the questions this post tends to raise.



