MTEB Leaderboard Lies: Embedding Recall in Production

MTEB rank measures average competence across a fixed public battery; production recall measures how well a model finds the right documents on your specific corpus and queries. Those are different functions, and a model can be excellent at the first while merely average at the second. Recall@k, the share of the truly relevant documents that land in your top k results, is what actually determines whether your RAG answers are grounded, and no leaderboard measures it on your data.

The benchmark is broad by design. According to the original MTEB paper (Muennighoff et al., 2022), it spans 8 embedding tasks, 58 datasets, and 112 languages, evaluating 33 models at launch. According to MMTEB (Enevoldsen et al., 2025), the massively multilingual successor pushes that scope to over 500 quality-controlled tasks across 250-plus languages, which only widens the average further from any single production workload. ZeroEntropy describes today's leaderboard, originally introduced by HuggingFace and now community-maintained, as running 50-plus datasets across roughly 8 task categories. That breadth is the point and also the problem: a single averaged number rewards models that are good everywhere, which is not the same as being good on the narrow slice of language your users actually search.

Across the RAG pipelines we build for enterprise clients at Particula Tech, the leaderboard's biggest failure is one of framing. Teams treat rank as a linear quality scale, so they agonize over a model sitting two positions lower, when the honest read is that the top of the board is a cluster of strong models whose relative order says little about which one wins on a legal-contract corpus or a support-ticket archive. The rank is a filter, not a verdict. For the upstream question of which family to even shortlist, our guide to which embedding model to use for RAG and semantic search is the right starting point; this post is about not trusting the number once you have it.

Benchmark contamination is when a model has been trained, directly or through close paraphrases, on data that overlaps the evaluation corpus, so it scores by partial recall of seen text rather than by generalizing. This is the single most important reason to distrust a raw MTEB rank. ZeroEntropy's MTEB analysis states that many embedding models are partially trained on data overlapping MTEB datasets or close paraphrases, and that this contaminates the leaderboard.

Note the careful phrasing. The documented risk is general: there is no public roster of named "frontier" models proven to have trained on the eval, and you should not assume a specific vendor cheated. What you should assume is that contamination is common, undetected, and asymmetric. It inflates scores on the public benchmark and evaporates on private data the model has never seen, which is precisely your corpus.

The retrieval split makes this especially likely because of its lineage. According to the BEIR paper (Thakur et al., 2021), BEIR is a zero-shot retrieval benchmark of 18 publicly available datasets evaluated across 10 retrieval systems, and MTEB's retrieval portion is largely built on those same public BEIR datasets. Public datasets are the most likely to appear, in whole or in paraphrase, inside a large embedding model's training mix. The tell is quantitative. ZeroEntropy's rule of thumb, and it is a heuristic rather than a measured universal, is that a drop of 15 or more NDCG points moving from MTEB to a private domain signals contamination. NDCG, normalized discounted cumulative gain, is a ranking metric that rewards placing the most relevant documents highest. If a model that ranks near the top of the leaderboard falls off a cliff on your held-out queries, the leaderboard was measuring something you cannot use.

Only large gaps carry signal; small ones are noise you should not act on. Leaderboard points do not map linearly to production recall, so a fraction-of-a-point difference between two models is not evidence that one will retrieve better on your corpus. The honest version of "MTEB vs real-world retrieval" is that the benchmark orders models on average public performance, and the residual between that average and your specific workload is large enough to swamp the small gaps entirely.

Two supported observations make this concrete without inventing precision. First, the ordering is not stable over time. According to ZeroEntropy's MTEB analysis (2026), a model ranked top-5 six months ago is rarely top-5 now, which tells you how much churn lives inside the rankings even before you introduce your own data. Second, ZeroEntropy's practical guidance is to trust MTEB directionally, meaning large gaps mean something, but to evaluate on your data before committing. The corollary they draw is the diagnostic that runs through this whole post: if the leaderboard order does not predict your eval order, contamination is a likely culprit.

Here is how we translate leaderboard signals into decisions. Treat the table below as the reading key, not a formula.

The takeaway is discipline about what a point buys you. A large gap is a reason to include or exclude a model from consideration. A small gap is a reason to run your own test, not to choose.

Two models sitting a hair apart on the leaderboard can diverge substantially once you measure them on your own queries, because your corpus stresses a different distribution of language than the public average. The leaderboard's averaging is exactly what hides this: a model that is slightly better across 58 datasets can be meaningfully worse on the one distribution that matters to you, and the averaged rank never shows it.

The mechanism is domain shift. An embedding model's quality is not uniform across topics, jargon, query length, or document structure. A model tuned to shine on web-style question answering can under-retrieve on dense clinical notes or on short, keyword-like enterprise search queries, even when its public rank is a notch higher. When two models are near-tied on average, small per-domain differences that cancel out in the mean can point the same direction on your traffic and add up. This is why we insist on measuring the specific model on the specific corpus rather than reasoning from the aggregate.

This is also where embedding choice interacts with the rest of the stack. A near-tie at retrieval time can be decisively broken downstream, which is why we treat the embedding model, the index, and the reranker as one system rather than three independent picks. Our analysis of embedding quality versus the vector database unpacks where recall is actually won or lost, and if two candidates stay tied after your own eval, a reranker often settles it: the tradeoffs are laid out in our comparison of reranker models across Cohere, Voyage, Jina, and BGE on latency and NDCG. The point stands: near-tied on MTEB is not near-tied on your problem.

MTEB's retrieval split scores on binary relevance, and re-scoring the same datasets with graded relevance changes the model ranking, because binary labels are blind to how well a model orders results it already found. Binary relevance marks each query-document pair as relevant or not, a single bit. Graded relevance distinguishes highly relevant from marginally relevant from tangentially relevant, which is closer to how a human judges a results page.

Why does this move the leaderboard? Under binary scoring, a model that surfaces the single best document in position three scores the same as a model that surfaces it in position one, as long as both retrieve it inside the cutoff. The difference your users feel, the best answer at the top versus a passable answer at the top, is invisible to the metric. Swap to graded relevance and models that are good at fine-grained ordering pull ahead of models that merely find relevant documents somewhere in the top k.

ZeroEntropy re-annotated 28 of MTEB's retrieval datasets with graded relevance and reports that the resulting model ranking shifts meaningfully against the binary leaderboard. (The same page promotes ZeroEntropy's own embedding model on graded relevance, which is vendor self-promotion, so treat the model claim separately from the methodological point about binary versus graded labels, which stands on its own.) The practical lesson is about which metric you optimize. If your product rewards precise ranking, and most RAG and search products do, weight graded metrics like NDCG@10 over binary recall when you evaluate, because the binary leaderboard is systematically silent about the thing you care about most.

Sample real queries from your production traffic, annotate the retrieved documents with graded relevance, measure recall@k and NDCG@k on a held-out set, and compare the model ordering to the leaderboard ordering. That is the whole method, and it is the only reliable way to choose an embedding model for production. Everything above is the argument for why you cannot skip it.

Here is the concrete procedure we run when we build RAG pipelines for clients:

This is the same discipline we detail in our guide to RAG evaluation and retrieval-accuracy testing, applied specifically to embedding selection. It also determines a build decision most teams make too early: whether to self-host or call an API. Once you can measure recall on your own corpus, the tradeoffs in our breakdown of self-hosted versus API embeddings across Qwen3, EmbeddingGemma, Gemini, and Voyage become a data-driven call instead of a guess anchored to a leaderboard position.

Use the MTEB leaderboard as a shortlist and nothing more, then choose on a graded eval of your own traffic. That single rule resolves every failure mode above. Trust large leaderboard gaps to include or exclude models, ignore small gaps entirely, screen every candidate for the 15-plus NDCG contamination drop, and weight NDCG over binary recall because ranking quality is what your users experience.

Concretely, here is what to do and what to skip. Do take the top five to ten models on MTEB's retrieval split as your candidate set, because a high rank reliably signals broad competence. Do build a frozen, graded, traffic-grounded eval and let it make the final call. Do treat contamination as the default suspicion whenever leaderboard order and eval order disagree. Skip agonizing over sub-point leaderboard differences, skip binary recall as your only metric, and skip any decision that rests on a "current" leaderboard snapshot, since the ordering churns enough that a top-5 model from six months ago is rarely top-5 now. For where all of this fits in a full retrieval stack, our RAG systems pillar guide maps the connected decisions from chunking through reranking.

The leaderboard is not lying about competence; it is lying if you read it as production recall. The models near the top are strong, and any of them can be the right pick, but which one is right for your corpus is a question only your own graded eval can answer. When we build retrieval systems, that eval is the first artifact we stand up and the last authority we defer to, and it is exactly the harness Particula Tech builds for clients who need to choose on the recall their users will feel rather than the rank a public benchmark advertises. Build the eval, freeze the test set, and let your traffic, not the leaderboard, pick the model.