Why Metadata Filters Collapse Your Vector Search Recall

The single most dangerous word in this topic is selectivity, because it is used in opposite directions by different sources and neither camp flags it.

A February 2026 cross-engine preprint from the University of California, Merced defines it formally (arXiv 2602.11443): global filter selectivity quantifies the baseline prevalence of the filter across the entire dataset, computed as the number of vectors satisfying the predicate divided by the total, in the interval (0,1]. A July 2026 vendor benchmark uses the identical definition, "the measured fraction of points passing the filter". Under both, selectivity 0.01 means the filter keeps 1% of your data.

Plenty of practitioner writing runs the term the other way, where a "highly selective" filter is one that eliminates almost everything. The result is that a sentence containing "high selectivity" carries no reliable information at all, and a reader who resolves it backwards will tune their index in exactly the wrong direction.

So no figure below is stated in terms of selectivity. Every one is stated as the concrete percentage of rows that pass the filter. When you write your own runbook, do the same, because the cost of getting it backwards is an index configuration optimized for the regime you are not in.

Every number in this section comes from a July 2026 engineering benchmark published by Qdrant. Read it with the conflict in view. Qdrant measured Qdrant on one machine, configured four ways, with no other engine present, and the configuration that wins is Qdrant's own filterable HNSW beating the ACORN implementation Qdrant also ships. To its credit the post discloses that scope in its own text, along with the version (v1.18.2), the dataset (deep-image-96, 1,000,000 points) and the search parameter (hnsw_ef=64), which is more disclosure than most vendor benchmarks offer. The relative ordering is still a vendor's ordering.

Four configurations, single-field filter, recall at latency:

Read the latency column of the first three rows. 1.6ms, 1.7ms, 1.6ms. The plain graph falls from 62.9% to 0.1% recall between the first and third rows without moving its latency by a tenth of a millisecond. There is no operational signal here. A p99 dashboard, an error rate, a result-count check: all green, all the way down to 0.1%.

The second observation is that the fix is real and cheap. Building the payload index that turns the plain graph into a filterable HNSW build takes recall from 0.1% to 99.8% at the 1% pass rate while lowering latency from 1.6ms to 1.0ms, because the filtered subgraph is smaller to traverse. If you are running a graph index with metadata filters and no payload index, that is the first thing to change.

An unreviewed preprint posted to arXiv in July 2026 (arXiv 2607.00768) reports the same cliff from the other side, stating that ACORN-1 "suffers connectivity instability below 5% selectivity and recall collapse below 1%". It measures ACORN-1 recall at 0.45 to 0.72 when the filter passes 1% of rows and 0.03 to 0.10 at 0.3%, with its own method recovering to 0.70 to 0.96 and 0.77 to 0.98 respectively. Two caveats belong in the same breath: the preprint has no venue and no acceptance note, and those ACORN-1 figures are the authors' own measurement of the baseline their method is designed to beat. The paper treats low pass rates as the hard regime, and its 1% and 0.3% anchors line up with the fraction-passing convention used throughout this post.

Three different execution strategies produce three different failure shapes. Knowing which one your engine picked tells you which number above applies to you.

The fourth row of the table is the finding most teams have never seen, and it inverts a common piece of design advice.

At a filter passing 10% of points, the plain graph scored 20.6% recall when the filter was uncorrelated with the query and 88.4% when it was correlated. Same index, same pass rate, same latency (1.7ms both times), and the entire distance between 20.6% and 88.4% comes from nothing but the relationship between what the user asked and what the predicate keeps.

That reorders the risk assessment for access-control filters. The intuition that tenant filters are dangerous because they are restrictive is half wrong: a tenant's queries are overwhelmingly about that tenant's own documents, which is the correlated, easier case. The predicates that actually hurt are the ones that cut across topic rather than along it:

The unreviewed preprint above evaluated exactly this case, constructing deliberately negative correlation using K-means clusters, and measured ACORN-1 recall falling to 0.08 to 0.41 while its own method held 0.80 to 0.98 with a 5x to 9x latency advantage over HNSW. Treat those as directional given the self-baseline caveat, but the direction is corroborated by the correlated-versus-uncorrelated gap in the vendor table, which is an independent measurement of the same effect.

The operational takeaway for anyone running retrieval behind an authorization boundary: the recall risk lives in the shape of your predicate, not its restrictiveness, so a per-index recall number tells you nothing. Measure per predicate shape.

Payload indexes are built per field. They are not built per combination. The vendor benchmark states the mechanism directly: the engine builds them per field, never per combination, so an AND filter lands on an intersection that no single field's edges cover.

Same four configurations, two keyword predicates combined with AND:

The filterable HNSW column is the story. The configuration that returned 99.8% on a single filter passing 1% of points returns 70.8% on two filters passing 1% combined, and 63.7% at a 4% combined pass rate. Adding a payload index per field and assuming you are covered is a specific, measurable mistake.

The last row is the honest bottom of the curve. At a combined pass rate of 0.012%, roughly 120 points out of a million, three of the four configurations return effectively nothing: 0.5%, 0.6%, 1.8%. Only the planner path, which at that point is doing something other than approximate graph search, returns 100%, and it does so in 1.3ms. That is the clearest argument in the dataset for exact search below a threshold. When the surviving subset is 120 vectors, brute force is not a compromise, it is the fast path.

Nothing above is free, and the costs land in three different budgets.

Build time. From the benchmark: indexing one million points took 171 seconds without payload indexes and 448 to 560 seconds with them, a 2.6x to 3.3x cost. That is index build time on one million points. It is not a memory multiplier, not a storage multiplier, and not a query cost, and it will be misread as one of those if you quote the multiple without the unit. For a nightly rebuild it is noise. For a corpus that reindexes on ingest, it is a capacity plan change, and the cost per QPS math for large vector deployments is where that lands.

Query latency. The ACORN path costs 3.5ms to 4.7ms against 1.0ms to 1.2ms for filterable HNSW on the same rows, roughly 3x to 5x. Quote the raw milliseconds rather than the multiple, which moves depending on which row you pick. The absolute numbers are small, and for most RAG pipelines a 3ms difference is invisible next to generation latency. It stops being invisible inside an agent loop firing dozens of queries per task.

A gate worth copying. The engine's planner applies ACORN only when the filter passes 40% of points or fewer, controlled by a max_selectivity setting. That is the single most decision-relevant configuration number on the page, because it encodes the correct intuition: above roughly 40% pass rate the plain graph is close enough that predicate-aware traversal is paying latency for recall you already have. If you are building your own routing logic, that threshold is a reasonable starting default.

None of the numbers above transfer to another engine. Different implementation, different machine, different dataset. For cross-engine claims the only source in this stack is the UC Merced work, and it is an unreviewed preprint.

Its setup is unusually well specified: FAISS 1.12.0 on CPU with both HNSW and IVFFlat, Milvus v2.6.6, and pgvector 0.8.1 on PostgreSQL 16, over the MoReVec dataset with 768-dimension L2-normalized embeddings at three cardinalities (roughly 10K, 100K and 551K movies, with 247K, 1.5M and 2.6M reviews). The query load is stated as 1000 x 7 x 4 = 28,000 filtered k-NN queries per dataset, sweeping k across 1, 10, 40 and 100 and pass rates down to 0.0003, meaning filters keeping three rows in ten thousand.

Per-cell recall values are not extractable from the published tables, so this post pairs no number with the following. Three qualitative findings from the abstract, quoted because the phrasing matters:

For pgvector specifically, the documented lever is iterative scans. From the project's own documentation: starting with 0.8.0 you can enable iterative index scans, which will automatically scan more of the index until enough results are found. hnsw.iterative_scan accepts strict_order or relaxed_order, where strict ensures results are in the exact order by distance, and IVFFlat has the equivalent ivfflat.iterative_scan with ivfflat.max_probes. Three defaults bound the effort: hnsw.ef_search at 40, hnsw.max_scan_tuples at 20,000, and hnsw.scan_mem_multiplier at 1 (a multiple of work_mem). Hit any of those ceilings on a restrictive predicate and the scan stops with whatever it has found, which is a bounded failure rather than no failure.

-- pgvector: bound the damage on filtered queries
SET hnsw.iterative_scan = strict_order;
SET hnsw.max_scan_tuples = 100000;   -- default 20000
SET hnsw.scan_mem_multiplier = 4;    -- default 1, multiple of work_mem
SET hnsw.ef_search = 200;            -- default 40

Raising these is not a fix, it is a wider net. Verify with the harness below that the net is now wide enough for your worst predicate.

Every source in this stack reports recall against pass rate on public datasets. Not one of them tells you where your own filters sit on that curve. These two artifacts close that gap, and they are the reason the vendor conflict of interest above stops mattering: you are not trusting anyone's benchmark, you are running your own.

-- What fraction of rows does this predicate actually pass?
SELECT
  rows_passing,
  total_rows,
  rows_passing::float / NULLIF(total_rows, 0) AS pass_fraction
FROM (
  SELECT
    count(*) FILTER (
      WHERE tenant_id = $1
        AND doc_type = 'contract'
        AND jurisdiction = 'DE'
    ) AS rows_passing,
    count(*) AS total_rows
  FROM documents
) counts;

import numpy as np

def exact_topk(vectors, mask, query, k):
    """Ground truth: exact top-k over only the rows the predicate passes."""
    ids = np.flatnonzero(mask)
    if ids.size == 0:
        return np.empty(0, dtype=np.int64)
    scores = vectors[ids] @ query          # L2-normalized: dot == cosine
    take = min(k, ids.size)
    top = np.argpartition(-scores, take - 1)[:take]
    return ids[top[np.argsort(-scores[top])]]

def recall_at_k(ann_ids, truth_ids, k):
    if truth_ids.size == 0:
        return 1.0
    hit = len(set(ann_ids[:k]) & set(truth_ids[:k]))
    return hit / min(k, truth_ids.size)

def audit(queries, masks, vectors, ann_search, k=10):
    """Report recall grouped by measured pass rate, not as one average."""
    bands = {}
    n = len(vectors)
    for q, mask in zip(queries, masks):
        pass_rate = int(mask.sum()) / n
        band = next(b for b in (0.4, 0.1, 0.05, 0.01, 0.001, 0.0)
                    if pass_rate >= b)
        truth = exact_topk(vectors, mask, q, k)
        got = ann_search(q, mask, k)
        bands.setdefault(band, []).append(recall_at_k(got, truth, k))
    return {b: (float(np.mean(v)), len(v)) for b, v in sorted(bands.items())}

Two research directions are worth tracking rather than adopting. A January 2026 preprint, Curator (arXiv 2601.01291), takes the partition route with a dual index inside a clustering tree, reporting up to 20.9x latency reduction on low-pass-rate queries at 5.5% construction-time and 4.3% memory overhead. Its filters are labels such as dates and price ranges and it does not address multi-tenancy or access control, so do not recruit it for an authorization boundary. And the July 2026 preprint above adds an adaptive exact fallback, reporting recall 1.00 with 20x to 75x speedup at one million vectors and a pass rate of 0.1% or below, which is the same instinct as the last row of the table: stop approximating when the subset is small enough to enumerate.

Filtered recall is where retrieval quality and access control meet, and in a regulated deployment the failure is not "bad search results". It is a clinician, an underwriter or a case lawyer being told, implicitly and with a straight face, that a document they are entitled to see does not exist. That failure is indistinguishable from a correct empty answer unless somebody measured. If you are combining dense retrieval with keyword predicates, hybrid dense and sparse search changes the shape of this problem without removing it, and engine choice trade-offs are laid out in our Pinecone and Qdrant comparison. The rest of the stack is in the RAG systems pillar.

Particula Tech builds filtered-recall harnesses into on-premise retrieval deployments as a standing gate: pass rates measured per predicate shape from real query logs, ground-truth recall floors per band, and a re-test on every index rebuild. The index configuration is the easy half. Knowing which band your predicates land in is the half nobody runs.