Visual RAG vs OCR: ColPali for PDFs, Tables, Charts

The OCR-then-chunk pipeline drops answers whenever meaning lives in layout or pixels rather than in a linear stream of text: multi-column tables, charts, diagrams, checkboxes, stamps, and scanned handwriting. OCR gives you characters, not structure, and everything downstream inherits that loss.

The standard pipeline runs in four stages. A parser or OCR engine converts each page to text, a chunker splits that text into passages, an embedding model turns each passage into a vector, and a retriever finds the closest vectors to the query. Every stage assumes the document is fundamentally a sequence of words. That assumption holds for a contract and breaks for a financial statement.

Consider what happens to a table. OCR reads it left to right, top to bottom, and emits a flat run of numbers with the column and row relationships gone. "Q3 revenue" and the figure that belongs to it end up in different chunks, or in the same chunk with no signal about which cell pairs with which header. A chart is worse: the data lives in the geometry of bars and axes, so OCR returns axis labels and a legend while the actual values never become text at all. A scanned form compounds it, because OCR error on degraded scans corrupts exactly the fields that matter.

None of this is a parser-quality problem you can fix by swapping engines. Even the best modern parsers (we cover the landscape in our guide to document parsing for RAG with Reducto, LlamaParse, and Docling) still have to make an irreversible choice: reduce a two-dimensional page to a one-dimensional token stream. Whatever meaning lived in the second dimension is gone before the chunker sees it, and no amount of clever chunking to preserve context recovers information that the OCR step already discarded. That is the silent failure. Retrieval quality looks fine on your text-heavy eval set and collapses on the visual documents nobody benchmarked.

There are three practical patterns for retrieving over visual documents, and they trade accuracy against storage and operational cost. From lightest to heaviest: caption-and-index, single-vector multimodal embeddings, and page-as-image late interaction.

Caption-and-index keeps your existing text pipeline and bolts a vision-language model onto the front. For every image, chart, or figure, you generate a text caption ("bar chart of quarterly revenue, Q3 highest at roughly 40M"), then index that caption alongside the OCR'd text. It is cheap and needs no new infrastructure. It is also the weakest, because a caption is a lossy summary written before anyone knows the query. On ViDoRe, the best caption-based baseline (Unstructured plus captioning plus BGE-M3) reached only 67.0 nDCG@5, well behind visual approaches.

Single-vector multimodal embeddings embed the whole page (text and visuals together) into one dense vector, then retrieve with a normal nearest-neighbor search. This is the operationally simplest true visual approach: one vector per page, so your vector database and query path barely change. Cohere Embed 4 and voyage-multimodal-3 are the production options here, and the broader case for treating vision and text in one system is something we unpack in multimodal AI with vision and text models for business.

Page-as-image late interaction is ColPali's approach and the most accurate. Instead of compressing a page to one vector, it keeps many vectors per page (one per image patch) and scores queries against all of them at search time. That granularity is why it wins on hard documents, and the multi-vector storage is why it costs the most. Choosing between these three is partly an embedding-model decision, and our guide on which embedding model to use for RAG covers the single-vector side of that choice in depth.

ColPali beats OCR-based retrieval by 14.3 to 15.2 nDCG@5 points on ViDoRe, depending on which baseline you compare against, a relative gain of roughly 21% to 23%. According to the ColPali paper (Faysse et al., ICLR 2025), ColPali reaches 81.3 average nDCG@5 on ViDoRe, versus 67.0 for the strongest text baseline and 66.1 for an OCR-plus-BGE-M3 dense retrieval pipeline.

ColPali is a document retrieval model that embeds each page as an image and scores queries against image patches with late interaction, skipping OCR entirely. Architecturally, it runs a SigLIP-So400m/14 vision encoder inside PaliGemma-3B. Each page is split into a 32x32 grid, producing 1,024 patch embeddings, and each embedding is projected down to 128 dimensions. At query time, the model embeds the query tokens and scores them against every page's patches using late-interaction MaxSim.

Late interaction is a retrieval scoring method that keeps one embedding per token or patch and compares every query token against every document token at search time, rather than compressing each side to a single vector first. MaxSim is the specific scoring function: for each query token, take the maximum dot-product against all of the page's patch vectors, then sum those maxima across query tokens. The effect is that each part of the query can "find" the patch of the page that best answers it, which is exactly what you want when the answer is a single cell in a table or one bar in a chart.

Here is the full ViDoRe V1 picture from the paper, which is more useful than any single delta:

Two things stand out. First, the jump from raw zero-shot SigLIP (51.4) to fine-tuned ColPali (81.3) shows that the architecture matters as much as the vision backbone: the late-interaction training is doing real work. Second, the text baselines cluster tightly between 59.6 and 67.0 regardless of whether they use OCR, captioning, BM25, or a dense retriever. That clustering is the tell. Once you have flattened the page to text, the retrieval method you pick barely moves the ceiling, because the ceiling was set by what OCR threw away.

One honest caveat on the benchmark itself. According to the ViDoRe V2 benchmark (Macé et al., 2025), V1 had saturated with top models exceeding 90% nDCG@5, which is why V2 adds harder, more realistic retrieval across four multilingual datasets. As single-vector multimodal models have matured, the gap to late-interaction models has narrowed on easier documents, while the hardest visual-reasoning pages remain where multi-vector approaches earn their cost. Treat the 81.3-vs-66.1 delta as directionally solid and document-dependent, not a fixed law.

The price of ColPali's accuracy is storage and search cost: about 1,024 vectors per page instead of one, and a MaxSim comparison that touches all of them at query time. A multi-vector embedding is exactly that, many vectors stored per document instead of a single compressed one.

The storage math is blunt. A conventional pipeline stores one embedding per chunk, often 1,024 or 1,536 dimensions. ColPali stores 1,024 vectors of 128 dimensions per page, which is 131,072 values, on the order of 100x more raw vector data per page before any compression. For a 10,000-page corpus that is the difference between a vector index you barely notice and one that dominates your infrastructure bill. It is the central operational fact of the approach, not a footnote.

Latency scales with it. A single-vector search is one dot product per candidate. MaxSim is a max-and-sum across 1,024 patch vectors per candidate page, per query token. Naively, that is expensive enough to matter at scale. Three mitigations make it production-viable:

Vector databases have caught up to this. Qdrant supports multi-vector ColPali and ColQwen2 indexes with MaxSim scoring natively, which is where most teams run these models rather than building the storage layer themselves. ColQwen2, for the record, is a ColPali-family late-interaction retriever built on the Qwen2-VL-2B vision-language model, part of the ColVision model line, and it stores and scores the same multi-vector way.

Multimodal RAG has three recurring failure modes we see across production systems, and naming them helps you diagnose a pipeline that retrieves the wrong page. We label them caption drift, modality leakage, and dominant-modality bias. These are our own taxonomy, but they line up with the systematic failure modes that document-centric benchmarks now report.

Caption drift is when a generated caption or extracted summary describes an image in a way that loses the specific detail a query needs, so the index no longer matches the real content. A caption that says "revenue chart" cannot answer "what was Q3 gross margin," because the number was in the pixels, not the sentence. This is the fundamental weakness of the caption-and-index pattern, and it gets worse the more the caption model tries to be concise.

Modality leakage is when retrieval scores are dominated by whichever modality embeds most cleanly, so a strong text match on an irrelevant caption outranks the actual chart that holds the answer. You see it in fused pipelines where a page's OCR text and its image are both indexed: a keyword-heavy but off-topic text region beats the correct figure on pure embedding similarity.

Dominant-modality bias is the systematic version of that leakage: a joint or fused retriever that consistently prefers one modality, usually text, regardless of where the answer actually lives. It is a bias baked into the retriever, not a per-query accident, and it is the reason "just fuse text and image scores" often underperforms a clean single approach.

According to UNIDOC-BENCH (Salesforce AI Research, 2025), the document-centric benchmark built to expose exactly these problems, the evaluation spans roughly 70,000 real-world PDF pages across eight domains and 1,600 multimodal QA pairs, comparing four retrieval paradigms head to head: text-only, image-only, text-image fusion, and joint multimodal retrieval. When we build RAG pipelines for enterprise clients with mixed corpora, the first thing we do is separate the visual documents from the text ones and evaluate them independently, because a single blended accuracy number hides all three failures.

The OCR pipeline still wins when documents are clean, text-dense, and high-volume, and when storage, latency, and the availability of extracted text matter more than the last few points of accuracy on visual content. This is a large share of real corpora, so do not throw OCR out.

Text is the faithful representation for contracts, policies, transcripts, manuals, code, and most born-digital prose. For those, a single embedding per chunk retrieves accurately at roughly 1/100th the storage of a multi-vector page index, and the retrieval-method clustering in the ViDoRe table (59.6 to 67.0 across every text approach) tells you the visual model would buy you little on genuinely textual documents.

There is also a capability the page-as-image approach does not give you for free: the extracted text itself. If your application needs to highlight the matching passage, redact sensitive fields, or pull structured values into a database, you need the text, and OCR produces it as a byproduct. A pure ColPali index retrieves the right page image but hands you no characters, so you often run OCR anyway on the retrieved page. At very high ingestion volume, the multi-vector storage and MaxSim latency can also be disqualifying on their own. The honest framing is not "visual RAG replaces OCR" but "visual RAG is the right tool for the document types where OCR loses information, and OCR remains the default everywhere else."

Choose your pipeline per document type, not per corpus: default to OCR plus dense retrieval, and switch to visual RAG for the specific document classes where OCR measurably drops answers. Trying to pick one global approach for a mixed corpus is how you end up with a system that is mediocre on both.

Here is the decision guidance we apply:

The opinionated version: for peak accuracy on hard visual documents, use ColPali or ColQwen2 with a two-stage retrieval setup (single-vector first pass, MaxSim rerank) on a store like Qdrant, and budget for the storage. When you want one index for a mixed corpus without operating a multi-vector store, use Cohere Embed 4, which per Cohere's April 2025 launch embeds up to a 128K-token context (around 200 pages) into a single Matryoshka vector you can truncate from 1,536 down to 256 dimensions for cost. Skip the caption-and-index pattern as a primary strategy; at 67.0 ViDoRe it is the weakest option, though it is a fine cheap supplement. And watch the frontier: page-as-image retrieval is generalizing beyond PDFs. According to PixelRAG (Berkeley SkyLab, 2026), rendering even web pages to screenshot tiles and embedding them with a LoRA-tuned Qwen3-VL-Embedding-2B model improves retrieval accuracy by up to 18% over text-based baselines on SimpleQA.

If you are staring at a corpus of scanned forms, financial PDFs, and slide decks and wondering whether to rip out your OCR stack or bolt a vision model beside it, that per-document-type call is exactly what Particula Tech's RAG architecture audits pressure-test: we benchmark visual and OCR retrieval on your own documents so the decision is grounded in your accuracy and cost numbers, not a public leaderboard. The broader map of where these choices fit lives in our RAG systems pillar guide. Start by measuring retrieval quality on your visual documents separately from your text ones. If the two numbers are far apart, you already know which pages need to become images.