Contextual Retrieval vs Late Chunking: Which One to Use

Contextual retrieval, published by Anthropic on 19 September 2024, is a prompt. For every chunk you send the whole document plus that chunk to an LLM and ask for a situating sentence:

<document>{{WHOLE_DOCUMENT}}</document>
Here is the chunk we want to situate within the whole document
<chunk>{{CHUNK_CONTENT}}</chunk>
Please give a short succinct context to situate this chunk within the overall
document for the purposes of improving search retrieval of the chunk. Answer
only with the succinct context and nothing else.

The 50 to 100 tokens that come back get prepended to the chunk, and the concatenation is what you embed and index for BM25. There is no endpoint, parameter or SDK method for this. It is a technique plus a cookbook notebook, so the implementation quality is entirely yours.

Late chunking, published two weeks earlier as arXiv:2409.04701 and revised through July 2025, changes nothing about the text. You run the whole document through the encoder first, apply your chunk boundaries to the resulting token embeddings, then mean-pool each span. No additional training, no extra model, no rewriting. Your boundaries stay where you put them.

It works because an encoder-only transformer has a fully connected attention matrix: every token representation in the window is already conditioned on every other token there. Pool a span out of that and the vector carries the document's context without anyone writing it down. The same property explains the constraint: an endpoint returning one pooled vector per input has already thrown away the intermediate state the technique needs.

The published illustration is a Wikipedia article about Berlin, where later chunks say "the city" rather than naming it. Cosine similarity to the query "Berlin" moves from 0.708 to 0.825 on the second chunk and 0.753 to 0.850 on the third. The first chunk, which names Berlin outright, barely moves: 0.849 to 0.850. That is the shape of the technique. It pays where the pronoun is, and nowhere else.

One caveat governs the table. These two families have almost never been run on one eval set by one set of authors: Anthropic's numbers come from an internal evaluation that does not mention late chunking, and the late chunking paper does not mention contextual retrieval. Two third-party papers put them on a shared benchmark, and they disagree about the size of the gap.

Read the two head-to-heads as bounds, not as one answer. The ConTEB evaluation used Qwen-2.5-7B-Instruct served with vLLM on an 80GB A100 as the contextualiser and modernbert-embed-large as the embedder, and found an 11.4-point average gap. The NFCorpus comparison (arXiv:2504.19754) used Phi-3.5-mini-instruct in 4-bit on an RTX 4090 and found 0.008 nDCG@5, which rounds to nothing. Its conclusion is the useful sentence: contextual retrieval preserves semantic coherence more effectively but needs greater computational resources, while late chunking is more efficient and tends to sacrifice relevance.

Anthropic's own figures need unpacking every time they are quoted. At top-20 chunks: contextual embeddings alone cut retrieval failures by 35%, from 5.7% to 3.7%. Adding contextual BM25 took that to 49%, or 2.9%. Adding a reranker over an initial top-150 retrieval reached 67%, or 1.9%. The 67% that circulates as "contextual retrieval" includes hybrid search and a commercial reranker, both of which you should be running anyway. If you are deciding whether the LLM rewriting pass earns its keep, the number is 35%, and the other two components have their own posts: hybrid dense and sparse embedding search and when you actually need a reranker in RAG.

Late chunking's own cross-dataset result is similarly modest. Averaged over SciFact, NFCorpus, FiQA and TRECCOVID across three encoders, the paper reports 54.3 nDCG@10 against 52.4 for naive chunking in the sentence-boundary setting: 1.9 points absolute, which the authors frame as 3.63% relative. The double-digit gains quoted elsewhere trace back to one dataset in a single-model evaluation, NFCorpus, 23.46 to 29.98.

Late chunking is not a free upgrade over naive chunking, and three published results make that concrete. Quora came out identical at 87.19 nDCG@10. ConTEB's COVID-QA subset fell 21 nDCG@10 points against plain non-contextualised embedding. And on MSMarco with Stella-V5, early chunking scored 0.630 nDCG@5 and 0.501 MAP@5 while late chunking scored 0.503 and 0.340, a 20% relative drop.

The pattern behind all three is document length. The technique conditions each chunk's vector on the other tokens inside the same encoder window, so a document that is one or two chunks long offers almost nothing to condition on. Short-passage corpora are the worst case: you pay the long-context forward pass and get a vector slightly noisier than the one you would have got by embedding the passage alone.

The second constraint is the encoder's own context window. When a document is longer than the window you cannot embed it in one pass, and the paper's answer is what it calls long late chunking: split into macro chunks of l_max tokens with an overlap parameter, process them sequentially, then pool by chunk boundary as usual. The macro chunk is capped by the encoder, 8,192 tokens for the paper's models. A very long document is therefore several separate encoder passes, and any relationship reaching past the overlap is out of reach exactly as it would have been without late chunking.

Say plainly what late chunking is not. It does not move your boundaries, it is not sliding-window chunking, and it is not overlap by another name. If your splitter cuts a table in half, both techniques inherit that cut. Contextual retrieval can at least paper over it with a sentence describing what the fragment belongs to. Late chunking cannot, because it never touches the text.

Anthropic's published $1.02 per million document tokens is stale by roughly a factor of four. It was computed on claude-3-haiku-20240307, deprecated on 19 February 2026 and retired on 20 April 2026, with claude-haiku-4-5-20251001 as the documented replacement. Any 2026 article quoting $1.02 as a current price is pricing a model you cannot call.

Here is the same arithmetic on the Claude Haiku 4.5 rate card as of 13 August 2026, using Anthropic's stated assumptions: 800-token chunks, 8,000-token documents, 50 tokens of instruction and 100 tokens of generated context per chunk, with prompt caching so the source document is written to cache once and read once per chunk. Ten chunks per document.

That is $3.94 per million document tokens, or about $1.97 through the Batch API, which discounts input and output 50% and stacks with the caching multipliers. This is our calculation, not an Anthropic number. Re-run it on your own numbers: document length drives the cache-write share, and generated-context length drives the output share, which at $5 per million is a sixth of the total.

The number that bites is not the first index, it is the second. Late chunking re-indexes for the price of a forward pass. Contextual retrieval re-indexes at the full LLM cost unless you cache the generated strings and key them by document version, which most reference implementations do not do for you. Cache them and your context sentences are pinned to an older model and an older document revision. Do not cache them and every chunk-size change, every embedding-model swap and every weekly corpus refresh re-runs the entire pass.

Storage is where late chunking wins without argument. Neither technique adds a vector: 100,000 documents at 16 chunks each is 1.6 million vectors either way, roughly 4.9 GB. Late chunking adds no stored text on top of that. Contextual retrieval adds 50 to 100 generated tokens per chunk to whatever you persist, and you do have to persist them, because you need them again on re-embed.

For regulated corpora the binding constraint is not price. Contextual retrieval sends every document you own to a text-generation model, in full, once per chunk. If that model is hosted by a third party, this is a bulk export of the entire knowledge base described as an indexing job, and in most finance, healthcare and defence-supplier reviews that ends the conversation regardless of retrieval scores.

On-premise the question becomes GPU hours rather than dollars. Taking the published 1890.94 ms per document at face value, contextualising a million documents is about 525 A100-hours, roughly 22 days on one 80GB A100 running flat out. Late chunking at 15.81 ms per document is about 4.4 hours for the same corpus. One is a capacity and scheduling problem, the other is an afternoon. Embedding-serving economics on your own hardware are a separate calculation, worked through in self-hosted versus API embeddings.

Sizing the contextualiser rests on two published data points: an 80GB A100 running Qwen-2.5-7B-Instruct under vLLM, and roughly 20GB of VRAM running Phi-3.5-mini-instruct in 4-bit on an RTX 4090. Both are research setups, neither tells you what a 70B contextualiser costs, and no published run pairs a frontier contextualiser against late chunking on one benchmark. Do not read the ConTEB gap as what a better contextualiser would produce.

Check the licence before the benchmark. The two models most associated with late chunking carry cc-by-nc-4.0 on their Hugging Face cards: jinaai/jina-embeddings-v3 and jinaai/jina-embeddings-v5-text-small. Non-commercial means non-commercial, and a bank running either in production is out of compliance however good the numbers are. The jinaai/jina-embeddings-v4 card declares no licence field at all, its own procurement problem. Commercially usable long-context encoders the pooling trick works on include nomic-ai/nomic-embed-text-v1, nomic-ai/modernbert-embed-base, lightonai/modernbert-embed-large and Alibaba-NLP/gte-modernbert-base, all Apache-2.0.

One question belongs to your compliance team rather than your architects. Contextual retrieval stores LLM-written sentences inside a record that will be retrieved, quoted back to users and quite possibly disclosed: derived data with no human author and no review step, in the same field as the source text. Nothing published covers how that interacts with document-level access control, or with an erasure request against a source document once the derived sentence exists. Ask before you index 40 million chunks, not after.

On the Jina Search Foundation API, late_chunking is a nullable boolean on the v3 and v4 request schemas, documented as concatenating all inputs and processing them as one sequence before splitting. Send your chunks as an ordered list in one request:

curl https://api.jina.ai/v1/embeddings \
  -H "Authorization: Bearer $JINA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "jina-embeddings-v3",
    "task": "retrieval.passage",
    "late_chunking": true,
    "truncate": true,
    "dimensions": 1024,
    "input": ["chunk one text", "chunk two text", "chunk three text"]
  }'

Two operational notes. truncate defaults to false, so without it an oversized batch returns INPUT_TOKEN_LIMIT_EXCEEDED with a 400 rather than a partial result, which matters because late chunking wants big batches by construction. And the ceiling that stops a bulk index is tokens, not requests: 1M tokens per minute on the free tier against 500 requests per minute, then 10M and 100M TPM on tiers 1 and 2.

Self-hosted, the shortest path is chonkie's MIT-licensed LateChunker, which needs sentence-transformers:

# pip install "chonkie[st]"
from chonkie import LateChunker

chunker = LateChunker(
    embedding_model="nomic-ai/modernbert-embed-base",  # the default, Apache-2.0
    chunk_size=2048,
    min_characters_per_chunk=24,
)
chunks = chunker.chunk(document_text)

The constructor accepts only a string or a SentenceTransformerEmbeddings instance and raises ValueError on anything else, and chunk() returns the base Chunk type rather than a distinct late-chunk class.

For evaluation, the reference implementation is the Apache-2.0 jina-ai/late-chunking repository, which has had no commit since December 2024. Treat it as a paper artifact, not a dependency. Its entrypoint takes these flags:

python run_chunked_eval.py \
  --model-name jinaai/jina-embeddings-v2-small-en \
  --strategy fixed \
  --chunk-size 256 \
  --task-name SciFactChunked \
  --eval-split test \
  --long-late-chunking-embed-size 8192 \
  --long-late-chunking-overlap-size 256

--long-late-chunking-embed-size defaults to 0, which disables long late chunking, so a document longer than the model's window is truncated unless you set it. --strategy accepts semantic, fixed or sentences, and --n-sentences (default 5) applies only to the last.

On the contextual retrieval side, LlamaIndex ships DocumentContextExtractor in llama_index.core.extractors:

from llama_index.core.extractors import DocumentContextExtractor

extractor = DocumentContextExtractor(
    docstore=docstore,
    llm=llm,                     # required in practice, see below
    max_context_length=64000,    # the default is 1000
    max_output_tokens=256,
    key="context",
    prompt=DocumentContextExtractor.SUCCINCT_CONTEXT_PROMPT,
    oversized_document_strategy="warn",
)

Three details from the source, each worth a debugging session. The llm parameter is typed optional with a None default, but the constructor asserts hasattr(llm, "achat") before falling back to the global setting, so passing nothing raises. max_context_length defaults to 1000 tokens while the class docstring's own example passes 64000, and no oversize strategy truncates: warn logs and sends the document anyway, ignore sends it silently, only error stops the run. And nodes that already carry the metadata key are silently skipped, which is convenient for resuming a partial index and confusing when you change the prompt and expect a re-run. The class exposes ORIGINAL_CONTEXT_PROMPT, Anthropic's wording, and SUCCINCT_CONTEXT_PROMPT, which asks for comma-separated keywords with pronouns resolved.

Start where Anthropic starts: if your knowledge base is under about 200,000 tokens, roughly 500 pages, put the whole thing in the prompt and skip both techniques. A large share of regulated-industry pilots sit under that line and are building a retrieval pipeline they do not need. The ceiling on that advice is accuracy rather than context size, which is the subject of long-context degradation and context rot.

Above it, document length, corpus stability, re-index frequency and deployment boundary decide:

That last row is the one most teams should act on. Contextual embeddings alone bought 35%; hybrid search and a reranker on top bought the rest of the way to 67%. If BM25 does not yet sit alongside your dense vectors, or the generator gets whatever the ANN index returns without a reranking pass, those two changes are cheaper, better understood and helpful on every corpus shape. Neither technique here removes the need for either.

This week, do two things. Compute the median and 90th-percentile token length of your documents, because that one measurement disqualifies an option for most corpora. Then pick 50 queries where you already know the right passage and score nDCG@10 for naive chunking, late chunking and contextual retrieval on the same 50, using the harness in RAG evaluation and retrieval accuracy testing. The two published head-to-heads disagreed with each other about the size of the gap, which tells you the gap belongs to the corpus and not to the techniques. Fifty labelled queries is a morning's work and it beats every number in this post, including ours. The rest of the pipeline around it is mapped in our RAG systems pillar.