How to Chunk Documents for RAG Without Losing Context

Why Arbitrary Chunking Breaks RAG Systems

Most RAG implementations split documents using fixed token counts—512 tokens per chunk, 1024 tokens per chunk—whatever fits the embedding model's context window. This approach treats documents like they're uniform text blocks that can be divided anywhere. They're not.

When you split a policy document at token 512, you might break mid-sentence: "Employees are entitled to remote work privileges" ends up in chunk 1, while "provided they maintain availability during core business hours and complete the required paperwork" lands in chunk 2. Someone searches for remote work policies and gets back incomplete information that's technically accurate but functionally wrong.

The damage compounds when dealing with complex documents. Technical specifications reference earlier sections. Legal documents have clauses that modify previous statements. Financial reports present data in tables that need surrounding explanation. Split these at arbitrary boundaries and you lose the relationships that make information useful.

Companies I've worked with discover this during user acceptance testing. Their RAG system works perfectly in controlled tests with simple queries but falls apart with real questions. A healthcare provider's AI assistant would retrieve medication dosage information without the accompanying contraindications listed in the next paragraph. The chunking strategy made this inevitable—they optimized for uniform chunk sizes instead of semantic completeness.

Semantic Chunking: Following Document Structure

The solution isn't smaller chunks or bigger chunks—it's smarter chunks. Semantic chunking splits documents at natural boundaries where meaning actually breaks: paragraphs, sections, logical topic shifts. This approach preserves the internal coherence that makes text understandable.

Start by respecting document structure. If you're processing reports with clear section headings, those headings signal topic boundaries. Split there. Each chunk becomes a semantically complete unit about a specific topic, not an arbitrary slice of tokens.

For documents without explicit structure, identify logical breaks. Paragraph boundaries work for most prose. Topic sentences often signal shifts in meaning. In technical documentation, code blocks and their explanatory text should stay together. In legal documents, keep clauses with their modifiers and exceptions.

I worked with a financial services company that switched from fixed-token chunking to semantic chunking based on report sections. Their investment analysis RAG system went from retrieving disconnected statistics to returning complete analyses with context. The improvement wasn't better embeddings or a faster database—it was simply keeping related information together.

Implementation requires parsing document structure before chunking. For PDFs, extract heading hierarchies. For HTML, use the DOM structure. For plain text, identify paragraph breaks and topic transitions. Most document processing libraries (PyMuPDF for PDFs, BeautifulSoup for HTML) provide structure information you can use to guide chunking decisions.

The trade-off: semantic chunks vary in size. You might have a 200-token chunk followed by an 800-token chunk. That's fine. Consistency in meaning matters more than consistency in length. Your embedding model can handle variable-length inputs, and the retrieval quality improvement justifies the irregularity.

Chunk Overlap Strategies That Preserve Context

Even with semantic chunking, information at chunk boundaries risks context loss. The sentence that starts a new chunk might reference concepts from the previous chunk. Overlap solves this by including portions of adjacent chunks in each segment.

The standard approach: overlap chunks by 10-20% of chunk size. If your chunks average 500 tokens, include the last 50-100 tokens from the previous chunk at the start of each new chunk. This creates redundancy that ensures context carries across boundaries.

But smart overlap does more than duplicate text. Tag overlapping sections with metadata indicating they're shared between chunks. When your retrieval system pulls two adjacent chunks, it can identify the overlap and present them as a continuous passage instead of repeating the overlapping section twice.

For documents with strong hierarchical structure, use hierarchical overlap. Include the section heading in every chunk from that section. If a document has three levels of headings (Part > Chapter > Section), prepend each chunk with all three relevant headings. A user searching for specific information gets back chunks that include the full context path.

I implemented this for a legal tech company processing case law. Each chunk included the case name, year, and relevant section headings as metadata. When their system retrieved passages about precedent, lawyers immediately understood which case and section the information came from, even if the passage itself didn't mention it. The overlap strategy provided navigational context automatically.

The challenge with overlap: it increases storage requirements and processing time. A 1000-chunk document might become 1200 chunks with 20% overlap. For most applications, the accuracy improvement justifies the cost. If storage becomes an issue, reduce overlap percentage but never eliminate it entirely. Even 5% overlap helps preserve continuity at chunk boundaries. When dealing with sensitive information, make sure to review our guide on preventing data leakage in AI applications.

Optimal Chunk Sizes for Different Document Types

No single chunk size works for all document types. Technical specifications need different chunking than customer support transcripts. The optimal size depends on information density and how users query the content.

Handling Tables, Lists, and Structured Data

Standard text chunking breaks down when documents contain structured data. Tables lose meaning when split across rows. Lists get separated from their introductory context. Code blocks become incomprehensible when divided mid-function.

For tables, extract them as complete units with metadata indicating they're structured data. Store table content in a format that preserves structure—CSV for simple tables, JSON for complex nested data. Create a text description of the table for semantic search, but keep the structured data available for precise retrieval.

When users query for information that lives in a table, your RAG system should recognize this and return the structured data, not just text fragments. A query about "Q4 regional sales breakdown" should retrieve the actual table, not a sentence that says "sales varied by region."

For lists, keep the list with its heading or introductory sentence. A bulleted list of product features makes no sense without the sentence "Key features include:" that precedes it. Treat the introduction and list as a single semantic unit.

Code blocks require special handling in technical documentation. Keep code with its explanation. If a document shows example code followed by line-by-line analysis, that entire section stays together. Developers searching your documentation need to see both the code and the explanation, not one without the other.

I consulted with a SaaS company whose product documentation mixed explanatory text, code examples, and configuration tables. We implemented specialized chunking: text paragraphs used semantic chunking, code examples stayed complete with their surrounding explanation (even if large), and configuration tables were extracted as structured data with searchable descriptions. Their developer support tickets dropped because the AI documentation assistant started returning complete, useful information instead of text fragments.

Preserving Document Hierarchy in Chunks

Documents have hierarchical structure—chapters contain sections, sections contain subsections, subsections contain paragraphs. Flatten this structure during chunking and you lose critical context about how information relates.

Preserve hierarchy by encoding it in metadata. Each chunk should know its position in the document structure: which section it belongs to, what level heading it falls under, whether it's part of a nested list or table.

For a business report with structure like "Executive Summary > Financial Performance > Revenue Growth", every chunk from the Revenue Growth section should carry metadata: section_1: "Executive Summary", section_2: "Financial Performance", section_3: "Revenue Growth". When your retrieval system finds relevant chunks, it can group them by section and present them with appropriate context.

Hierarchical metadata enables smarter retrieval strategies. You can retrieve at different levels: find all chunks from "Financial Performance" when users ask broad questions, or narrow down to specific subsections for detailed queries. This multi-level retrieval isn't possible without preserving document structure.

I implemented this for an enterprise knowledge base that contained hundreds of policy documents with complex nested structures. Each chunk included full hierarchical metadata showing its path through the document structure. When employees queried policies, the system could answer "According to Section 3.2.1 of the Remote Work Policy..." because we preserved that structural information. Users trusted answers more when they understood exactly where information came from. For more on source attribution, see our guide on how to fix RAG citations.

Metadata Enrichment for Context Preservation

Chunks need metadata that provides context lost during splitting. Beyond document structure, capture information that helps retrieval systems understand what each chunk contains and how it relates to other chunks.

Essential metadata for every chunk includes document title, creation date, author, document type, chunk position (chunk 15 of 200), the section heading, and preceding context. This metadata serves two purposes: helps filter and rank retrieval results, and provides context when presenting chunks to users or LLMs.

Add domain-specific metadata based on your content. Medical documents need patient demographics and case types. Legal documents need case names and jurisdiction. Financial documents need fiscal periods and report types. This specialized metadata enables filtering that dramatically improves retrieval precision.

Create chunk summaries as metadata. Before embedding each chunk, generate a 1-2 sentence summary using an LLM. Store this summary alongside the full text. During retrieval, you can search both the full chunk and its summary, improving semantic matching for queries that use different terminology than the source document.

Link related chunks through metadata. If chunks 45 and 46 are sequential, store those relationships: previous_chunk: 45, next_chunk: 46. For documents with cross-references, capture those: references: [12, 89, 134]. This turns your chunk collection into a graph of related information that retrieval systems can navigate intelligently.

Testing Chunking Strategies With Real Queries

The only way to validate your chunking approach: test it against real user queries. Create a test set of 50-100 questions that represent actual searches against your documents. For each question, identify which chunks should be retrieved to answer it correctly.

Run retrieval tests with different chunking strategies: fixed token counts at various sizes, semantic chunking with different overlap percentages, specialized handling for structured content. Measure retrieval accuracy—how often does each strategy return the correct chunks in the top-5 results?

Pay attention to questions that require information from multiple chunks. If you ask "What are the prerequisites and deadlines for the remote work program?", the answer might span three paragraphs in the original document. Does your chunking strategy keep that information in adjacent, related chunks that can be retrieved together? Or does it scatter pieces across unrelated chunks?

Test edge cases specifically: questions about table contents, queries that reference list items, searches for code examples, questions about exceptions and caveats that modify earlier statements. These reveal whether your chunking preserves the structural and semantic relationships that matter for your domain.

I've run these tests dozens of times across different implementations. The pattern holds: semantic chunking with 15-20% overlap consistently outperforms fixed-token chunking for complex documents. The improvement ranges from 25% to 50% better retrieval accuracy, depending on document type and query complexity. For more details on model selection, see our guide on which embedding model to use for RAG and semantic search.

When to Re-Chunk Documents in Production

Chunking isn't a one-time decision. As your document collection grows and user query patterns evolve, you'll need to adjust your strategy.

Re-chunk when retrieval metrics degrade. If your RAG system's accuracy drops over time, the problem might be that new document types need different chunking than your original approach. Monitor precision and recall metrics, and watch for patterns in failed queries.

Re-chunk when document formats change. If you initially processed plain text documents and later added PDFs with complex layouts, your chunking strategy probably needs updating to handle the new structure.

Re-chunk when user feedback indicates context problems. If users consistently report that retrieved information feels incomplete or out of context, your chunks might be too small or splitting at inappropriate boundaries.

Re-chunk when you upgrade embedding models. Different models have different optimal input lengths and context handling. When switching from a model with 512-token context to one with 8000-token context, you can use larger chunks that preserve more context in single units. For more on this, see our article on long context LLMs and performance issues.

The practical challenge: re-chunking and re-embedding large document collections takes time and compute resources. Plan for this during system design. Build your document processing pipeline so you can re-run chunking without rebuilding everything. Store original documents separately from processed chunks so you can regenerate chunks with new strategies without re-ingesting source files. Learn more about when to re-embed documents in your vector database.

Chunking for Multi-Modal Documents

Documents increasingly contain multiple modalities—text, images, charts, diagrams, embedded videos. Standard text chunking doesn't work for these hybrid documents.

For images and diagrams within documents, extract them separately and generate text descriptions using vision models. Create chunks that include both the image reference and the generated description. Store the actual image alongside the text chunk so your system can present both to users.

For charts and graphs, extract the underlying data if possible and store it as structured data. Generate a text description of what the visualization shows. During retrieval, return both the description and the original chart image.

For documents with embedded videos or audio, transcribe the media and chunk the transcription using the same semantic approach as text. Link transcription chunks back to timestamps in the original media so users can jump to relevant sections.

Handle mixed-modality pages (text with embedded images and tables) by segmenting them into separate chunks by modality but maintaining linkage. A page with two paragraphs, an image, and a table becomes four related chunks with metadata indicating they're from the same page and their sequence.

Building a Robust Chunking Pipeline

A production-ready chunking system needs more than just a splitting algorithm. Build a pipeline that handles document preprocessing, chunk generation, metadata enrichment, validation, and version control.

Your pipeline should parse documents to extract structure, clean and normalize text, identify semantic boundaries for chunking, generate overlapping segments, enrich chunks with metadata, create searchable summaries, validate chunk quality, embed chunks for vector storage, and store chunks with versioning information.

Implement quality checks at each stage. Flag chunks that are too short (might be incomplete) or too long (might need further splitting). Detect chunks with malformed text from parsing errors. Identify chunks that lost critical context.

Build monitoring into the pipeline. Track chunk size distributions, measure processing time per document, log errors and edge cases, and monitor retrieval performance over time.

Make the pipeline rerunnable and versioned. When you improve chunking logic, you need to reprocess documents. Track which chunking version created each chunk so you can gradually migrate to new strategies without breaking existing functionality.

Chunk Smarter to Retrieve Better

Document chunking determines whether your RAG system retrieves meaningful information or disconnected text fragments. Fixed-token chunking might be simple to implement, but it destroys the semantic relationships that make documents useful. Semantic chunking, intelligent overlap, and metadata enrichment preserve context and enable accurate retrieval.

The key insight: treat documents as structured, meaningful content, not as uniform text to be divided arbitrarily. Respect document structure, maintain hierarchical relationships, preserve tables and lists as complete units, enrich chunks with contextual metadata, and test against real queries to validate your approach.

Most RAG implementations spend enormous effort optimizing embeddings and vector databases while using naive chunking strategies. That's backward. Start with sophisticated chunking that preserves context. Everything else becomes easier—better embeddings have better input to work with, retrieval systems find more relevant results, and users get answers that actually make sense.

If your RAG system returns fragmented information or misses context, audit your chunking strategy before anything else. The problem usually starts there, and fixing it delivers immediate, measurable improvements in retrieval accuracy.