RAG Chunking Strategy: How Chunk Size, Overlap, and Document Structure Affect Recall

When teams say "our retrieval is weak," the problem is often not the retriever first. It is the unit of retrieval. Chunking decides what can be found, what gets diluted, what gets duplicated, and what never reaches the model as one coherent piece of evidence.

That is why chunking is one of the highest-leverage decisions in production RAG. If the answer-bearing unit is split across boundaries, buried inside a huge section, or duplicated so heavily that top-k loses diversity, recall drops before reranking or prompting even gets a fair chance.

This guide explains how chunk size, overlap, and document structure affect recall, where chunking failures show up in traces, and how to choose a strategy with evals instead of intuition.

Why chunking is a recall lever, not an ingestion detail

Retrieval can only return what the index made retrievable. If your chunking strategy does not preserve the unit that actually answers the query, high-quality embeddings and rerankers will still struggle.

In practice, recall in RAG is not only "did we retrieve the right document?" It is often closer to: did we retrieve a chunk that contains the decisive evidence in a usable form?

• Boundary loss: the answer is split across two chunks, so neither chunk is strong enough on its own
• Semantic dilution: the answer is buried inside a very broad chunk with many other topics
• Duplicate saturation: overlap creates too many sibling chunks, reducing top-k diversity
• Structure loss: headings, tables, steps, or exception clauses are detached from the text that gives them meaning

That is why teams sometimes think they have an embedding problem when the real issue is chunk construction.

The three chunking decisions that matter

Most chunking strategies reduce to three choices:

1. Chunk size: how much text belongs in one retrievable unit
2. Overlap: how much neighboring text is repeated across boundaries
3. Document structure: whether splitting respects headings, lists, tables, code blocks, and section hierarchy

These interact. A fixed-size window can look fine on a short FAQ but fail badly on a policy document with nested exceptions. A large chunk can preserve coherence but reduce retrieval focus. Overlap can save clause continuity but flood retrieval with near-duplicates.

Chunk size: too small vs too large

When chunks are too small

Small chunks often improve targeting for narrow questions, but they break easily when answers depend on qualifiers, exceptions, or sequence.

Common failure modes:

• policy rule is retrieved without the exception that changes the answer
• parameter description is separated from allowed values or default behavior
• troubleshooting step is retrieved without prerequisites or warnings
• a table row is separated from the header that defines each column

In those cases, topic-level recall can look acceptable while full-answer recall is poor. The system finds the topic but not the complete evidence.

When chunks are too large

Large chunks preserve more surrounding context, but they often weaken retrievability. One vector now represents several ideas at once. The answer-bearing span becomes less salient inside the embedding, and the model may receive broad context with weak local precision.

Common failure modes:

• top-k returns broad sections that mention the topic but not the answer
• reranking has to work harder to isolate the right passage
• final context includes unnecessary text, raising token cost and contradiction risk
• specific literals like SKUs, version constraints, or error codes get buried inside generic prose

Overlap: what it preserves, and what it breaks

Overlap exists to reduce boundary damage. It helps when the useful unit crosses a split point: long sentences, section intros, parameter blocks, exception clauses, or multi-step instructions.

But overlap is not free. Heavy overlap creates sibling chunks that are almost identical, which can crowd out diversity in the candidate set and waste context tokens later.

Overlap usually helps when:

• sentences or clauses frequently run across chunk boundaries
• headings need a few following lines to become meaningful
• numbered steps depend on the previous or next step
• the parser cannot preserve structure perfectly, so some continuity must come from text repetition

Overlap usually hurts when:

• top-k is dominated by near-duplicate siblings from the same source
• reranker cost grows without material recall gain
• final context repeats the same paragraph with slightly shifted boundaries
• the system loses document diversity and misses the one different source that mattered

For many corpora, around 10 to 20 percent overlap is a better starting point than aggressive 30 to 50 percent overlap. But even that should be validated, not assumed.

Document-aware chunking usually beats fixed windows

Fixed token windows are easy to implement, but enterprise documents are structured objects, not flat text streams. Headings, bullets, warnings, tables, examples, and exception clauses are semantic boundaries. Ignoring them is one of the fastest ways to lower effective recall.

Document-aware chunking should try to preserve:

• heading + immediate explanatory body
• rule + exception in the same retrievable unit when possible
• table header + relevant rows
• parameter name + type + constraints + example
• procedure title + prerequisites + step group

In practice, this often moves retrieval quality more than changing from one reasonable embedding model to another. The biggest win is not a magic token count. It is preserving the answer-bearing unit the way the source actually expresses it.

How chunking failures show up in production traces

If you log retrieval candidates and final context, chunking problems have recognizable signatures:

These are often visible long before answer quality dashboards tell you what is wrong. If the same sibling chunks repeat everywhere, or the retrieved unit consistently lacks the qualifier that changes the answer, you already know where to look.

The eval that actually picks the right chunking strategy

The mistake is choosing chunking by feel. The right way is to compare candidate strategies on a labeled set of real queries and inspect failure cohorts.

Step 1: build a gold set with answer-bearing spans

For each query, label the source and the specific passage, clause, row, or step group that makes the answer correct. This matters because document-level relevance is too coarse for chunking decisions.

Your labels should capture:

• whether the answer exists in the corpus at all
• the gold document or section
• the answer-bearing span that must be preserved
• query cohort, such as policy, API, troubleshooting, pricing, or compliance

Step 2: compare strategies on more than recall@k

Recall@k matters, but it is not enough. Compare chunking strategies on the same corpus and measure:

• Candidate recall@k: did the gold source appear?
• Full-span recall: did at least one retrieved chunk contain the decisive span intact?
• Selection recall: did the useful chunk make it into final context?
• Duplicate rate: how many candidates are near-identical siblings?
• Context precision: how much of final context is useful instead of repetitive or broad?
• Token cost / latency: how much retrieval and context inflation did the strategy introduce?

Full-span recall is the metric many teams miss. A strategy can look fine on candidate recall while still fragmenting the decisive evidence.

Step 3: inspect failures by document cohort

One global chunking strategy is rarely optimal across every corpus. Policies, API docs, and troubleshooting manuals behave differently. Compare performance by cohort before you declare a winner.

If one strategy wins on API docs and loses on policies, the correct answer may be document-type routing rather than one compromise setting for everything.

Queries
  ↓
Gold source + answer-bearing span labeled
  ↓
Run strategy A / B / C on same corpus
  ↓
Measure recall@k + full-span recall + duplicate rate
  ↓
Inspect failures by document type
  ↓
Choose strategy by evidence, not token folklore

Starting defaults by document type

These are starting points, not universal truths. Use them to seed the eval, not to skip it.

When to use parent-child or hierarchical retrieval

Parent-child retrieval is useful when you need both precision and surrounding context. The retriever searches small child chunks, then expands to a larger parent section for the final prompt.

This is usually worth it when:

• documents are long and strongly structured
• answers live in one narrow span, but interpretation needs nearby section context
• fixed overlap keeps producing duplicates without solving boundary loss
• you want small retrieval units without forcing the model to reason from tiny isolated fragments

A common starting pattern is: retrieve 150 to 250 token child chunks, then attach an 800 to 1,500 token parent section if that improves grounded answer rate without blowing up context precision.

What to fix before you touch embeddings

Many retrieval projects jump to a new embedding model too early. Before that, verify the basics:

• parsing is not destroying headings, tables, lists, or code blocks
• duplicate or near-duplicate documents are controlled
• stale versions are not competing with current versions
• metadata filters are applied correctly by product, region, plan, or version
• you have a labeled eval set that can prove chunking improvements

If those are broken, embedding changes mostly add movement, not clarity.

Want us to diagnose chunking and retrieval?

A good chunking strategy is not a number. It is an evidence-backed choice tied to your corpus, your queries, and your failure modes. If recall is weak, users say "it's in the docs," or your top-k is full of near-duplicates, chunking deserves a real audit.