Run RAG Chunking Experiments That Answer a Real Question

Chunking is a retrieval hypothesis

A chunk is not simply a fixed number of tokens. It is a claim about which pieces of a document should stay together so a retriever can find and a model can use them. Policy documents, API references, support articles, and long contracts have different semantic boundaries.

Start with a baseline that is easy to describe: a token target, a modest overlap, a document-aware splitter, and preserved source metadata. The baseline gives you something to challenge. Choosing a number because it appears in a tutorial does not.

Create a retrieval set before tuning

Collect real questions where possible, then record the source passages that a strong answer should cite. Include questions that require one passage, multiple passages, a rare term, a recent update, and a request that should not be answered from the corpus.

Evaluate retrieval separately from final answer quality. If the right passage never enters the context window, a better prompt cannot repair it. Inspect recall at a chosen top-k, duplicate results, source diversity, and whether chunk boundaries cut through the evidence needed to answer.

Change one variable at a time

Compare a small set of chunk sizes and overlaps while keeping embedding model, metadata, query processing, and evaluation questions stable. If everything changes together, the result becomes a story rather than an experiment.

Record index size, ingestion time, average retrieved context, and qualitative failures alongside retrieval metrics. A larger chunk may improve one multi-step question but bury a precise definition. A smaller chunk may increase recall while producing too many near-duplicates.

Preserve context deliberately

Use headings, document identifiers, page numbers, source URLs, timestamps, and parent-child relationships when they exist. Metadata is often more reliable than trying to make one chunk contain every possible explanation. It also gives the application a way to cite and filter results.

Overlap is useful when a thought crosses a boundary, but excessive overlap mostly creates duplicates. The planner can estimate the duplication cost. Your evaluation set should decide whether that cost bought a measurable improvement.

Use the result to choose a product boundary

Some questions are too broad for a retrieval assistant to answer confidently from a document set. A good system shows the source, asks for clarification, or says that the corpus does not support a conclusion. That behavior is more valuable than a fluent answer with weak evidence.

The final choice is a tradeoff among relevance, context cost, index size, and maintainability. Document the chosen baseline and the cases it does not solve. Future changes can then improve a known limitation instead of repeating the first experiment.

Checklist

• A representative question set has expected source evidence.
• Retrieval is evaluated before final answer quality.
• Only one chunking variable changes per comparison.
• Chunks preserve useful source metadata.
• The chosen baseline documents its known weak cases.