Chunking decides which sentences can ever be retrieved together. Get it wrong and the symptom shows up three layers away, as a model that looks like it is hallucinating.
Chunking is the least glamorous decision in a retrieval pipeline and quietly the highest-leverage one. It is also the one whose failures are hardest to recognise, because a bad chunk boundary does not produce an error. It produces a confident answer built from a passage that was missing its second half.
Before touching the prompt or the model, check whether the right chunk was retrieved at all. If it was not, nothing downstream can fix it — you are debugging generation for a retrieval bug, and you can rule it in or out in about ten minutes.
What chunking actually decides
A chunk is the smallest unit your system can retrieve. If the sentence that answers the question and the sentence that gives it meaning end up in different chunks, no reranker, no bigger model and no better prompt puts them back together.
The strategies, and when each is right
| Strategy | Cuts on | Right when | Breaks when |
|---|---|---|---|
| Fixed size | A token count | Uniform prose, and you need a baseline today | The document has any structure at all |
| Fixed + overlap | Token count, windows overlapping | Prose you cannot parse structurally | Still splits tables; index grows with the overlap |
| Recursive | A separator hierarchy: sections, then paragraphs, then sentences | Mixed prose — the sensible default | Headings get orphaned from the text they govern |
| Structure-aware | The document’s own units | Anything with headings, tables or code | Needs a real parser, and scanned PDFs fight back |
| Semantic | Embedding distance between adjacent sentences | Long narrative that shifts topic without headings | Slow and costly to build, and often no better than recursive |
| No chunking | Nothing — whole documents | Short documents, generous context budget | Cost per query, and the answer diluted among noise |
If you want one default: recursive, escalating to structure-aware for any document type that carries meaning in its layout. Semantic chunking is the one people reach for because it sounds principled, and it is the one that most often fails to beat a well-configured recursive splitter on a real eval set.
What the size and overlap numbers actually do
Roughly 512 tokens with 10–15% overlap is a reasonable starting point for prose, but it is a starting point, not a finding. What matters is what each knob trades.
Smaller chunks
Sharper embeddings, because one chunk is about one thing — retrieval precision goes up. But context gets stripped away: a paragraph that says “this does not apply to enterprise accounts” is useless once separated from what “this” was. You retrieve the right topic and the wrong meaning.
Larger chunks
Context survives, and the embedding blurs. A chunk covering four subjects sits in the average of all four and ranks well for none of them. You also pay for every token you retrieve, on every call, forever.
Overlap is insurance against exactly the failure in the diagram: a boundary landing mid-passage. It is not free — 15% overlap is roughly 15% more vectors, 15% more storage and more near-duplicate results competing in the top-k.
Structure beats token counts
The single biggest improvement most pipelines can make is to stop treating a document as a stream of tokens.
- Never split a table. Half a table is worse than no table — the model reads the rows it got as complete. Keep it whole, however long, or convert it to text per row.
- Never split a code block or a function. Same reason, and the half it kept will look valid.
- Carry the heading path into every chunk. A chunk that begins “It must be requested within 30 days” is unanswerable; prefixed with Refunds › Enterprise › Exceptions, it is precise.
- Keep list items with their stem. An orphaned bullet is a sentence fragment with an embedding.
Document processing covers getting to a parseable structure in the first place, which is where the actual weeks go on real corpora.
Metadata is half the win
What you attach to a chunk is as important as where you cut it. Source, title, heading path, dates, and — the one people forget — whatever the permission filter needs. A chunk without a tenant or role marker cannot be filtered at query time, and that is not a retrieval quality problem, it is a data leak. See chunk metadata and indexing.
How to tell whether yours is wrong
Measure retrieval separately from generation, or you will keep attributing chunking failures to the model.
# Fifty real questions, each tagged with the passage that should answer it.
# This measures retrieval alone -- no generation, no judge, no ambiguity.
def recall_at_k(questions, k=6) -> float:
hits = 0
for q in questions:
retrieved = retrieve(q.text, k=k)
if any(q.gold_passage in chunk.text for chunk in retrieved):
hits += 1
return hits / len(questions)
for name, index in {"fixed_512": a, "recursive": b, "structure": c}.items():
print(f"{name:>12} recall@6 = {recall_at_k(EVAL, k=6):.0%}")
# fixed_512 recall@6 = 61%
# recursive recall@6 = 78%
# structure recall@6 = 91% <- tables and headings stopped being shredded
Two rules for that eval set. Write the questions from real user queries, never from the documents — questions written while reading a passage retrieve that passage far too easily. And when recall is already high but answers are still poor, stop tuning chunking: the problem has moved downstream to reranking or generation. RAG evaluation goes further.
Chunking sets the ceiling on everything after it. Cut on the document’s own boundaries rather than a token count, never split a table or a function, carry the heading path into the chunk, and measure recall on real questions before you touch anything downstream. Teams that do this find most of their “hallucination problem” was a boundary in the wrong place.
Next: chunking in depth, retrieval and reranking for the stage after this one, and RAG vs fine-tuning if the question underneath is whether retrieval is the right tool at all.