RAG Chunking Strategies — Which One, and How to Tell Yours Is Wrong

A bad boundary raises no error. It produces a confident answer built from half a passage, so it looks like hallucination three layers downstream. Before you touch the prompt or the model, check whether the right chunk came back at all: if it did not, nothing after retrieval can fix it (RAG Architecture calls these gates 1 and 2).

Choose what to cut on

Semantic chunking sounds principled, which is why teams reach for it. A semantic splitter embeds every sentence with its neighbours to find the cuts, then embeds the chunks again: on Acme’s 20 million tokens that is $10.40 instead of $2.60 at $0.13 per million. The one systematic comparison, Qu et al., “Is Semantic Chunking Worth the Computational Cost?”, found no consistent gain over fixed-size chunks. LLM-guided chunkers report gains on their own benchmarks; adopt one only if it wins on yours.

Let the content pick the unit

Half a table is worse than none: the model reads the rows it got as the complete table. When a table must be split, cut between rows and repeat the header row in each piece; Docling’s HybridChunker and Unstructured’s chunk_by_title do that by default. Getting typed elements (heading, table, list item) out of a PDF is the job of Document Processing, and it is where most of the weeks go on a real corpus.

Size the chunk for the embedder, not for Claude

The embedding model reads a fixed window, counted by its own tokenizer, and it reads the heading path, the overlap and the new text together. Past the window a truncating embedder drops the tail without an error: search can never find those sentences, yet the model still reads them when the chunk is retrieved. Hosted windows run from 2,048 tokens (Google gemini-embedding-001) to 128,000 (Cohere embed-v4.0), and some small local models stop at 512 (Embeddings).

Defaults as read in the library sources on 2026-09-28 (Chunking, step 2). Tokens at about 4 characters each.

The defaults disagree on size and on overlap (2% to 50%), so a default is a convention, not a finding. Start near 512 tokens in all with 10 to 15% overlap for prose, and let the measurement in the last step move you. Overlap is not free: with chunk text C that includes overlap O, the index grows by C ÷ (C − O), so 464 ÷ 400 = 1.16× here and 2× at OpenAI’s 800/400.

Cut in order: structure, size, overlap

1. StructureSplit on headings into sections; keep each section’s heading path.
2. SizeInside one section, split to the window minus the path: 488 tokens.
3. OverlapRepeat 64 tokens between neighbours, never across a heading.

from langchain_text_splitters import (
    MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter)

WINDOW, PATH_BUDGET, OVERLAP = 512, 24, 64   # embedder tokens
by_head = MarkdownHeaderTextSplitter(
    [("#", "h1"), ("##", "h2"), ("###", "h3")])
by_size = RecursiveCharacterTextSplitter(
    chunk_size=WINDOW - PATH_BUDGET,      # 488, overlap included
    chunk_overlap=OVERLAP,
    length_function=count_tokens)        # the EMBEDDER's tokenizer

def chunk(markdown: str) -> list[dict]:
    out = []
    for sec in by_head.split_text(markdown):              # 1 structure
        path = " > ".join(sec.metadata.values())
        for body in by_size.split_text(sec.page_content):  # 2 size, 3 overlap
            out.append({"heading": path, "text": f"{path}\n{body}"})
    return out

Runs on langchain-text-splitters 1.1.3 (checked 2026-10-21). count_tokens is your embedding model’s tokenizer; the splitter’s chunk_size includes the overlap, so it is the window minus the path.

Keep the path in its own field as well as in the text: the field is what Metadata & Filters filters and displays, and the text copy is what the embedding sees.

Search small, return big

1. Index childrenSplit each 500-token parent into four 125-token children; embed only the children.
2. Search childrenSmall chunks make sharp vectors: the top 3 hits are children.
3. Send parentsLook up each hit’s parent by id, once each, in rank order.

Small chunks rank well and read badly; big ones read well and blur the vector. Parent-child retrieval gets both: 3 hits in 2 parents send 1,000 tokens where the children alone were 375 (2.7×), and the index holds 4 vectors per parent. A fetch by id skips the search filter, so apply the tenant and permission filter again on the parents (Chunking, step 4). LlamaIndex’s sentence-window and auto-merging retrievers are variants of the same idea.

Fix the chunk that cannot stand alone

Context per chunk is Anthropic’s Contextual Retrieval: on its own 2024 data it cut top-20 retrieval misses by 35% with contextual embeddings, 49% with contextual BM25 added and 67% with a reranker on top. With the document cached it costs about 38 cents per million document tokens on Claude Haiku 5.5, the smallest current model. Take the heading path first; add context where your golden set shows misses the path cannot fix.

Stamp every chunk

Source and page

What a citation points at, and what a reader clicks to check it.

Heading path

Embedded with the text and kept as a field for display and filters.

Tenant and permissions

Applied inside the search, never in the prompt. A chunk without them cannot be filtered: that is a data leak, not a quality bug.

Chunker and embedder version

"chunker": "headings-v3" and the embedding model id, so a re-chunk or a mixed index is visible (Vector Indexes & Stores).

A new chunker is a new index: every chunk id changes and everything is re-embedded. Build it beside the old one and cut over only when the golden set and the permission tests pass (RAG Best Practices).

Measure the cut before you upgrade it

def recall_at_k(golden, search, k=5):
    """golden: (question, gold excerpt) pairs from real user queries."""
    norm = lambda s: " ".join(s.split())
    def hit(q, gold):
        return any(norm(gold) in norm(c["text"]) for c in search(q, k))
    return sum(hit(q, g) for q, g in golden) / len(golden)

# one search function per candidate index, same questions, same k
for name, search in {"recursive-488": a, "headings-v3": b}.items():
    print(f"{name:>14}  recall@5 = {recall_at_k(golden, search):.2f}")

95% margins for a pass rate of 0.85; RAG Evaluation owns the table and the paired comparison.

Score gold excerpts, the passage that answers each question, not chunk ids, which change with every chunker. Take the questions from real user queries: questions written while reading a passage reuse its words and retrieve it far too easily. Bigger chunks hold more, so compare chunkers at equal prompt tokens, not only equal k. And when recall@k is already high but answers are still wrong, stop tuning chunking: the loss is downstream, at the reranker cut or in generation (Retrieval & Reranking, LLM evaluation metrics).

Next: Chunking: Cut on Structure, Size for the Embedder — the full lesson, with the interactive window and the parent-child figure. Wondering whether retrieval is the right tool at all? RAG vs fine-tuning.