“Six stages — chunk, embed, index, retrieve, rerank, generate. The interesting decisions are at the two ends.
Chunking: I start at about 512 tokens with 10–15% overlap for prose, but I don’t chunk structured documents on a token count at all. I chunk them on their own boundaries — a section, a whole table, a function. The failure everyone hits once is splitting a table so that neither half means anything.
Retrieval: dense alone misses exact strings — part numbers, error codes, names. So hybrid. BM25 and vector search in parallel, fused with reciprocal rank fusion, then a cross-encoder rerank over the top 50 down to the five that go in the prompt.
And the number I actually track isn’t similarity score, it’s whether the chunk containing the answer made it into the context. I measure recall on a labelled set before I tune anything else.”
The deep dive — diagrams, tradeoff tables, and the follow-up trap
Where quality is lost
Six stages — but quality is lost in three of them
Chunk on the boundary, not on the number
512 tokens is a default for prose and wrong for everything else.
Content
Chunk on
Why
Policies, prose docs
heading + paragraph, ~512 tok, 64 overlap
keeps a claim with its qualifier
API reference
one endpoint, whatever its size
half an endpoint is worse than none
Tables, sheets
the whole table + caption
a split row loses its header row
Code
a whole function or class
a signature without its body retrieves nothing
Transcripts
turn windows, 6–10 turns
pronouns need their antecedent in scope
Retrieve wide, rerank narrow
Each stage buys recall; the rerank spends a little of it back to buy prompt space.
Stage
Returns
Added p50
Answer chunk present
BM25 only
50
12 ms
71%
Vector only
50
28 ms
78%
Hybrid, fused with RRF
50
34 ms
91%
+ cross-encoder → top 5
5
170 ms
88%
The last row gives up three points of recall and cuts prompt tokens roughly tenfold. That is the trade, and it is usually worth taking.
REAL SYSTEM
An enterprise policy corpus — ~240K chunks, heavy on tables. Recall@5 sat at ~61% and the agent’s answers were right ~68% of the time. Two changes: stop splitting tables (re-chunk on document structure) and add BM25 alongside the vector search with a cross-encoder rerank. Recall@5 went to ~88% and answer accuracy to ~84%, with ~200 ms added to p50. No prompt or model change was involved.
FOLLOW-UP TRAP
“Context windows are huge now — why not skip retrieval and paste everything in?” — cost scales linearly with tokens on every single call, attention degrades in the middle of a long context, and you lose the two things retrieval gives you for free: a citation, and a place to enforce per-user permissions. Retrieval is an access-control boundary, not only a cost trick.
Learn it in full:Chunking: Cut on Structure, Size for the Embedder — cutting on the document’s own units, fitting heading path and overlap in the embedder’s window, and measuring the cut by recall@k.
