1:1 mentoring with Big Tech AI engineers
AgenticPrep.ai
Interview PlanRoadmapLLM & AgenticRAGMCPSystem DesignInterview PrepBlogDiscussAI System Design Studio
Q6

Walk me through your RAG pipeline. What chunk size and overlap, and why?

Fundamentals & Architecture

RAGEmbeddingsRetrievalArchitecture

Asked at Sierra · Databricks · Glean

How to Answer

“Six stages — chunk, embed, index, retrieve, rerank, generate. The interesting decisions are at the two ends.

Chunking: I start at about 512 tokens with 10–15% overlap for prose, but I don’t chunk structured documents on a token count at all. I chunk them on their own boundaries — a section, a whole table, a function. The failure everyone hits once is splitting a table so that neither half means anything.

Retrieval: dense alone misses exact strings — part numbers, error codes, names. So hybrid. BM25 and vector search in parallel, fused with reciprocal rank fusion, then a cross-encoder rerank over the top 50 down to the five that go in the prompt.

And the number I actually track isn’t similarity score, it’s whether the chunk containing the answer made it into the context. I measure recall on a labelled set before I tune anything else.”

The deep dive — diagrams, tradeoff tables, and the follow-up trap

Where quality is lost

Six stages — but quality is lost in three of them
The pipeline, left to right chunk embed index retrieve rerank generate Chunking loses it first a table split in half means nothing a claim without its qualifier inverts fix: chunk on the document’s own boundaries, not on a token count Dense retrieval loses exact terms embeddings blur part numbers, error codes and proper nouns together fix: BM25 and vector in parallel, fused with reciprocal rank fusion The prompt loses the answer score order is not usefulness order a long context buries its middle fix: cross-encoder rerank, then cut to the five that fit The metric that matters is not similarity score — it is whether the answer-bearing chunk is in the context at all.

Chunk on the boundary, not on the number

512 tokens is a default for prose and wrong for everything else.

ContentChunk onSizeWhy
Policies, prose docsheading + paragraph~512 tok, 64 overlapkeeps a claim with its qualifier
API referenceone endpointwhatever it ishalf an endpoint is worse than none
Tables, spreadsheetsthe whole table + captionwholea split row loses its header row
Codefunction or classwholea signature without its body retrieves nothing
Transcriptsturn windows6–10 turnspronouns need their antecedent in scope

Retrieve wide, rerank narrow

Each stage buys recall; the rerank spends a little of it back to buy prompt space.

StageReturnsAdded p50Answer chunk present
BM25 only5012 ms71%
Vector only5028 ms78%
Hybrid, fused with RRF5034 ms91%
+ cross-encoder → top 55170 ms88%

The last row gives up three points of recall and cuts prompt tokens roughly tenfold. That is the trade, and it is usually worth taking.

REAL SYSTEM

An enterprise policy corpus — ~240K chunks, heavy on tables. Recall@5 sat at ~61% and the agent’s answers were right ~68% of the time. Two changes: stop splitting tables (re-chunk on document structure) and add BM25 alongside the vector search with a cross-encoder rerank. Recall@5 went to ~88% and answer accuracy to ~84%, with ~200 ms added to p50. No prompt or model change was involved.

FOLLOW-UP TRAP

“Context windows are huge now — why not skip retrieval and paste everything in?” — cost scales linearly with tokens on every single call, attention degrades in the middle of a long context, and you lose the two things retrieval gives you for free: a citation, and a place to enforce per-user permissions. Retrieval is an access-control boundary, not only a cost trick.

Previous

When do you fine-tune instead of adding retrieval or changing the prompt?

Next

How do you get reliable structured output? What happens when the JSON doesn't parse?