Scenario: RAG System Design with Component-Level Evaluation

"Design a RAG system for Q&A over private documents. Walk through each component and explain how you would evaluate each step."

This scenario is covered comprehensively in Section 16: RAG Pipelines — Production Deep Dive. Here we add the interview-specific framing and complete evaluation plan.

5-Minute Interview Walkthrough

RAG System Design — Interview Walkthrough

MINUTE 1: CLARIFY

"Before I design, let me ask:

What type of documents? (PDF/HTML/structured data?)
How many documents and how often do they change?
Who are the users? (internal/external, tech-savvy?)
Latency requirement? (real-time chat vs async report?)
Compliance? (HIPAA/SOC2/GDPR?)
What's acceptable for 'I don't know' answers?"

MINUTE 2-3: ARCHITECTURE

"I'll design in two pipelines: offline ingestion + online retrieval."

[Draw the architecture from Section 16 — ingestion → chunking → embedding → indexing | query → retrieve → rerank → generate]

MINUTE 4: EVALUATION PLAN

"The key insight: evaluate each component independently. I can't fix what I can't diagnose."

[Walk through the 5-stage evaluation from Section 16.8]

MINUTE 5: NON-FUNCTIONALS

"Safety: hallucination detection, citation enforcement, PII gateway. Monitoring: freshness, retrieval quality, user satisfaction. Scale: semantic cache, model tiering, async for batch."

Complete Evaluation Plan (Interview Format)

Component	Metric	Method	Target	When to Run
Ingestion/Chunking	Chunk completeness	Human audit of 100 random chunks: "Does this chunk contain a complete, useful piece of information?"	> 90% pass rate	Quarterly, or after pipeline changes
Ingestion/Chunking	Table parsing accuracy	Compare parsed tables vs original. Check cell values match.	100% for digital PDFs, > 95% for scanned	On every parser update
Retrieval	Recall@5	Golden set: 200 queries with labeled relevant docs. Measure % of relevant docs in top 5.	> 85%	Nightly regression
Retrieval	MRR (Mean Reciprocal Rank)	Same golden set. Where does the first relevant doc appear?	> 0.7	Nightly regression
Reranking	Recall@5 improvement	Compare retrieval quality with and without reranking on same golden set	Reranker adds > 5% recall improvement, else remove it	A/B test when adding/tuning reranker
Generation	Faithfulness	LLM-as-judge: extract claims, verify each against retrieved chunks	> 90%	On 10% sample, real-time
Generation	Answer relevance	LLM-as-judge: "Does this answer the question asked?" (1-5 scale)	> 4.0 avg	On 10% sample, real-time
Generation	Citation accuracy	Verify cited source actually contains the claimed information	100% (citations must be real)	On every response
End-to-end	User satisfaction	Thumbs up/down widget + optional comment	> 80% positive	Every interaction
End-to-end	Task completion rate	Did the user's follow-up indicate success?	> 75%	Weekly analysis

Safety & Guardrails

class RAGGuardrails:
    def check_retrieval(self, query, retrieved_chunks):
        # If retrieval quality is low, don't generate — fail safely
        if not retrieved_chunks:
            return {"action": "refuse",
                    "message": "I couldn't find relevant information to answer this."}

        # Check if best result is actually relevant
        best_score = retrieved_chunks[0].score
        if best_score < 0.5:  # cosine similarity threshold
            return {"action": "hedge",
                    "message": "I found some potentially related information, "
                               "but I'm not confident it fully answers your question."}
        return {"action": "proceed"}

    def check_generation(self, answer, chunks, query):
        # Citation check: every claim must reference a chunk
        claims = extract_claims(answer)
        unsupported = [c for c in claims
                       if not any(supports(chunk, c) for chunk in chunks)]

        if len(unsupported) / max(len(claims), 1) > 0.3:
            return {"action": "regenerate",
                    "reason": "Too many unsupported claims"}

        # PII check on output
        if contains_pii(answer):
            return {"action": "redact",
                    "reason": "PII detected in output"}

        return {"action": "serve"}