RAG Monitoring
What to log, what to watch with no labels, how big a judged sample an alert needs, how to tell drift kinds apart, how to measure freshness and deletes, and how to send a bad answer to the gate that lost it.
Last updated
After this section you can
- Log a RAG request so any answer traces to its chunks, scores and version stamps
- Size a judged sample so an alert threshold means what it says
- Triage a bad answer to one of the five gates and turn it into a regression case
13
RAG Monitoring: Catching Silent Regressions
A golden set proves the day you shipped; production moves without a deploy. Log enough to explain an answer, and watch what needs no labels.
Key idea
A RAG system fails silently: documents change, questions drift and model aliases move, and no code changes. Log each retrieval, watch label-free signals, judge a sized sample, probe known answers, and send each bad trace to its gate.
You cannot label production, so watch it at four speeds
Every speed reads one logged request. Wider than your screen? Scroll sideways.