1:1 mentoring with Big Tech AI engineers
LLM & Agentic

Metrics Framework

Observability metrics for LLM applications: latency, token usage, cost tracking, and quality scoring dashboards.

Last updated

Production5 min readFirst readThe Agent Loop

After this section you can

  • Track the five categories of metric and say what decision each one drives
  • Calculate the tool-call and reliability metrics that reveal agent health
  • Assemble a dashboard someone would actually act on
48

Observability: Complete Metrics Framework

You can't improve what you don't measure. Here's every metric category, how to calculate each, and how it drives decisions.

WHICH EVAL SECTION IS THIS?

This is the production metrics reference — what to graph, how to compute each metric, and what to alert on. For the eval process (golden datasets, LLM-as-judge, A/B tests, CI regression) see Eval & Observability: The Full Stack. For grading agent trajectories specifically, see Grading Agents: Agentic Evaluation.

Observability Pipeline Architecture

Storage and Viz

⚙ Processing

Data Sources

Agent Runtime
OTel spans

LLM Gateway
Token counts

Tool Calls
Latency, errors

User Feedback
Thumbs up/down

Collector
OTel Collector

Eval Pipeline
Nightly golden set

Cloud Trace
Request traces

Cloud Monitoring
Dashboards

BigQuery
Analytics

Alerting
PagerDuty

Related

More in LLM & Agentic

Get full access to all 74+ sections with code examples, diagrams, and interactive animations.

Unlock Premium