1:1 mentoring with Big Tech AI engineers
Q23Premium

Token spend tripled overnight and nobody shipped a prompt change. How do you find it?

Scale & Cost

Cost OptimizationObservabilityDebuggingProduction

Asked at Datadog · OpenAI · Perplexity

How to Answer

“Three-x overnight with no deploy means volume, retries, or context growth — and I can tell which within about ten minutes if the traces carry the right attributes.

First question: did request count go up, or did cost per request go up? If it’s per-request, it isn’t a traffic story at all. Then I split tokens per request into prompt and completion. Prompt tokens climbing against a flat request count is almost always context growth — a retrieval change, a memory system that started including more history, or a prefix cache that stopped hitting.

The one that catches people is retries. A tool starts timing out, the agent retries, every retry replays the whole conversation, and spend triples while request count doesn’t move at all. That’s why I want tokens attributed per step and per trace, not a monthly total on a provider dashboard.

And the cheap safeguard is a per-trace token budget with an alert, so this is a page at two-x rather than a surprise on the invoice.”

The deep dive — diagrams, tradeoff tables, and the follow-up trap

Loading…