1:1 mentoring with Big Tech AI engineers
LLM & Agentic

Agent Cost Control

Control what an agent costs: effort levels, task budgets vs session budgets vs max_tokens, prompt-cache economics, model routing, and per-turn token accounting.

Last updated

Staff7 min readFirst readPrompt CachingAdaptive Thinking & Effort

After this section you can

  • Separate max_tokens, task budgets and session budgets by scope, unit, and what happens when each is hit
  • Use effort as the primary cost dial, and stop reaching for it to control output length
  • Split an agent’s bill into cached input, uncached input and output before choosing a fix
36

Agent Cost Control

An agent’s bill is not one number you cap. It is a per-turn choice about depth, a prefix you either reuse or re-pay for, and three ceilings that behave nothing alike.

THE CENTRAL IDEA

Every lever here answers a different question. Effort decides how much work a turn does. Caching decides how much of the input you pay full price for. Routing decides which model does the reading. Budgets decide what happens when a run overruns — and the three things called a budget stop the run in three different ways.

Three ceilings that are not the same ceiling
ONE RUN — WHERE EACH LIMIT BITES max_tokens one response model cannot see it lost cut off mid-answer stop_reason: max_tokens task_budget one agentic loop model sees a countdown wraps up finishes on its own terms advisory · tokens · min 20,000 session budget one whole session checked before each call pauses, keeps its state hard · dollars · budget_reached A guillotine, a deadline the model can see, and a spend cap the platform enforces. Only the middle one changes how the model behaves.

Related

More in LLM & Agentic

LLM Lifecycle

Preview

Complete lifecycle of large language models from pre-training through fine-tuning, RLHF, and deployment — with architecture diagrams and production considerations.

How LLMs Call Tools

Preview

How LLMs use function calling and tool use — the mechanics behind tool-calling agents, from prompt engineering to structured output.

Prompt Engineering Foundations

Preview

Where an instruction belongs — tool description, system prompt, or user turn — why emphasis that worked on older models now causes over-triggering, and the prompt patterns that have expired.

Tool Surface Design

Preview

When an action deserves its own tool instead of bash, how to write a description that raises selection accuracy, and how to scale past a few dozen tools without destroying the prompt cache.

Get full access to all 74+ sections with code examples, diagrams, and interactive animations.

Unlock Premium