1:1 mentoring with Big Tech AI engineers
Q29Premium

How do effort and adaptive thinking change your latency and cost, and how do you pick a setting?

Scale & Cost

LatencyCost OptimizationEvaluationOptimization

Commonly asked at Anthropic · Cursor · Perplexity

How to answer — what you would actually say

“On current Claude models thinking is adaptive: the model decides how much to reason on each turn, and output_config.effort — low, medium, high, xhigh, max — bounds it. There’s no token budget for thinking any more.

Effort moves three things. Cost, because thinking is billed as output at five times the input price whether I display it or not. Latency, because thinking runs before the first text token. And behaviour: lower effort makes fewer, more consolidated tool calls; higher effort explores and checks more, which can sometimes mean fewer turns overall. And max_tokens has to fit thinking plus the answer.

I pick it per route from a sweep on my own evals, judged by cost per completed task, not cost per call: low for classification and subagents, medium or high for most work, xhigh for coding agents. A useful pattern is to run low and re-run only the failures at high, if I have a checker that catches them.

Two rules: set it explicitly after a model change — Opus 5.5 defaults to medium, Sonnet 5.5 to high — and pin it per route, because changing effort between requests invalidates the cache.”

The deep dive — diagrams, tradeoff tables, and the follow-up trap

Loading…