The Paradigm Shift
What changes when the model picks its own path, the five classic parts that become agent parts, and the Google Cloud product that holds each one, with the failure rules that do not change.
Last updated
After this section you can
- Decide when a fixed workflow beats an agent, by counting calls and input tokens
- Map each classic part to the agent part that replaces it, and to the Google Cloud product that holds it where one does
- Name the rules that do not change, and the loop guard a run still needs
The Paradigm Shift
When the model picks the path, five classic parts take on new jobs, and Google Cloud holds three of them.
A workflow is code that fixes the path. An agent lets the model choose the path at runtime, so cost and latency become ranges. Five classic parts take on new jobs: the gateway, the memory, the durable run, the cache and the guard.
Blue request, amber decision or model, purple state, green output, red stop, teal the loop. Names as of 2026-04-22; AWS and Azure equivalents are in Agent Tech Stack. Pans sideways on a phone.
What changes when the model picks the path
- Classify, retrieve, draft
- Input: 22,500 tokens
- Path set by code
- Same ticket
- Input: 33,000 tokens
- Path set by the model
- Same ticket, longer path
- Input: 73,500 tokens, 2.2×
- Path set by the model
Rule of thumb: start with the workflow. Add autonomy only where the steps cannot be predicted.
Illustrative, uncached input: a 6,000-token prefix, then 1,500 more tokens per step, re-read on every call.
Anthropic's test (2024-12-19) is an open-ended problem whose steps cannot be predicted or hardcoded. Each call re-reads everything before it, so input grows faster than the call count, and tests assert outcomes, not steps.
Latency compounds too: each of ten sequential calls has a 1% chance of landing in the slowest 1%, so at least one does with probability 1 − 0.9910 = 9.6%. See The Agentic Spectrum.
The load balancer becomes an AI gateway
- Identifythe tenant and its key
- Check budgetbefore the call is made
- Screenthe input with Model Armor
- Routeto a model, with a fallback
- Logthe model, stop reason and usage
A load balancer spreads equal requests across identical servers (System Design 101). A gateway is the one policy point in front of the model. On Google Cloud, Agent Gateway fills that role, and Model Armor screens. Claude comes from Model Garden, where a regional endpoint costs 10% more than a global one (as of 2026-10-08).
Give it short-lived identity, not a long-lived key. Workload Identity Federation has accepted short-lived tokens since 2026-05-04.
Outage failover belongs in the gateway: server-side fallbacks cover classifier refusals only, and a 429, a 529 or a 5xx comes back to you unchanged. See Model Routing.
Screening lowers the noise but does not make injection safe. See Prompt Injection Defense.
The database becomes memory
- Working memory
- The context window. Each call re-reads all of it; the 5.5 models have a 1M-token window.
- Session state
- The transcript you send with each call. The Messages API keeps no conversation between calls.
- Long-term memory
- Facts loaded into the window when a task needs them.
A database answers queries. The model reads only what is in the window, so a fact not loaded for this call does not exist. On Google Cloud, Agent Runtime keeps state for days, backed by Memory Bank.
When the window fills, let the server compact it (beta), not you. An early change breaks the cache, and on accounts created on or after 2026-08-31 an edit before a thinking block returns a 400 by default.
The queue becomes a durable run
- Acceptthe request, and return a run id at once
- Runa worker advances the loop one step at a time
- Checkpointafter every step, with the transcript
- Resumefrom the last checkpoint
Pub/Sub ack deadline: 10 s by default, 600 s at most. Cloud Tasks HTTP dispatch: 15 s to 30 min. Read from the API definitions, 2026-10-08.
A durable run holds state: the request, the steps and the next one. On Google Cloud, Pub/Sub, Cloud Tasks or a Cloud Run job holds the run. Delivery is at least once, so key each side effect yourself. The Messages API has no idempotency key. See Agent Durability.
The cache starts with the prompt cache
- Matches the same prefix
- Reads at 0.05× on Sonnet 5.5 and Opus 5.5
- Breaks on an earlier change
- Risk: a silent miss
- Matches a similar question
- A hit skips the model call
- A loose threshold admits errors
- Risk: a fast, wrong answer
Rule of thumb: keep the prompt cache first. Add a semantic cache only with per-tenant keys and a measured wrong-hit rate.
A 5-minute prompt-cache write costs 1.25× the input price, and on Sonnet 5.5 and Opus 5.5 a read costs 0.05× (as of 2026-10-08). A prefix under 512 tokens is not cached on Opus 5.5, Sonnet 5.5 or Haiku 5.5, and the usage fields show zero.
Similar is not compatible: a stored answer can be wrong for this tenant or this date. See Semantic Caching and Prompt Caching.
The circuit breaker gains a guard
- Stops calls to a failing service
- Opens on repeated 529s and timeouts
- Caps steps, dollars and tool scope
- Fires even when every call succeeds
Rule of thumb: keep both. A breaker protects you from a dependency; a guard protects you from the run.
A loop can spend money with no failed call. $50 buys 125 uncached Opus 5.5 calls, each re-reading 100,000 tokens. From the cache, that context costs $0.02, not $0.40, so caching helps. The guard still needs a dollar cap.
What does not change
Timeouts, retries with jitter, idempotency keys, breakers on 429 and 529, rate limits and least privilege carry over unchanged. The SDK and the application each retry twice, so 3 × 3 makes 9 HTTP calls per tier.
A breaker must not retry a spend-cap 429: it has no retry-after, and API use pauses until 00:00 UTC on the 1st. Retry budgets across agents are in Multi-Agent in Production.
- Reading a multi-agent system as microservices. Anthropic measured about 15× the chat tokens (2025-06-13).
- Using old names. Vertex AI is now the Gemini Enterprise Agent Platform; Agent Engine is Agent Runtime.
“Design an agent for support tickets on Google Cloud. What runs the loop, and what protects the budget?”
Test first: can you predict the steps? If not, make only that step an agent. In order: a load balancer spreads equal traffic, and an AI gateway is the one policy point for keys, budgets, fallbacks and logs.
Memory is what you load into the window, not a table you query. A run is a durable record of steps, resumed from the last checkpoint. The prompt cache is an economics lever; the semantic cache is a correctness risk.
A breaker protects you from a dependency; a guard protects you from the run. The trap is treating the breaker as the step budget. You need both.
A support pipeline classifies, retrieves, drafts, and a person approves before sending. Workflow or agent, and which node, if any, is an agent?
A workflow. The steps are known, and the approval is a human gate. Only an open-ended sub-task becomes an agent node.
A loop may run 25 or 50 steps, adding 1,500 tokens per step after a 6,000-token prefix. What happens to uncached input at 50?
It rises from 600,000 to 2,137,500 tokens, 3.6 times as much: $8.55 against $2.40 on Opus 5.5, before output.
A tool run takes 40 minutes. Which managed piece holds the run, and why not Cloud Tasks?
Agent Runtime, which keeps long-running agents' state for days. A Cloud Tasks HTTP dispatch lasts at most 30 minutes, so a 40-minute call would time out first.
- A workflow fixes the path in code; an agent lets the model choose it. Compare them by calls and input tokens.
- Five classic parts change jobs: the gateway, the memory, the durable run, the cache and the guard.
- Google Cloud holds three: Agent Gateway, Sessions and Memory Bank, and the durable run.
- Timeouts, retries, idempotency and breakers on 429 and 529 do not change.