A pilot proves the model can do the task. Production proves the organisation can run it every day. Pilots stall on the same six missing pieces: an owner, an eval set, real data access, a security sign-off, a cost bound and a way to roll back. Treat each as a gate with an artifact, and pass them in order.
What the numbers say
Sources: Gartner, 25 June 2025; McKinsey, The state of AI in 2025 (another 39% were experimenting); Gartner, 19 October 2026 (a Q2 2026 survey of more than 2,000 CIOs and technology executives).
Gartner's prediction is often quoted as fresh 2026 research; it is from June 2025, and it names three causes: “escalating costs, unclear business value or inadequate risk controls”. McKinsey's 2026 survey shows movement: 40% of respondents at large companies (revenue over $1 billion) now report scaling agents, up from 27% a year earlier, while the share at smaller companies stayed at 22%. None of these causes is the model.
| Cause | Missing | Gate |
|---|---|---|
| Unclear business value | no owner, metric or real-case evidence | 1, 2 |
| Inadequate risk controls | no data scoping, permission review or undo | 3, 4, 6 |
| Escalating costs | no per-task price or ceiling | 5 |
Gate 1: a named owner and one metric
- sponsor: an innovation budget
- success: a good demo
- after the demo: nobody's job
- sponsor: the workflow's business owner
- success: one number, such as minutes per ticket or cases closed without rework
- after launch: on someone's on-call rota
The test: ask “who gets paged when this is wrong at 2 a.m.?” If nobody answers, you are still in the pilot.
For a forward-deployed engineer this gate has a sharp edge. Gartner predicts (29 September 2026) that 70% of enterprises will abandon agentic AI built by vendor forward-deployed engineering by 2028, citing cost and a lack of internal capability to evolve what was built. The fix is in your plan from week one: a customer engineer pairs with you, owns the eval set and runs the release. More on the role in What Is a Forward Deployed Engineer?
Gate 2: an eval set before more features
“It looked good in the demo” is a common reason a pilot cannot be promoted: nobody can say whether version 2 is better than version 1. The eval set turns opinions into a pass rate the owner signed off on.
- CollectReal cases from the workflow, including the ugly ones.
- GradeThe owner writes what a correct outcome is.
- Set the barA pass rate, plus cases that must never fail.
- Gate CIEvery prompt, tool or model change runs the set.
results = [grade(case, run_agent(case.input)) for case in load_cases("evals/golden.jsonl")]
pass_rate = sum(r.passed for r in results) / len(results)
critical = [r.case.id for r in results if r.case.critical and not r.passed]
if pass_rate < BAR or critical: # BAR agreed with the owner, e.g. 0.90
sys.exit(f"eval gate failed: {pass_rate:.0%} pass, critical misses: {critical}")Grade agents on the outcome and on the path they took: Grading Agents. Building the golden set and wiring it into CI: The Eval Stack. Which metrics to report: LLM Evaluation Metrics That Actually Matter.
Gate 3: production data, scoped
Most pilots run on an export: a CSV of last quarter's tickets, a copy of the wiki. Production runs against live systems, and that changes three things at once.
| Pilot | Production | Artifact |
|---|---|---|
| a static export | live APIs, with rate limits and outages | tool timeouts and error results |
| your admin login | a service account per agent, least privilege | a scoped credential, reviewed |
| clean sample records | personal data, other tenants, stale rows | a data-handling note: what is read, kept, logged |
Start the access request in week one; in large enterprises it is often the longest lead time in the project. If you want to reuse production traces to improve the agent later, agree consent and retention now, not after launch (Data Flywheel).
Gate 4: the security review
A security team cannot approve “an agent”. It can approve a list of tools, what each can change, and what happens when the model is tricked. Bring those, and the review has something concrete to approve.
| They will ask | You bring | Go deeper |
|---|---|---|
| What can it do? | every tool, marked read, write or irreversible | action gates |
| What if a document tells it to misbehave? | which inputs are untrusted, and what they cannot reach | guardrail checkpoints |
| Who approves risky actions? | the approval step for refunds, emails, deletes | approval gates |
| Can we reconstruct what it did? | a trace per run: inputs, tool calls, outputs | tracing |
Gate 5: a cost bound
Pilot bills are invisible because pilot volume is tiny. Price one task from real usage, then multiply by production volume before the CFO does it for you. A support-triage agent on Sonnet 5.5, with an assumed task size:
| Per task | Tokens | Cost |
|---|---|---|
| Input written to cache ($2.50/M) | 20% of 150K | $0.075 |
| Cache hits ($0.10/M) | 80% of 150K | $0.012 |
| Output ($10/M) | 4K | $0.040 |
| One task | $0.127 |
The 200-case pilot cost $25.40. At 2,000 tasks a day the same agent costs $254 a day, about $7,620 a month, and a run that loops and reads 10× the tokens costs $1.27 on its own. Put a ceiling per task (say $1.00) and per day in the loop, alert on both, and decide with the owner what happens when one trips. Effort levels, caching and routing to a smaller model are in Agent Cost Control.
Gate 6: rollout with a way back
The last gate is the one that lets everyone say yes to the others: if it goes wrong, how fast can you undo it? Ship the agent behind a flag, with the old path still running.
- ShadowRuns on live traffic; humans still do the work; compare.
- CanaryA small share of real cases, with rollback triggers set in advance.
- WidenStep up only while the eval set and live metrics hold.
- Keep the switchOff sends work back to the human queue in seconds.
Name the triggers before launch (error rate, escalation rate, cost per task), not after the first incident. The full pattern is in Deployment & Rollout; promoting a new prompt or model through the same eval gate is in Promotion and the Eval Gate; and what to watch in the first day is The First 24 Hours.
Adding features to the pilot instead of building the eval set. Requesting data access after the demo. Sending security a slide instead of a tool list. Quoting pilot cost as production cost. Launching to 100% with no flag, so the only rollback is a redeploy. Leaving the customer with a system nobody on their side can change.
“Your agent pilot went well. Why isn't it in production, and what do you do?” Walk the gates in order and name the artifact for each: owner and metric, eval set and bar, scoped service account, signed-off tool list, cost per task with a ceiling, flag and rollback triggers. Then say which one you would start first: data access, because it usually has the longest lead time.
The pilot scored well with the sponsor, but the operations team refuses to adopt it. Which gate was skipped?
Gate 1. The team that will live with the agent never owned it or agreed what success means. Make their lead the owner, agree one metric, and let them grade the eval set.
Security has had your review request for six weeks. What do you send to unblock it?
Concrete artifacts: every tool marked read, write or irreversible; the untrusted inputs and what they can reach; the approval step for risky actions; and a sample trace showing what is logged per run.
A model upgrade looks better in spot checks. How do you decide whether to ship it?
Run the eval set: ship only if the pass rate meets the agreed bar and no critical case fails. Then roll it out like any change: canary first, with the rollback triggers already defined.
- Gartner predicts over 40% of agentic AI projects will be cancelled by end of 2027, for cost, unclear value and weak risk controls. None of those is the model.
- Six gates turn a pilot into production: owner, eval set, scoped data, security sign-off, cost bound, rollback.
- Each gate leaves an artifact you can show: a name and metric, a graded set, a credential, a tool list, a budget, a switch.
- Start data access and the eval set in week one; they take longest and unblock the rest.
- Build for handover: the customer's own engineers must be able to run and change it.