“Almost always, it isn’t the model. A demo proves the model can do the task; production proves the organisation can run it every day. ‘Almost ready’ for three months usually means two things: nobody can say whether this week’s version is better than last week’s, and some gate nobody planned for is quietly blocking.
So I diagnose by asking for an artifact per gate. Who gets paged when it’s wrong at 2 a.m., and what one number do they care about? What’s the pass rate on real cases, against a bar the owner signed off? Does it run on live systems with a scoped service account, or on a CSV export? Has security seen a tool list marked read, write or irreversible? What does one task cost at production volume, and what’s the ceiling? And how do we turn it off in seconds?
Whatever has no artifact is the work. I’d stop adding features and build the eval set first: a couple of hundred real cases, graded by the owner, gating every prompt and model change in CI. Then the agent goes behind a flag, in shadow mode on live traffic, then a canary with rollback triggers written down before launch.
And I’d plan the handover from week one: an engineer on the customer’s side owns the eval set and runs the release, or it stalls again the day I leave.”