“I’d evaluate it like a new engineer’s first month, except replayable. I build the task set from our own history: take merged PRs, check out the parent commit, give the agent the issue text, and hold the PR’s tests back as a hidden grader. Tests that failed before must now pass, and the existing suite must stay green. That grades what the agent did to the repo, not what its PR description says.
Then I grade the diff and the path. Did it stay inside the files the task allows? Did it get to green by deleting, skipping or loosening a test? Did it add a dependency or touch CI config? How many steps and dollars did it take? A passing suite reached by editing the tests is a failure.
I run every task several times, because opening PRs on its own needs it to succeed every time, so I look at pass^k, not best-of-k. And I keep sandbox timeouts out of the score.
Then shadow mode: it opens draft PRs on real tickets, humans review them as they would a colleague’s, and I track merge rate, review time and reverts. It earns the right to open PRs one task class at a time, starting with something like dependency bumps, and required checks and human review still apply.”