The agent says the tests pass. That statement leaves several questions open: which tests, against which revision, using whose interpretation of the requirement, and with what omitted cases?
In the Northstar case, sequential retry tests passed while concurrent requests could still create two orders. The test runner was not lying. The evidence was narrower than the claim.
Evaluation is the work of making that boundary visible. It has two related purposes: deciding whether a particular change satisfies its contract, and deciding whether a workflow is reliable enough for a class of future tasks. One successful patch does not answer the second question.
Choose the oracle before trusting the output
An oracle is the reference used to judge an outcome. It may be an invariant, a known result, an independently written test, or accountable human judgment. Independence is about how the expectation was established, not simply whether a second model produced it.
| Evidence layer | Northstar example | What it cannot establish alone |
|---|---|---|
| Contract | ADR-17 defines key scope and expiry | That code implements the decision. |
| Deterministic check | Concurrent requests create at most one order | All production timing and infrastructure behavior. |
| Adversarial case | Same key across accounts cannot expose another account’s response | Complete authorization correctness. |
| Human review | Reviewer examines transaction and recovery assumptions | Exhaustive execution coverage. |
| Operational observation | Controlled pilot records replay, conflict, and failure signals | Safety under all future workloads. |
Compilation, linting, and tests establish particular properties under stated conditions. Reserve “proof” for an actual proof and its assumptions. A green check is useful without being omniscient.
- Check the exact revision
- Challenge concurrency, access, and crash behavior
- Review limits and recovery
- Use representative tasks and held-out cases
- Repeat trials with a recorded workflow version
- Count failures and interventions
New failures feed regression cases; changed graders require independent review.
Build a task set from work and failures
Start with a bounded category and representative historical tasks, sanitized for the approved environment. Include ordinary work, ambiguous requests, missing context, rejected actions, and previously observed failures. Record the expected behavior before running the workflow.
Include sequences as well as isolated tickets. In practice an agent works on a codebase its own earlier changes have shaped, so run related tasks in order against the accumulated result. Check whether later changes stay correct and whether the code remains maintainable as earlier ones pile up. A workflow that scores well on fresh, independent tasks can still degrade a repository over many merges.
For ORD-42, hold back a concurrency and cross-account case from the implementation prompt while keeping the required invariants public. Hidden tests should test the contract, not invent secret requirements. Once a failure becomes a published regression, add new held-out cases to preserve an independent check of generalization.
Anthropic’s agent-evaluation guidance distinguishes tasks, trials, graders, and outcomes, and recommends combining evaluation methods. That distinction is useful here: a fluent explanation is an output, while the persisted order state is part of the outcome we actually care about.
Inspect the trajectory when the outcome is suspicious. A task that passed after disabling a check or using forbidden credentials is a failure of the workflow even if the final response looks correct.
Inspect the task and grader too. OpenAI’s analysis of SWE-bench Verified identifies test-design and contamination problems that complicate frontier comparisons. A familiar benchmark score cannot replace fresh, representative tasks from the workflow being authorized.
Measure a workflow version
Record the model identifier available to the team, harness revision, task revision, context manifest, tool versions, permission policy, environment, and grader version. Keep sensitive traces behind the same access controls as their source data.
Make the environment entry concrete: CPU and memory limits, concurrency, timeouts, and available services. Anthropic’s infrastructure-noise investigation shows that resource configuration can materially change coding-evaluation results with the model and harness held fixed. Preserve those settings before attributing a score change to model quality.
Repeat representative trials when nondeterminism matters. Report the number of attempts, successes, invalid runs, and intervention requirements. Do not quietly exclude the runs where the environment broke; separate infrastructure failures so readers can see the operational cost.
A small pilot cannot establish a rare-event safety rate. If no severe failure appears in a handful of tasks, say exactly that. Do not translate it into “safe for production.” Compare with the existing process using similar task classes and account for selection differences.
An independent reviewer should inspect changed evaluation criteria. Otherwise the system can improve its apparent performance by making the exam easier. A failing test may indeed be wrong, but fixing it requires separate evidence and approval, not a passing patch’s preference.
A release packet a person can inspect
For a consequential change, supply:
- The accepted specification and exact code revision.
- A summary of behavior changed, including interfaces and state.
- Reproducible check commands and observed results, with failures retained.
- Scope and provenance of the context used.
- Known limitations and remaining human decisions.
- Rollout, pause, reconciliation, and recovery instructions with named owners.
For Northstar, the packet explicitly says that no external payment side effects were tested. That limitation is more valuable than another paragraph claiming confidence.
Admit work at the speed of verification
If four patches arrive each day and reviewers can responsibly assess two, more generation expands the queue. Set a work-in-progress limit and stop admitting new execution when review capacity is exhausted. Expedite incidents through an explicit path, not by secretly skipping the queue.
Prefer feedback that returns quickly enough to change the next attempt. Slow security or integration checks may need earlier placement. Do not remove them because they make the dashboard look slow.
Some evidence cannot arrive quickly. A customer may need to complete a monthly workflow before a change shows its effect, and a quiet week says little about peak load. Before rollout, define the observation window, the criteria for the decision, and the conditions that justify acting early, such as a serious correctness failure. Treat missing telemetry as unknown, not as success, and let confidence reflect how much the relevant users have actually exercised the change. Reacting to every early fluctuation makes each intervention impossible to interpret. The target is the shortest time to evidence strong enough for the decision at hand.
Promote and revoke by task class
A team’s promotion record should state the task boundary, evaluated workflow version, evidence window, unresolved failure classes, allowed actions, and revocation triggers. Documentation fixes and stateful order changes should not inherit the same permission merely because the same model handles both.
Start with preparation-only authority. Expand only after the accountable owner has accepted the evidence and the recovery path has been exercised. A changed model, tool, policy, or environment triggers proportionate re-evaluation. An unauthorized action or serious unexplained regression can contract authority immediately.
Report the share of incoming work that qualifies for the class beside its success rate. A workflow that succeeds on every eligible task while admitting only the easiest few percent of requests has achieved something narrower than one handling most of the queue. Show the denominator, the sample size, and what was turned away. Path to the Dark Factory puts this measure in a fuller scorecard for one class of work.
Track first-pass acceptance by task scope as well as by class. The useful frontier is the largest coherent task class the workflow completes consistently at a stated quality threshold, with inference, preparation, review, and recovery costs included. The largest task an agent has completed once is a demonstration, not a frontier. Scope alone is a weak measure: it rewards bundling unrelated work, and some high-risk changes should stay deliberately small. Keep two kinds of improvement apart in the ledger. Execution acceleration makes an existing task faster or cheaper. Scope expansion lets one validated attempt replace several handoffs, removing context reconstruction and integration work rather than speeding it up. Both are worth having, and they need different evidence.
Your first artifact is a small evaluation ledger: five representative tasks, expected outcomes, independent checks, observed attempts, and a decision about the next experiment. Five is a manageable starting exercise, not a statistical certification threshold.