Diff Review Can’t See Your Agent Regress. A Standing Suite Can.

Every PR to your coding agent last quarter passed review, and none of those reviews could have caught what Chen, Zaharia, and Zou measured. Under the same model name, GPT-4’s directly executable code rate fell from 52% to 10% between March and June 2023, with no diff to review (Chen, Zaharia, Zou: How is ChatGPT’s behavior changing over time?¹). The likely cause was mundane (the June version added non-code text to its generations). The authors also name why it hurts: a formatting shift like that is hard to detect when generated code sits inside a larger software pipeline. Your coding agent is that pipeline, and per-diff review was never built to see it move.

Diff Review Checks the PR, Not the System

A quarter of green reviews is evidence that each change looked right to the person who read it. It is not evidence that the agent still does everything it did in June. The per-diff reviewer’s framework is deliberately scoped to one diff, because the next question needs a different instrument. Anthropic names that instrument. A regression eval asks, “Does the agent still handle all the tasks it used to?” and should hold a nearly 100% pass rate (Anthropic Engineering: Demystifying evals for AI agents²).

Here is one quarter of change log for one team’s coding agent. It recurs through the rest of the post.

Author’s judgment. This change log is an illustrative composite, not one real team’s history. CL-2, CL-3, and CL-4 are modeled on sourced events, cited in the trigger section; the other entries are typical of the agent-system changes the sources describe.

CL-1  Jul 08  CLAUDE.md: add "prefer the repo's test helpers in tests/helpers/"  review: approved
CL-2  Jul 22  harness: add a code-search subagent to the agent's tools          review: approved
CL-3  Aug 05  config: model alias silently resolves to a newer snapshot        review: none, no diff
CL-4  Aug 19  vendor: serving-side change at the model provider                review: none, no diff
CL-5  Sep 02  capability: the agent now writes database migrations             review: approved
CL-6  Sep 16  incident: PR #2214 reported "tests pass"; rollback never written  review: approved
CL-7  Sep 23  evals: loosen the golden baseline on a flaky migration case      review: approved

Read it the way the reviewers did and nothing is wrong. CL-1 is reasonable context engineering, CL-2 adds a tool, CL-5 ships a feature, and CL-7 fixes a flaky test. CL-3 and CL-4 never produced a diff, so review on them was trivially green. CL-6 is the only entry that looks like a failure, and it shipped through review too; a reviewer caught the missing rollback days later. The question none of those reviews could answer is therefore: does the agent still handle everything it handled last month?

Google hit the same wall with ordinary software. At one point, more than 80% of Google Web Server production pushes contained user-affecting bugs that had to be rolled back. The fix was not a sharper reading of each change. It was a policy that all new code changes include tests and that those tests run continuously. Within a year, emergency pushes dropped by half (Software Engineering at Google: Testing Overview³).

Learned systems widen the gap. Sculley and colleagues argued that unit and end-to-end tests are valuable, “but in the face of a changing world such tests are not sufficient to provide evidence that a system is working as intended” (Sculley et al.: Hidden Technical Debt in Machine Learning Systems⁴). Google’s ML Test Score separates a system that worked at launch from one that keeps working. It awards full credit for a test only when a system runs it automatically on a repeated basis (Breck et al.: The ML Test Score⁵). OpenAI’s cookbook on multi-agent systems compresses the split:

The core lesson is simple: agent-level evals tell us which local behaviors look risky, while macro evals tell us what those risks become at system scale.

Per-diff review is the agent-level eval of your engineering process. The rest of this post builds the system-level one.

Trigger on Changes Nobody Diffs

The standard advice is to rerun your evals when you swap models, which covers one row of the change log out of seven. Wire the suite to every change to the agent system: prompt and CLAUDE.md, harness and tools, model alias or version. Then add a scheduled run for the vendor-side changes you will never see a diff for. OpenAI’s guidance is to “Set up continuous evaluation (CE) to run evals on every change” (OpenAI: Evaluation best practices⁷). Anthropic puts automated evals in CI/CD, “running on each agent change and model upgrade as the first line of defense against quality problems” (Anthropic Engineering: Demystifying evals for AI agents²).

CL-1 looks like documentation, but it is configuration that changes behavior. Gloaguen and colleagues found that “providing context files does not generally improve task success rates, while increasing inference cost by over 20% on average” (Gloaguen et al.: Evaluating AGENTS.md⁸). The old ML-systems rule applies: configurations “should undergo a full code review and be checked into a repository” (Sculley et al.: Hidden Technical Debt in Machine Learning Systems⁴). Review is half of it; the other half is a test run. One Google team found that running temporary environments on presubmit “prevented 95% of broken servers from bad configuration” (Software Engineering at Google: Continuous Integration¹²).

CL-2 is a harness change, and one tool can be a model-upgrade-sized event. Zhang and colleagues report that adding the WarpGrep search subagent on identical infrastructure “adds 2.1 to 2.2 points across models, comparable to a routine model upgrade and sufficient to flip the MiniMax 2.5 vs Claude Opus 4.6 ordering” (Zhang et al.: Stop Comparing LLM Agents Without Disclosing the Harness⁹). The broader harness-versus-model variance numbers live in the leaderboard noise post.

CL-3 and CL-4 matter most, because nothing in your repository changes. An alias like OpenRouter’s ~anthropic/claude-fable-latest “updates whenever the author publishes a new version” (OpenRouter: AI Agent Regression Testing After a Prompt or Model Change¹⁰). Serving changes leave you nothing to pin at all. In the August to early September 2025 incidents Anthropic’s postmortem describes, 16% of Sonnet 4 requests were affected in the worst hour. Approximately 30% of Claude Code users who made requests during the period had at least one message routed to the wrong server type (Anthropic Engineering: A postmortem of three recent issues¹¹).

These issues exposed critical gaps that we should have identified earlier. The evaluations we ran simply didn’t capture the degradation users were reporting…

Their remedy is the last row of the table: run evaluations “continuously on true production systems.” The wiring:

• Path-filter the agent’s own files. Trigger the per-change tier on CLAUDE.md, .claude/**, harness/**, and the model config, not only on application code.
• Pin the alias and check it. Record the concrete snapshot every green run used, and fail CI when the alias resolves to something new.
• Schedule a nightly run. The cron is the only trigger CL-4 will ever fire.
• Keep both. Autonoma’s Tom Piaggio names the failure on each side. Path-filtered only, “a silent model swap goes unnoticed”; nightly only, a bad prompt edit sits “for up to 24 hours before anyone finds out” (Autonoma: Agent Regression Testing¹³).

Run Capability and Regression as Two Suites

CL-5 is where a single pass rate starts lying. The agent now writes database migrations, the team adds migration cases, and the headline number rises while something else quietly breaks. Keep two suites with opposite targets. Capability evals “should start at a low pass rate, targeting tasks the agent struggles with,” while regression evals should have “a nearly 100% pass rate” (Anthropic Engineering: Demystifying evals for AI agents²).

The numbers below are illustrative, for the week CL-5 merged:

Bad: “Agent eval pass rate went from 81% to 84% after CL-5. Migrations work now. Ship it.”

Good: “Capability, migrations: 3/12 to 5/12, still climbing. Regression: 150/150 to 149/150, with reg/handler-error-mapping-422 failing since CL-5. Block on the regression case; the migration gain ships with it, not instead of it.”

The blended number averages a new win against a lost behavior, and a lost behavior is the one thing a regression suite exists to refuse. Capability at 5/12 says the migration work is real and unfinished, which is the normal state of a capability suite. Regression at 149/150 says the change cost the agent something it already had.

Cases move between the suites on purpose:

After an agent is launched and optimized, capability evals with high pass rates can ‘graduate’ to become a regression suite that is run continuously to catch any drift.

Note the “can” and the “after.” A migration case graduates once it passes reliably across trials, not the first week it goes green. Anthropic is explicit about what a regression suite at 100% buys you: “An eval at 100% tracks regressions but provides no signal for improvement” (Anthropic Engineering: Demystifying evals for AI agents²). Traffic also runs the other way. The macro-evals cookbook recommends that teams “promote the clearest lower-level eval failures into a regression suite” (OpenAI Cookbook: Macro Evals for Agentic Systems⁶), which is the intake path two sections down.

Tier Each Eval by Cost and Stakes

Running every eval on every change is a footgun with a bill attached. Give each eval its own cadence from three inputs: what it costs to run, whether it is saturated, and what a miss costs.

Weigh the cost and how saturated the eval is against the business value of catching the error.

Their example table spans the range. A cheap code assertion runs “on every change b/c its incredibly cheap,” and a mid-cost judge runs “somewhat frequently (e.g. nightly).” An expensive high-stakes judge runs “prior to each release,” and a saturated one is retired or run every two weeks. For the per-change tier they are blunter: “Favor assertions or other deterministic checks over LLM-as-judge evaluators” (Hamel Husain, Shreya Shankar: Q: How are evaluations used differently in CI/CD vs. monitoring production?¹⁵). Google’s CI chapter draws the same split: “CI should optimize quicker, more reliable tests on presubmit and slower, less deterministic tests on post-submit” (Software Engineering at Google: Continuous Integration¹²).

Here is the change-log team’s suite as a manifest, each eval declaring its inputs and resulting tier, plus the CI wiring that reads it:

suite_version: 2026.09.3
evals:
  - id: reg/handler-error-mapping-422
    suite: regression
    kind: code_assertion          # deterministic, no model call
    cost_per_run_usd: 0.02
    trials: 1
    stakes: high
    pass_rate_30d: 1.00
    owner: payments-api
    tier: per_change

  - id: reg/repo-wide-agent-run
    suite: regression
    kind: agent_run               # full task, real repo, live model endpoint
    cost_per_run_usd: 3.80
    trials: 5
    stakes: high
    pass_rate_30d: 0.98
    owner: eval-platform
    tier: nightly                 # the run that catches CL-4

  - id: cap/migrations/reversible-migration
    suite: capability
    kind: agent_run
    cost_per_run_usd: 4.10
    trials: 5
    stakes: high
    pass_rate_30d: 0.42
    owner: data-platform
    tier: nightly

  - id: reg/pr-description-tone-judge
    suite: regression
    kind: llm_judge
    cost_per_run_usd: 0.60
    trials: 3
    stakes: low
    pass_rate_30d: 1.00
    saturated: true               # harden first, then retire
    owner: devex
    tier: harden_then_retire

on:
  pull_request:
    paths: ["CLAUDE.md", ".claude/**", "harness/**", "agent.config.yaml", "evals/**"]
  schedule:
    - cron: "0 3 * * *"
jobs:
  per-change:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: ./scripts/run-evals.sh --tier per_change
  nightly:
    if: github.event_name == 'schedule'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: ./scripts/run-evals.sh --tier nightly --record-suite-version

The path filter from the trigger section fires only the cheap tier, so CL-1 and CL-2 pay cents per PR. Multi-trial agent runs wait for the cron, and the nightly reg/repo-wide-agent-run against the live endpoint is what catches CL-4, the change no PR ever touched. Where the checker runs and why an LLM judge stays off the blocking gate are covered in the done-criteria post.

If you want the tiering done for you, eval-cadence¹⁶ reads a manifest like this one. It prices every eval’s full run against your daily budget, flags evals that are unowned, flaky, saturated or in the wrong tier, and writes the workflow above from the manifest.

Saturated evals get one more step before retiring: “you should first try to make the eval more difficult so its not saturated to begin with” (Hamel Husain, Shreya Shankar: Q: How often should I run my evals?¹⁴). Flaky evals get no grace period. Google’s testing chapter warns that “As you approach 1% flakiness, the tests begin to lose value” (Software Engineering at Google: Testing Overview³). One Google team with flaky presubmits saw, on some days, more than 50 code changes bypass and ignore the test results (Software Engineering at Google: Continuous Integration¹²). Multi-trial agent runs are noisy by construction, which is one more reason they belong in the nightly tier rather than on the PR.

Price the Full Run Before You Schedule It

Put a dollar figure on one full multi-trial run, and let that number decide its tier instead of habit. Kapoor and colleagues priced one. SWE-Agent’s authors set a cost limit of USD 4 per task, so the full benchmark “could cost over USD 8,000 for a single evaluation run” (Kapoor et al.: AI Agents That Matter¹⁷). The consequence is the part that should worry you:

The high cost makes it infeasible to run evaluations multiple times, and perhaps as a result, agent evaluations are rarely accompanied by error bars.

The same group’s Holistic Agent Leaderboard cost about $40,000 in total. It skipped one model on one benchmark because the team estimated that run alone at about $20,000 (Kapoor et al.: Holistic Agent Leaderboard¹⁸). Your suite is smaller, but the arithmetic has the same shape.

Route Every Failure Through Intake

CL-6 is what a miss looks like. The agent reported “tests pass” on PR #2214, Add refund_reason column to payments, and the migration’s rollback step was never written. No test exercised migrate down, so the self-report was technically true. No case written in advance would have flagged it, and that is not a gap you close by writing more cases up front.

Wei Wu’s eight-week study of one production agent runtime makes the limit precise. Of 22 incidents with full root-cause postmortems, the team audited 15 retrospectively. Ex-ante prevention was 0 of 15 (0%), and ex-post regression blocking was 13 of 15 (87%). In 12 of 15 (80%), the root cause of the miss was “a dimension the audit had never conceived.” Roughly 70% of silent failures were caught by human user-view observation of system output, “not by unit tests, health checks, or governance audits” (Wei Wu: When Errors Become Narratives²⁰). One runtime, so the frequencies don’t generalize, but the shape does. In Wu’s own phrase, “audits are regression engines, not prediction engines.” The suite blocks failures you already had and predicts none of the new ones. So keep humans watching output (Wu’s team made it a weekly 30-minute observation ritual, no coding allowed), and run every miss through intake.

Unless we have some formalized process of learning from these incidents in place, they may recur ad infinitum.

The SRE book adds the step most teams skip: “define postmortem criteria before an incident occurs so that everyone knows when a postmortem is necessary.” Adapted from its trigger list, set these now:

• User-visible degradation. An agent PR shipped a defect that a reviewer or user found after merge. CL-6 qualifies.
• Data loss of any kind. A migration, deletion, or overwrite the agent wrote or ran incorrectly.
• Human intervention. Someone reverted an agent PR or took a task back by hand.
• Resolution time over threshold. An agent task ran past its time or token budget.
• Monitoring failure. The agent’s self-report said done and a human found otherwise. The SRE list notes this usually implies manual incident discovery, which is exactly how CL-6 surfaced.

Then convert each triggered incident into a case:

1. Reproduce it from the trace. The trace schema post covers which fields make a run replayable.
2. Minimize it. Shrink to the smallest repo state and task that still produce the failure.
3. Add it as a regression case with a deterministic check: reg/migrations/rollback-present asserts every up migration has a down and that migrate down exits 0.
4. Link the incident. Put incident: INC-0916 in the case metadata, so the next person to touch the case knows why it exists.
5. Prove it bites. Confirm the case fails on the pre-fix agent and passes on the fix before you merge it.

Anthropic says to “look at your bug tracker and support queue” and convert user-reported failures into test cases (Anthropic Engineering: Demystifying evals for AI agents²). Husain and Shankar say to “Always analyze after incidents, user complaint spikes, or metric drift” (Hamel Husain, Shreya Shankar: Q: How often should I re-run error analysis?²²), then “add representative examples to your CI dataset. This mitigates regressions on new issues” (Hamel Husain, Shreya Shankar: Q: How are evaluations used differently in CI/CD vs. monitoring production?¹⁵). The practice predates LLMs. In Shankar’s interview study, ML engineers “reported processes to analyze live failure modes and update the validation datasets.” One described every failed prediction landing in a queue that three people worked through once a week (Shankar et al.: Operationalizing Machine Learning²³). OpenAI’s improvement-loop cookbook calls it a flywheel: “Traces show what happened, feedback explains what mattered, evals make those expectations reusable” (OpenAI Cookbook: Build an Agent Improvement Loop with Traces, Evals, and Codex²⁴).

Without intake, incident reports are noise you can’t attribute. Anthropic’s postmortem admits that even as reports climbed, “we lacked a clear way to connect these to each of our recent changes” (Anthropic Engineering: A postmortem of three recent issues¹¹). A versioned suite with linked incidents is that connection.

Name an Owner and Version the Suite

CL-7 is the suite regressing, not the agent. A week after CL-6 became reg/migrations/rollback-present, the case started flaking (the test database occasionally timed out on migrate down), and someone fixed it:

 id: reg/migrations/rollback-present
 incident: INC-0916
-check: every up has a down, and `migrate down` exits 0
+check: every up has a down
 trials: 3
-pass_threshold: 3/3
+pass_threshold: 2/3

The PR was approved as a flake fix. It also deleted the half of the check that would have caught PR #2214. Nobody who owned the migration behavior saw it, because nobody owned it.

Split ownership on purpose. Anthropic’s version: “What proved most effective was establishing dedicated evals teams to own the core infrastructure, while domain experts and product teams contribute most eval tasks” (Anthropic Engineering: Demystifying evals for AI agents²). Shankar’s interviews found that higher-stakes ML applications “created separate teams to manage the dynamic evaluation process.” So did the opposite failure: “every engineer working on a particular model had a cloned version of the main evaluation notebook, with a few changes” (Shankar et al.: Operationalizing Machine Learning²³). The platform function owns the harness; the team that owns migrations owns reg/migrations/rollback-present and the definition of correct inside it. Then review every suite change the way Autonoma’s practitioners do:

The teams that keep this running treat the golden set the way they’d treat a schema migration: reviewed in a pull request, not edited silently.

The suite drifts even where measurement is the whole job. METR’s Time Horizon 1.1 grew from 170 to 228 tasks (73 added, 15 removed, 53 updated). METR found the trend “somewhat sensitive to task composition” (METR: Time Horizon 1.1²⁵). Criteria drift too: “users need criteria to grade outputs, but grading outputs helps users define criteria” (Shankar et al.: Who Validates the Validators?²⁶). That study is about individual graders, so extending it to a suite is an analogy. It still explains why Husain describes strong teams that “treat evaluation criteria as living documents” (Hamel Husain: A Field Guide to Rapidly Improving AI Products²⁷). The ML Test Score names the golden-file version of the problem: such tests are hard to maintain over time without blindly updating the golden file (Breck et al.: The ML Test Score⁵). CL-7 is a blind update with a PR attached.

Tian Pan supplies the gate: “Make ‘who reviewed the evals?’ a release-gate question,” and “Version the eval set and tie it to a date” (Tian Pan: The Eval Suite That Became the Spec Nobody Agreed To²⁸). Owning the suite also means owning where it lives. OpenAI’s hosted Evals platform becomes read-only for existing users on October 31, 2026, and is scheduled to shut down on November 30, 2026 (OpenAI: Evaluation best practices⁷).

Start While Requirements Still Map to Cases

The cheapest time to build the suite is while product requirements still translate directly into test cases.

Evals get harder to build the longer you wait. Early on, product requirements naturally translate into test cases. Wait too long and you’re reverse-engineering success criteria from a live system.

Harder is not impossible. Anthropic’s own late-start example is the Bolt AI team, which “started building evals later, after they already had a widely used agent.” It still needed a quarter: “In 3 months, they built an eval system” (Anthropic Engineering: Demystifying evals for AI agents²). The same post concedes that some teams reasonably add evals only once they reach scale. So the claim here is about difficulty, not a compounding-cost law. OpenAI is less forgiving, listing “waiting until you ship before implementing any evals” under its vibe-based-evals anti-pattern (OpenAI: Evaluation best practices⁷).

The practical argument is about memory. Today, the team that added migration support can write cap/migrations/reversible-migration in an afternoon, because the requirement is still in a ticket. In six months, the same case means reading old PRs to work out whether the agent was ever supposed to write the down step. If you have no suite yet, build the first one from your own agent’s real failures. The leaderboard noise post covers sizing it and the task, trial, and outcome vocabulary. Everything above is what keeps that first suite standing.

The Standing Suite, Compressed

Author’s judgment. The “failure if skipped” column is my synthesis. Each cell follows from the sourced premises in its section, but no source states the row as written.

Use this as the charter for your suite: one row per section, the decision it forces, and what the change log looked like without it.