Stop Picking Your Coding-Agent Model Off a Leaderboard

Your platform team is about to swap coding-agent models over a two-point leaderboard gap. Anthropic measured six percentage points of movement on Terminal-Bench 2.0 from infrastructure configuration alone — model, harness, and task set all held fixed (Anthropic: Infrastructure Noise¹). The swap is a real production change with a real rollback cost, and the evidence line justifying it is smaller than the measurement error of the instrument that produced it.

Read the Gap Against the Noise Floor

A two-point gap on an agentic coding leaderboard is not a small advantage over the row below it. It is a number smaller than every published source of variance in the measurement that produced it.

The artifact here is docs/decisions/0007-swap-default-coding-model.md — the swap memo your platform team circulated on Monday. Evidence line one reads: Model B leads Model A by 2.1 points on SWE-bench Verified. Everything downstream in ADR-0007 hangs off that line: the migration plan, the cost delta, the rollout window, the two weeks of engineering time.

Anthropic ran six compute-resource configurations on GKE against the same model, the same harness, and the same task set. The only variable was how much CPU and memory the eval container was allowed to use.

Agentic coding benchmarks like SWE-bench and Terminal-Bench are commonly used to compare the software engineering capabilities of frontier models—with top spots on leaderboards often separated by just a few percentage points. Infrastructure configuration alone can produce differences that exceed those margins.

The gap between the most- and least-resourced setups on Terminal-Bench 2.0 was 6 percentage points (p < 0.01), and the infrastructure error rate fell monotonically from 5.8% under strict enforcement to 0.5% when uncapped (Anthropic: Infrastructure Noise¹). None of that movement is capability. All of it lands in the same column ADR-0007 is reading as capability.

Evan Miller’s error-bar work supplies the arithmetic underneath. Detecting an absolute difference of 0.03 at least 80% of the time requires an eval containing at least n≈969 independent questions. SWE-bench Verified holds 500 (Evan Miller: Adding Error Bars to Evals²; OpenAI: Introducing SWE-bench Verified³). Miller also shows clustered standard errors running over 3X larger than naive ones. Failing to adjust for clustered sampling “may lead an unsuspecting analyst to suppose that the measurement of the overall eval score is much more precise than it actually is.”

Run-to-run variance closes the remaining distance. A 2026 stability study ran three SWE-bench-family agents five times each and reports standard deviations of 2.0, 2.2, and 3.4 percentage points. Its own best agent posts a Pass@5 of 70.0% against an All@5 of 40.0% — a thirty-point spread between “solved it at least once” and “solved it every time” (Guo et al.: SWE-Doctor⁴). At the level of a single task the instability is starker: the same model on the same SaaS workflow scored 0.000, 0.214, and 0.679 across three runs, which the authors call “a qualitative, not merely quantitative, difference” (Shi et al.: SaaS-Bench⁵).

None of this is exotic. It is why serious evaluators publish intervals instead of ranks — Epoch AI runs most models 16 times on GPQA Diamond and 8 times on MATH Level 5 and plots one standard error around the mean (Epoch AI: About Our Benchmarking⁶). METR reports a 50%-time horizon of 2h17m with a 95% confidence interval of 65m to 4h25m (METR: Evaluation of OpenAI GPT-5⁷). When rank confidence intervals are computed and displayed, orderings collapse. Neuhof and Benjamini find cases where “the rank CIs indicate that the observed ranking cannot be trusted and all models are statistically interchangeable,” and note in companion work that “rankings are typically presented as single values, without any quantification of their uncertainty” (Neuhof & Benjamini: Rank Intervals for Leaderboards⁸; Neuhof & Benjamini: Quantifying Ranking Uncertainty⁹).

Author’s judgment. No single source states “a 2.1-point gap on an agentic coding leaderboard is a statistical tie.” That verdict is my composite of four sourced premises: infrastructure alone moves the score 6pp, run-to-run SD is 2.0-3.4pp, the benchmark is a 500-sample set well under the n≈969 needed to resolve a 3-point delta, and rank intervals on other benchmarks routinely overlap across the top of the table.

Read against that floor, ADR-0007’s evidence line does not say Model B is better. It says the two models are tied and one of them got the better draw. The decision question is therefore: is the gap you are citing larger than the noise the benchmark generates on its own?

Check Whether the Benchmark Measures Anything

Verify the instrument before you read the dial. ADR-0007’s metric column names SWE-bench Verified, and SWE-bench Verified is a benchmark its own publisher has stopped evaluating on.

OpenAI audited it and found “at least 59.4% of the audited problems have flawed test cases that reject functionally correct submissions” — 35.5% with tests strict enough to enforce specific implementation details, 18.8% checking functionality the problem statement never described, and 5.1% miscellaneous (OpenAI: Why We No Longer Evaluate SWE-bench Verified¹⁰). The contamination finding is worse than the test-quality finding: every frontier model OpenAI tested could reproduce the original human-written bug fix used as the ground-truth reference. Scores on the benchmark rose from 74.9% to 80.9% over six months while OpenAI concluded the rise meant nothing about coding ability.

This means that improvements on SWE-bench Verified no longer reflect meaningful improvements in models’ real-world software development abilities. Instead, they increasingly reflect how much the model was exposed to the benchmark at training time.

This is not a benchmark that went bad recently. SWE-bench Verified exists because the original SWE-bench was already broken. OpenAI annotated 1,699 random samples and filtered out 68.3% of them “due to underspecification, unfair unit tests, or other issues” — 38.3% flagged for underspecified problem statements, 61.1% flagged for unit tests that may unfairly mark valid solutions as incorrect (OpenAI: Introducing SWE-bench Verified³). The 500-sample instrument in ADR-0007’s metric column is the salvage operation, and the salvage operation has now been audited and set down.

The exploitability findings are the part that should end the conversation. Berkeley RDI audited eight major agent benchmarks and reports that “every single one can be exploited to achieve near-perfect scores without solving a single task” (Berkeley RDI: How We Broke Top AI Agent Benchmarks¹¹). One exploit is a conftest.py file of ten lines of Python that resolves every instance on SWE-bench Verified by hooking pytest to report passes. This is not hypothetical: the same audit found a real leaderboard entry, IQuest-Coder-V1, where “24.4% of its trajectories simply ran git log to copy the answer from commit history.” It also cites reward hacking in 30%+ of evaluation runs for o3 and Claude 3.7 Sonnet. The companion paper surfaces “219 distinct flaws across the eight classes” (Wang et al.: BenchJack¹²).

Even honest runs are inflated by memorization and leakage. Frontier models reach “up to 76% accuracy in identifying buggy file paths using only issue descriptions” without access to repository structure, falling to “only up to 53% on tasks from repositories not included in SWE-Bench” (Liang et al.: The SWE-Bench Illusion¹³). An independent audit found that “32.67% of the successful patches involve cheating as the solutions were directly provided in the issue report or the comments,” and that filtering suspicious passes dropped one agent’s resolution rate from 12.47% to 3.97% (Aleithan et al.: SWE-Bench+¹⁴). On the ranking side, researchers documented “undisclosed private testing practices benefit a handful of providers who are able to test multiple variants before public release and retract scores if desired,” identifying 27 private Llama variants tested before the Llama-4 release (Singh et al.: The Leaderboard Illusion¹⁵). The platform disputes the finding, arguing “boosts in a model’s score due to pre-release testing are minimal” (Arena: Our Response¹⁶). Cite that one with the dispute attached.

ADR-0007 reports a 2.1-point reading from a retracted gauge. That is not a small evidentiary problem you can round past — it means the memo’s central number has no defined unit.

The Harness Moves the Score More Than the Model

Attribute a leaderboard delta to the harness before you attribute it to the model, because the controlled measurement says the harness wins. A 3x3 factorial study of long-horizon agent tasks decomposed the variance and found average harness variance of 18.48 pp² against average model variance of 2.37 pp² — a ratio of 7.80x.

in this regime, performance variance is governed more by harness configuration than by model choice, and current evaluation protocols therefore systematically misattribute harness-level gains to model improvements

The magnitudes are not marginal. The same paper reports Terminal-Bench 2 moving 69.7% to 77.0% from a harness change alone, SWE-bench Pro moving 45.9% to 55.4% for Claude Opus 4.5 under different harnesses, and cross-scaffold gaps on the HAL leaderboard of 34 points for one model and nearly 48 points for another (Zhang et al.¹⁷). Epoch AI’s analysis of what SWE-bench Verified actually measures lands in the same place: “A good scaffold can increase performance by up to 20%.” The conclusion follows directly — “performance on SWE-bench Verified reflects the sophistication of the scaffold as much as the capability of the underlying model” (Epoch AI: What Skills Does SWE-bench Verified Evaluate?¹⁸). A pre-registered controlled comparison found that “scaffold choice alone moves measured GAIA accuracy by as much as 28 percentage points within a single model” (Starace: Scaffold Effects on GAIA¹⁹).

This has been true since the founding work. The SWE-agent paper’s own ablation reports that “SWE-agent solves 10.7 percentage points more issues than the baseline agent that uses just the default Linux shell” — same model, different agent-computer interface (Yang et al.: SWE-agent²⁰). You can watch it happen live on a public leaderboard: Aider scores the same models under different edit formats, and gemini-exp-1206 posts 80.5% on whole against 69.2% on diff, while o1-mini posts 70.7% against 61.1% (Aider: Code Editing Leaderboard²¹). One model, one benchmark, one protocol change, ten points. Forecasting work treats elicitation as its own axis for exactly this reason, projecting 54% on SWE-bench Verified for low-elicitation agents against 87% for state-of-the-art ones in the same period (Pimpale et al.: Forecasting Frontier LM Agent Capabilities²²). METR’s elicitation guidelines concede outright that “it is hard to upper-bound what might be possible with clever prompting and tooling” (METR: Guidelines for Capability Elicitation²³).

So when a swap proposal puts two leaderboard rows side by side, read the pair in this order:

• When the two rows name different scaffolds, treat the delta as harness-attributable until proven otherwise — the variance decomposition puts the prior at roughly 7.8-to-1 against the model.
• When the two rows name the same scaffold but different edit formats, tool sets, or retry policies, you are still reading a harness delta; the Aider table moves ten points on edit format alone.
• When neither row discloses its harness, the comparison is unreadable and the correct move is to stop citing it, not to discount it.
• When both rows disclose a harness and neither is the agent runtime you operate, the delta transfers to your stack with unknown sign — a scaffold that suits Model B on someone else’s rig can invert on yours.

The disclosure your memo needs, and that almost no leaderboard row provides, is small enough to fit in a config block:

model:              ____   # exact version string, not family name
scaffold / agent:   ____   # SWE-agent, mini-SWE-agent, vendor-internal, ...
edit format:        ____   # whole | diff | udiff | tool-call
tool set:           ____   # shell only, or file viewer + search + linter
retry / trial policy: ____ # pass@1, pass@5, best-of-n
max turns:          ____
compute per task:   ____   # CPU/memory ceiling of the eval container

Every line you cannot fill in is a variable that moved the score for reasons unrelated to the model. And the harness is the one layer in this whole argument that you own outright — it is deterministic, inspectable, and yours to hold fixed, which is the case I make in the harness audit. The leaderboard measured someone else’s harness. You are deploying a third one that nobody measured.

Demote the Leaderboard to Shortlist Duty

The leaderboard still has one honest job, and it is upstream of the decision rather than at it: nominating two or three models worth trialing. That job is real work — you cannot eval seven models against your workload, and something has to cut the field.

What the leaderboard cannot do is tell you the model is your bottleneck.

I suggest not thinking of switching model as the main axes of how to improve your system off the bat without evidence. Does error analysis suggest that your model is the problem?

Husain and Shankar are direct about the ordering: “Error analysis is the most important activity in evals,” because it decides which evals are worth writing in the first place (Hamel Husain & Shreya Shankar²⁴). The leaderboard line and the error analysis answer different questions, and only one of them is about your codebase. Husain’s earlier framing is blunter still: “Don’t rely on generic evaluation frameworks to measure the quality of your AI. Instead, create an evaluation system specific to your problem” (Hamel Husain: Your AI Product Needs Evals²⁵).

Author’s judgment. The claim that most agent failures trace to task design, context, or harness before model capability is my inference, not a sourced statistic. It follows from two premises above: harness variance measures at 7.8x model variance in controlled conditions (Zhang et al.¹⁷), and error analysis is the activity that decides whether the model is the problem at all (Husain & Shankar²⁴).

Translate the memo’s evidence line before you argue about it:

• When a proposal cites a leaderboard gap as the reason to swap, rewrite the line as “Model B is a shortlist candidate” and see whether the rest of the memo still stands. In ADR-0007 it does not — nothing else in the document is independent evidence.
• When nobody has read the last month of failed agent runs, stop the swap and do error analysis first; a model change cannot fix a task-design failure or context rot in a bloated session.
• When the failures cluster in one repo, one layer, or one task shape, the fix is task decomposition or harness configuration, and the shortlist can wait.
• When error analysis does implicate raw capability, the leaderboard has done its job: take the top two or three rows and move to the eval gate.

That distrust-the-self-report discipline is the same one I apply one level down, at the individual diff, in evaluating AI coding agent output — an agent’s claim that it succeeded is a hypothesis, and so is a vendor’s. Ranking content promises a shortcut around that work, which is why the internet ranks models endlessly and why I keep pushing back on it.

Gate the Swap on a Workload-Weighted Eval

Build the eval suite from your own agents’ failures, not from the benchmark’s task list. Anthropic’s guidance sets the floor lower than most teams assume: “We see teams delay building evals because they think they need hundreds of tasks. In reality, 20-50 simple tasks drawn from real failures is a great start” (Anthropic: Demystifying Evals for AI Agents²⁶). The same guidance supplies the vocabulary — a task is a single test with defined inputs and success criteria, each attempt at it is a trial, and the outcome is the final state in the environment rather than what the agent said it did.

The gate that would earn ADR-0007 an approval:

1. Pull 20-50 tasks from real failures. Mine the last quarter of agent runs in your repos, not curated GitHub issues. Each task gets defined inputs and a success criterion checked against environment state.
2. Weight the suite by workload frequency. If 40% of your agent traffic is test authoring and 5% is migrations, the suite reflects that split — a model that wins on the tasks you rarely run is not a win.
3. Hold the harness fixed and vary only the model. Same scaffold, same tools, same retry policy, same resource ceiling.
4. Run multiple trials per task and set the regression bar before you look. Write down the pass threshold and the per-task regressions you will not tolerate, then run. A bar set after the results is not a bar.
5. Stage the rollout with a revert path. One team, one repo, one week, with the old model one config flag away — keep the blast radius small enough that a bad swap costs a rollback rather than a quarter.

for model in model-a model-b; do
  ./eval/run.sh \
    --suite tasks/workload-weighted/ \
    --harness prod \
    --trials 5 \
    --model "$model" \
    --out "results/${model}.json"
done

./eval/compare.sh results/model-a.json results/model-b.json --bar regression.yaml

Five trials per task is not ceremony. It is the direct consequence of the 2.0-3.4pp run-to-run standard deviations above — a single trial per model reproduces the exact defect the leaderboard has.

Treat the swap as a production change because vendors treat it that way internally. After a run of quality complaints, Anthropic’s postmortem on Claude Code committed to run “a broad suite of per-model evals for every system prompt change to Claude Code, continuing ablations to understand the impact of each line.” One of those evals showed a 3% drop for both Opus 4.6 and 4.7 (Anthropic: An Update on Recent Claude Code Quality Reports²⁷). The root causes are instructive for anyone who thinks the model is the variable that matters: a default reasoning-effort change from high to medium and a caching bug that cleared thinking on every turn — runtime configuration, not model weights, producing user-visible quality regressions.

The regressions also do not respect the direction of the upgrade. One eval vendor measured a customer agent’s prompt-injection resistance dropping from 94% to 71% on their harness after a GPT-4o to GPT-4.1 upgrade. Their conclusion is the sentence to paste into the memo: “Run your own tests on your own data. Third-party numbers are a starting point, not a finish line” (Promptfoo: Your Model Upgrade Just Broke Your Agent’s Safety²⁸). A newer model is not a strictly better model, and capability gains do not carry safety or format-compliance gains with them.

Rewritten, ADR-0007’s evidence line stops referencing the leaderboard at all. It reads: On 34 workload-weighted tasks drawn from our own failed runs, five trials each, harness held fixed, Model B resolved 71% against Model A’s 62%, with no per-task regressions above the bar. That is bounded, verifiable work, produced on the system you actually operate — and it is the only version of the memo that survives review.

The whole gate — entry condition through staged rollout, each predicate with its source attached — is a checklist you can paste straight into the swap ADR: model-swap-gate²⁹.

The Swap Decision, Compressed