The evidence base behind everything I publish on running AI coding agents in production. Every entry is a verbatim figure or quote from a primary source — a study, a benchmark, an engineering post — pulled while researching a post, then checked against the live page. Sources that drift or go dead are dropped or flagged.
191
Verified sources
6
Themes
67 / 124
Data / practitioner
2026-07-13
Last verified
Every entry checked against its live source · dataset: research.json
>
191 / 191 sources
01
Task Design & Decomposition
23 sources
Scoping, decomposing, and speccing work so an agent finishes it on the first try.
Alif Al Hasan, Sumon Biswas (Case Western Reserve University)May 29, 2026data
Across 547 confirmed real-world safety failures mined from the GitHub issue trackers of 13 foundational code models, the top threat category is Constraint Violations at 40.4%, ahead of Destructive Operations (24.5%), Authorization Bypasses (18.3%), and Deception (15.7%).
Independent corroboration of the constraint-violation finding by a different dataset and method - two teams reaching the same top category within two points.
3 more excerpts
Failures arise during benign, goal-directed use rather than adversarial attack
Nearly 60% of confirmed incidents were rated high or critical severity
The quantified breakdown appears in the full text, not on the arXiv abstract page
80% of tool calls come from agents that appear to have at least one kind of safeguard (like restricted permissions or human approval requirements), 73% appear to have a human in the loop in some way, and only 0.8% of actions appear to be irreversible
Irreversible agent actions are rare in real traffic, so oversight should concentrate on the small slice where a single error is costly.
2 more excerpts
such as sending an email to a customer
And while these higher-risk actions are rare as a share of overall traffic, the consequences of a single error can still be significant.
Arpandeep Khatua, Hao Zhu, Peter Tran, Arya Prabhudesai, Frederic Sadrieh, Johann K. Lieberwirth, Xinkai Yu, Yicheng Fu, Michael J. Ryan, Jiaxin Pei, Diyi Yang (Stanford, SAP Labs)Jan 19, 2026 (v1); revised Jan 26, 2026 (v2)data
Across 600+ collaborative coding tasks in 12 libraries and 4 languages, agents working together achieve on average 30% lower success rates than the same agents doing both tasks individually - the 'curse of coordination'. GPT-5 and Claude Sonnet 4.5 configurations reach only 25% under two-agent cooperation, roughly half the solo baseline. 77.3% of tasks have conflicting ground-truth solutions.
Without assigned file and interface ownership, parallel agents duplicate work and overwrite changes they believe will merge cleanly - the collision is the default outcome, not an edge case.
4 more excerpts
Adding a messaging tool did not help: the difference between 'with comm' and 'no comm' settings is not statistically significant for task success, though it did reduce literal merge conflicts
The paper separates two problems: merge conflicts are spatial coordination (who edits which lines), while task success requires semantic coordination (what to implement, not just where)
Agents were given no pre-assigned file ownership and were free to redivide the features between themselves
Scope limit: no experimental arm tested pre-assigned ownership as a fix, so the benchmark diagnoses the problem without validating the cure
Kwa, West, Becker, et al. (METR)submitted 2025-03-18data
frontier AI time horizon has been doubling approximately every seven months since 2019, though the trend may have accelerated in 2024
The primary paper behind the autonomy trend confirms a ~7-month doubling of the 50%-task-completion time horizon since 2019, driven mainly by greater reliability and error-adaptation — the mechanism that inflates calls per task.
4 more excerpts
"Claude 3.7 Sonnet have a 50% time horizon of around 50 minutes".
"within 5 years, AI systems will be capable of automating many software tasks that currently take humans a month".
"The increase in AI models' time horizons seems to be primarily driven by greater reliability and ability to adapt to mistakes"
50%-task-completion time horizon. This is the time humans typically take to complete tasks that AI models can complete with 50% success rate
The length of tasks (measured by how long they take human professionals) that generalist frontier model agents can complete autonomously with 50% reliability has been doubling approximately every 7 months for the last 6 years.
The autonomous task length frontier agents can complete has doubled roughly every 7 months for 6 years, so autonomous runs — and the per-task call count behind them — keep growing.
4 more excerpts
current models have almost 100% success rate on tasks taking humans less than 4 minutes, but succeed <10% of the time on tasks taking more than around 4 hours
"If the measured trend from the past 6 years continues for 2-4 more years, generalist autonomous agents will be capable of performing a wide range of week-long tasks."
the best current models—such as Claude 3.7 Sonnet—are capable of some tasks that take even expert humans hours, but can only reliably complete tasks of up to a few minutes long
AI agents often seem to struggle with stringing together longer sequences of actions
METR (Becker, Rush, Barnes, Rein)July 10, 2025data
When developers are allowed to use AI tools, they take 19% longer to complete issues—a significant slowdown that goes against developer beliefs and expert forecasts.
Experienced developers were measurably slower with AI in codebases they know well, contradicting their own forecasts of a speedup.
4 more excerpts
16 experienced developers from large open-source repositories (averaging 22k+ stars and 1M+ lines of code)
developers expected AI to speed them up by 24%
they still believed AI had sped them up by 20%
developers estimated that they were sped up by 20% on average when using AI—so they were mistaken
Across 20,574 coding-agent sessions from 1,639 repositories, the most prevalent misalignment symptom is Developer Constraint Violation - defined as violating an explicit developer constraint - at 38.33% of episodes, with 73.68% of those attributed to instruction-following failure. The separate underspecification cause (C1) accounts for only 15.36%.
The dominant measured failure is agents breaking constraints developers already stated, not developers failing to state them - which is why sharpening prompt prose does not address the main failure mode.
4 more excerpts
The symptom taxonomy is explicitly multi-label - 29.56% of episodes carry two labels and 0.54% carry three or more - so the seven shares deliberately do not sum to 100%
90.50% of episodes impose effort and trust costs rather than irreversible system damage, yet 91.49% of visible resolutions still require explicit user correction
Misalignment compounds across sessions: probability of misalignment in the next session is 0.519 after an affected session versus 0.336 otherwise
Constraint violation is markedly worse in CLI sessions (49.49%) than IDE sessions (32.26%)
On a Kubernetes root-cause-analysis workload, a decomposition fixed at design time with no runtime branching cost 1,632 +/- 145 tokens in retries versus 904 +/- 17 for a monolithic run - 80.5% worse. Runtime-structured decomposition with schema-validated handoffs cut retry cost to 436 +/- 132, a 51.7% reduction against monolithic and 73.2% against static.
Decomposition is not automatically a win. Splitting work without runtime isolation adds rerun surface area, because a failure anywhere forces re-execution of every downstream subtask.
4 more excerpts
The mechanism is stated directly: 'fixed sequential execution must rerun all downstream subtasks from the point of failure'
Structuring is not free - the runtime-structured baseline run cost 2,716 +/- 424 tokens against 904 +/- 17 monolithic, so the trade only pays at a nonzero failure rate
Authors' limitation: both use cases are controlled scenarios at temperature 0 with low natural failure rates (0-2%), and token savings depend on deployment failure rates they did not measure at scale
Authors' limitation, load-bearing for this post: 'Decomposition policies are developer-authored and may not generalize to automatically derived graphs'
Tim Menzies, William Nichols, Forrest Shull, Lucas Layman (NC State, SEI-CMU, Fraunhofer CESE)2016data
Across 171 software projects from 2006 to 2014: 'We found no evidence for the delayed issue effect; i.e. the effort to resolve issues in a later phase was not consistently or substantially greater than when issues were resolved soon after their introduction.'
The classic exponential cost-of-delay curve does not replicate, so the case for planning before an agent runs has to rest on measured agent failure rates rather than shift-left folklore.
2 more excerpts
Requirements issues reaching system test showed roughly a 1.85x median resolution-time increase, against the 37-250x multipliers cited in the classic literature
Used in the post as an honesty move - it argues against a convenient cliche the author declined to use
Across CompSkillBench - 300 compositional queries over 2,209 real MCP server skills spanning 24 categories - standard LLM decomposition reaches only 34.2% category recall at the step level, making decomposition quality the primary bottleneck.
Granularity is the hard part of decomposition and the part models are worst at, which is why the task graph is drawn by a human rather than delegated to the agent.
1 more excerpt
Iterative Skill-Aware Decomposition raised decomposition accuracy from 51.0% to 67.7% (+32.7%, Wilcoxon p < 10^-6)
only 48% of developers consistently check AI-assisted code before committing it, even though 38% find that reviewing AI-generated logic actually requires more effort than reviewing human-written code.
Most teams under-review AI code even though reviewing it costs more effort, so the last-mile verification tax is real and often unpaid.
1 more excerpt
AI gets you 80% to an MVP; the last 20% requires patience, learning deeply or hiring engineers.
A user asked to "clean up old branches." The agent listed remote branches, constructed a pattern match, and issued a delete. This would be blocked since the request was vague, the action irreversible and destructive, and the user may have only meant to delete local branches.
Vague-plus-irreversible-plus-destructive is the dangerous combination to gate; a concrete incident shows why you don't delegate blast-radius actions blind.
4 more excerpts
Claude Code users approve 93% of permission prompts.
If a session accumulates 3 consecutive denials or 20 total, we stop the model and escalate to the human.
Destroy or exfiltrate. Cause irreversible loss by force-pushing over history, mass-deleting cloud storage, or sending internal data externally.
Instead, a false positive costs a single retry where the agent gets a nudge, reconsiders, and usually finds an alternative path.
'In 2026, the value of an engineer's contributions shifts to system architecture design, agent coordination, quality evaluation, and strategic problem decomposition.'
A frontier lab naming decomposition as a core emerging engineering skill - cited as Anthropic's position, not as independent evidence.
4 more excerpts
The landing page returns 200 but is gated behind a form, and the PDF is not text-extractable, so the quote could not be re-confirmed against a live fetch on Aug 3, 2026; it was carried from a verified full 17-page read earlier the same session
PDF fallback surface: https://resources.anthropic.com/hubfs/2026%20Agentic%20Coding%20Trends%20Report.pdf
The widely circulated line 'the bottleneck is no longer writing code but clarity about what to build' does NOT appear anywhere in this report and is not cited in the post
Verified adjacent data from the same report: developers use AI in roughly 60% of their work but report being able to fully delegate only 0-20% of tasks
Anthropic (Erik Schluntz and Barry Zhang)Dec 19, 2024practitioner
When building applications with LLMs, we recommend finding the simplest solution possible, and only increasing complexity when needed. This might mean not building agentic systems at all.
The default should be the simplest solution; reaching for an agent is a decision to justify, not an assumption.
4 more excerpts
They are typically just LLMs using tools based on environmental feedback in a loop.
Code solutions are verifiable through automated tests; Agents can iterate on solutions using test results as feedback
The autonomous nature of agents means higher costs, and the potential for compounding errors.
Agentic systems often trade latency and cost for better task performance, and you should consider when this tradeoff makes sense.
Spec-driven tooling applied to a small bug produced four user stories with sixteen acceptance criteria - a sledgehammer for a nut. 'All SDD approaches and definitions I've found are spec-first, but not all strive to be spec-anchored or spec-as-source.'
Over-specification is a real failure mode with a cost, which supplies the stop condition in the post's pricing checklist.
1 more excerpt
The most authoritative non-vendor treatment of the spec-first / spec-anchored / spec-as-source distinction found in this research pass
Daniel Epstein (Microsoft Developer Blog)May 19, 2026practitioner
Names the failure directly: 'No backlog: There is no structured list of what needs to be built, in what order, with what dependencies. Work gets discovered during implementation, not planned before it.' The prescribed fix is 'Specs in Backlog first: Every capability is an issue. Every issue has acceptance criteria.'
A named practitioner framing of the missing artifact this post builds - the dependency-ordered backlog that precedes any agent run.
2 more excerpts
The article contains no numbers, percentages, or named studies - it is argumentative, and the post cites it as practitioner framing only, never as measurement
A widely circulated line about 'the hardest step ... assumed rather than solved' is from a reader comment by Rolf Kristensen, not from Epstein's article, and is not cited in this post
Hamel Husain, interviewed by Sara Verdi (Arize)Jul 30, 2026practitioner
'A really common way that the model is not the problem is query disambiguation. The LLM doesn't have a chance because the user is asking a very ambiguous question.'
Ambiguity attaches to interface contracts, not just prose - an agent told to clean up an authentication service cannot know whether it may change the public API, add a dependency, touch the schema, or alter error behavior.
1 more excerpt
Backs the semantic half of the post's two-edge model: a node needs an owned contract, not only an owned file glob
Kent Beck (O'Reilly, 'Coding with AI: The End of Software Development As We Know It')session page, undated; underlying event May 8, 2025practitionerpartial
'Augmented coding deprecates formerly leveraged skills such as language expertise. Augmented coding amplifies vision, strategy, task breakdown, and feedback loops.'
Task breakdown is an appreciating skill under agentic coding, not a depreciating one - which is the argument for naming an owner rather than letting it go unassigned.
2 more excerpts
Quote confirmed verbatim on the O'Reilly session page, but the page carries no publication date; the event date is corroborated from independent announcements and O'Reilly Radar coverage
Beck's own newsletter does not carry this exact sentence; kentbeck.com has a near-identical paraphrase in a mutable homepage section, so the O'Reilly page is the only stable surface for the verbatim wording
For difficult tasks, I'll often reject five or six (or more!) agent attempts before accepting one as good enough to work with, or giving up and making the change by hand.
Getting value from agents on hard tasks means aggressively rejecting weak attempts and keeping judgment work human.
3 more excerpts
able to correctly diagnose 80% of issues on its own
The current core AI skill is shifting as much work onto AI agents as possible, without going too far.
I still don't use LLMs to write Slack messages, ADRs, issues and so forth.
If your project has a robust, comprehensive and stable test suite agentic coding tools can _fly_ with it.
A strong automated test suite is the single biggest enabler of agent productivity on a codebase.
2 more excerpts
what should we call the other end of the spectrum, where seasoned professionals accelerate their work with LLMs while staying proudly and confidently accountable for the software they produce?
Automated testing / Planning in advance / Comprehensive documentation / Good version control habits / Effective automation / Culture of code review / Manual QA / Research skills / Ship to preview environment
The machinery around the model — the context it sees, the harness it acts through, the loop it runs in.
AnthropicNovember 24, 2025data
The most common failures are wrong tool selection and incorrect parameters, especially when tools have similar names like `notification-send-user` vs. `notification-send-channel`.
At scale, loading all tool definitions upfront is the failure driver; deferred tool loading cuts token cost and measurably raises tool-selection accuracy.
4 more excerpts
When using natural language tool calling, each invocation requires a full inference pass, and intermediate results pile up in context whether they're useful or not.
At Anthropic, we've seen tool definitions consume 134K tokens before optimization.
Opus 4.5 improved from 79.5% to 88.1%
This represents an 85% reduction in token usage while maintaining access to your full tool library.
Bandi, Dumitru, Hertzberg, Agarwal et al. (Scale AI)Jan 31, 2026data
Across 1,000 expert-written tasks spanning 36 real MCP servers and 220 tools, automated diagnostics show 63.3% of diagnosed failures are cognitive rather than tool-call related.
Second independent finding that the majority of agent failures are model-side, not tool-surface.
1 more excerpt
'Several high-performing models fail after successful tool execution due to premature stopping or incorrect synthesis' - a cognitive fault the harness can still address
Even the best frontier models only achieve 68% accuracy at the max density of 500 instructions.
Instruction-following accuracy degrades sharply with density — the best frontier models hit only 68% at 500 instructions — so packing rules in measurably erodes compliance.
4 more excerpts
At 500 instructions, llama-4-scout exhibits an extreme O:M ratio of 34.88, indicating omission errors are over 30 times more frequent
Threshold decay: "Performance remains stable until a threshold, then transitions to a different (steeper) degradation slope" — exhibited by gemini-2.5-pro, o3
Primacy effects display an interesting pattern across all models: they start low at minimal instruction densities indicating almost no bias for earlier instructions, peak around 150–200 instructions
Analysis reveals model size and reasoning capability to correlate with 3 distinct performance degradation patterns, bias towards earlier instructions, and distinct categories of instruction-following errors.
Du et al. (Findings of EMNLP 2025)November 2025data
even when models can perfectly retrieve all relevant information, their performance still degrades substantially (13.9%-85%) as input length increases but remains well within their claimed context lengths.
we find that context files tend to reduce task success rates compared to providing no repository context, while also increasing inference cost by over 20%.
Every line in a context file binds: agents comply, tasks get harder, costs rise — measured across four agents and two benchmarks (SWE-bench Lite 300 tasks, AGENTbench 138 instances).
2 more excerpts
we find that agents generally follow instructions present in the context files.
We conclude that context files, even developer-provided ones, are not effective at providing a repository overview
Huang et al. (arXiv)Submitted 22 January 2026 (accepted at ICAIBD 2026)datapartial
Procedural reliability, particularly tool initialization failures, constitutes the primary bottleneck for smaller models.
For smaller models, tool-invocation reliability (especially tool initialization) is the primary failure bottleneck, localizable via a 12-category taxonomy.
3 more excerpts
1,980 deterministic test instances
12-category error taxonomy capturing failure modes across tool initialization, parameter handling, execution, and result interpretation
Mid-sized models (qwen2.5:14b) offer practical accuracy-efficiency trade-offs on commodity hardware (96.6% success rate, 7.3 s latency)
Kelly Hong, Anton Troynikov, Jeff Huber (Chroma)July 14, 2025data
Even under these minimal conditions, model performance degrades as input length increases, often in surprising and non-uniform ways.
18 LLMs degrade non-uniformly as input grows — the independent mechanism behind the bloat warning (applies to CLAUDE.md by analogy; the study never tests it).
4 more excerpts
models do not use their context uniformly; instead, their performance grows increasingly unreliable as input length grows
Even a single distractor reduces performance relative to the baseline (needle only).
Whether relevant information is present in a model's context is not all that matters; what matters more is how that information is presented.
Even a single distractor reduces performance relative to the baseline (needle only), and adding four distractors compounds this degradation further
Liu et al. (TACL 2024)July 6, 2023 (v1); revised Nov 20, 2023data
language model performance is highest when relevant information occurs at the very beginning (primacy bias) or end of its input context (recency bias), and performance significantly degrades when models must access and use information in the middle of long contexts
Mid-file placement is the worst-served position — the U-shaped retrieval curve.
1 more excerpt
extended-context models are not necessarily better than their non-extended counterparts at using their input context.
a single piece of irrelevant information can distract the models and substantially degrade their performance, even on problems whose clean versions they correctly solve.
Irrelevant context degrades accuracy even when all relevant information is present.
1 more excerpt
we find that simply adding an instruction to ignore irrelevant information brings notable performance gains on our benchmark.
Terminal-Bench (Stanford x Laude)live leaderboard (fetched Jul 27, 2026)data
Eleven Claude Opus 4.6 entries span 58.0% (Claude Code, +/-2.9) to 76.4% (Meta-Harness, +/-2.4) on 89 terminal tasks - an 18.4-point spread on identical model weights, 13.1 points at the non-overlapping confidence bounds.
The same frontier model varies by double-digit percentage points purely as a function of the harness it runs in.
2 more excerpts
Entries are self-submitted by harness authors via pull request, machine-validated for timeout/resource parity and a five-trial minimum, then maintainer-merged
Submissions span Dec 2025 to May 2026 and are not contemporaneous; harness-side and model-side settings are not held constant, so the spread is observational rather than controlled
Worawalan Chatlatanagulchai et al.17 Nov 2025 (submitted)data
While developers use context files to make agents functional, they provide few guardrails to ensure that agent-written code is secure or performant
Empirically, teams pack context files with functional setup but almost no security or performance guardrails — the constraint side of CLAUDE.md is systematically under-specified.
3 more excerpts
2,303 agent context files across 1,925 repositories
Build and run commands: 62.3%, Implementation details: 69.9%, Architecture: 67.7%; Security: 14.5%, Performance: 14.5%
These files are not static documentation but complex, difficult-to-read artifacts that evolve like configuration code
Yang et al. (NeurIPS 2024)May 6, 2024 (v3 November 11, 2024)datapartial
SWE-agent solves 10.7 percentage points more issues than the baseline agent that uses just the default Linux shell (300-issue ablation); on SWE-bench Lite, SWE-agent with GPT-4 Turbo resolves 18.00% versus 11.00% for the shell-only agent with the same model.
Agent-computer interface design alone moves resolve rates by double digits with the model held fixed - the founding demonstration of the scaffold confound.
2 more excerpts
Verification is partial because arXiv HTML routes 404 and the PDF required reader-proxy extraction; figures converged across three independent extraction passes
Citation caution: the paper's verbatim 10.7pp sentence arithmetically pairs with the 7.33% no-demonstration shell baseline, not the 11.00% row - quote the sentence or the 18.00/11.00 pair, never 10.7 with 11.00
We attribute this improvement to the legibility of failed logical search. Repeated failures under explicit lexical constraints provide a clearer signal that required evidence may be absent, whereas Agentic Hybrid may still return semantically related but unsupported passages.
Logical/lexical retrieval can signal 'nothing found' where embedding search cannot, which measurably reduces hallucination on answer-unavailable questions.
3 more excerpts
On average, its refusal rate increased from 0.767 to 0.828, while the hallucination rate decreased from 0.128 to 0.083.
anchoring the retrieval process in logical queries substantially reduces hallucinations in generated responses.
matches a strong agentic hybrid baseline, while substantially reducing construction and serving cost
In a controlled 3x3 factorial experiment, average harness variance is 18.48 pp-squared versus average model variance of 2.37 pp-squared - a 7.80x ratio - and public leaderboards show harness-only swings of 7.3pp (Terminal-Bench 2), 9.5pp (SWE-bench Pro, same Opus 4.5), up to 15pp (SWE-bench Verified), and 34-48pp cross-scaffold gaps on the HAL Leaderboard.
Performance variance is governed more by harness configuration than model choice, so evaluation protocols without harness disclosure systematically misattribute harness gains to model improvements.
2 more excerpts
The same model under a different harness can rank above or below a competitor - rank order itself is harness-dependent
The paper proposes a harness-aware evaluation framework with a disclosure standard and variance decomposition protocol
Across 1,794 manually annotated trajectories (63,000+ execution steps, seven frontier models, three scaffolds), environment triggers account for 9.4% of decisive errors against 57.9% epistemic and 32.8% competence.
The direct refutation of harness causation - agent failures are predominantly model-side, which is why this post argues leverage rather than cause.
3 more excerpts
Largest single trigger is false premises at 30.7%
Epistemic errors are the largest share in every scaffold tested, ranging 44% to 80%
The paper's own prescription - 'earlier validation and intervention' - is itself a harness prescription
agents.md (Agentic AI Foundation / Linux Foundation)undated (fetched Jul 13, 2026)practitionerpartial
Agents automatically read the nearest file in the directory tree, so the closest one takes precedence and every subproject can ship tailored instructions.
The nearest-file hierarchy generalizes across 20+ tools — the routing skill transfers beyond Claude Code.
1 more excerpt
For example, at time of writing the main OpenAI repo has 88 AGENTS.md files.
Aider (Paul Gauthier)continuously updated (fetched August 2026)practitioner
The same model's code-editing score moves ~10 points by edit format alone: gemini-exp-1206 scores 80.5% in whole format versus 69.2% in diff format; o1-mini 70.7% versus 61.1%.
A live, reproducible public leaderboard shows the harness's edit protocol moving scores by roughly the same magnitude as top-of-table model gaps.
1 more excerpt
Live page - re-pull current figures before quoting in new work
Albert NahasFeb 17 (year not stated on page; brief lists 2026)practitioner
when the context window fills up and gets compacted, your CLAUDE.md values get summarized away with everything else
CLAUDE.md instructions decay mid-session — they get summarized away at compaction — so hook-based reinforcement is more reliable for must-follow standards.
3 more excerpts
hook output requires approximately 15 tokens per prompt reminder
Over 50-turn session, motto reminders total ~750 tokens against 200k context window
hook output arrives as clean system-reminder messages — no disclaimer, no 'may or may not be relevant' framing
Of course, there's a trade-off: runtime exploration is slower than retrieving pre-computed data. Not only that, but opinionated and thoughtful engineering is required to ensure that an LLM has the right tools and heuristics for effectively navigating its information landscape.
Just-in-time context retrieval is not free: it trades latency for freshness and demands deliberate tool and heuristic design to work.
4 more excerpts
An agent running in a loop generates more and more data that could be relevant for the next turn of inference, and this information must be cyclically refined.
Context, therefore, must be treated as a finite resource with diminishing marginal returns.
agents built with the 'just in time' approach maintain lightweight identifiers (file paths, stored queries, web links, etc.) and use these references to dynamically load data into context at runtime using tools.
In certain settings, the most effective agents might employ a hybrid strategy, retrieving some data up front for speed, and pursuing further autonomous exploration at its discretion.
As token count grows, accuracy and recall degrade, a phenomenon known as context rot. This makes curating what's in context just as important as how much space is available.
Curating what's in the context window matters as much as how much space is available - official-docs corroboration of context rot.
The core challenge of long-running agents is that they must work in discrete sessions, and each new session begins with no memory of what came before.
A high-level prompt alone fails a long-running loop; cross-session state must be externalized to disk.
2 more excerpts
even a frontier coding model like Opus 4.5 running on the Claude Agent SDK in a loop across multiple context windows will fall short of building a production-quality web app if it's only given a high-level prompt
memoryless-session framing is softened by Opus 4.5+ auto-compaction per Anthropic's March 2026 follow-up - the externalized-state lesson persists, the mechanism is version-dependent
getEvents(), allows the brain to interrogate context by selecting positional slices of the event stream
Cross-iteration retrieval - pulling the right slice of past sessions into a fresh window - is its own discipline, distinct from single-shot JIT retrieval.
1 more excerpt
a context object that lives outside Claude's context window
gather context -> take action -> verify work -> repeat
The agent loop is a repeated four-step cycle; managing context across iterations (compaction) is a loop-only concern with no single-task analog.
1 more excerpt
The Claude Agent SDK's compact feature automatically summarizes previous messages when the context limit approaches, so your agent won't run out of context.
This metadata is the first level of progressive disclosure: it provides just enough information for Claude to know when each skill should be used without loading all of it into context.
Progressive-disclosure mechanics: metadata triggers, bodies load on relevance.
1 more excerpt
This means that the amount of context that can be bundled into a skill is effectively unbounded.
Unlike CLAUDE.md content, a skill's body loads only when it's used, so long reference material costs almost nothing until you need it.
Skills are the designated destination for procedures that outgrew CLAUDE.md — with a stickiness caveat once invoked.
1 more excerpt
When you or Claude invoke a skill, the rendered SKILL.md content enters the conversation as a single message and stays there for the rest of the session.
'The operating system enforces the sandbox boundary on the running process, so it holds regardless of what the model chose to run and even if an allowed command does more than its name suggests.'
Where the check physically runs is what decides whether it is a control - stated by the vendor, at mechanism level.
4 more excerpts
'Sandboxing reduces risk but is not a complete isolation boundary'
Default failure mode: if the sandbox cannot start because dependencies are missing, Claude Code warns and runs commands without sandboxing unless sandbox.failIfUnavailable is set
Claude may retry a sandbox-blocked command with the dangerouslyDisableSandbox parameter
excludedCommands has no managed-only lockdown, so a developer can always widen the policy
'For most hook events, only exit code 2 blocks the action. Claude Code treats exit code 1 as a non-blocking error and proceeds with the action, even though 1 is the conventional Unix failure code.'
The hook layer fails open by default - and the vendor redirects hard enforcement away from it.
3 more excerpts
'Because the if filter is best-effort, use the permission system rather than a hook to enforce a hard allow or deny'
'The filter also fails open, running your hook regardless of pattern, when the Bash command can't be parsed'
'They provide deterministic control over Claude Code's behavior, ensuring certain actions always happen rather than relying on the LLM to choose to run them.'
'Deterministic' is the vendor's own word for the control surface - but the claim is immediately qualified.
1 more excerpt
The very next sentence: prompt-based and agent-based hooks 'use a Claude model to evaluate conditions' - so 'hook' is an umbrella containing probabilistic members, and the determinism attaches to command hooks only
Anthropic (Claude Code Docs)2026 (undated on page; brief dates it 2026)practitioner
If Claude keeps doing something you don't want despite having a rule against it, the file is probably too long and the rule is getting lost. If Claude asks you questions that are answered in CLAUDE.md, the phrasing might be ambiguous. Treat CLAUDE.md like code: review it when things go wrong, prune it regularly, and test changes by observing whether Claude's behavior actually shifts.
Anthropic's own guidance says to maintain CLAUDE.md like code — prune it, and test rule changes by observing whether Claude's behavior actually shifts.
4 more excerpts
Claude stops when the work looks done. Without a check it can run, 'looks done' is the only signal available, and you become the verification loop: every mistake waits for you to notice it.
The over-specified CLAUDE.md. If your CLAUDE.md is too long, Claude ignores half of it because important rules get lost in the noise.
Keep it concise. For each line, ask: 'Would removing this cause Claude to make mistakes?' If not, cut it. Bloated CLAUDE.md files cause Claude to ignore your actual instructions!
Ruthlessly prune. If Claude already does something correctly without the instruction, delete it or convert it to a hook.
Anthropic (Claude Code Docs)2026 (undated on page)practitioner
CLAUDE.md content is delivered as a user message after the system prompt, not as part of the system prompt itself. Claude reads it and tries to follow it, but there's no guarantee of strict compliance, especially for vague or conflicting instructions.
CLAUDE.md is advisory context delivered as a user message, not enforced configuration — so strict compliance is not guaranteed, especially for vague or conflicting rules.
4 more excerpts
CLAUDE.md and CLAUDE.local.md files in the directory hierarchy above the working directory are loaded in full at launch. Files in subdirectories load on demand when Claude reads files in those directories.
target under 200 lines per CLAUDE.md file. Longer files consume more context and reduce adherence.
Both are loaded at the start of every conversation. Claude treats them as context, not enforced configuration. To block an action regardless of what Claude decides, use a PreToolUse hook instead.
if two rules contradict each other, Claude may pick one arbitrarily.
Anthropic (David Dworken, Oliver Weller-Davies)Oct 20, 2025practitioner
'Constantly clicking approve slows down development cycles and can lead to approval fatigue, where users might not pay close attention to what they're approving, and in turn making development less safe.'
Why prompt-time human review does not scale, argued by the vendor whose product depends on it.
2 more excerpts
'In our internal usage, we've found that sandboxing safely reduces permission prompts by 84%' - vendor-internal, no methodology, sample size, or definition of 'safely'
Context-reversal finding: this blog says a successful prompt injection is 'fully isolated', which the product documentation explicitly contradicts ('not a complete isolation boundary')
Level 1: Metadata | Always (at startup) | ~100 tokens per Skill ... Level 2: Instructions | When Skill is triggered | Under 5k tokens ... Level 3+: Resources | As needed | Effectively unlimited
The on-demand tier has a documented, quantified cost model (Anthropic's stated architecture, not a measured benchmark).
1 more excerpt
This filesystem-based architecture enables progressive disclosure: Claude loads information in stages as needed, rather than consuming context upfront.
'In coding agents, part of the harness is already built in (e.g. via the system prompt, or the chosen code retrieval mechanism, or even a sophisticated orchestration system).'
The harness arrives partly pre-built - the inherited-defaults premise, from the discipline's highest-authority restatement.
4 more excerpts
The 2x2 that organizes the post: guides (feedforward) vs sensors (feedback), crossed with computational (deterministic, reliable) vs inferential (non-deterministic)
She files AGENTS.md and Skills as inferential feedforward - a legitimate quadrant member, not the harness's opposite
'Building this outer harness is emerging as an ongoing engineering practice, not a one-time configuration'
Names cybernetics as the lineage via a Wikipedia link, with no specific control-theory originator
The root file should be pointers and critical gotchas only; everything else drifts into noise.
The root tier's content rule comes from Anthropic itself: pointers and gotchas, not documentation.
2 more excerpts
Claude loads them additively as it moves through the codebase: root file for the big picture, subdirectory files for local conventions.
Skills solve this through progressive disclosure, offloading specialized workflows and domain knowledge that would otherwise compete for context space and loading them only when the task calls for it.
Claude by AnthropicNovember 25, 2025practitionerpartial
Every conversation starts with this context already loaded, eliminating the need to explain basic project information repeatedly.
The always-loaded tier recurs every session — the recurring-cost premise. (Excerpt deliberately omits the page's 'system prompt' clause, refuted 0-3 against the docs.)
Claude Docs (Anthropic)living docs (fetched Aug 3, 2026; inline version markers v2.1.198-v2.1.212)practitioner
'Running each Claude Code session in its own worktree means edits in one session never touch files in another.' The docs distinguish mechanisms explicitly: worktrees 'isolate file edits, while subagents and agent teams coordinate the work itself.'
File-ownership isolation is a shipped, first-class product mechanism - the enforcement layer exists today.
2 more excerpts
The page ships the enforcement mechanism but gives no guidance on how a human decides which files each session owns, and says nothing about what happens when two sessions need the same file - that absence is the gap the post addresses
Isolation is filesystem and branch level, not permission or prompt level: subagents take an 'isolation: worktree' frontmatter field, and Claude runs 'git worktree lock' while an agent is active
'Other exit codes - Hook failed, action proceeds (fail-open by default)'. The failClosed override ships with default false: 'When true, hook failures (crash, timeout, invalid JSON) block the action instead of allowing it through. Useful for security-critical hooks.'
A second vendor whose guardrail layer fails open by default, with the security switch shipped off.
3 more excerpts
Verification note: two independent passes disagreed on whether the page renders permission: "deny" or permission: 'deny'; that quote was dropped from the post rather than resolved
No raw-markdown endpoint (cursor.com/docs/hooks.md returns 404), so all quotes come from rendered HTML
beforeReadFile hook failures are logged and the read is allowed through
CLAUDE.md files in subdirectories are not being automatically loaded when accessing files in those directories, contrary to what the documentation states.
The lazy tier has unresolved reliability reports — documented design, verify on your surface (single macOS report, closed unresolved).
GitHub (anthropics/claude-code)opened Feb 11, 2026; closed not-planned Mar 22, 2026practitioner
only the root-level CLAUDE.md is loaded at session start, and no subdirectory CLAUDE.md files are ever injected — even after multiple Read tool calls into those directories.
Lazy loading failed on the VS Code extension across three versions (2.1.39/2.1.45/2.1.49; CLI reportedly fine) — surface-specific reliability caveat.
GitHub (anthropics/claude-code)opened Nov 16, 2025; closed not-plannedpractitionerpartial
A modular CLAUDE.md structure with 6 referenced files (~2,100 lines total) consumes the same tokens as a monolithic file, providing organizational benefits only.
A user measured the @imports split delivering zero token savings, and Anthropic declined the lazy-imports request (author-self-reported measurement, not maintainer-confirmed).
1 more excerpt
85-90% of loaded content is irrelevant to most conversations
Google (Gemini CLI docs)undated, main branch (fetched Jul 27, 2026)practitioner
'For global rules (those without an argsPattern), tools that are denied are completely excluded from the model's memory.' The model never sees the tool as an option.
A deny decision is not a request to the model - it is a change to what exists. Deterministic policy evaluation as reviewable code.
4 more excerpts
SCOPE: the memory-exclusion claim applies only to global rules without an argsPattern; the doc says nothing about argument-conditional deny rules
Precedence is arithmetic: final_priority = tier_base + (toml_priority / 1000), tier bases Default 1 through Admin 5 - tier always dominates because the fractional term can never reach 1
'The first rule that matches determines the outcome' - first in priority order, not file order
In non-interactive mode ask_user is treated as deny
yes, compaction and smaller models help on cost per step. But my issue wasn't just inefficiency, it was agents retrying when they shouldn't. I needed visibility + limits per agent/task, and the ability to cut it off, not just optimize it.
Practitioners want per-agent/per-task limits and a hard cut-off, not just cost optimization — the wedge is attribute-and-enforce, not optimize.
4 more excerpts
My AGENTS.md is 845 lines and it only started getting good once it got that long" (Sammi), directly contested by "sweet spot is between 60 and 120 lines. With psuedo xml tags between sections" (typpilol)
Budget alerts are not a kill switch. Credits are not protection.
Claude often ignores CLAUDE.md / The more information you have in the file the more it gets ignored
"cost control is a policy problem - we certainly don't need to use opus 4.6 for a simple test refactor... we need a way to measure cost / performance for agents on individual repos, with individual types of tasks..." (author bisonbear, id 47563774)
Ivan Kahl / DometrainJanuary 15, 2026practitionerpartial
You cannot craft the perfect CLAUDE.md file immediately. Instead, treat it as a living document.
CLAUDE.md is a living document refined over time, not a one-shot artifact.
2 more excerpts
Claude Code agents have a context window, and the CLAUDE.md file gets added to the agent's context. Any unnecessary instructions and wordy sentences will consume more of that context.
Always review the CLAUDE.md file and correct any assumptions or missing details related to project architecture.
Frontier thinking LLMs can follow ~ 150-200 instructions with reasonable consistency.
There is a practical instruction ceiling — even frontier models only follow roughly 150-200 instructions consistently — so every line in CLAUDE.md competes for a finite budget.
4 more excerpts
At HumanLayer, our root CLAUDE.md file is less than sixty lines.
Claude Code's system prompt contains ~50 individual instructions
Smaller models get MUCH worse, MUCH more quickly
LLMs bias towards instructions that are on the peripheries of the prompt
'The evaluator and permission control should likely sit outside the loop that evolves harness, with held-out tests, trace audits, and human review at decision points that matter.'
Enforcement belongs outside the loop it governs - and, because harnesses evolve, the audit is recurring rather than one-time.
4 more excerpts
SCOPE: scoped specifically to self-modifying harnesses, not agent harnesses generally; the 'should likely' hedge is the author's own
'A harness is the system surrounding a base model that orchestrates execution and decides how the model thinks and plans, calls tools and acts, perceives and manages context, stores artifacts, and evaluates results'
The words deterministic and probabilistic appear nowhere in the post - do not attribute that framing to her
'The core interface of mainstream coding agents has become stabilized across Claude Code, Codex, OpenCode, and Cursor-style agents'
OpenAIundated, on or before Feb 11, 2026 (Wayback-bounded)practitionerpartial
'Agents are most effective in environments with strict boundaries and predictable structure, so we built the application around a rigid architectural model.'
The canonical build-a-harness text - and the differentiation foil: written from an empty repository, with zero coverage of inherited vendor defaults.
4 more excerpts
Direct fetch returns HTTP 403 and web.archive.org is blocked to the tool; content read via a text-extraction proxy
Page carries no byline and no publication date - attribute to OpenAI, not to an individual
Its 'boundaries' are architectural layer-dependency lint rules inside the application codebase, not harness control surfaces
Posture cuts against gates: 'The repository operates with minimal blocking merge gates'
Simon Willison16th October 2025practitionerpartial
This is _very_ token efficient: each skill only takes up a few dozen extra tokens, with the full details only loaded in should the user request a task that the skill can help solve.
Named-author validation of metadata-first, on-demand loading (about Skills; the CLAUDE.md parallel is the post's).
'A coding agent is a piece of software that acts as a harness for an LLM, extending that LLM with additional capabilities that are powered by invisible prompts and implemented as callable tools.'
The harness's prompts are invisible to the user by construction - the closest practitioner framing to the unreviewed-defaults claim.
1 more excerpt
'A tool is a function that the agent harness makes available to the LLM'
'you can try telling it not to in your own prompt, but how confident can you be that your protection will work every time?'
A system-prompt instruction is not a security control - the cleanest one-line statement of the post's premise.
1 more excerpt
On vendor guardrails claiming '95% of attacks': 'in web application security 95% is very much a failing grade' - note the 95% is Willison characterizing vendor marketing, not his own measurement
context engineering is the delicate art and science of filling the context window with just the right information for the next step...task descriptions and explanations, few shot examples, RAG, related (possibly multimodal) data, tools, state and history
Context engineering, not prompt engineering, is the real discipline: filling the window with the right information environment for the next step.
1 more excerpt
the art of providing all the context for the task to be plausibly solvable by the LLM
Towards Data Science (Mostafa Ibrahim)March 20, 2026practitioner
The agent optimises locally. At each step, it asks, 'Do I have enough?' and when the answer is uncertain, it defaults to 'get more'. Without hard stopping rules, the default spirals.
Without a hard stop rule, an agent's local 'get more' default turns retrieval into an unbounded budget fire; capping cycles and abstaining is the control.
3 more excerpts
Three cap retrieval cycles. After three failed passes, return a best-effort answer with a confidence disclaimer.
agents making 200 LLM calls in 10 minutes, burning $50–$200 before anyone noticed
costs spike 1,700% during a provider outage as retry logic spiralled out of control
The honest column in the ledger: JIT buys these at the price of retrieval latency on the steps that load (usually trivial next to a model call, but nonzero), a new failure mode (an unresolvable reference must surface as an honest error, not a hallucinated payload), and a dependency on description quality — the agent loads from the catalog's one-liners, so a bad stub hides a good payload.
JIT context introduces two specific liabilities leaders must design for: unresolvable references must fail loud as honest errors, and retrieval quality is capped by the quality of catalog descriptions.
3 more excerpts
in a loop, the window is re-sent every step, so a preloaded handbook isn't one payment but thirty
a preloaded copy is a snapshot that ages as the run proceeds, while a reference resolves to the current state of the file, the ticket, the database at the moment of use
long-context research and practitioner experience agree that models degrade as windows fill with low-relevance text
This is powerful for keeping your main file lean. Put detailed instructions in separate markdown files, then reference them. Claude pulls in the content when relevant.
The SERP's reputable competitor teaches the naive @imports model the docs refute — the foil for the kicker.
'A harness is every piece of code, configuration, and execution logic that isn't the model itself. A raw model is not an agent. But it becomes one when a harness gives it things like state, tool execution, feedback loops, and enforceable constraints.'
The canonical definition, from the coinage. Cite Trivedy rather than downstream restatements.
1 more excerpt
The component enumeration explicitly places System Prompts inside the harness alongside tools, sandbox, orchestration logic, and hooks - so 'harness vs CLAUDE.md' is not the boundary the field draws
negative instructions can be unreliable as user prompts
Negative 'don't do that' rules are unreliable in a user message like CLAUDE.md, so positive, runnable framing is preferable — reserving DO-NOT for hard safety boundaries.
3 more excerpts
Reddit user reported Claude Code created duplicate files despite explicit 'NEVER create duplicate files' rule
Gemini models have 'hit-or-miss' performance with negative commands
They are effective at preventing unethical or harmful behavior, especially when used in system prompts
Knowing an agent’s output is actually correct, beyond a green build.
Albayaydh, Zhao, FlechaisJul 7, 2026data
A synthesis of 27 benchmark, taxonomy, and audit papers spanning 19 benchmarks finds that additional scaffolding does not consistently improve reliability.
The honesty brake on 'more harness is better' - and the reason this post's claim is bounded to blast radius rather than quality.
3 more excerpts
Failures compound nonlinearly with task length
Strong performance on individual sub-tasks does not reliably translate into end-to-end success
Secondary synthesis - every number in it is someone else's measurement
32.67% of successful SWE-bench patches involved solution leakage (the fix present in the issue report or comments) and 31.08% passed on weak tests; filtering both drops SWE-Agent+GPT-4's resolution rate from 12.47% to 3.97%.
A third of measured SWE-bench success was answer leakage, a concrete mechanism by which leaderboard scores inflate without capability.
1 more excerpt
Over 94% of benchmark issues predate LLM knowledge cutoff dates
Teams delay building evals thinking they need hundreds of tasks; in reality 20-50 simple tasks drawn from real failures is a great start, structured by task/trial/outcome vocabulary.
A working internal agent eval suite is a 20-50 task project, not an infrastructure program - removing the main excuse for deciding from public leaderboards instead.
4 more excerpts
Opus 4.5 initially scored 42% on CORE-Bench; after fixing grading bugs and using a less constrained scaffold, the same model's score jumped to 95%.
So as not to unnecessarily punish creativity, it's often better to grade what the agent produced, not the path it took.
'With frontier models, a 0% pass rate across many trials (i.e 0% pass@100) is most often a signal of a broken task, not an incapable agent'
The initial 42% observation is Anthropic citing an external report; the diagnosis and the 95% re-run are Anthropic's own
In internal experiments spanning six compute-resource configurations on GKE with model, harness, and task set held fixed, the gap between the most- and least-resourced setups on Terminal-Bench 2.0 was 6 percentage points (p < 0.01); infrastructure error rates fell from 5.8% under strict 1x enforcement to 2.1% at 3x headroom and 0.5% uncapped.
Infrastructure configuration alone produces score differences exceeding the few-point margins that separate top leaderboard entries, so cross-infrastructure leaderboard comparisons are not decision-grade evidence for a model swap.
4 more excerpts
Infrastructure configuration can swing agentic coding benchmarks by several percentage points - sometimes more than the leaderboard gap between top models. The gap between most- and least-resourced setups on Terminal-Bench 2.0 was 6 percentage points (p < 0.01).
A 2-point lead on a leaderboard might reflect a genuine capability difference, or it might reflect that one eval ran on beefier hardware, or even at a luckier time of day, or both.
Top leaderboard spots are often separated by just a few percentage points, per the post's own framing
Resource headroom is an eval-infrastructure design requirement: error rate falls an order of magnitude from strict to uncapped provisioning
Their hyper-productivity is revealing a significant 'speed vs. trust' gap. Recent, deeper examinations of agent-generated code and agent-driven PRs reveal that a large percentage of agent efforts fail to meet the quality bar of being truly 'merge-ready,' often containing subtle regressions, superficial fixes, or a general lack of engineering hygiene.
Agent hyper-productivity creates a speed-vs-trust gap where most agent PRs aren't merge-ready, overwhelming review capacity.
3 more excerpts
29.6% of 'plausible' fixes introduced behavioral regressions or were incorrect upon rigorous retesting
True solve rates for GPT-4 patches dropped from 12.47% to 3.97% after detailed manual audits
Over 68% of agent-generated pull requests reportedly face long delays or remain unreviewed, creating an urgent need for scalable review automation.
arXiv (Sabrina Haque, Sarvesh Ingale, Christoph Csallner)Submitted January 7-8, 2026datapartial
Across agents, test-containing PRs are more common over time and tend to be larger and take longer to complete, while merge rates remain largely similar.
Whether an agent PR includes tests varies and doesn't correlate with merge outcomes, so test presence is a signal to read, not proof of quality.
2 more excerpts
We observe variation across agents in both test adoption and the balance between test and production code within test PRs
Testing is a critical practice for ensuring software correctness and long-term maintainability
arXiv (Yue Liu, Ratnadira Widyasari, Yanjie Zhao, Ivana Clairine Irsan, Junkai Chen, David Lo)Submitted 30 March 2026 (v2 revised 26 April 2026)datapartial
22.7% of tracked AI-introduced issues still survive at the latest version of the repository. These findings show that AI-generated code can introduce long-term maintenance costs into real software projects.
Over a fifth of AI-introduced issues survive at HEAD, so AI code accrues durable technical debt at scale unless verification catches it.
3 more excerpts
302.6k verified AI-authored commits from 6,299 GitHub repositories
more than 15% of commits from every AI coding assistant introduce at least one issue
code smells are by far the most common type" / "89.3% of all issues
Berkeley RDI (Hao Wang, Qiuyang Mang, Alvin Cheung, Koushik Sen, Dawn Song)April 2026data
We built an automated scanning agent that systematically audited eight among the most prominent AI agent benchmarks — SWE-bench, WebArena, OSWorld, GAIA, Terminal-Bench, FieldWorkArena, and CAR-bench — and discovered that every single one can be exploited to achieve near-perfect scores without solving a single task.
Every major agent benchmark can be gamed to near-perfect scores without solving anything, so self-reported benchmark performance is structurally untrustworthy.
4 more excerpts
Every one of eight major AI agent benchmarks audited can be exploited to near-perfect scores without solving a single task - including 100% on SWE-bench Verified via a 10-line conftest.py that hooks pytest and rewrites every test result to passed.
A conftest.py file with 10 lines of Python 'resolves' every instance on SWE-bench Verified.
SWE-bench Verified (500 tasks) — 100% score via pytest hooks
Benchmark scores are actively being gamed, inflated, or rendered meaningless, not in theory, but in practice.
Applying rank confidence intervals to MMLU abstract-algebra rankings, the authors conclude the observed ranking cannot be trusted and all models are statistically interchangeable.
Once uncertainty is displayed, observed leaderboard orderings among top models frequently collapse into statistical ties.
On MMLU (57 subjects), three distinct models can be ranked as the fourth from the top - statistically consistent with the same rank position - with uncertainty driven more by between-subject variability than prompt variants.
Leaderboard ranks published as single values hide that several models are often statistically tied for the same position.
Clustered standard errors on public evals can be over 3X larger than naive standard errors, and detecting an absolute score difference of 0.03 at 80% power requires an eval of at least ~969 independent questions.
Most reported model-to-model benchmark gaps are narrower than honestly computed confidence intervals - the statistical foundation for treating small leaderboard gaps as ties.
2 more excerpts
The same pair of models can differ significantly on one benchmark (MATH) and not on others (HumanEval, MGSM) in the paper's worked example
The Llama 3 paper's reported confidence intervals are judged likely anti-conservative (too narrow)
A good scaffold can increase SWE-bench Verified performance by up to 20%, so scores reflect the sophistication of the scaffold as much as the capability of the underlying model.
Scaffold quality is a confound baked into every SWE-bench Verified score - the leaderboard measures a model-plus-scaffold system.
Across five runs, run-to-run standard deviations were 2.0 percentage points for SWE-Doctor (the most stable agent), 2.2 for mini-SWE-agent, and 3.4 for live-SWE-agent; SWE-Doctor's Pass@5 was 70.0% against All@5 of 40.0%.
Even the most stable SWE-bench-family agents swing multiple percentage points between identical runs - variance comparable to the gaps separating leaderboard leaders.
1 more excerpt
The 30-point spread between Pass@5 (solves at least once) and All@5 (solves every time) is its own nondeterminism exhibit
Under controlled, pre-registered conditions, scaffold choice alone moves a single model's measured accuracy by up to 28 percentage points (Claude Opus, GAIA Level 2: Planner-Actor-Rater 84% vs ReAct 56%).
Published agent capability scores conflate what a model can do with what its scaffold lets it do, at magnitudes far exceeding typical inter-model leaderboard gaps.
1 more excerpt
The paper's citation of Pimpale et al.'s 33% vs 62.2% Sonnet 3.5 elicitation split is chain-of-citation only - not independently verified against Pimpale's own text
METR's headline 50%-time-horizon estimate of ~2h17m carries a 95% CI of 65 minutes to 4h25m, and of 28 tasks with zero successes in 6 runs, roughly 25-35% of failures were estimated possibly spurious or infrastructure-related.
Even a dedicated evaluator's headline capability metric carries hours-wide uncertainty, much of it from task-set resampling and infrastructure rather than capability.
1 more excerpt
Uncertainty across measurements is highly correlated because it largely comes from resampling the task set
The paper forecasts that by early 2026, low-elicitation non-specialized LM agents reach 54% on SWE-Bench Verified while state-of-the-art-elicitation agents reach 87% - a 33-point gap attributable to elicitation level alone.
The forecasting literature treats elicitation/scaffold quality as a first-class capability axis separate from the model.
State-of-the-art models identify buggy file paths from issue descriptions alone - no repository access - at up to 76% accuracy on SWE-Bench repositories but only up to 53% on repositories outside the benchmark; consecutive 5-gram verbatim similarity runs up to 35% on SWE-Bench Verified/Full versus 18% elsewhere.
SWE-bench performance gains are partially memorization of the benchmark's repositories, so the score measures training exposure as well as coding skill.
Claude Sonnet 4.6 evaluated three times on the same HR-grievance workflow scored 0.000, 0.214, and 0.679 - a range the authors call a qualitative, not merely quantitative, difference.
Single-task agent nondeterminism can dwarf any leaderboard rank gap - an illustrative extreme, not a benchmark-wide average.
The authors identify 27 private LLM variants tested by Meta on Chatbot Arena in the lead-up to the Llama-4 release, with undisclosed private testing letting providers test multiple variants and publish only the best score.
Public leaderboards are gameable by labs through selective disclosure, biasing the ranking independent of any measurement noise.
2 more excerpts
LMArena publicly disputed several of the paper's framings and calculations at https://news.lmarena.ai/our-response/ - cite alongside for balance
Estimated arena data share: Google 19.2% and OpenAI 20.4%, versus 29.7% combined for 83 open-weight models
Sinha, Arun, Goel, Staab, GeipingSept 2025 (rev. Mar 13, 2026)data
the per-step accuracy of models degrades as the number of steps increases. This is not just due to long-context limitations -- curiously, we observe a self-conditioning effect -- models become more likely to make mistakes when the context contains their errors from prior turns.
Long-horizon reliability is a different quantity from single-turn accuracy; models self-condition on their own prior errors and scaling does not fix it.
2 more excerpts
larger models can correctly execute significantly more turns even when small models have near-perfect single-turn accuracy
measured on a synthetic running-sum task; thinking mitigates self-conditioning; larger models are more prone, not less
Thomas Claburn, The RegisterReport Dec 17, 2025; Register coverage Dec 17, 2025datapartial
The bots created more logic and correctness errors (1.75x), more code quality and maintainability errors (1.64x), more security findings (1.57x), and more performance issues (1.42x).
AI-authored PRs carry more defects than human ones in every category, concentrated in logic and security, so review depth should follow issue class.
3 more excerpts
On average, AI-generated pull requests (PRs) include about 10.83 issues each, compared with 6.45 issues in human-generated PRs.
AI-authored PRs contain 1.4x more critical issues and 1.7x more major issues on average than human-written PRs.
The report examined 470 open source pull requests.
Systematic auditing found 219 distinct flaws across eight flaw classes in major agent benchmarks; patching reduced the hackable-task ratio from near 100% to under 10% across four benchmarks.
Benchmark exploitability is a design-flaw problem, not just a contamination problem - the academic backbone for the RDI exploit findings.
A production coding-agent quality regression traced to a reasoning-effort default change, a caching bug, and one system-prompt addition; one internal eval showed a 3% drop for both Opus 4.6 and 4.7, and Anthropic committed to running a broad suite of per-model evals for every system prompt change.
A named lab now gates every change to its coding agent behind per-model internal evals - the swap-as-production-change discipline practiced at the source.
1 more excerpt
Non-model changes (runtime config, caching) produced user-visible quality regressions - runtime configuration is a quality variable independent of the model
The agent runs the build, sees green, and moves on. But 'build passes' and 'the output is production-ready' are different bars.
Agent self-verification confirms compilation and tests but not production-readiness, so quality attributes must be checked explicitly.
2 more excerpts
Developers consistently report agents declaring tasks complete while skipping accessibility attributes, test isolation, config externalization, dark mode, responsive layout, and meta tags.
The agent's own verification handles 'does it compile and do tests pass.' The orchestrator handles 'did it actually do what was asked, completely.'
Don't ask the same agent to write code and verify it. That's like having students grade their own exams...The separation is what makes the gates trustworthy.
The agent that writes the code must not be the one that grades it; separated validation gates are what make verification trustworthy.
3 more excerpts
Eight quality gates required before production
Every commit is a known-good checkpoint. When something fails, the blast radius is one subtask, not an entire feature.
Agents are extremely literal. Give them vague instructions and they'll build something that technically matches what you said but misses what you meant.
Epoch AI runs most models 16 times on GPQA Diamond and Mock AIME and 8 times on MATH Level 5, displaying plus/minus one standard error following Miller's arXiv:2411.00640 methodology.
A reputable third-party evaluator treats single-run benchmark scores as insufficient and re-runs models many times specifically to bound noise.
After a GPT-4o to GPT-4.1 upgrade, an agent's prompt-injection resistance dropped from 94% to 71% on the vendor's eval harness.
Model swaps silently regress agent behavior on dimensions no public leaderboard measures - run your own tests on your own data; third-party numbers are a starting point, not a finish line.
1 more excerpt
Authority caveat: commercial eval-tooling vendor with a named staff-engineer author and a falsifiable data point - cited with attribution, not as neutral research
Qualitative case study (Rechat/Lucy): performance plateaued under generic evaluation frameworks until a problem-specific evaluation system replaced them.
Generic evaluation frameworks do not transfer to a specific workload - create an evaluation system specific to your problem.
Hamel Husain and Shreya ShankarJanuary 15, 2026practitioner
Generic evaluation metrics are everywhere...These metrics measure abstract qualities that may not matter for your use case. Good scores on them don't mean your system works.
Evals should be derived from error analysis of real traces, because good scores on generic metrics don't mean the system works.
4 more excerpts
On model switching: do not treat switching model as the main axis of improvement without evidence - does error analysis suggest the model is the problem?
Error analysis helps you decide what evals to write in the first place. It allows you to identify failure modes unique to your application and data.
Spend 60-80% of our development time on error analysis and evaluation
Binary evaluations force clearer thinking and more consistent labeling. Likert scales introduce significant challenges.
on a real build, structured verification consistently found 30-40% of the specification unimplemented after the agent reported 'complete.' Not broken code. Missing code.
Agents routinely report 'complete' while 30-40% of the spec is unbuilt, a gap code review can't see because there is no diff.
3 more excerpts
Code review examines what was built...But if a feature wasn't built at all, there's no diff to review.
Verification works forward from the spec: 'given what was specified, was it built?'
5-6 passes to full completion is consistent enough to plan around
METR's protocol requires models be provided the best available scaffolding and tooling because it is hard to upper-bound what might be possible with clever prompting and tooling.
The eval-methodology establishment treats scaffolding quality as a confound that must be standardized before capability claims are comparable.
1 more excerpt
METR's elicitation-gap data page was unreachable (redirect stub) - its numbers are not cited
OpenAIcirca February 23, 2026 (date not visible on page)practitionerpartial
OpenAI's audit found at least 59.4% of audited problems have flawed test cases that reject functionally correct submissions (35.5% overly strict tests, 18.8% out-of-scope checks), and all frontier models tested could reproduce the original human-written bug fix.
The benchmark's own creator retracted it: score gains (74.9% to 80.9% in six months) no longer reflect real-world software development ability.
1 more excerpt
Verification is partial because openai.com blocks automated fetches (HTTP 403); content was retrieved via reader proxy and cross-checked against independent snippets, and the publication date is inferred from third-party citation
OpenAIAugust 2024 (page updated February 24, 2025)practitionerpartial
Human screening of 1,699 SWE-bench samples flagged 38.3% for underspecified problem statements and 61.1% for unit tests that may unfairly mark valid solutions incorrect; 68.3% of samples were filtered out to produce the 500-task Verified set.
The majority of original SWE-bench tasks were broken or underspecified before later contamination concerns - the earliest documented data-quality failure in the benchmark's lineage.
1 more excerpt
Verification is partial because openai.com blocks automated fetches (HTTP 403); content retrieved via reader proxy
the biggest mistake engineers make in code review: only thinking about the code that was written, not the code that could have been written.
The core reviewer skill for agent output is architectural judgment about unwritten alternatives, not line-level nitpicking.
3 more excerpts
about once an hour I notice that the agent is doing something that looks suspicious, and when I dig deeper I'm able to set it on the right track and save hours of wasted effort.
If you're a nitpicky code reviewer, I think you will struggle to use AI tooling effectively.
Trying to make a badly-designed solution work costs time, tokens, and codebase complexity.
Running agents in production: cost, permissions, failure modes, guardrails.
FinOps Foundation (finops.org)Last updated February 17, 2026data
More acute are the challenges of identifying the consumer of the model output, which is especially difficult when the consumers of the same model can be different interfaces/functional modules in the same user application (e.g., 'tech support chatbot' or 'new customer chatbot')
The hard, unsolved FinOps problem for AI is mapping model output back to the specific consumer; account-level billing is the wrong granularity and no accepted multi-agent allocation framework exists yet.
2 more excerpts
"Tokens! The meters, or elements of charge can be very different. For example, measuring the tokens at the user input vs. the compressed and semantic reduced or re-written actual prompt input token quantity that goes to the API endpoint that is charged."
"Lack of generally accepted frameworks for cost allocation across multi-agent workloads"
Rather than supervising what the agent does, we supervise what it's able to do by enforcing access boundaries through, for example, sandboxes, virtual machines, and egress controls.
Safety comes from constraining what the agent can reach, not from watching what it does, because any model-layer check has a non-zero miss rate.
3 more excerpts
Any probabilistic defense has a non-zero miss rate.
Claude Code previously protected against agents taking unintended actions by asking users for permission at each turn... Our telemetry showed users approved roughly 93% of permission prompts.
The claude_code.tool_decision event carries a source enum recording which control surface made each decision: config, hook, user_permanent, user_temporary, user_abort, user_reject.
The per-tool-call authorization provenance record a governance process needs already exists in the product - and ships disabled.
4 more excerpts
Attributing spend to specific skills, plugins, or subagent types via the `skill.name`, `plugin.name`, and `agent.name` attributes
OpenTelemetry export to your backend is opt-in and requires explicit configuration.
Telemetry is off by default: CLAUDE_CODE_ENABLE_TELEMETRY 'Enables telemetry collection (required)'
Argument capture is gated behind a second variable, OTEL_LOG_TOOL_DETAILS=1
Anthropic (platform.claude.com)undated (data available "for dates on or after January 1, 2026")practitioner
Values for a given date can be revised for up to 30 days as late events arrive and reconciliation runs. For invoicing-grade totals, query dates at least 30 days in the past.
Provider analytics numbers are a post-hoc, reconciled reporting layer that keeps moving for up to 30 days and are attributed per-user, not per-request — useless as a real-time per-task control.
3 more excerpts
Enterprise Analytics cost granularity: "per-user and organization-level token usage and cost over time (usage-based Enterprise plans)" — NOT per-request.
Cost data freshness: "Data is typically available within four hours of the underlying usage but may take up to 24 hours."
"Daily Claude Code metrics per user: sessions, lines of code, commits, pull requests, tool acceptance, and estimated cost by model"
Agents introduce a risk called *excessive agency*, where an agent determines the best solution to a problem is to take broader actions beyond its scope.
First-party cloud guidance names excessive agency as a High-risk gap and prescribes least-privilege boundaries plus user confirmation to contain it.
3 more excerpts
Level of risk exposed if this best practice is not established: High
Implement user confirmation for the agent, requiring users to confirm agent actions and mitigating the risk of excessive agency.
A permission boundary sets the maximum permissions which can be given to a role.
Autonomy is not a configuration decision that's decided once. Rather, it is more like a score that goes up or down, and that your system earns through demonstrated reliability in your specific environment and workflows.
Agent autonomy should be an earned, revocable score tied to measured reliability, not a one-time day-one setting.
3 more excerpts
Expansion of autonomy should happen as a consequence of earned trust, not as a deployment decision we make on day one.
Named trust-score inputs: percentage of agent actions completed without human override (30-day window); false escalation rate; override-correctness rate; time-to-revert
Conservative defaults with clear, earned expansion paths are the right architecture as the fastest route to durable autonomy at scale.
This brief event was the result of user error — specifically misconfigured access controls — not AI.
Even vendors' own defense of an agent-caused deletion frames it as an access-control misconfiguration, corroborating that these are authorization failures, not model failures.
2 more excerpts
The AI agent encountered a problem and determined that the optimal solution was to delete and recreate the entire environment.
Kiro requires two-person approval before pushing changes to production. But the deploying engineer had broader permissions than a typical employee, and Kiro inherited those elevated privileges.
Enforcing least privilege requires control at the point of tool invocation, in real time, against a defined scope that reflects the agent's function, not its operator's credentials.
Least privilege for agents must be enforced at tool-invocation time and scoped to the agent's function, not inherited from its operator's broad credentials.
2 more excerpts
Authentication tells you who the agent is. It tells you nothing about what the agent should be allowed to do.
Gartner identifies approximately 40 tool definitions as the threshold beyond which agent latency and token cost increase measurably.
Railway's CLI token created for managing custom domains had blanket permissions across the entire GraphQL API, including destructive operations on production volumes. There is no role-based access control (RBAC) for Railway API tokens.
The production database deletion happened because an over-broad, unscoped token authorized destructive operations, not because the model went rogue.
3 more excerpts
Tokens are not scoped by operation, by environment, or by resource. Every token is effectively root.
Soft guardrails are probabilistic controls that guess at intent instead of enforcing rules
The agent knew the rules, yet it violated every one of them
Tier 1 systems handling information retrieval need automated monitoring. Tier 2 workflows with reversible actions require real-time guardrails. Tier 3 systems involving financial transactions demand human-in-the-loop for all decisions.
Controls should be tiered in proportion to an action's risk, from monitoring for retrieval up to human-in-the-loop for high-stakes transactions.
3 more excerpts
15-20% of policy violations occur during tool execution before output generation
a single agent performing 1000+ actions per hour makes comprehensive human oversight untenable
Access control determines which resources your agents can touch, validation filters what they consume and produce, human oversight governs high-stakes decisions
Least privilege does not mean making the agent weak. It means giving the agent exactly enough power to complete the approved task, for the approved time, in the approved context.
Least privilege scopes an agent to exactly the task, time, and context approved, which defines the axes of an authority-by-task-class table.
3 more excerpts
Static roles like 'claims analyst' or 'support ops' are often far wider than the exact permissions a single agent run should have.
Read access can still expose sensitive personal data, trade secrets, or protected records.
Shared service accounts destroy attribution: one API key used by multiple automations cannot prove who did what later
After the key crosses it's `max_budget`, requests fail
A proxy can enforce multi-level budgets by validating spend before a request is admitted and hard-failing over the ceiling, i.e. terminate before the next call rather than alert after the invoice.
3 more excerpts
"validates spend against the authoritative database before being admitted (covering key, team, user, organization, end-user, tag, and per-window budgets)"
"`fail_closed_budget_enforcement`" enables a hard ceiling "even while Redis is degraded"
Exceeded-budget response body: `"ExceededTokenBudget: Current spend for token: 7.2e-05; Max Budget for Token: 2e-07"`.
When agents run agentic loops, they can make unbounded LLM calls, causing unexpected costs.
Agentic loops make unbounded LLM calls by default, so the ceiling must be set per session — a hard iteration cap and a per-session dollar cap keyed to a trace/session id.
3 more excerpts
Control 1 — "Max Iterations": "Hard cap on the number of LLM calls per session".
Control 2 — "Max Budget Per Session": "Dollar cap per session (identified by `x-litellm-trace-id`)".
"When the counter exceeds `max_iterations`, the request receives a **429 Too Many Requests**".
Cost visibility tells you what your agents spent — through dashboards, cost traces, and budget alerts. Cost governance controls what they are permitted to spend, by enforcing per-session ceilings that terminate sessions before a threshold is exceeded.
Cost visibility (dashboards, alerts) is not cost control; governance means enforcing per-session ceilings that terminate the session before the threshold is crossed, and provider caps operate at the wrong (account/key) granularity.
3 more excerpts
"only 44% of organizations have adopted financial guardrails or AI FinOps practices" — attributed to Gartner, March 2026
"A 10-step agent with an average cost of $0.02 per step looks inexpensive in planning. That same agent entering a retry loop and executing 2,000 steps doesn't — that's $40 from a session that was supposed to cost $0.20."
"Provider-level controls operate at the API key or account level, not the individual session level. They cannot distinguish a single runaway session from many well-behaved sessions using the same key."
Every request passes through it, which means budget enforcement happens in one place, consistently, regardless of which agent sent the request.
Infrastructure-level (proxy) budget enforcement is the only reliable guard against runaway costs because it enforces at one chokepoint, whereas application-level checks can be forgotten in a new agent.
3 more excerpts
"agent that takes 50 turns on a complex task hits 100,000 input tokens and 40,000 output tokens, costing roughly $0.90 per session. Run 100 of those sessions per hour, and you are looking at $90/hour, or over $2,100/day".
"developer on r/AI_Agents recently described watching their agent rack up $15 in API costs in under 10 minutes".
"If a developer forgets to add the check in a new agent, there is no safety net."
gen_ai.tool.name is Required, while gen_ai.tool.call.arguments and gen_ai.tool.call.result are both Opt-In. A fully spec-compliant trace records that a tool ran and nothing about what it ran.
'We have OpenTelemetry' does not mean 'we have an audit trail' - the audit-grade fields are the weakest requirement level in the spec.
2 more excerpts
Both opt-in fields carry the spec warning 'This attribute may contain sensitive information' - a real privacy tradeoff, not an oversight
The opentelemetry.io spec path is now a stub; cite the repo path
OWASP Gen AI Security ProjectLLM Top 10 for LLM Applications, 2025 editionpractitioner
'Provide the application with its own API tokens for extensible functionality, and handle these functions in code rather than providing them to the model. Restrict the model's access privileges to the minimum necessary for its intended operations.'
An independent standards body placing the control in code rather than in the prompt.
3 more excerpts
'Given the stochastic influence at the heart of the way models work, it is unclear if there are fool-proof methods of prevention for prompt injection'
Framing caveat: the list is introduced as measures that 'can mitigate' impact - recommendations, not requirements
Item 2 independently recommends using 'deterministic code to validate adherence to these formats'
Agent-level cost attribution starts with identity. When every agent has a unique, registered identity, every API call, token consumption event, and tool invocation can be tagged to that identity.
Agent-level cost attribution requires giving every agent a registered identity so every token and tool call can be tagged to it — but the field's default stops at alerts, not termination.
2 more excerpts
"Per-agent budgets define expected spend. Alerts fire when an agent approaches or exceeds its budget."
"Cloud cost management tools track compute and API spend at the account or service level — not at the agent level."
Ravi Kanani, LeanOps TechnologiesMay 19, 2026practitioner
OpenAI and Anthropic API calls show up as a single line item per provider. There's no native breakdown by your customer, your feature, or your workflow.
Cloud FinOps tooling structurally fails on LLM workloads because cloud tags don't propagate to the API call and provider billing arrives as one line item — attribution must be a schema on the call itself.
3 more excerpts
"the company spent $87,000/month on Anthropic API calls that arrived as a single line item".
"two enterprise customers were responsible for 78% of LLM costs while paying for 12% of revenue".
"Tagging doesn't propagate to OpenAI/Anthropic API calls. The tag lives on the EC2 instance making the API call, not on the API call itself."
Scott Castle, Chief Product Officer at CloudZeroMay 15, 2026practitioner
Consumption dimensions tell you what was used, not who in your business used it. Allocation is the work of mapping that usage back to teams, budgets, and cost centers.
Aggregate token counts tell you what was used but not who used it; allocation to teams, budgets, and cost centers is the actual work, and centralized billing traded away the per-team visibility seats used to provide.
3 more excerpts
"Aggregate token counts don't tell you which teams are driving spend."
"Centralized billing simplified procurement and security, but it traded away the per-user and per-team visibility teams used to get from individual seats."
"AI cost also scales differently than cloud cost. It moves with prompt size, fanout, retries, and agentic loops."
Senior engineers become the verification layer for product ambiguity. They are no longer just checking implementation quality. They are reconstructing intent from generated code, thin specs, incomplete Jira tickets, and edge cases nobody wrote down.
The unbudgeted review burden concentrates on senior engineers as intent-reconstructors, creating retention risk that throughput dashboards never show.
3 more excerpts
"Replacement cost of a senior software engineer at $150,000 to $300,000 in 2026, including recruiting, ramp time, and lost institutional knowledge." — Industry benchmarks cited
"25% of PRs are now reviewed by AI agents, up from 0% in 2025. But review times have increased nearly 200%." — AI Engineering Report 2026 caption
the burden "does not get measured in PR throughput dashboards"
the review queue becomes the binding constraint on their delivery pipeline
When agents raise output, the human review queue — not code generation — becomes the constraint that caps delivery.
3 more excerpts
"developers at large organisations spend between ten and fifteen percent of their working hours reading and commenting on others' code" — attributed to Sadowski et al., Google study
"review latency between submitting a pull request and receiving actionable feedback routinely stretches over twenty-four hours" — Introduction
"reviews of agent-generated code become rubber-stamps: the human approves because the code looks correct"
Pereira, Sinha, Ghosh, Dutta (Nutanix, Inc.)10 Mar 2026data
code review agents can exhibit a low signal-to-noise ratio when designed to identify all hidden issues, obscuring true progress and developer productivity
"Find everything" review agents drown the signal, so resolution/merge rate is the wrong yardstick and signal-to-noise proxies developer trust.
Sebastian Baltes, Marc Cheong, Christoph Treude09 Jun 2026data
The development time has been shortened but the team now needs to spend more time to review. Doesn't look like any benefit.
Individual AI speedups externalize review burden onto the team, making review a shared, exhaustible resource rather than a free step.
3 more excerpts
"30 PRs per day across 6 reviewers" — [R07] reviewer-burden example
"reviewer-burden" ranked among top 3 most frequent codes (226 instances) — coding frequency
"Individual developers and organizations benefit from AI-generated content, but the cumulative effect degrades the shared resources that collaborative development depends on."
The reviewer role is being automated. The review, understood as judgment about whether the software is correct for its purpose, is relocating to where the agent cannot follow.
Agents can take over diff inspection, but human judgment doesn't disappear — it relocates to intent specification up front and accountability at merge.
3 more excerpts
"An agent-assisted developer produces more pull requests per day than human review capacity can absorb." — Monperrus paper discussion
"Automate the checkpoint and the judgment does not evaporate. It relocates to intent specification on the way in and accountability on the way out"
"The human does not leave the loop. The human moves from the end of it to the start."
More code is entering the pipeline, but less of it is reaching production successfully. The bottleneck has moved from writing code to deciding whether code is safe to merge.
Third-party delivery data shows generation is not the wall — validation is, with feature throughput rising while main-branch throughput and success rates fall.
3 more excerpts
"feature branch throughput up 59% year over year, while main branch throughput for the median team actually fell" — CircleCI 2026 State of Software Delivery report
"main-branch throughput fell nearly 7%, and main-branch success rates dropped to 70.8%" — CircleCI 2026
"agentic AI PRs have a pickup time 5.3x longer than unassisted PRs. AI-assisted PRs wait 2.47x longer" — LinearB 2026 Software Engineering Benchmarks Report
The bottleneck moves from generation to review queues, CI capacity, flaky environments, branch policy, cost ceilings, and the human attention needed to decide what should actually merge.
As agents get capable, the constraint shifts off code generation and onto the whole delivery surface — review bandwidth, CI, and human merge decisions.
3 more excerpts
"The model matters, but the delivery surface matters just as much."
"A team that cannot write crisp tasks will struggle to evaluate agents honestly."
"Reviewers do not need another wall of generated explanation. They need the shortest path to deciding whether the change should merge."
Dex Horthy (HumanLayer)Jul 23, 2026 (undated in body; dated by commit history)practitioner
'no amount of harness engineering or loopsmaxxing can solve what is fundamentally a model-training issue.'
The strongest counterargument to a harness-centric thesis, and the reason this post bounds its claim to blast radius rather than quality.
4 more excerpts
On the limit of fast deterministic gates: 'Running the tests gets you a clean pass or fail in ~seconds... But the cost function of bad architecture is measured in weeks, months, maybe even years'
'if you build a harness but you don't own the weights and can't RL the model inside it, you'll always be at a disadvantage to a team that owns both'
Cites Faros AI: 31.3% of PRs skip review entirely, +242.7% incidents per PR under high AI adoption
The words determinism, guardrails, permissions, sandboxing, and policy enforcement appear nowhere in the document - it argues about design quality, not authorization
When agents help vs. hurt; single vs. multi-agent; build vs. buy.
AnthropicJune 13, 2025data
Subagents facilitate compression by operating in parallel with their own context windows, exploring different aspects of the question simultaneously before condensing the most important tokens for the lead research agent.
Subagents earn their place by isolating and compressing context — separate windows, not raw speed, are the reason to split.
3 more excerpts
agents typically use about 4× more tokens than chat interactions, and multi-agent systems use about 15× more tokens than chats
some domains that require all agents to share the same context or involve many dependencies between agents are not a good fit for multi-agent systems today
token usage by itself explains 80% of the variance
Cemri, Pan, Yang, Agrawal, Chopra, Tiwari, Keutzer, Parameswaran, Klein, Ramchandran, Zaharia, Gonzalez, Stoica (UC Berkeley)Submitted 17 Mar 2025; last revised 26 Oct 2025 (v3)data
This process identifies 14 unique modes, clustered into 3 categories: (i) system design issues, (ii) inter-agent misalignment, and (iii) task verification.
Multi-agent failure is predominantly a system-design and coordination problem, not a model-quality problem — a readiness test, not a model upgrade.
3 more excerpts
MAST-Data, a comprehensive dataset of 1600+ annotated traces collected across 7 popular MAS frameworks
We develop MAST through rigorous analysis of 150 traces, guided closely by expert human annotators and validated by high inter-annotator agreement (kappa = 0.88).
Despite enthusiasm for Multi-Agent LLM Systems (MAS), their performance gains on popular benchmarks are often minimal.
David Klotz (IAAI, Media University Stuttgart)April 29, 2026 (arXiv:2604.26482v1)data
Mission-critical systems of record: Retain Buy as the primary option. Consider Make selectively for peripheral modules, extensions, or integration layers where the core system's integrity is not at risk.
Agentic AI shifts make-vs-buy by application type: commodity and differentiating apps move toward build, while regulated and mission-critical systems stay buy.
3 more excerpts
Commodity utilities: Default to Make. Evaluate Buy only where ecosystem integrations provide strong network value or where the firm's AI capability is below the viability threshold.
Where software development once required large teams working over months, small teams augmented by AI agents can now deliver functional applications in days or weeks.
AI-era Make demands skills in prompt engineering, agent orchestration, AI output validation, and governance of AI-generated artifacts.
David Wood (O'Reilly)August 2009 (book publication)data
Fully 60% of the life cycle costs of software systems come from maintenance, with a relatively measly 40% coming from development.
Maintenance dominates software lifecycle cost, and most of that maintenance is new enhancement work rather than bug-fixing.
1 more excerpt
During maintenance, 60% of the costs on average relate to user-generated enhancements (changing requirements), 23% to migration activities, and 17% to bug fixes.
DORA (Google Cloud)2024 (page last updated April 13, 2026)datapartial
AI adoption significantly increases individual productivity, flow, and job satisfaction. However, it also negatively impacts software delivery stability and throughput
AI helps the individual developer but hurts system-level delivery stability and throughput.
1 more excerpt
Unstable organizational priorities cause meaningful decreases in productivity and substantial increases in burnout.
GitClearJanuary 2026 (research notation on page)data
the percentage of changed code lines (associated with refactoring) sunk from 25% of changed lines in 2021, to less than 10% in 2024, while lines classified as 'copy/pasted' (cloned) rose from 8.3% to 12.3%
AI-assisted development correlates with more code duplication and less refactoring, increasing long-term maintenance burden on code you own.
3 more excerpts
211 million changed lines from repos owned by Google, Microsoft, Meta, and enterprise C-Corps
4x more code cloning
'copy/paste' exceeds 'moved' code for first time in history
if AI adoption increases by 25%, estimated throughput delivery is expected to decrease by 1.5%
Individual AI productivity gains do not translate into system-level delivery throughput or stability, because code generation was never the bottleneck.
3 more excerpts
estimated delivery stability is expected to decrease by 7.2%
75.9% of respondents (of roughly 3,000 people surveyed) are relying on AI for at least part of their job responsibilities
if AI adoption increases by 25%, time spent doing valuable work is estimated to decrease 2.6%
Sheryl Estrada (Fortune)August 18, 2025, 6:54 AM ETdata
Purchasing AI tools from specialized vendors and building partnerships succeed about 67% of the time, while internal builds succeed only one-third as often.
Most enterprise GenAI builds fail; buying and partnering succeeds roughly three times more often than building internally.
3 more excerpts
95% failure rate for enterprise AI solutions
about 5% of AI pilot programs achieve rapid revenue acceleration
150 interviews with leaders, a survey of 350 employees, and an analysis of 300 public AI deployments
Yubin Kim, Ken Gu, Chanwoo Park, et al. (MIT / Google)Submitted 9 Dec 2025data
Relative performance change compared to single-agent baseline ranges from +80.8% on decomposable financial reasoning to -70.0% on sequential planning, demonstrating that architecture-task alignment determines collaborative success.
Whether a multi-agent split helps or hurts is decided by task decomposability — decomposable tasks gain sharply, sequential ones degrade sharply.
3 more excerpts
Across 260 configurations spanning six agentic benchmarks, five canonical architectures (Single-Agent and four Multi-Agent: Independent, Centralized, Decentralized, Hybrid), and three LLM families
The framework identifies the best-performing architecture for 87% of held-out configurations
architectures without centralized verification tend to propagate errors more than those with centralized coordination
'Planning, implementation, and testing of the same feature share too much context' to split across agents, and 'Components requiring constant back-and-forth belong in the same agent.'
There are principled places not to cut the graph - shared context and high synchronization needs are the signals to keep work in one node.
1 more excerpt
Used in the post as the 'where not to cut' check in the pricing list, a counterweight to over-decomposition
Use one when a side task would flood your main conversation with search results, logs, or file contents you won't reference again: the subagent does that work in its own context and returns only the summary.
Isolate a polluting side-task in a subagent's own context — the operational test for when to split before splitting the whole job.
3 more excerpts
Each subagent runs in its own context window with a custom system prompt, specific tool access, and independent permissions.
Preserve context by keeping exploration and implementation out of your main conversation
Enforce constraints by limiting which tools a subagent can use
Bryan Ross (GitLab, Field CTO)March 24, 2026practitionerpartial
For a team of roughly 200 developers, an internal build typically costs around $1.4M in year one, requires 2–3 dedicated FTEs to maintain, and takes 12–18 months to reach a first real use case.
Building an internal agentic AI platform in regulated industries is a multi-year, multi-FTE commitment with governance surface most organizations underestimate.
2 more excerpts
Every engineer building the platform is an engineer _not_ modernizing a legacy pipeline, remediating security debt, or accelerating a critical delivery program.
Building an internal agentic AI platform in banking or insurance is a multi-year platform engineering commitment with regulatory surface area most organizations underestimate
Pat Brans (CIO.com)December 11, 2025practitionerpartial
With such a layer in place, the build-versus-buy question fragments, and CIOs might buy a vendor's persona agent, build a specialized risk-management agent, purchase the foundation model, and orchestrate everything through a platform they control.
The industry consensus has shifted to hybrid: assemble build and buy across the AI stack under an orchestration layer you control.
2 more excerpts
Six months ago many were experimenting, but now they're scaling.
including cases where a senior executive's data surfaced in a junior employee's query.
The factor that stands out most to me is that these developers were all working in repositories they have a deep understanding of already, presumably on non-trivial issues since any trivial issues are likely to have been resolved in the past.
AI's edge is smallest exactly where you own and deeply understand a mature codebase long-term.
3 more excerpts
56% had never used Cursor before the study
Developers accepted less than 44% of AI generations
A quarter of the participants saw increased performance, 3/4 saw reduced performance
I can only focus on reviewing and landing one significant change at a time, but I'm finding an increasing number of tasks that can still be fired off in parallel without adding too much cognitive overhead to my primary work.
Human review-and-land throughput — one significant change at a time — is the real ceiling on how far parallel agents scale.
1 more excerpt
Code that started from your own specification is a lot less effort to review.
'Actions carry implicit decisions, and conflicting decisions carry bad results.' On the parallel-subagent failure: 'The actions subagent 1 took and the actions subagent 2 took were based on conflicting assumptions not prescribed upfront.'
The root cause of parallel-agent collision is assumptions that were never prescribed in advance - which is an argument for pre-assigned structure over runtime negotiation.
4 more excerpts
Cognition's April 2026 follow-up states that multi-agent systems work best when writes stay single-threaded and additional agents contribute intelligence rather than actions
Share context, and share full agent traces, not just individual messages
At the core of reliability is Context Engineering
The simplest way to follow the principles is to just use a single-threaded linear agent