Audit Your Agent Harness: The Deterministic Layer Nobody Reviews

You have reviewed your CLAUDE.md a dozen times, and you have never once opened the file that decides whether the agent’s next rm -rf actually runs. Those are different files, governed by different software, and only one of them can stop a command — the other is a request the model is free to reinterpret.

Most of Your Harness Arrived Pre-Configured

Before you write a single new rule, inventory the ones that shipped. The harness governing your agents is mostly inherited, and the inherited part is the part with teeth.

Start with a call so unremarkable that nobody has ever written a rule about it. The agent proposes rm -rf ./build in a repo where ./build is a symlink into /srv/shared/artifacts — routine cleanup by the text, a shared volume by the filesystem. Right now, today, name the piece of software that decides whether that command runs. If you cannot, that is not a discipline failure — it is what the word “harness” has come to mean.

“A harness is every piece of code, configuration, and execution logic that isn’t the model itself. A raw model is not an agent. But it becomes one when a harness gives it things like state, tool execution, feedback loops, and enforceable constraints.”

That definition includes the pieces you authored and the far larger set you did not. Birgitta Böckeler makes the inheritance explicit: in coding agents, part of the harness is already built in — via the system prompt, the chosen code retrieval mechanism, or an orchestration system you never configured (Birgitta Böckeler: Harness Engineering²). Simon Willison’s definition puts the same point at the center: a coding agent extends an LLM with capabilities “powered by invisible prompts and implemented as callable tools” (Simon Willison: How Coding Agents Work³). Invisible by construction, in the vendor’s product, on your laptop.

For the rm -rf ./build call, four surfaces are candidates, and every one of them is at a value someone else picked:

Each of those defaults is the vendor’s own documentation rather than my inference — fail-closed permission matching (Claude Code Docs: Security⁴), the exit-code semantics (Claude Code Docs: Hooks Reference⁵), the unsandboxed fallback (Claude Code Docs: Sandboxing⁶), and the telemetry switch (Claude Code Docs: Monitoring⁷). The audit is not an exercise in imagining what could go wrong. It is reading four documented values and writing down which ones you chose.

This layer goes unreviewed because a different layer absorbed all the attention. Böckeler splits harness controls two ways. Computational controls are “deterministic and fast, run by the CPU” — tests, linters, type checkers, structural analysis. Inferential controls cover “Semantic analysis, AI code review, ‘LLM as judge’,” run on a GPU or NPU, where “results are more non-deterministic” (Birgitta Böckeler: Harness Engineering²). She crosses that with guides, which steer the agent before it acts, and sensors, which observe after. AGENTS.md and Skills land in the inferential-feedforward cell. That cell is exactly where the industry’s review effort has gone, including my own post on structuring CLAUDE.md. You have been reviewing one quadrant of four, and it is the probabilistic one.

Böckeler names the members of two cells. The other two are where the mechanisms in this audit land:

The bottom-left cell has a thousand blog posts and a linting ecosystem. The top-left cell decides whether rm -rf ./build runs, and most teams have never opened it.

The canonical harness-engineering essay does not close the gap either, because it is greenfield by construction. OpenAI’s account of building with Codex starts from an empty repository with no pre-existing human-written code, and its enforcement apparatus — “strictly validated dependency directions,” custom linters, structural tests — is authored by the team. Their framing: “Agents are most effective in environments with strict boundaries and predictable structure, so we built the application around a rigid architectural model” (OpenAI: Harness Engineering⁸). Excellent advice for a system you are building from scratch. Silent on the agent runtime you installed last quarter and have been running ever since.

The Harness Is the Variable You Control

Hold the model fixed and the scoreboard still moves by double digits. That is the whole reason this audit earns a slot on a staff engineer’s week — not because the harness is exotic, but because it is the input with the widest measured range that you can actually change.

On the Terminal-Bench 2.0 leaderboard, Claude Opus 4.6 appears many times over, once per harness submission:

Same weights, 18.4 points apart, and the confidence intervals do not overlap — the worst-case reading still leaves 13.1 points (Terminal-Bench 2.0 Leaderboard⁹). Read that table with the caveat it deserves: entries are submitted by the harness authors themselves, the submission dates span February to May 2026, and nothing holds harness-side settings constant across them. It is an observational spread, not a controlled experiment, and anyone who cites it as one is overreaching.

The controlled companion comes from a frontier lab measuring against its own interest. Anthropic varied nothing but infrastructure resourcing and re-ran the benchmark:

“Infrastructure configuration can swing agentic coding benchmarks by several percentage points—sometimes more than the leaderboard gap between top models.”

Their own number: the gap between the most- and least-resourced setups on Terminal-Bench 2.0 was 6 percentage points, at p < 0.01. A second Anthropic post supplies the vivid version. Opus 4.5 initially scored 42% on CORE-Bench; after a researcher fixed grading bugs, ambiguous task specs, and stochastic tasks, “Opus 4.5’s score jumped to 95%” (Anthropic: Demystifying Evals for AI Agents¹¹). One model, two scaffolds, fifty-three points.

The interesting question is therefore not which model you standardized on, but which harness you were measuring when you decided.

Concede the Causation Point First

Author’s judgment. The move from “the harness has leverage” to “audit the harness” is my inference, not a claim any source makes. It follows from three sourced premises: agent failures are predominantly model-side, a fixed model’s behavior still varies widely with its harness, and the weights are not yours to retrain.

Do not claim the harness causes most agent failures. The best trajectory evidence available says the opposite, and a post that overclaims here loses every reader who has already read the paper. Zhao and colleagues collected 3,843 execution trajectories across seven frontier models and three scaffolds, then filtered to 1,794 complete trajectories for manual annotation over more than 63,000 execution steps. Across 1,184 failed trajectories, environment triggers account for 9.4% of decisive root causes against 57.9% epistemic:

“Our findings show that coding-agent failures are predominantly driven by epistemic errors, typically begin within the first few execution steps, and often remain hidden until recovery is no longer possible, suggesting that improving coding-agent reliability requires earlier validation and intervention rather than relying solely on final-outcome evaluation.”

Their definitions make the split sharper than the labels suggest. Epistemic triggers arise “when the information needed to avoid the error is already available but is ignored, forgotten, or misinterpreted,” while environment triggers “originate from external blockers that are outside the agent’s reasoning process.” The single largest trigger of all is a false premise, at 30.7%. Scale AI’s MCP-Atlas benchmark reaches the same verdict from a different direction across 1,000 expert-written tasks spanning 36 real MCP servers and 220 tools: “Automated diagnostics show that 63.3% of diagnosed failures are cognitive rather than tool-call related” (Bandi et al.: MCP-Atlas¹³). The agent misreads a premise, or stops early, or synthesizes the wrong answer from a tool call that worked perfectly.

Both findings are true, and neither one touches the argument. You cannot patch the model’s priors — the weights ship the way they ship. You can change what the runtime is permitted to execute, and you can change it this afternoon. The trajectory paper’s own prescription points the same way: earlier validation and intervention is a harness prescription, because validation and intervention are things the harness does. The harness is not where most failures start. It is the part of the system whose behavior you own.

Where the Check Runs Decides Whether It Controls

Ask one question of every control in your harness: where does this check physically execute? If the answer is “inside the model’s context,” you are holding a suggestion, and suggestions have no failure mode you can reason about.

Bad: A line in CLAUDE.md reading “Never delete paths outside the repository.” The check runs inside the model’s context, competing for attention with everything else in the context window, and it evaluates rm -rf ./build as a string the model has already decided looks fine. Good: A deny rule the runtime evaluates before the call is dispatched, plus an OS sandbox that confines where the process can write once it starts — so the symlink into /srv/shared/artifacts is out of bounds whether or not anyone anticipated it.

{
  "permissions": {
    "deny": ["Bash(rm -rf /srv/**)"]
  },
  "sandbox": {
    "enabled": true,
    "failIfUnavailable": true,
    "allowUnsandboxedCommands": false
  }
}

The vendor draws the distinction more precisely than most security teams do:

“Claude Code evaluates permission decisions before a command runs, based on the command string and, in auto mode, a separate classifier’s judgment about whether the command is safe. The operating system enforces the sandbox boundary on the running process, so it holds regardless of what the model chose to run and even if an allowed command does more than its name suggests.”

“Even if an allowed command does more than its name suggests” is the entire argument for rm -rf ./build in nine words. The permission layer judges a string; a string cannot tell you where a symlink resolves. Note also what the runtime does not special-case: rm and rmdir commands targeting /, your home directory, or other critical system paths still trigger a prompt, and ./build is none of those. The footgun is not the dramatic command. It is the boring one whose blast radius is a property of the filesystem rather than of the text.

Standards bodies land in the same place. OWASP’s prompt-injection entry recommends that you “Provide the application with its own API tokens for extensible functionality, and handle these functions in code rather than providing them to the model,” and restrict the model’s privileges to the minimum necessary. Both are framed as measures that can mitigate impact, alongside the admission that “it is unclear if there are fool-proof methods of prevention for prompt injection” (OWASP: LLM01:2025 Prompt Injection¹⁴). Willison puts the practitioner version in one line: you can try telling it not to in your own prompt, “but how confident can you be that your protection will work every time?” (Simon Willison: The Lethal Trifecta¹⁵).

There is a stronger form of enforcement than blocking a call, and one runtime documents it plainly. In Gemini CLI’s policy engine, a deny is not a refusal delivered to the model — it is a change to what the model can perceive. For global rules, meaning those without an argsPattern, tools that are denied are “completely excluded from the model’s memory,” so “the model will not even see the tool as an option, which is more secure and saves context window space” (Gemini CLI: Policy Engine¹⁶). That scope qualifier matters: the docs make the claim for blanket deny rules and say nothing about argument-conditional ones.

The audit question is therefore: where does this check physically execute — in the model’s context, in the runtime before dispatch, or in the kernel? Which authority tier a given task class deserves is a separate question, and the permission tiering post answers it in full. This post is about the mechanism underneath that policy: whether the tier you assigned is enforced by anything at all.

Your Hook Layer Is Not an Enforcement Boundary

The PreToolUse hook you wrote to catch destructive commands crashed on a malformed argument, exited 1, and rm -rf ./build proceeded anyway. That is not a bug in your script. It is the documented default in every major agent runtime, and two vendors say so in their own reference docs.

“For most hook events, only exit code 2 blocks the action. Claude Code treats exit code 1 as a non-blocking error and proceeds with the action, even though 1 is the conventional Unix failure code.”

The same page goes further and redirects you off the layer entirely. Because the if filter is best-effort — it fails open and runs your hook regardless of pattern when the Bash command cannot be parsed — the docs instruct you to “use the permission system rather than a hook to enforce a hard allow or deny.” OpenAI’s Codex documentation is blunter still:

“Some specialized tool paths can opt out of the default hook path. Treat tool hooks as a useful guardrail, not a complete enforcement boundary.”

The named hole behind “specialized tool paths” is concrete: hosted tools such as WebSearch do not use the local function-tool hook path at all. Codex also continues the tool call when a PreToolUse hook returns an unsupported field, and it launches multiple matching command hooks concurrently, “so one hook can’t prevent another matching hook from starting.” Cursor documents the same posture with an escape hatch attached. Unexpected exit codes mean “Hook failed, action proceeds (fail-open by default)”. The per-script failClosed option that reverses this defaults to false, described by the vendor as “Useful for security-critical hooks” (Cursor Docs: Hooks¹⁸). The vendor anticipated your security use case and shipped the switch off.

One more qualifier, because the word “deterministic” is doing heavy lifting in this space. Anthropic’s hooks guide does say hooks “provide deterministic control over Claude Code’s behavior, ensuring certain actions always happen rather than relying on the LLM to choose to run them.” The next sentence offers prompt-based and agent-based hooks “that use a Claude model to evaluate conditions” (Claude Code Docs: Automate Actions with Hooks¹⁹). “Hook” is an umbrella that now contains probabilistic members. Never audit it as a single category.

When you see one of these in your config, run the matching move:

Hooks are still worth having. They are the right layer for formatting, notification, telemetry, and the fast feedback that helps an agent land bounded, verifiable work on the first try. They are the wrong layer to carry a policy whose failure mode you cannot tolerate.

The Tool-Call Audit Trail Ships Disabled

Turn on the record of which surface authorized each call before you need it to reconstruct an incident. Afterward is too late, because the record is not buffered somewhere waiting to be enabled — at defaults, it was never written.

The artifact already exists and it is better than most teams expect. Claude Code emits a claude_code.tool_decision event carrying a source attribute that names the control surface responsible for each decision: "config", "hook", "user_permanent", "user_temporary", "user_abort", and "user_reject". The docs define the second one exactly as you would want: "hook" means “A PreToolUse or PermissionRequest hook returned the decision” (Claude Code Docs: Monitoring⁷). That is the answer to “which layer let rm -rf ./build through,” recorded automatically, for free — once two environment variables are set.

# Nothing is emitted until this is set; the docs mark it "(required)"
export CLAUDE_CODE_ENABLE_TELEMETRY=1

# Without this, you learn that Bash ran. Not what it ran.
export OTEL_LOG_TOOL_DETAILS=1

The second variable is the one teams miss. tool_parameters — the actual command string — is emitted only when OTEL_LOG_TOOL_DETAILS=1. And the gap is not vendor-specific: in the OpenTelemetry GenAI semantic conventions, gen_ai.tool.name carries requirement level Required, while gen_ai.tool.call.arguments (“Parameters passed to the tool call”) and gen_ai.tool.call.result are both Opt-In (OpenTelemetry: GenAI Spans²⁰). A fully spec-compliant agent trace tells you that Bash ran and nothing whatsoever about what it ran. That is a real privacy tradeoff rather than an oversight, which is why it will not fix itself by upgrading.

Know the blind spot you are buying, too. When source is "config", the event covers project settings, personal allow and deny rules, managed policy, CLI flags, the active permission mode, a session-scoped grant, or the tool simply being inherently safe. The docs are explicit about the limit: “The event doesn’t indicate which of these sources matched.” You learn the decision was automatic. You do not learn which rule made it.

This matters because the vendor has explicitly handed you the review job: “Claude Code only has the permissions you grant it. You’re responsible for reviewing proposed code and commands for safety before approval” (Claude Code Docs: Security⁴). Responsibility you cannot reconstruct after the fact is not responsibility you are discharging.

This Bounds Blast Radius, Not Code Quality

Author’s judgment. Separating the two axes — maintainability versus blast radius — is my framing, not a distinction the counter-sources draw. It follows from their own scope: the strongest critiques of harness engineering measure design quality, and the controls in this audit do not touch design quality.

The strongest argument against spending a week on your harness config is that it will not make the code better. It won’t, and pretending otherwise is how this practice gets sold as a quality program and then fails to deliver one.

“What I’m gonna try to convince you is that no amount of harness engineering or loopsmaxxing can solve what is fundamentally a model-training issue.”

Fast deterministic gates measure what is fast to measure: “Running the tests gets you a clean pass or fail in ~seconds.” Meanwhile “the cost function of bad architecture is measured in weeks, months, maybe even years,” and “there is no penalty for eroding codebase maintainability.” No arrangement of hooks and permission rules detects a design that will be expensive in eighteen months. The academic synthesis agrees from the other side: across 27 benchmark, taxonomy, and audit papers spanning 19 distinct benchmarks, “additional scaffolding does not consistently improve reliability” (Albayaydh et al.: Beyond the Leaderboard²²).

Both are aimed at a different axis than this audit. They measure whether the agent produces good software; the audit measures how far a wrong call can reach. Sell the harness audit as blast-radius control and never as a quality program — the first claim is well-supported, the second is refuted, and bundling them means losing the argument on the weaker half.

Even the blast-radius claim has a documented ceiling, and the honest move is to source it from the vendor’s own contradiction. Anthropic’s engineering blog says “Sandboxing ensures that even a successful prompt injection is fully isolated, and cannot impact overall user security” (Anthropic: Beyond Permission Prompts²³). The product documentation for the same feature says “Sandboxing reduces risk but is not a complete isolation boundary,” then enumerates the escape routes (Claude Code Docs: Sandboxing⁶):

The marketing surface and the reference surface disagree, and only one of them is the thing that runs.

The Audit Is Recurring, Not One-Time

Schedule this against your runtime’s release notes rather than your compliance calendar, because the object you audited last quarter has since been edited by someone who does not work for you. Böckeler names the shape directly: building the outer harness “is emerging as an ongoing engineering practice, not a one-time configuration” (Birgitta Böckeler: Harness Engineering²).

The drift rate is measurable on a single page. The Claude Code sandboxing reference carries version-gated behavior notes at seven separate point releases between v2.1.187 and v2.1.218 (Claude Code Docs: Sandboxing⁶). The list: credential protection, network session grants, TLS termination, symlink resolution on protected settings paths, plan-mode approval scope, filesystem-layer disabling, and classifier routing for rm. Thirty-one point releases, seven documented changes to what the enforcement layer does. An annual review against that cadence is decorative.

Lilian Weng arrives at the structural version of this from a narrower case — self-improving harnesses, meaning harnesses that edit their own components. Her position on those: “The evaluator and permission control should likely sit outside the loop that evolves harness, with held-out tests, trace audits, and human review at decision points that matter—how much oversight can be scaled up and automated remains an open research area” (Lilian Weng: Harness Engineering for Self-Improvement²⁴). Note the hedge and the scope; she is writing about self-modifying systems, not agent harnesses in general. The transferable principle is small and useful anyway: a check that lives inside the thing it governs moves when that thing moves. The same logic applies to the iteration structure the harness runs, which the loop engineering post covers as its own problem.

The Harness Audit: Seven Questions

Run these against your own config this week and write down the one you cannot answer. That is your finding — not a policy document, not a maturity score, just the specific question about rm -rf ./build that your organization currently has no answer to.

1. Name the software that decides. Open the settings files at every scope and list which permission rules you actually wrote versus which arrived with the install. If you cannot say which rule matches rm -rf ./build, you are running inherited defaults, which is a choice you have not yet made (Claude Code Docs: Security⁴).
2. Ask where each check physically executes. Sort every control into three buckets: the model’s context, the runtime before dispatch, and the kernel. Anything in the first bucket is a guide. Anything you assigned an authority tier in the permission tiering framework needs a mechanism in one of the other two.
3. Check whether your sandbox can quietly not exist. Confirm sandbox.failIfUnavailable is true and allowUnsandboxedCommands is false, so the dangerouslyDisableSandbox retry is ignored. Then read excludedCommands at every scope — that key has no managed-only lockdown, so any developer can append to it (Claude Code Docs: Sandboxing⁶).
4. Grep your hook scripts for exit 1. Every one you find is a policy that fails open. Convert the ones that must block to exit 2, set failClosed: true on Cursor’s security-critical scripts, and move anything that must hold to the permission system (Claude Code Docs: Hooks Reference⁵).
5. Turn on the decision record before the incident. Set CLAUDE_CODE_ENABLE_TELEMETRY=1 and OTEL_LOG_TOOL_DETAILS=1, then verify a claude_code.tool_decision event actually lands with a populated source and tool_parameters (Claude Code Docs: Monitoring⁷).
6. Re-run this against release notes, not the calendar. Diff your runtime’s changelog for behavior changes to the enforcement surfaces, and re-baseline anything you measured. When the harness changes, the same model can move double digits, so the numbers you decided on belong to a system that no longer exists (Terminal-Bench 2.0 Leaderboard⁹).
7. Say out loud what this buys. Blast radius, not code quality. If anyone in the room expects the audit to improve the agent’s architecture, correct them before you start, or the whole exercise gets judged against a promise it never made.

If you would rather not open five settings files by hand, harness-audit²⁵ reads them for you and prints the seven answers — including the ones it cannot determine, which are the ones worth writing down.

Do this at the config layer rather than the prompt layer because prompt-time review does not scale, and the vendor building the approval flow says so. Constantly clicking approve “can lead to ‘approval fatigue’, where users might not pay close attention to what they’re approving, and in turn making development less safe” (Anthropic: Beyond Permission Prompts²³). The seventh approval of the day is not review. It is a reflex with a keyboard shortcut.

Answer the seven questions and you have converted a shrug into a specification. Then go decide which authority tier each task class actually deserves — that is the next question, and it is a much better one to argue about once you know your answers are enforced by something.