Self-Improving AI Software Factory: Govern the Harness Loop

A self-improving AI software factory is a loop that edits the harness your agents run on. For a buyer or a platform team, the question is not whether the loop runs. It is who controls three things: what the loop is graded on, what it is allowed to touch, and who decides what goes live.

Factory raised $200M in September to “create a world where software builds and improves itself” (Factory: $200M at $5B¹).

Its Signals post is the most concrete public description of that loop. It validates a shipped fix with a before-and-after reading of a friction signal, one the loop itself mines for work (Factory: Signals²).

Now take away the optimizer’s view of the test. When researchers split Terminal-Bench 2.1 so the optimizer never saw the test tasks, the evolved harness averaged exactly flat across three models (Wang et al.: Rethinking Harness Evolution³).

Draw the Loop That Edits Your Harness

Every vendor draws the same self-improving loop. None of them marks which parts the proposer can write to.

Port’s loop has four steps: capture the run, grade it, find the pattern, and write the fix back into the skill or prompt, with a person on the write-back (Port: Self-Improving AI Software Factory⁴).

Autoheal’s loop has two agents. “The Evaluator agent scores every worker agent’s run,” and a Healer agent “fixes low scoring worker agents by opening pull requests that improve skills, prompts, tools, or model selections” (Autoheal: $7.9M seed⁵).

Those descriptions are accurate. They also skip the part that decides whether the loop improves anything:

A self-improvement loop optimizes whatever signal it is given.

So before you buy or build one, draw the loop as five roles: proposer, evaluator, gate, record, live harness. Then mark every role the proposer can write to.

The map of writable roles is the map of what will get optimized.

Here is Factory’s Signals loop drawn that way, from the post alone:

# Five roles. writable_by_proposer: "not stated" means the post is silent.
proposer:
  who: Droid
  does: "picks up those tickets, implements fixes"
evaluator:
  who: friction analyzer
  reads: [error events, repeated rephrasing, escalation in tone]
  writable_by_proposer: not stated
gate:
  - who: Droid review        # the post: Droid "reviews its own PRs"
    writable_by_proposer: same agent
  - who: human PR approval   # "A human still approves the PR before merge."
    writable_by_proposer: no
record:
  what: [Linear ticket, PR description]
  written_by: the loop       # "it files Linear tickets automatically"
live_harness:
  what: Droid as users run it
  changes_on: merge

Draw the Signals loop as roles and two things stand out:

• One gate the proposer cannot influence: the human who “still approves the PR before merge” (Factory: Signals²).
• One evaluator with unstated write access: the post never discusses who can write to it.

That is not a claim about Factory’s internals. It is the question the diagram makes you ask.

The research literature draws the same map with different labels. One audit sorts harness components into five roles: Execution, Evaluation, Selection, Recording, Propagation. It reports that “Harness tampering is observed across all evaluated self-improving systems” (Wang, Zhang, Shao: Auditing Harness Tampering⁷).

The proposer also reads more than you might expect. In Meta-Harness, “Each evaluated harness contributes a directory containing its source code, scores, and execution traces” for the next proposal (Lee et al.: Meta-Harness⁸). Anything the proposer reads, it can learn to move.

The loop drawn here is the outer loop. The loop engineering guide covers the inner loop of a single agent run: caps, ledgers, gates. The outer loop rewrites the harness those inner loops run on.

Ask for the Number on Unseen Tasks

“The factory improved itself” is a claim about tasks it has not seen yet. Most published evidence measures it on tasks it already has.

The paper that tested this most directly draws the line. A gain on the tasks that drove the change is adaptation, not improvement:

Moreover, when the same set of tasks is used for both harness evolution and final evaluation, reported gains overstate true harness improvements, as they reflect adaptation to specific instances rather than generalizable harness design.

So ask where each gain was measured. Line up the studies published in 2026 that way, and the held-out evidence runs from mixed to flat:

RRSI names the pattern: “The separation appears out of distribution, and there the ranking inverts” (Xia et al.: RRSI⁹).

Giummarra et al. state their own result in one line: “validation gains may reverse on held-out evaluation” (Giummarra et al.: Reflective Prompt and Skill Optimization¹⁰).

Meta-Harness is upfront about its setup. It audits for leakage by hand and by regex. That is a check, not a held-out number (Lee et al.: Meta-Harness⁸).

The counter-evidence belongs in the same conversation. Two results point the other way.

The Rethinking paper reports that “task-specific harness evolution improves over the search baseline by 80% on ARC-AGI-3” and by 11% on EdgeBench games (Wang et al.: Rethinking Harness Evolution³).

Weco’s research-agent run selected on private held-out grading. On a task family outside the selection benchmark, its reward-hacking rate fell “from 55% to 32% during the run, 7 percentage points below the human-engineered agent” (Srikanth et al.: Recursive Self-Improvement of AI Research Agents¹¹).

The positive case shares one thing with the negative ones: the number that matters was measured on data the optimizer did not train on.

Two caveats. Every study here is a benchmark study, not a production factory, so hold the transfer to your traces loosely. And small deltas on agent benchmarks sit inside run-to-run noise more often than people expect, which the leaderboard noise post covers.

Now apply the same standard to Factory’s own number. After a change to how Droid handles ambiguous requests, “the ‘repeated rephrasing’ friction rate dropped by thirty percent within forty-eight hours” (Factory: Signals²).

Repeated rephrasing is one of the signals the friction analyzer scans for. So that number is a before-and-after reading of a signal inside the loop. It may be a real improvement. But the post does not report what the metric did on sessions the loop never mined. That is the number to ask for.

The claim test is therefore: what moved on tasks the optimizer never saw?

Split the Evals Before the Optimizer Runs

Author’s judgment. The third, release-only holdout below is my inference, not a design any single source tested. It follows from three premises: Self-Harness’s held-out split drives selection and its authors say higher-stakes changes need stronger gates, Giummarra et al. call for a fresh external holdout, and Henry Pan says his own gate lacked a held-out panel.

Split your evals three ways:

• Proposer split. The proposer learns from it.
• Gate split. The promotion gate scores against it. The proposer never sees it.
• Release holdout. Only a person runs it, at release.

The first two come from Self-Harness, where “the held-out split is never shown to the proposer and is used only by the automatic promotion gate” (Zhang et al.: Self-Harness¹²). Its acceptance rule is strict about trade-offs:

# evals/proposer/  sessions that crossed the threshold: traces, scores, failures visible to the proposer
# evals/gate/      frozen panel never mined for tickets: scores visible to the gate only
# evals/release/   holdout a named person runs at release: the loop has no read access

def accept(candidate, incumbent):
    d_in  = candidate.passes("proposer") - incumbent.passes("proposer")
    d_out = candidate.passes("gate") - incumbent.passes("gate")
    # "A candidate is accepted only if it improves at least one split
    #  without degrading the other"
    return (d_in > 0 or d_out > 0) and d_in >= 0 and d_out >= 0

The d_in >= 0 and d_out >= 0 clause is the one teams drop. Dropping it is the footgun. “Proposals that only trade off one split against the other are rejected, even if their total pass count increases” (Zhang et al.: Self-Harness¹²).

Say a candidate gains six proposer tasks and loses two gate tasks. Its total goes up by four. It still does not ship.

The third split exists because a gate that selects on its holdout slowly turns that holdout into a training set. Every accept or reject leaks a little about the gate split back into the lineage.

Four sources flag the same limit:

The RSEA authors state their own gate’s limit: “The strict gate guarantees safety on held-out val, not on every test draw when val is small” (Nguyen, Nguyen, Vuong: Held-Out Selection¹³).

Giummarra et al. say “independent confirmation additionally requires a fresh external or study-level holdout” (Giummarra et al.: Reflective Prompt and Skill Optimization¹⁰).

Henry Pan ran more than a thousand experiments. Of his own setup, he says: “A stricter version would require a held-out task panel before declaring general improvement” (Henry Pan: 1,000+ Harness Experiments¹⁴).

Self-Harness notes that “Higher-stakes harness changes would require stronger acceptance gates than pass-rate non-regression alone” (Zhang et al.: Self-Harness¹²).

Here is how the three splits map onto the Signals loop:

• Proposer split: the sessions whose friction crossed the threshold. The loop already mines these.
• Gate split: a frozen panel of sessions never mined for tickets.
• Release holdout: a set no Droid touches. A person runs it before a promoted change reaches every user.

Factory’s post describes the first of these and neither of the others (Factory: Signals²).

Designing the suite those splits come from (triggers, tiers, failure intake) is covered in the standing agent eval suite. This section only asks who can see which split.

Expect Most Candidates to Die at the Gate

Plan for rejection as the normal outcome. A gate that rarely says no is not gating.

Henry Pan’s footnotes give the clearest practitioner number: “768 of 1,231 non-baseline candidates were discarded because a baseline-solved task regressed” (Henry Pan: 1,000+ Harness Experiments¹⁴). That is about 62% rejected on regressions alone. And that is before you check whether the survivors generalize.

Take the gate away. RSEA’s ALFWorld ablation runs selection without a held-out gate. The result: “a perfect 100% in-sample selection score but only 66.7% on test” (Nguyen, Nguyen, Vuong: Held-Out Selection¹³). Compare that with the alternatives:

The ungated loop looked perfect to itself. On test, it barely beat plain ReAct with no evolved harness at all.

A weak gate also costs you later. Weco names it: “a falsely accepted rewrite becomes the new incumbent” (Srikanth et al.: Recursive Self-Improvement of AI Research Agents¹¹). Every later candidate is then measured against that bad incumbent.

Freeze the Evaluator and Everything It Reads

A “DO NOT CHANGE” comment is a request. A self-improving loop selects for the edits that ignore it.

Move these out of the editable surface entirely: the evaluator, the eval data, the budget, the sandbox flag, the model config, and the logs the evaluator reads.

Bad: STOP’s improver protected its sandbox with a comment on the scoring line, score = utility(solution, use_sandbox=True) # DO NOT CHANGE use_sandbox=True. Generated GPT-4 improvers turned the sandbox off in 0.42% of cases without the warning and in 0.46% of cases with it. The authors’ verdict: “the difference was not statistically significant for either model” (Zelikman et al.: STOP¹⁵).

Good: AHE’s boundary is a permission, not a comment: “the Evolve Agent writes only inside the harness workspace, while the runs directory, tracer, verifier, and LLM configuration are read-only” (Lin et al.: Agentic Harness Engineering¹⁶).

The comment does not fail because the model misread it. It fails because of selection. STOP explains: “First, when not strictly enforcing budget constraints, initially most generated programs would follow them, but because those that ignored the constraints performed better, these would be identified by the improver as better” (Zelikman et al.: STOP¹⁵).

The loop needs no intent. A violating candidate only has to score higher once. Selection does the rest.

Henry Pan enforced the same boundary from a supervisor outside the agent. His improvement agent may edit three files: config/harness_config.json, src/harness/core.py and tests/test_core.py. Everything else stays fixed, including “model, provider, task panel, budgets, retries, concurrency, context ratio, and trial count.” A launch check the agent “cannot escape” holds it there (Henry Pan: 1,000+ Harness Experiments¹⁴).

The agent harness audit covers where checks physically run. The point here is narrower: the boundary is a file permission or a supervisor check, never an instruction.

In the Signals loop, the evaluator is the friction analyzer. So can a Droid open a PR against the code that computes “repeated rephrasing”? The post does not say (Factory: Signals²). Make it the first question you put to the vendor.

Protect the Inputs, Not Just the Scorer

Freezing the evaluator is not enough. Freeze the logs and libraries it reads too. Agents reach the score through its inputs.

Sakana caught the DGM case through “a transparent, traceable lineage of every change,” not a hidden evaluator. Even so, Sakana wrote that “more work is needed to prevent the model from attempting to cheat in this way in the first place” (Sakana AI: Darwin Gödel Machine¹⁷).

METR’s prescription points the same way: “patch the exploit in the scoring function rather than punishing the model” (METR: Recent Frontier Models Are Reward Hacking¹⁹).

Mind the settings. METR is 2025 frontier-model evaluation. Baker et al. is 2025 RL training. Treat both as evidence of how agents reach scores, not as harness-loop rates.

Treat the Change Log as a Claim

The proposer writes its own change record. That makes the record a claim about the change, not evidence for it. Read it as something to check. Never let it act as the gate.

Two rules follow. Require every change to cite a failure that an independent check confirms. And measure regressions instead of trusting the predicted risk.

AHE tested this directly. It asked its evolve agent to predict which tasks each edit would fix and which it would break, then scored those predictions:

Fix predictions sat roughly 5x above random. Regression predictions sat “only about 2x above their random baselines,” so “most upcoming regressions go unforeseen” (Lin et al.: Agentic Harness Engineering¹⁶).

So the change record is useful for one thing: what the change was meant to do. For what it will break, it is close to useless.

The failure case is worse than silence. Give a proposer nothing to fix, and it can invent a failure, then guard against it.

Phantom Guardrails built a micro-lab where the correct action was to do nothing. The proposer “invents a failure: in 15/60 runs, versus 0/60 on featureless input,” then adds a guard for it (Wang et al.: Phantom Guardrails²⁰). Eleven of the fifteen came from one proposer model in one synthetic domain. Read it as a mechanism, not a cross-model rate.

Accept-if-not-worse lets those guards pile up. Under that rule, phantom guards grew “from 1/60 trajectories after round one to 11/60,” because “Removing a no-op changes the proxy by exactly zero, so the suppression signal never demerits it” (Wang et al.: Phantom Guardrails²⁰).

A warrant-aware rule excludes them. It credits a guard only when an oracle confirms the failure the guard cites. In the lab, it kept the phantom out of every round and still adopted the real fixes in 60 of 60 runs (Wang et al.: Phantom Guardrails²⁰).

Those two rules turn into four checks:

• A change cites a failure. Replay the cited episode against an oracle before crediting it. If the failure does not reproduce, reject the change, however plausible the PR description reads.
• A change predicts “no regressions.” Run the gate split anyway. A prediction 2x above random is not a measurement.
• A change adds a guard, check, or instruction. Ask which confirmed episode it fixes. Add-only changes are where phantom fixes accumulate, because a suppression metric never penalizes them.
• A change summarizes its own test results. Re-run the tests from the gate, outside the loop’s process, and diff the results.

Now read the Signals loop against that list. The system “files Linear tickets automatically,” and “Droid picks up those tickets, implements fixes, and reviews its own PRs” (Factory: Signals²). The ticket, the fix, the PR description and the first review are each a self-report.

Only one step sits outside the loop: the human approval. That raises the next question: what is that human shown?

Put the Promotion Decision Outside the Loop

A reviewer inside the loop feels the same pressure as the proposer. So the decision about what goes live has to come from something the loop cannot influence.

That takes two things:

• A mechanically enforced veto. It ignores the benchmark score entirely.
• A named approver. This person reads the gate evidence and holds stop and rollback.

Roesner and Kohno show why. They poisoned a self-modifying coding agent’s benchmark and put an LLM security reviewer in the loop. In the first generation, “the security reviewer itself was influenced by the benchmark’s pressure” and approved the change. The cost: “After this generation, half of held-out tasks were solved with vulnerable code” (Roesner, Kohno: Reflections on Trusting Trust, Revisited²¹).

The reviewer prevailed in the next generation. So the lesson is not that reviewers fail. It is that they compete with the score, and sometimes the score wins. The authors’ proposed fix:

Security review could instead be integrated as more than a nudge (e.g., with a mechanistically enforced veto that disregards benchmark scoring).

Three deployments show where promotion authority can sit:

Ouroboros is the most developed of the three. Its design principle is the one to copy: “separate the substrate being evolved from the authority that decides whether a mutation can become the next live version” (Razzhigaev et al.: Ouroboros²²).

Warp’s loop is the plain version: “A human reviews, approves, and merges, and the next run of the triage skill inherits the new knowledge” (Segner: How Warp Builds Self-Improving Agents²³).

Both fit Weng’s framing that “Humans should move up the stack, not be removed from the loop” (Lilian Weng: Harness Engineering for Self-Improvement⁶). The stakes of that human step scale with blast radius: “an edit to a shared skill changes how every team that installs it works” (Port: Self-Improving AI Software Factory⁴).

Factory and Autoheal both put a human on the merge. Autoheal says “Every behavior change is version-controlled in git and requires engineer approval,” and that the system “verifies those changes against historical benchmarks for regressions before engineer review” (Autoheal: $7.9M seed⁵).

A human on the merge is necessary but not enough. The question is what evidence that human sees: the Droid’s own description and review, or a gate result on a split the loop never touched.

An approver reading the proposer’s summary is signing off on the loop’s claim about itself. At the volume a self-improving loop generates, that is the review that gets skimmed first.

Keep the Last Promoted Harness as Fallback

Author’s judgment. Treating fallback as the default state, rather than an incident action, is my inference. It follows from RSEA’s fallback-to-base mechanic and from Roesner and Kohno’s finding that cleaning contamination out of an evolved agent is difficult.

Keep the last promoted harness as the default. When a candidate cannot beat it, fall back to it automatically.

RSEA builds this into selection: “if no candidate strictly improves, RSEA returns the empty state and reduces exactly to ReAct,” which is why it “never significantly underperforms the base agent on any benchmark” (Nguyen, Nguyen, Vuong: Held-Out Selection¹³). Doing nothing is a legal outcome of an improvement round. The loop is never forced to ship its best bad candidate.

Make the fallback automatic, not an on-call action. Cleanup after the fact is unreliable.

Roesner and Kohno found that decontaminating a poisoned self-modifying agent “is difficult, particularly if the defender does not already have knowledge of the design of the poisoned benchmark” (Roesner, Kohno: Reflections on Trusting Trust, Revisited²¹). Bad changes also stick around. The tampering audit found that “Uncorrected early modifications can also persist across dozens of iterations, being carried through to the final best agents” (Wang, Zhang, Shao: Auditing Harness Tampering⁷).

If you cannot reliably remove a bad change from an evolved harness, don’t try. Return to a version that predates it.

Four steps support that return:

1. Commit one logical change at a time. AHE’s convention: “Each logical edit becomes one commit on the workspace’s git history, which yields file-level diffs and rollback granularity for free” (Lin et al.: Agentic Harness Engineering¹⁶).
2. Tag every promoted version. The fallback target is the last tag that passed the gate and the release holdout, not the last commit.
3. Make fallback the default state. A round with no candidate that clears the gate ends on the incumbent, with no human action required.
4. Run rollback on a path the loop cannot touch. In Ouroboros, “Rollback restores an earlier reviewed state and follows a separate recovery path” (Razzhigaev et al.: Ouroboros²²).

The Self-Improving Factory, Compressed

Hand this table to a vendor or to your own platform team. If a factory cannot answer every row, it is changing itself. It is not improving itself.

For your own loop, I packaged the gate rows of this table as a script: harness-promotion-gate²⁴. It does four things:

1. It rejects a candidate that touches the evaluator or its inputs.
2. It applies the improve-one-degrade-neither rule across the proposer and gate splits.
3. It refuses any cited failure an oracle did not confirm.
4. It prints the evidence your named approver should read, instead of the change’s own summary.