Where a Decision Model Belongs: Placing Jev in Your Stack

Jev’s launch numbers answer a question you do not have. The tier it occupies has been running in production at Anthropic, Meta and OpenAI for over a year, and those teams published what it costs: roughly 25 percent inference overhead when a small LLM does the filtering (Anthropic Alignment Science: Cost-Effective Constitutional Classifiers¹), and 19.3ms per classification for a 22M-parameter purpose-built model (Chennabasappa et al.: LlamaFirewall²). So the question a new model class raises is not whether it is fast. It is which decisions move down a tier, which ones should never have left deterministic code, and who owns the threshold once they do.

Read the Response Shape Before the Benchmark Table

The launch post’s multiples tell you nothing about where the model goes in your architecture. The API reference does, and it takes about ninety seconds to read.

Take the narrowest decision anyone is actually buying this for: should this agent’s rm -rf run? Written as a Noul question against the documented endpoint, the gate looks like this.

POST https://api.typesafe.ai/v1/systemone

{
  "state": {
    "pending_command": "rm -rf ./build",
    "cwd": "/srv/app",
    "requested_by": "agent-session-4412"
  },
  "model": "jev-latest",
  "questions": {
    "rm_is_safe": {
      "type": "noul",
      "instructions": "Is this removal confined to build output this session created?",
      "criteria": {
        "true": "Target is disposable build output",
        "false": "Target may contain source or data"
      }
    }
  }
}

What comes back is not an allow or a deny. It is an answers object keyed by your question id, carrying a noul field that the docs define as “The yes/no answer on a scale from 0 (no) to 1 (yes),” plus a usage block and the model id that actually answered (TypeSafe: API reference³). Nothing in the payload executes anything. You still have to write the if.

Two details in that payload are load-bearing and neither one appears in a benchmark table. The vendor’s own example response returns "model": "jev-1.13.0" for a request that asked for jev-latest, which means alias resolution is visible in the payload of every call you make (TypeSafe: API reference³). And state accepts “A plain string for text, or structured data (object/array) for things like chat logs, records, or the current state of your application,” which is the seam where your context window budget and the model’s stop being the same number.

Author’s judgment. The observation that this request shape diverges from the OpenAI chat-completions convention is mine. It follows from reading the two schemas side by side: TypeSafe’s body is state plus a map of named typed questions, where a chat-completions body is a message array. Neither vendor states the contrast, so the integration cost that follows from it is an inference, not a published claim.

That divergence is the part the speed comparison hides. Swapping one chat-completions provider for another is a base-URL change; adopting a System One model means writing question definitions, naming them, banding their outputs, and logging them, in a shape no other model in your stack uses. That is integration work, not a model swap.

The hosts do not even agree on the budget you design against. TypeSafe’s own Models page publishes “64k tokens per request; 32k tokens for state plus the longest question” (TypeSafe: Models⁴), while OpenRouter lists 32K context for typesafe/jev-1.13 (OpenRouter: Typesafe API and Models⁵) and Cloudflare publishes a 32,000-token context window on its model page (Cloudflare Docs: Jev⁶). Pick whichever you like; just know you picked one.

And before you read the launch page, read the concepts page, because the vendor has already limited its own claim.

Calibration is measured across groups of predictions; it does not guarantee that an individual answer is correct.

The launch post, on the same domain, says “The model never makes type errors” (TypeSafe: Introducing System One Models & Jev⁸). Both statements are true and they are about different things. One is a guarantee about the shape of the answer. The other is an explicit refusal to guarantee the content of it. The placement question is therefore: what do you do with a number that is well calibrated in aggregate and unverified in the individual case?

The Tier Already Ships at Anthropic, Meta and OpenAI

Before you price Jev, price whatever is answering rm_is_safe in your system today. You already run something in that slot, even if it is a regex, and the replacement cost is the difference between the two, not the vendor’s multiple against a frontier LLM.

There are three published tiers, and your current answer is one of them.

Those are not projections. LlamaFirewall reports the first-layer numbers from internal production deployments, and notes that “For the remaining 10% of cases requiring deeper inspection, end-to-end latency can exceed 300 milliseconds” (Chennabasappa et al.: LlamaFirewall²). The shape of a shipped guardrail stack is a cheap tier that resolves nearly everything and an expensive tier that sees the residual.

The tax on that slot has been public since January 2025. Anthropic’s first-generation constitutional classifiers shipped with the bill attached: “These classifiers also maintain deployment viability, with an absolute 0.38% increase in production-traffic refusals and a 23.7% inference overhead” (Sharma et al.: Constitutional Classifiers⁹). A year later the same team cut that to roughly 1 percent additional compute by cascading, with a 0.05 percent refusal rate on harmless queries measured over a month of Sonnet 4.5 traffic (Anthropic: Next-generation Constitutional Classifiers¹⁰). The cascade design is the part worth stealing:

Since exchanges flagged by the first stage are escalated rather than refused, the first-stage classifier can flag a higher proportion of production traffic without incurring an excessive refusal rate.

In deployment that first-layer probe escalated approximately 5.5 percent of traffic to the second stage, for approximately a 40x reduction compared to the single exchange classifier (Cunningham et al.: Constitutional Classifiers++¹¹). Escalating instead of refusing is what buys the cheap tier its false-positive budget, and it is the reason a gate’s band design matters more than its headline accuracy.

OpenAI says the same thing from the other direction, comparing its reasoning-based safety model against the classifiers it sits behind: “Traditional classifiers have lower latency and cost less to sample from than gpt-oss-safeguard” (OpenAI and ROOST: User guide for gpt-oss-safeguard¹²). These tiers are complements. Nobody shipping this at volume treats them as substitutes.

Read a cost claim against the baseline it declines to name

Jev’s published multiples are measured against frontier LLMs, not against the tier above. The launch post’s numbers come from “Workflow evals” in which, in the vendor’s words, “we test how they compare to the average of the smartest models (in this case, Astra and Fable)” (TypeSafe: Introducing System One Models & Jev⁸). DataCamp traced the headline accuracy figure to the same construction: “Jev averages 67.8% agreement with the reference answers,” where the scoring is done “against the average predictions of GPT-6 Astra and Anthropic’s Fable” (Matt Crabtree: Jev: TypeSafe’s System One Model Explained¹³). That is agreement with a synthetic consensus of two other vendors’ models, not agreement with ground truth. The same piece says plainly that “All the numbers below come from TypeSafe’s own evaluation tables, and no large-scale independent reproduction has surfaced yet, so treat them as vendor-reported.”

I have written elsewhere about why a vendor-run eval is not a procurement signal (why leaderboard movement is mostly noise); the point here is narrower. A 40x speedup against a frontier LLM is not a speedup against a 19.3ms classifier, and no published number compares the two.

The slot may also be temporary. sean goedecke argues that a generic decision model carries weights for tasks you do not have: “Generic classifiers have to encode knowledge of all kinds of irrelevant things in their weights, so they can address lots of different tasks. That makes them larger, slower, and more expensive to run,” and concludes that “For serious work, a specific hand-built classifier will always be cheaper and faster than Jev” (sean goedecke: System One models like Jev can train their own replacements¹⁴). He flags this as speculation, writing “I expect this to be a common pattern,” and he notes the bespoke path still needs ML expertise, just deferred until after the feature is validated. Treat it as a hypothesis about the shape of the market rather than a shelf life you can put in a spreadsheet. It still changes the buy decision: you may be renting the tier while you find out whether the feature is worth owning.

Deterministic Rules First, the Model on the Residual

Resolve every action you can state as a rule before any of it reaches a model. This is not a preference. It is the documented, shipped decision order for the one pre-execution gate with a public spec, and the model sits in third position.

Claude Code’s auto mode puts it in writing: “Each action goes through a fixed decision order. The first matching step wins.” Allow, ask and deny rules resolve immediately; read-only actions and file edits in your working directory are auto-approved; “Everything else goes to the classifier” (Claude Code: Choose a permission mode¹⁵). Applied to rm_is_safe, most removals never become a model call at all.

The hierarchy is not clean, and the exceptions are the interesting part. On the same page, writes to protected paths “route to the classifier even when an allow rule matches, and so do rm and rmdir removals targeting a critical path in Claude Code v2.1.218 and later.” Broad allow rules that grant arbitrary code execution, including blanket Bash(*) and wildcarded interpreters like Bash(python*), are dropped on entering auto mode, while “Narrow rules like Bash(npm test) stay in effect” (Claude Code: Choose a permission mode¹⁵). In other words: the deterministic layer wins by default, and the model overrides it precisely where a rule is too coarse to be trusted.

Inventorying that deterministic layer is its own exercise, and most teams have never done it (audit the harness before you blame the model). A decision model is one candidate implementation of the coordination layer that belongs outside each agent’s prompt (coordination logic belongs in an architecture layer), but it is only ever the residual half of that layer. OpenAI’s guidance draws the same line: “Rules alone handle deterministic cases well (e.g., keyword matches, metadata thresholds), but they can struggle with satire, coded language, or nuanced policy boundaries” (OpenAI and ROOST: User guide for gpt-oss-safeguard¹²). Rules are not weaker than the model. They are stronger, on a narrower set.

Prefer a deny rule over a model-mediated boundary

Bad: Tell the agent “don’t touch ./data, only clean ./build” and let rm_is_safe enforce it on every removal for the rest of the session.

Good: Write the constraint into the harness, and let rm_is_safe see only what the rules cannot resolve.

{
  "permissions": {
    "deny": ["Bash(rm -rf ./data*)"],
    "allow": ["Bash(rm -rf ./build)"]
  }
}

The difference is not stylistic. A stated boundary lives in the transcript, and the transcript is a lossy medium:

Boundaries are not stored as rules. The classifier re-reads them from the transcript on each check, so a boundary can be lost if context compaction removes the message that stated it. For a hard guarantee, add a deny rule instead.

That is the footgun in the whole category. A model-mediated constraint degrades silently with context pressure, and nothing in your logs marks the moment it stopped applying. If you need a guarantee, the guarantee has to live somewhere the context window cannot evict it.

Your Threshold Moves on Two Independent Axes

The number you compare rm_is_safe against is a parameter you own, and it decays along two axes that need different instruments. Miss either one and the gate keeps returning confident answers against a threshold that no longer means what it meant when you set it.

Author’s judgment. The three-band act / review / escalate split, with the edges set by what a wrong answer costs, is not mine: Flavio Copes published it first (Flavio Copes: A deep dive into Jev, TypeSafe’s System One model¹⁶). What follows is the layer underneath it. TypeSafe’s build guide describes a two-outcome pattern and shows a two-sided uncertainty window in code; Hendrickx et al. formalize the reject option as a single threshold with two outcomes; Constitutional Classifiers++ supplies the escalate-rather-than-refuse principle. The synthesis I am claiming is narrower: the two-axis decay, and a separate instrument for each axis. No source states it.

This is old ground. “Machine learning models always make a prediction, even when it is likely to be inaccurate,” and rejection as a response to that was, in the authors’ words, “already studied in 1970” (Hendrickx et al.: Machine Learning with a Reject Option¹⁷). Banding a probability is not new architecture. It is a fifty-year-old technique with a known price, and the price is coverage: Geifman and El-Yaniv showed “an unprecedented 2% error in top-5 ImageNet classification can be guaranteed with probability 99.9%, and almost 60% test coverage” (Geifman and El-Yaniv: Selective Classification for Deep Neural Networks¹⁸). Guaranteeing the error rate cost roughly forty percent of the answers. Whatever band you set on rm_is_safe, you are buying the same trade, and the vendor’s accuracy table does not show it to you.

Axis one is the model. Flavio Copes has already covered this half, and covered it well: on version pinning he writes that “The response reports the versioned ID that answered, so log it, and pin that ID once you’ve tuned thresholds against it” (Flavio Copes: A deep dive into Jev, TypeSafe’s System One model¹⁶). He also reads the vendor’s own numbers carefully, noting that “the docs use 0.5 as a review floor and 0.9 before a destructive action, as examples, not defaults.” The vendor agrees that the alias moves: “An alias moves when a new release ships, so the answers behind it can change without a change on your side” (TypeSafe: Models⁴).

Author’s judgment. Calling jev-latest and omni-moderation-latest floating aliases is my characterization. TypeSafe’s “an alias moves” line supports it directly. OpenAI’s moderation docs never use the words “alias” or “floating”; they describe the mechanism and its consequence, and I am naming the pattern the two share.

That pattern has four years of precedent in a shipping product of exactly this shape, and the vendor has spent those years telling customers the threshold is theirs and that it will move. OpenAI’s moderation endpoint returns per-category scores between 0 and 1, and the docs are explicit about who owns the decision:

Treat moderation scores as signals for your application’s policy, not as an automatic blocking decision.

The same page warns that “We plan to continuously upgrade the moderation endpoint’s underlying model. Therefore, custom policies that rely on category_scores may need recalibration over time.”

Axis two is your traffic, under a fixed model version, and this is the one nobody has written down. A pinned jev-1.13.0 does not save you if the distribution of commands hitting rm_is_safe changes. Ovadia et al. found across image, text and tabular classifiers under dataset shift that “traditional post-hoc calibration does indeed fall short, as do several other previous methods” (Ovadia et al.: Can You Trust Your Model’s Uncertainty?²⁰). That is 2019 work on non-LLM classifiers under synthetic corruption, so reading it onto a synthetic-data-trained decision model in 2026 is my extrapolation, not their finding. The routing literature names the same hazard more precisely. Confidence-based deferral “often works remarkably well in practice,” and post-hoc mechanisms beat it specifically where “downstream models are specialists that only work well on a subset of inputs,” where samples carry label noise, and where “there is distribution shift between the train and test set” (Jitkrittum et al.: When Does Confidence-Based Cascade Deferral Suffice?²¹). Confidence routing is not broken. It has three named ways to break, and an agent gate whose command mix changes with every new tool is sitting on the third one.

The vendor, to its credit, tells you to do the work yourself: “Test thresholds by plotting confidence against accuracy on your data” (TypeSafe: How to build with TypeSafe²²). That is the whole procedure in one sentence and almost nobody runs it. Here is the longer version:

1. Pin the version. Send jev-1.13.0, never jev-latest, and record the model string the response returned next to every answer.
2. Label a sample of your own traffic. A few thousand real rm_is_safe calls with a human verdict attached beats any published accuracy table, because it is drawn from your command distribution rather than someone’s workflow eval.
3. Plot confidence against accuracy on that sample and read the reliability curve, not the single accuracy number.
4. Read the coverage price off the curve before you set the band. Pick the error rate you can live with, then look at how many decisions you just gave up.
5. Scale the band edges to what being wrong costs, which is a risk-tiering question you should already have answered (tier the agent’s authority by cost of being wrong), not a property of the model.
6. Re-run on a cadence with a test built for repeated looking. Checking once is not monitoring, and checking naively is worse than not checking.
7. Alarm on the two axes separately, per the table above. They have different fixes.

Step 6 is the one that catches people. Classical calibration metrics “provide static calibration assessment but do not address sequential monitoring with false alarm control.” The naive alternative is actively misleading: “A practitioner who checks calibration daily with a p<0.05 threshold will, over a year of monitoring, almost certainly observe spurious alarms even if calibration remains stable” (Farran: When Your Model Stops Working²³).

All of that has to be legible per decision, which means the gate writes a row, not a boolean.

{
  "gate": "rm_is_safe",
  "model_requested": "jev-1.13.0",
  "model_returned": "jev-1.13.0",
  "noul": 0.93,
  "band": "review",
  "band_edges": {"act": 0.97, "escalate": 0.80},
  "band_set_id": "2026-09-14-traffic-sample-4k",
  "ground_truth": null
}

Every number in that record is illustrative: the probability is invented and the band edges are placeholders for whatever your own reliability curve produces, because no published source has a number for your traffic. The fields are the point: model_returned makes a moved alias a diff instead of an incident, band_set_id tells you which calibration run a given decision was made under, and ground_truth stays null until a human or a downstream outcome fills it in. That last field is what turns a log into the labelled sample step 2 asks for.

One more reason to treat calibration numbers as conditional: the only structured non-TypeSafe comparison that exists is a competitor’s model card, and it is unusually honest about its own limits. It reports a 0.081 expected calibration error against Jev’s 0.246, and then immediately discloses that “Laya achieves its 0.081 ECE after domain temperature fitting” and that “Before temperature scaling, the base checkpoint has higher raw ECE (0.213 vs 0.144)” (ConvAI Innovations: Laya model card²⁴). Before the fitting step, Jev is the better calibrated of the two. The card also states that “Jev figures are third-party published, never measured here (no TypeSafe API access); sample sizes and prompts differ.” A calibration number is a statement about a fitting procedure on a dataset, which is exactly why yours has to come from your data.

None of this is a complaint about the product. It is the deal, and one of the earliest critics named it precisely:

“At the end of the day, it delegates the hallucination problem a little bit to the user,” explained Ronacher. “The user has to say, okay, if this only comes back with 50% probability, maybe this is a coin toss, and I disregard it. But if it’s 95%, sure, then I can do something with it.”

Armin Ronacher, CTO of Earendil, is describing the transfer accurately. The model hands you a number and the judgment moves to your side of the boundary.

The Gate Fails on Coverage, Not on Schema

Audit what never reaches the model before you spend a day tuning what does. The only independent stress test of a shipped pre-execution gate found that more than a third of the dangerous actions were never evaluated at all, and no amount of threshold work touches that channel.

Ji et al. ran a 128-prompt adversarial DevOps benchmark against Claude Code’s auto mode and decomposed the failures by tier. The finding that matters for placement is not the headline error rate.

“Of the 253 actions, 93 (36.8%) were performed via Edit or Write tool calls, which are routed through Tier 2 and are not evaluated by the classifier” (Ji et al.: Measuring the Permission Gate²⁶). Every unsafe action in that row is a false negative by construction, because the gate never got a vote. The authors name the architectural assumption directly:

auto mode assumes dangerous actions transit the shell, but agents routinely achieve equivalent effects through file edits that the classifier does not evaluate.

Read the table honestly, though, because it does not say what the headline suggests. Coverage is 51 of 115 false negatives, the minority cause. Classifier misjudgment on actions it did evaluate accounts for 64, and “restricting to the 160 actions the classifier actually evaluates (Tier 3), the FNR is still 70.3%.” The authors are also explicit that their 81.0 percent end-to-end false-negative rate is not a rebuttal of the vendor’s number: “Neither rate is directly comparable to Anthropic’s reported 17% FNR on production traffic, which measures accidental overshoot on organic developer workloads.” This is one model, one threat category, four synthetic task families. It is a workload-sensitivity result, not a scandal.

Coverage is not the biggest failure channel; it is the invisible one. A misjudged Tier 3 action shows up in your gate’s own metrics as a decision that was made and was wrong. A Tier 2 action that routes around the gate produces no record anywhere, and your rm_is_safe dashboard stays green while the agent writes the same destruction into a file. The first question about a decision model is therefore not how accurate it is, but what fraction of the actions you care about it is even asked about. Vendor accuracy tables measure the numerator of a fraction you have not computed (why leaderboard movement is mostly noise).

Ask what the model can see relative to the actor it guards

Coverage has a second form, and it is measured in tokens rather than tool names. Zhou et al. identify “a critical blind spot arising from the mismatch between the limited inspection windows of guardrail models and the substantially larger context inference windows of downstream LLMs” (Zhou et al.: Prompt Overflow²⁷). Their measured case is stark: “most current guardrail models (e.g., Llama Prompt Guard 2) only operate with an effective context window of 512 tokens,” while “state-of-the-art LLMs support long input contexts (e.g., 400k tokens for GPT-5.1 model).” Against Prompt Guard 86M, “the bypass rate remains consistently above 99.5% across all tested densities (K=4…16).”

That population is lightweight binary prompt-injection detectors, not calibrated multi-class decision models, so none of it is a measurement of Jev and it would be dishonest to present it as one. Use it as the question to ask instead: Jev’s budget is 64k tokens per request with 32k for state plus the longest question (TypeSafe: Models⁴). What is the context window of the agent it is guarding, and what gets truncated out of state when the session runs long? If the answer is “whatever fits,” the gate is inspecting a summary of a situation the actor understands in full.

Stop treating schema conformance as a reliability property

Bad: “It cannot hallucinate, because the output is a typed probability.” The type system guarantees you receive a float between 0 and 1.

Good: “It cannot return malformed output, and it can absolutely return a confidently wrong number.” Design the band, the escalation path, and the coverage audit around the second sentence.

sean goedecke put the same objection more bluntly: “To me, this seems like a semantic dodge, since Jev can absolutely still pick the wrong choice (e.g. calling the sky “red”),” adding that “Still, all of this is also true about regular LLMs with structured outputs, and it doesn’t make Jev any more reliable in practice” (sean goedecke: Jev means structured output is interesting again²⁸). He is disputing the differentiation claim, not the product; he calls the speed a genuine advantage.

There is a formal version of this, and it cuts deeper than the marketing argument. Constraining a model’s output space does not merely fail to improve the answer. It can degrade which valid answer you get: constrained decoding techniques “can distort the LLM’s distribution, leading to outputs that are grammatical but appear with likelihoods that are not proportional to the ones given by the LLM, and so ultimately are low-quality” (Park et al.: Grammar-Aligned Decoding²⁹). Guaranteed-valid is not a synonym for correct; here the guarantee is bought with distortion. Which returns you to the vendor’s own concepts page, and its refusal to guarantee any individual answer.

Where the Decision Goes: A Placement Procedure

Author’s judgment. The ordering below is mine. Every gate in it rests on a sourced premise established above, but no source states the sequence, and the claim that running the gates out of order wastes the most effort is an inference from those premises rather than a published finding.

Run the placement in fixed order. Each gate is cheaper than the one after it, which is the only reason the order matters.

Can you state the action as a rule you would enforce every time? If yes, write an allow or a deny rule and stop. Per deterministic rules first, the model belongs on the residual, and a deny rule survives context compaction in a way a stated boundary does not.

Do you know what the decision costs today? If no, go price it before you price Jev. Per the tier already ships, your current answer is a ~60ms static pass, a 19.3ms small classifier, or roughly 25 percent inference overhead from an LLM filter, and the replacement cost is the delta against that, not against a frontier model.

Is what remains genuinely ambiguous? If no, the residual is still rules work and a decision model will just add a round trip to it. If yes, this is the slot, and per read the response shape what you are buying is a calibrated probability plus an integration shape unlike anything else in your stack.

Have you derived the band on your own labelled traffic, under a pinned version? If no, you do not have a threshold, you have someone else’s example value. Per two independent axes, pin the model, plot confidence against accuracy on your data, price the coverage you give up, and instrument version drift and traffic drift separately.

What fraction of the actions you care about does the gate actually see? If you cannot answer with a number, answer that first. Per coverage, not schema, 36.8 percent of state-changing actions in the one independent measurement never reached the classifier at all, and a gate that is never asked produces no evidence that it failed.

Answer those five for rm_is_safe and you end up somewhere concrete: deny rules on the paths you will never remove, narrow allow rules on the removals you can enumerate, auto-approval for the bounded, verifiable in-directory work, a decision model on the unresolvable targets with bands derived from your own labelled sample under a pinned version, and a coverage audit that counts the state-changing actions reaching the gate at all. That is a placement, not a verdict on Jev. The model is a component; the architecture is still yours to get right on the first try.