Rank Your Agent Trace Fields by What You Can Never Get Back

Anthropic’s managed-agents team had a live event stream from every session and still could not separate a harness bug from a dropped packet from a dead container, because all three presented the same (Anthropic Engineering: Scaling Managed Agents¹). No dashboard fixes that after the fact. The fields that would have told those three apart are decided at call time, written once, and the published conventions rank the model alias you sent above the model snapshot that actually answered you.

A Conformant Trace Proves Failure Without Locating It

Two weeks after a refund-batch job started producing wrong totals, the only surviving record of the model call at the center of it is a span that looks like this.

{
  "trace_id": "refund-batch-0e91",
  "span_id": "7f2a",
  "name": "chat claude-opus-4-1",
  "attributes": {
    "gen_ai.operation.name": "chat",
    "gen_ai.provider.name": "anthropic",
    "gen_ai.request.model": "claude-opus-4-1",
    "gen_ai.usage.input_tokens": 41822,
    "gen_ai.usage.output_tokens": 1104
  },
  "status": "OK",
  "duration_ms": 8431
}

That span is close to everything the OpenTelemetry GenAI conventions actually compel. gen_ai.operation.name and gen_ai.provider.name are Required, gen_ai.request.model is Conditionally Required if available, and the three attributes carrying why the model did anything, gen_ai.system_instructions, gen_ai.input.messages and gen_ai.output.messages, all sit at Opt-In (OpenTelemetry: GenAI client spans²). Span 7f2a is conformant. It is also close to useless for finding the step that broke.

That is not a rhetorical claim, it has been measured. A two-author preprint from East China Normal University built a deterministic synthetic corpus of 312 traces across ticketing, document and order domains, rendered each trace through several field views, then asked five models both to detect that a run failed and to name the step where the failure originated (Zhu and Pu: TelemetrySuffBench³).

“Metadata, OpenTelemetry-compatible, and OpenInference-compatible views retain 99.5% to 100% detection F1 while limiting origin-step accuracy to at most 0.5%”

Read that with the caveats the paper itself supplies. Those numbers are joint across all three restricted views. The Full view also detects failures at between 99.3% and 100.0% F1, so near-perfect detection is a property of this corpus rather than something the standards-shaped views uniquely preserve. The compatibility renderers are the authors’ own: “For the canonical fields available in this corpus, the two compatibility renderers use the same conservative generic field set.” So the study measured traces shaped like the published conventions. It did not test the OpenTelemetry or OpenInference registries themselves, and it cannot rank one convention against another. What it does isolate is which class of field carries the localization signal, and the ablation is blunt: removing decision content “reduces origin-step accuracy to zero for all five models.”

The question a trace has to answer is therefore not whether the run failed. It is which step made it fail, and that is the one question the identity-and-timing fields in span 7f2a cannot touch. A survey of execution provenance puts the same point in reader-facing terms: final-answer accuracy alone “cannot explain how an output was produced, which evidence supported each claim, whether tool calls were justified, how memory influenced later decisions, or where failures originated” (Wang et al.: From Agent Traces to Trust⁴). A single-author preprint frames the structural version, that a trace “records which steps executed and in what order, never what each step relied on” (Zhao: Grade⁵); treat it as an argued design position rather than a validated result, since it is un-peer-reviewed, but the gap it names matches what TelemetrySuffBench measured.

Pair the Synthetic Result With a Natural-Trace One

One deterministic synthetic corpus should not carry an argument this expensive on its own. The peer-reviewed leg comes from ACL 2026, where a failure-attribution benchmark built on natural multi-agent traces reports that “full traces improve attribution accuracy by up to 76.5% over a partial-observation counterpart, confirming that missing inputs obscure many failure causes” (Chen et al.: Seeing the Whole Elephant⁶). Report that 76.5% as what it is: a relative improvement over a partial-observation baseline, not an absolute accuracy. The same paper shows that the choice of analyzer moves step-level accuracy substantially on identical full traces, so completeness is a precondition for attribution rather than a substitute for good analysis.

Record the Snapshot, Not the Alias

Write the resolved model snapshot into the span at call time, because the string you sent is the one field you could always have reconstructed and the string that answered you is the one that expires. On the generic GenAI spans document, gen_ai.request.model is Conditionally Required with example gpt-4, while gen_ai.response.model is only Recommended with example gpt-4-0613 (OpenTelemetry: GenAI client spans²). On the OpenAI client operations document the same split is sharper: gen_ai.request.model is flatly Required, and gen_ai.response.model stays Recommended (OpenTelemetry: OpenAI client operations⁷).

Two honest qualifications before that becomes an accusation. gen_ai.operation.name is Required too, so request-side data is not uniquely privileged in these documents. And the gen_ai.* attributes carry Development status rather than Stable, so this is a moving target by design. The ordering still holds where it counts: the alias sits in your own source control forever, and the snapshot stops serving on a published clock.

Anthropic retired claude-opus-4-1-20250805 on August 5, 2026, two months after deprecating it, and retirement is absolute: “Retired: The model is no longer available for use. Requests to retired models will fail” (Anthropic: Model deprecations⁸). OpenAI publishes the same mechanic on a different clock (OpenAI: Deprecations⁹).

Anthropic’s published dates cover Anthropic-operated platforms only, since Amazon Bedrock and Google Cloud set their own schedules. And there is a genuine escape hatch worth naming: “In some cases, developers may be able to provision dedicated capacity for continued access after a model’s shutdown date.” It is sales-gated and case by case, so design the trace around the default rather than the exception.

Bad: the span records the string you typed into your client and nothing else about model identity.

"gen_ai.request.model": "claude-opus-4-1"

Good: the span records what you sent, what answered, which prompt was live, and when the thing that answered stops existing.

"gen_ai.request.model": "claude-opus-4-1",
"gen_ai.response.model": "claude-opus-4-1-20250805",
"agent.prompt.version": "14",
"model.shutdown_date": "2026-08-05"

Only the first two of those four are named in a published convention. model.shutdown_date is my own attribute name, though its value is machine-readable on OpenAI’s Model object, “The date when the model will shut down, or null if not announced” (OpenAI: Models¹⁰); on Anthropic you copy the retirement date from the deprecations page. On OpenAI clients, set a fifth: openai.response.system_fingerprint, which the conventions mark Recommended and describe as “A fingerprint to track any eventual change in the Generative AI environment.”

The prompt-version half has the same shape. Anthropic’s own managed-agents tutorial is direct about what happens when you skip it: “If callers are passing the bare agent ID instead of a pinned version, they’ll start using the new prompt on their very next session,” and “There’s no built-in approval workflow on agents.update. Any key in the workspace can call it” (Nowicki: Managed Agents prompt versioning and rollback¹¹). A hosted prompt pointer is a footgun for a second reason: pointers expire. OpenAI states that “Prompt creation will be de-emphasized beginning June 3, 2026, and v1/prompts is scheduled to shut down on November 30, 2026,” and its own migration advice is to “move the prompt content out of the managed prompt object and into your application code” (OpenAI: Migrate from prompt objects¹²). Record the resolved content or a hash of it, not a reference to a service with a scheduled end date.

This is the same move I argued for spend in You Can’t Cap What You Can’t Attribute: commit the dimensions on every call, as a schema, before you ship. Forensics has the harder deadline, because an invoice can be recomputed and a retired snapshot cannot.

Pinning the ID Is Not Enough

Recording the snapshot buys you identity, not reproduction. Anthropic states the limit plainly: “Model weights are fixed for a given ID, but the serving infrastructure around the model can change over time. This infrastructure includes components such as the request router, safety classifiers, and sampling logic,” and “Occasionally, infrastructure updates produce minor differences in observable behavior even when the model ID and weights have not changed” (Anthropic: Model IDs and versioning¹³). So the pinned ID in span 7f2a identifies the weights, not the router or the classifier stack that shaped the response. Capture whatever behavior-affecting infrastructure identifiers the vendor exposes, and treat a matching model ID as weak evidence of a matching environment rather than proof of one.

Scope the Alias Claim to the Right Generation

The alias problem is real, but it is generation-scoped, and overclaiming it costs you the argument. Anthropic is explicit that from the 4.6 generation onward, “A 4.6-generation ID such as claude-sonnet-4-6 is not an alias. It is the snapshot.” The floating behavior belongs to earlier models, where “An alias such as claude-sonnet-4-5 is a convenience pointer that resolves to the most recent dated snapshot for that minor version” (Anthropic: Model IDs and versioning¹³). Span 7f2a carries claude-opus-4-1, a pre-4.6 ID, so its request model genuinely is a pointer and the dated snapshot behind it is the thing that answered. On a 4.6-or-later ID the request and response strings converge and this specific gap closes. Every model ID, dated or dateless, still has its own retirement schedule, so the clock does not close with it.

The Published Menus Disagree About Requiredness

Decide requiredness yourself and pin it in your own schema, because the published menus assign sharply different obligations to the same field. OpenInference has no Required, Recommended or Opt-In tiering at all, and exactly one attribute is required across all spans: “The openinference.span.kind attribute is required for all OpenInference spans” (Arize AI: OpenInference Semantic Conventions¹⁴). The OWASP Agent Observability Standard runs the other way on a different axis, requiring the agent’s own instructions and version in every valid Agent object while leaving model and tools out of the required set entirely, and its Model object requires only id, name and provider (OWASP AOS: Schema¹⁵).

Row four is the AOS case stated fairly. AOS does track model version, in the AgBOM inventory, which updates whenever a model is “discovered, removed or changed capabilities,” because “AgBOM must dynamically adapt to reflect the rapid iteration and evolution of agent architectures” (OWASP AOS: Inspect with AgBOM¹⁶). That is a good design for knowing what is deployed now, and the wrong instrument for knowing what served a session in June, precisely because it adapts.

Three scoping notes so this stays fair. AOS required is a JSON entity-validity constraint and OpenTelemetry’s Opt-In is a span-capture policy, so these menus assign sharply different requiredness rather than contradicting one another. The AOS schema is served from a moving dev branch rather than a tagged release, so it can change under this claim. And OpenInference does publish llm.prompt_template.version, “The version of the prompt template,” so prompt versioning is not missing from the menus: it is present, unranked, and easy to leave unset. On the neighboring tool-argument move, where gen_ai.tool.name is Required and the call arguments are not, I made that case in Audit Your Agent Harness and will not re-derive it here.

The Specs Meet Unrecoverability and Never Name It

Read the tiering rationale rather than the tiers. OpenTelemetry’s requirement-level document sets its criteria as “attribute availability across instrumented entities, performance, security, and other factors,” and defines the bottom tier as “recommended for attributes that are particularly expensive to retrieve or might pose a security or privacy risk” (OpenTelemetry: Attribute requirement levels¹⁷). Its worked demotion example is the sharpest illustration available: “reading response stream to find http.response.body.size when Content-Length header is not available.” That value exists only in flight and is gone the moment the stream closes, and the document classifies it purely as an expensive read under a performance heading.

Do not stretch that into a claim about what the specs never do. The criteria list is left open twice, cardinality and security are named as genuine criteria elsewhere on the page, and the document is signal-general with zero GenAI or agent content anywhere in it. The defensible statement is narrower and still useful: the tiering is never framed as recoverability, so nothing in the rationale distinguishes a field you can re-derive from a field that dies with the call. Nor is this a complaint that nobody ranks anything. Arize’s agent observability guide ranks explicitly, telling you to “Observe the outcome it produced, the path it followed, the actions it attempted, and the context that informed its decisions. These are the core observation surfaces, not an exhaustive checklist” (Arize AI: Agent observability¹⁸). That is a real ranking on a diagnostic axis, meaning what tells you fastest that something broke. Survivability is a different axis, and it is the one nobody publishes.

Retention Deletes the Session Before You Need It

Check your retention ceilings, per-trace caps and flush behavior this week, because every one of them discards sessions on a schedule set before anyone knows which session mattered. Span 7f2a can be perfectly instrumented and still not be there when you go looking for it.

• Read your platform’s retention ceiling and count backwards from your incident review cycle. LangSmith’s SaaS tier “retains trace data for 180 days from ingestion. After that, traces are permanently deleted, with limited metadata retained for usage statistics” (LangChain: Observability concepts¹⁹).
• Check the per-trace run cap against your longest agent session. “Each trace is limited to a maximum of 25,000 runs. Once the trace reaches this limit, LangSmith will reject any additional runs that you send for that trace.” It is the tail that gets refused, and the tail is where a compounding failure usually lands.
• Call flush() before any short-lived process exits. Langfuse is explicit that “To avoid losing data, short-lived applications must explicitly call flush() before exiting” (Langfuse: Core Concepts²⁰). That is the CI agent, the cron job and the one-shot task, which between them cover most of the agent work people actually run in production.
• Confirm you are storing your own copy, because the provider is not holding one for you. Default application-state retention on OpenAI’s Responses and Chat Completions endpoints is listed as “None, see below for exceptions,” and “When Zero Data Retention is enabled for an organization, the store parameter will always be treated as false, even if the request attempts to set the value to true” (OpenAI: Data controls²¹). Abuse-monitoring logs are retained up to 30 days, and the page describes no customer-facing way to query them.
• Re-verify export after you customize your framework’s tracing. “Tracing is unavailable for organizations that use OpenAI’s APIs under a Zero Data Retention (ZDR) policy,” and replacing the default processors via set_trace_processors() means “traces will not be sent to the OpenAI backend unless you include a TracingProcessor that does so” (OpenAI Agents SDK: Tracing²²).

None of these are bugs. Every one is a documented default doing exactly what it says on the page. That is what makes them expensive: a retention default has a blast radius measured in sessions you have not run yet, and you find out its size on the day you need one of them.

Answer the Tail-Sampling Objection

The obvious response to all of this is to sample on the tail and keep the interesting traces. OpenTelemetry does prescribe tail sampling as the remedy to head sampling’s core limit, noting that you cannot ensure error-bearing traces survive with head sampling alone and then, in the very next sentence, “For this situation and many others, you need tail sampling” (OpenTelemetry: Sampling²³). The problem is what the tail-sampling processor’s defaults commit you to.

processors:
  tail_sampling:
    decision_wait: 30s      # default
    num_traces: 50000       # default
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }

Both of those values are the processor’s own documented defaults (OpenTelemetry: Tail Sampling Processor²⁴). A 30-second decision window against an agent session that runs for minutes or hours means the keep-or-drop call is still made on an incomplete trace. That is the same pre-commitment, moved thirty seconds later. The num_traces ceiling is harder: “If the collector is processing more traces in-memory than the num_traces configuration option allows, some will have to be dropped before they can be sampled,” so overflow is discarded before any policy evaluates it. There is also a topology requirement that quietly rules out naive horizontal scaling, since “All spans for a given trace MUST be received by the same collector instance for effective sampling decisions.”

Then there is the volume mismatch. OpenTelemetry’s own guidance says to sample when “You generate 1000 or more traces per second” and not to when “You generate very little data (tens of small traces per second or lower).” Most agent fleets sit in the second bucket, which means sampling gets reached for outside the range its own maintainers scope it to. The thing deleting your session is retention policy and export configuration, not the sampler.

Compliance Fixes Duration, Never Field Selection

Do not outsource the field list to the regulator, because the statutory floor is a duration and a purpose, not a schema. The EU AI Act says logs must exist, says how long a provider keeps them, and names specific fields exactly once, inside a carve-out (Regulation (EU) 2024/1689²⁵).

No other article in the operative text says which fields a log must contain.

So the binding language is real, the “shall” is not advisory, and it still leaves the entire question of this post open. Practitioners are feeling the consequence directly: one survey of public incidents counts “10 documented incidents across 6 AI coding tools in 16 months. Missing audit trails, no liability frameworks, no vendor postmortems” (Foley: Ten AI Agents Destroyed Production. Zero Postmortems.²⁶). That is a practitioner’s tally of public reports rather than a peer-reviewed dataset, but it names the right hole. A six-month floor on an empty log is six months of an empty log.

Rank Every Trace Field by What Survives

Author’s judgment. No source ranks published trace-menu fields by unrecoverability, and the ranking below is my synthesis rather than anyone’s specification. It follows from four sourced premises: the requirement-level inversion in the OpenTelemetry GenAI conventions, the vendors’ published retirement clocks, the vendors’ documented retention ceilings, and the measured gap between detecting a failure and locating it.

Rank your fields by unrecoverability before you ship, and capture the top of that list unconditionally rather than at whatever tier a convention assigned it. Concretely, on the next agent you instrument:

1. Set gen_ai.system_instructions, gen_ai.input.messages and gen_ai.output.messages explicitly, taking the privacy and cost tradeoff deliberately instead of inheriting the Opt-In default.
2. Set gen_ai.response.model on every call, and treat a span carrying only gen_ai.request.model as a defective span.
3. Record the resolved prompt content or its hash, plus the pinned version, rather than a hosted prompt pointer.
4. Copy the retirement date or shutdown_date value into the span at call time.
5. Assert all of the above in CI, so a missing survivability field fails the build rather than the postmortem.

Instrumentation is one of the few things in an agent system you have to get right on the first try, because there is no second call to the same snapshot. Re-running is not a fallback, and the vendors say so themselves. Anthropic notes that “even with temperature of 0.0, the results will not be fully deterministic” (Anthropic: Messages²⁷) and extends that “both to Anthropic’s first-party inference service and to inference through third-party cloud providers” (Anthropic: Glossary²⁸), which closes the “we run on a cloud provider so we can replay it” objection. OpenAI’s seed parameter is still Beta: “Determinism is not guaranteed, and you should refer to the system_fingerprint response parameter to monitor changes in the backend” (OpenAI: Create chat completion²⁹). Anthropic’s own operators put it in one line: “Agents make dynamic decisions and are non-deterministic between runs, even with identical prompts” (Anthropic Engineering: Multi-agent research system³⁰).

Which produces the ranking, one row per section above:

Every row above is a decision made at call time by whoever wrote the instrumentation, usually months before the incident that needs it. That is the same class of decision as tiering an agent’s production authority before it has any authority to misuse: cheap to make deliberately in advance, impossible to make retroactively.