Evaluating Investigator Across Three Failure-Attribution Benchmarks

Andy Xu, Calvin Huang · July 22 2026

Our core mission is to build the infrastructure that allows agents to self-improve in production. The first step to achieving this is to identify where exactly an agent failed during its run. Today, we're sharing how Investigator, our agent powering Kalmia's root-cause analysis, performs across three failure-attribution benchmarks.

Finding the error in the trace

When an agent fails, it always leaves a trace: a lengthy thread of model calls, tool calls, and observations, where one wrong step can impact everything downstream. Currently, teams manually read all the trace contents to find which step actually caused the failure. Kalmia's approach is to build a specialized agent that can determine this step effectively, allowing every failed run to get an investigation and a root-cause verdict.

There is a growing body of research that analyzes failure attribution (finding the decisive error step) and error auditing (enumerating every error with a category). Existing methods are strong but narrow: each one is built and evaluated against a single benchmark, and thus results mix method quality with per-benchmark specialization. The TrajAudit[1] agent is designed specifically for the coding-agent benchmark RootSE. Who&When[2] methods specialize in multi-agent conversations. AgentCompass[3], the agent leading the TRAIL[4] benchmark, is built on a proprietary backbone specialized for error auditing, the exact task TRAIL measures.

In production, users send traces spanning a diverse range of agent tasks and behaviors, so we wanted to determine how well one method would generalize. We ran Investigator on all three benchmarks and tested it against each benchmark's strongest methods under matched conditions where their code is public (same model, inputs, and scoring code), and against published numbers where it is not.

Main results summary

We evaluated Investigator against each benchmark's best method, on each benchmark's own headline metric:

BenchmarkWhat it testsMetricOursBest alternative
RootSECoding-agent tracesExact error step57.8%51.0% (TrajAudit)
TRAIL (GAIA)Exhaustive error auditsJoint accuracy0.3200.239 (AgentCompass)
TRAIL (SWE-bench)Exhaustive error auditsJoint accuracy0.1310.051 (AgentCompass)
Who&WhenMulti-agent conversationsExact mistake step39.1%34.8% (binary-search)

Figure 1: Investigator's score on each benchmark's headline metric, next to the best alternative method. RootSE numbers are on Claude Sonnet 4.5, TrajAudit's own paper backbone (see Figure 5 for Claude Opus 4.8); TRAIL and Who&When are on Claude Opus 4.8.

RootSE and Who&When are matched-backbone runs of Investigator against each benchmark's respective methods, and TRAIL is Investigator against SOTA. We detail the full conditions in each section below.

How Investigator works

Investigator is an agent with four core pieces:

  • A folded overview of the complete trace. Every span is rendered into one line, with content folded behind a marker. We do not discard any information, so each span id remains citable and retrievable. As a result, costs scale with what Investigator reads rather than with trace length.
  • Retrieval tools. Investigator has a suite of retrieval tools allowing it to read any span in full, read a contiguous window of spans, or search the full content of the trace.
  • Symptom-first backward tracing. Investigator first determines what the issue is with the outcome, then walks dependencies backward to whichever step introduced the issue. Investigator separates the source of the fault from its propagation with an explicit test: would fixing this exact step change the outcome?
  • A mandatory hypothesis slate. Investigator must register multiple competing candidate steps before any deep investigation. It explores at least one step from each third of the run, investigates all of them, and determines a verdict only after the rival hypotheses have been eliminated with quoted evidence. Investigator will not blame a span when it has not read its full content.

Investigator's Workflow

A verdict is filed only after every rival hypothesis is eliminated

1 · The whole trace in view: 312 spans, folded to one line each. Symptom first: establish what is wrong with the outcome, then walk dependencies backward toward the step that introduced the fault.

walk dependencies backward

symptom: the outcome is wrong

2 · Before any deep dive: three rival candidates, one from each third. Then each is read in full and put to the same question - would doing this step correctly have averted the failure?

span 41 · early third

Read in full. Even with this step corrected, the failure still happens - not the cause.

Eliminated

span 187 · middle third

Read in full. The earliest step where doing it correctly averts the failure.

Verdict

span 296 · late third

Read in full. Wrong output, but only because it inherited span 187’s fault - propagation, not the source.

Eliminated
Figure 2: An example workflow of one investigation. The trace and span numbers are illustrative.

When Investigator tackles a new benchmark, only two thin adapters change: one converts the benchmark's trace format into spans, the other shapes the final verdict (TRAIL asks for a list of every error; RootSE and Who&When ask for a single decisive step). The agent itself never changes, so all three results measure one method rather than three tuned variants.

One-shot judge vs. Investigator

Here we compare Investigator to a one-shot judge: the model reading an agent trace in a single call and giving a verdict. Every method in the chart and table below is powered by Claude Opus 4.8. The value of our agent is shown by the performance gap between the judge and Investigator, and this gap only increases with trace length. On RootSE's lengthy coding traces, the judge scores 49.0% exact-step while Investigator scores 61.8%. On Who&When's short multi-agent conversations, Investigator improves the judge's accuracy from 27.2% to 39.1%. On TRAIL, the one-shot judge reaches 0.216 joint accuracy on GAIA and 0.007 on SWE-bench against Investigator's 0.320 and 0.131, with 27 of the 148 traces actually exceeding the 1M-token context window of the one-shot judge.

Accuracy, one-shot judge vs. Investigator

Investigator beats one-shot judge on benchmarks

RootSE (long coding traces)
49.0%
61.8%
Who&When (short conversations)
27.2%
39.1%
TRAIL GAIA (very long traces)
21.6%
32.0%
TRAIL SWE-bench (mostly over-window)
0.7%
13.1%
One-shot judgeInvestigator
Figure 3: Accuracy of a one-shot judge and Investigator on all three benchmarks. RootSE and Who&When show exact-step accuracy; TRAIL shows joint accuracy (find each error's span and category), with the 27 over-window traces scoring zero for the one-shot judge.

On long traces, Investigator also improves on the one-shot judge's token usage, because its usage grows far more slowly than trace size. It pays a fixed overhead for the folded overview and then reads only what the investigation needs, while the one-shot judge pays for the whole trace. So on Who&When's short conversations, the judge is cheaper (12.1k vs. 26.5k tokens per trace) and the agent's value there is accuracy alone. On RootSE, Investigator is more accurate while reading half the tokens (55k vs. 104k). On TRAIL, it reads about a third of what the judge reads.

BenchmarkOne-shot judge input/traceInvestigator input/trace
RootSE104k55k
Who&When12.1k26.5k
TRAIL GAIA172k61k
TRAIL SWE-bench498k154k

Figure 4: Mean input tokens per trace for the one-shot judge and Investigator on Claude Opus 4.8.

RootSE

RootSE is 102 failed runs of three coding agents (SWE-agent, OpenHands, AutoCodeRover) on SWE-bench tasks, each annotated with the earliest decisive error step. We compared Investigator against TrajAudit, the method built by RootSE's own authors, running their unmodified code on the same backbone their paper uses (Claude Sonnet 4.5) as well as on Claude Opus 4.8.

RootSE exact-step accuracy

Investigator beats TrajAudit on its own benchmark

Claude Sonnet 4.5 (their paper)
51.0%
57.8%
Claude Opus 4.8
59.8%
61.8%
TrajAuditInvestigator
Figure 5: Exact-step accuracy on RootSE for Investigator and TrajAudit.

TRAIL

TRAIL contains 148 OpenTelemetry traces carrying 841 annotated errors across a 21-category taxonomy. Instead of one decisive error step, TRAIL scores exhaustive recovery: for every error, did the agent find both its span and category? Its headline metric, joint accuracy, counts an error as recovered only when both are right; location accuracy requires just the span, and categorization F1 scores which error types were detected at all. We evaluated Investigator on both the GAIA and SWE-bench splits of TRAIL, run on Claude Opus 4.8 and scored with TRAIL's own scoring code. AgentCompass's published results were achieved with its own fine-tuned proprietary backbone, so this is the one comparison that cannot be run on a shared model.

TRAIL results

Investigator leads on five of six metric-split cells

Joint accuracy, GAIA
0.239
0.320
Joint accuracy, SWE-bench
0.051
0.131
Location, GAIA
0.657
0.587
Location, SWE-bench
0.250
0.358
Categorization F1, GAIA
0.309
0.483
Categorization F1, SWE-bench
0.232
0.511
AgentCompass (SOTA)Investigator
Figure 6: TRAIL joint accuracy (the headline metric), span-location accuracy, and categorization F1 on both splits, for Investigator and the published state of the art.

Who&When

Who&When contains 184 failed multi-agent runs annotated with the responsible agent and the mistake step. With Who&When's three methods ported verbatim and every method run on the same backbone (Claude Opus 4.8), Investigator outperforms all of them on step localization:

MethodStep exactWithin ±3Agent
Investigator (ours)39.1%70.7%57.6%
Binary-search (paper)34.8%66.8%57.1%
Step-by-step (paper)32.6%64.1%54.9%
All-at-once (paper)27.2%62.5%67.9%

Figure 7: Step and agent attribution on Who&When for Investigator and the paper's three methods.

Who&When also scores another question: which agent made the mistake, regardless of step. Investigator reports the answer indirectly by naming which agent was acting at the step it blames, so even a near-miss on the step can result in naming the wrong agent. Other methods that skip the step and directly predict the agent perform better on this task.

One method that generalizes

One of the primary concerns regarding any benchmark result is specialization. A method tuned and optimized until it wins the benchmark offers zero evidence it would work anywhere else. Running Investigator across three benchmarks is our attempt to audit how generalizable its performance is. The three tasks are diverse, and yet Investigator leads on all three. We believe this reflects Investigator's ability to capture something general about diagnosing agent failures, not just something specific to a single benchmark's format or annotators.

Users send traces that don't resemble any benchmark. Different agents display different behaviors within their traces, and a method designed only to win on the distribution it was tuned for would not perform well in production. We have thoughtfully built Investigator to perform across this diversity, ensuring that the engine within Kalmia ingesting and analyzing traces will generalize beyond any single benchmark.

References

  • [1] Wang, Xie, and Huo (2026). TrajAudit: Automated Failure Diagnosis for Agentic Coding Systems (introduces the RootSE benchmark). arXiv:2605.26563
  • [2] Zhang et al. (2025). Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems (introduces the Who&When benchmark). ICML 2025. arXiv:2505.00212
  • [3] Kartik et al. (2025). AgentCompass: Towards Reliable Evaluation of Agentic Workflows in Production. arXiv:2509.14647
  • [4] Deshpande et al. (2025). TRAIL: Trace Reasoning and Agentic Issue Localization. arXiv:2505.08638