Evaluating Investigator Across Three Failure-Attribution Benchmarks
Andy Xu, Calvin Huang · July 22 2026
Our core mission is to build the infrastructure that allows agents to self-improve in production. The first step to achieving this is to identify where exactly an agent failed during its run. Today, we're sharing how Investigator, our agent powering Kalmia's root-cause analysis, performs across three failure-attribution benchmarks.
Finding the error in the trace
When an agent fails, it always leaves a trace: a lengthy thread of model calls, tool calls, and observations, where one wrong step can impact everything downstream. Currently, teams manually read all the trace contents to find which step actually caused the failure. Kalmia's approach is to build a specialized agent that can determine this step effectively, allowing every failed run to get an investigation and a root-cause verdict.
There is a growing body of research that analyzes failure attribution (finding the decisive error step) and error auditing (enumerating every error with a category). Existing methods are strong but narrow: each one is built and evaluated against a single benchmark, and thus results mix method quality with per-benchmark specialization. The TrajAudit[1] agent is designed specifically for the coding-agent benchmark RootSE. Who&When[2] methods specialize in multi-agent conversations. AgentCompass[3], the agent leading the TRAIL[4] benchmark, is built on a proprietary backbone specialized for error auditing, the exact task TRAIL measures.
In production, users send traces spanning a diverse range of agent tasks and behaviors, so we wanted to determine how well one method would generalize. We ran Investigator on all three benchmarks and tested it against each benchmark's strongest methods under matched conditions where their code is public (same model, inputs, and scoring code), and against published numbers where it is not.
Main results summary
We evaluated Investigator against each benchmark's best method, on each benchmark's own headline metric:
| Benchmark | What it tests | Metric | Ours | Best alternative |
|---|---|---|---|---|
| RootSE | Coding-agent traces | Exact error step | 57.8% | 51.0% (TrajAudit) |
| TRAIL (GAIA) | Exhaustive error audits | Joint accuracy | 0.320 | 0.239 (AgentCompass) |
| TRAIL (SWE-bench) | Exhaustive error audits | Joint accuracy | 0.131 | 0.051 (AgentCompass) |
| Who&When | Multi-agent conversations | Exact mistake step | 39.1% | 34.8% (binary-search) |
Figure 1: Investigator's score on each benchmark's headline metric, next to the best alternative method. RootSE numbers are on Claude Sonnet 4.5, TrajAudit's own paper backbone (see Figure 5 for Claude Opus 4.8); TRAIL and Who&When are on Claude Opus 4.8.
RootSE and Who&When are matched-backbone runs of Investigator against each benchmark's respective methods, and TRAIL is Investigator against SOTA. We detail the full conditions in each section below.
How Investigator works
Investigator is an agent with four core pieces:
- A folded overview of the complete trace. Every span is rendered into one line, with content folded behind a marker. We do not discard any information, so each span id remains citable and retrievable. As a result, costs scale with what Investigator reads rather than with trace length.
- Retrieval tools. Investigator has a suite of retrieval tools allowing it to read any span in full, read a contiguous window of spans, or search the full content of the trace.
- Symptom-first backward tracing. Investigator first determines what the issue is with the outcome, then walks dependencies backward to whichever step introduced the issue. Investigator separates the source of the fault from its propagation with an explicit test: would fixing this exact step change the outcome?
- A mandatory hypothesis slate. Investigator must register multiple competing candidate steps before any deep investigation. It explores at least one step from each third of the run, investigates all of them, and determines a verdict only after the rival hypotheses have been eliminated with quoted evidence. Investigator will not blame a span when it has not read its full content.
Investigator's Workflow
A verdict is filed only after every rival hypothesis is eliminated
1 · The whole trace in view: 312 spans, folded to one line each. Symptom first: establish what is wrong with the outcome, then walk dependencies backward toward the step that introduced the fault.
symptom: the outcome is wrong
2 · Before any deep dive: three rival candidates, one from each third. Then each is read in full and put to the same question - would doing this step correctly have averted the failure?
span 41 · early third
Read in full. Even with this step corrected, the failure still happens - not the cause.
Eliminatedspan 187 · middle third
Read in full. The earliest step where doing it correctly averts the failure.
Verdictspan 296 · late third
Read in full. Wrong output, but only because it inherited span 187’s fault - propagation, not the source.
EliminatedWhen Investigator tackles a new benchmark, only two thin adapters change: one converts the benchmark's trace format into spans, the other shapes the final verdict (TRAIL asks for a list of every error; RootSE and Who&When ask for a single decisive step). The agent itself never changes, so all three results measure one method rather than three tuned variants.
One-shot judge vs. Investigator
Here we compare Investigator to a one-shot judge: the model reading an agent trace in a single call and giving a verdict. Every method in the chart and table below is powered by Claude Opus 4.8. The value of our agent is shown by the performance gap between the judge and Investigator, and this gap only increases with trace length. On RootSE's lengthy coding traces, the judge scores 49.0% exact-step while Investigator scores 61.8%. On Who&When's short multi-agent conversations, Investigator improves the judge's accuracy from 27.2% to 39.1%. On TRAIL, the one-shot judge reaches 0.216 joint accuracy on GAIA and 0.007 on SWE-bench against Investigator's 0.320 and 0.131, with 27 of the 148 traces actually exceeding the 1M-token context window of the one-shot judge.
Accuracy, one-shot judge vs. Investigator
Investigator beats one-shot judge on benchmarks
On long traces, Investigator also improves on the one-shot judge's token usage, because its usage grows far more slowly than trace size. It pays a fixed overhead for the folded overview and then reads only what the investigation needs, while the one-shot judge pays for the whole trace. So on Who&When's short conversations, the judge is cheaper (12.1k vs. 26.5k tokens per trace) and the agent's value there is accuracy alone. On RootSE, Investigator is more accurate while reading half the tokens (55k vs. 104k). On TRAIL, it reads about a third of what the judge reads.
| Benchmark | One-shot judge input/trace | Investigator input/trace |
|---|---|---|
| RootSE | 104k | 55k |
| Who&When | 12.1k | 26.5k |
| TRAIL GAIA | 172k | 61k |
| TRAIL SWE-bench | 498k | 154k |
Figure 4: Mean input tokens per trace for the one-shot judge and Investigator on Claude Opus 4.8.
RootSE
RootSE is 102 failed runs of three coding agents (SWE-agent, OpenHands, AutoCodeRover) on SWE-bench tasks, each annotated with the earliest decisive error step. We compared Investigator against TrajAudit, the method built by RootSE's own authors, running their unmodified code on the same backbone their paper uses (Claude Sonnet 4.5) as well as on Claude Opus 4.8.
RootSE exact-step accuracy
Investigator beats TrajAudit on its own benchmark
TRAIL
TRAIL contains 148 OpenTelemetry traces carrying 841 annotated errors across a 21-category taxonomy. Instead of one decisive error step, TRAIL scores exhaustive recovery: for every error, did the agent find both its span and category? Its headline metric, joint accuracy, counts an error as recovered only when both are right; location accuracy requires just the span, and categorization F1 scores which error types were detected at all. We evaluated Investigator on both the GAIA and SWE-bench splits of TRAIL, run on Claude Opus 4.8 and scored with TRAIL's own scoring code. AgentCompass's published results were achieved with its own fine-tuned proprietary backbone, so this is the one comparison that cannot be run on a shared model.
TRAIL results
Investigator leads on five of six metric-split cells
Who&When
Who&When contains 184 failed multi-agent runs annotated with the responsible agent and the mistake step. With Who&When's three methods ported verbatim and every method run on the same backbone (Claude Opus 4.8), Investigator outperforms all of them on step localization:
| Method | Step exact | Within ±3 | Agent |
|---|---|---|---|
| Investigator (ours) | 39.1% | 70.7% | 57.6% |
| Binary-search (paper) | 34.8% | 66.8% | 57.1% |
| Step-by-step (paper) | 32.6% | 64.1% | 54.9% |
| All-at-once (paper) | 27.2% | 62.5% | 67.9% |
Figure 7: Step and agent attribution on Who&When for Investigator and the paper's three methods.
Who&When also scores another question: which agent made the mistake, regardless of step. Investigator reports the answer indirectly by naming which agent was acting at the step it blames, so even a near-miss on the step can result in naming the wrong agent. Other methods that skip the step and directly predict the agent perform better on this task.
One method that generalizes
One of the primary concerns regarding any benchmark result is specialization. A method tuned and optimized until it wins the benchmark offers zero evidence it would work anywhere else. Running Investigator across three benchmarks is our attempt to audit how generalizable its performance is. The three tasks are diverse, and yet Investigator leads on all three. We believe this reflects Investigator's ability to capture something general about diagnosing agent failures, not just something specific to a single benchmark's format or annotators.
Users send traces that don't resemble any benchmark. Different agents display different behaviors within their traces, and a method designed only to win on the distribution it was tuned for would not perform well in production. We have thoughtfully built Investigator to perform across this diversity, ensuring that the engine within Kalmia ingesting and analyzing traces will generalize beyond any single benchmark.
References
- [1] Wang, Xie, and Huo (2026). TrajAudit: Automated Failure Diagnosis for Agentic Coding Systems (introduces the RootSE benchmark). arXiv:2605.26563
- [2] Zhang et al. (2025). Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems (introduces the Who&When benchmark). ICML 2025. arXiv:2505.00212
- [3] Kartik et al. (2025). AgentCompass: Towards Reliable Evaluation of Agentic Workflows in Production. arXiv:2509.14647
- [4] Deshpande et al. (2025). TRAIL: Trace Reasoning and Agentic Issue Localization. arXiv:2505.08638