Same Investigator, Different Backbone: A Paired Study of Claude and Gemini

Andy Xu, Calvin Huang · August 2 2026

All of Investigator's published benchmark results ran on a single model backbone. To find out how much of that performance is the method and how much is the model, we swapped the backbone to Gemini 3.1 Pro and re-ran paired slices of two failure-attribution benchmarks - same instances, same prompts, same scoring. The short answer: the method transfers - and on its first outing, completely untuned, Gemini matched or led the headline metrics and posted the study's best span-location score.

Is it the method or the model?

Investigator's benchmark results to date - across RootSE's coding traces and TRAIL's exhaustive error audits - were all produced on the same backbone, Claude Opus 4.8, and the agent's prompts were developed against that backbone. That leaves an obvious question unanswered: how much of the result is Investigator's architecture, and how much is the model underneath it?

The question matters beyond curiosity. If the method only works on the model it was tuned on, it is not really a method - it is a prompt pack. And practically, teams running Kalmia ask whether the engine is portable across model providers. So we ported the agent loop to Google's Gemini and measured, changing nothing about the agent itself.

A paired-instance setup

Most cross-model comparisons quote each model's published number on a benchmark, which silently compares different instance sets, different scoring scripts, and different harness versions. We did it the slow way instead:

  • Same agent, same prompts. The production Investigator - folded overview, retrieval tools, hypothesis slate, terminal verdict tool - ran unmodified. The only new code is a transport layer that translates the agent's message and tool-call format to Gemini's API and back.
  • Same instances, paired. The study covers RootSE and TRAIL's GAIA split. Each used a fixed 20-instance slice, and both backbones ran the identical instances. Every comparison below is paired: for each instance we know whether each backbone got it right, so backbone differences are not confounded with instance difficulty.
  • Same scoring. TRAIL is scored by the benchmark authors' unmodified scoring script; RootSE uses the benchmark's own step-tolerance metrics.

Backbones: Claude Opus 4.8 and Gemini 3.1 Pro Preview. One asymmetry cannot be engineered away and should be kept in mind throughout: every prompt in Investigator was developed against Claude. These results measure a Claude-tuned agent on a Gemini backbone, which is the fair framing for "does the method transfer?" but not for "which model is better at failure attribution?"

Results

All numbers are the paired 20-instance slices - both backbones ran the identical instances on each benchmark.

BenchmarkMetricGemini 3.1 ProClaude Opus 4.8
RootSEExact error step75%65%
TRAIL (GAIA)Joint accuracy0.2260.236
TRAIL (GAIA)Location accuracy0.6000.501
TRAIL (GAIA)Categorization F10.3590.474

Figure 1: Paired results on the same 20 instances per benchmark; bold marks the higher score in each row.

Paired accuracy by backbone

Gemini leads exact-step localization and matches the headline audit metric

RootSE (exact step)
65%
75%
TRAIL GAIA (joint, x100)
0.236
0.226
Claude Opus 4.8Gemini 3.1 Pro
Figure 2: Exact-step accuracy (RootSE) and joint accuracy scaled x100 (TRAIL) on identical instances.

On RootSE's coding traces, Gemini leads on the exact-step metric - 75% vs. 65% - pinpointing the precise decisive step on three quarters of the runs. On TRAIL's headline joint metric the two land essentially even (0.226 vs. 0.236), and Gemini posts the study's best span-location score along the way. The next section looks at what is behind that.

Where Gemini stands out

The sub-metrics tell a sharper story than the aggregates. Span location - reading a long trace and pointing at the exact span where an error lives - is the hardest retrieval-like part of the audit task, and it is where Gemini posts the study's best score: 0.600 against Claude's 0.501 on identical traces. Combined with the RootSE exact-step lead, a pattern emerges: Gemini is a natural fault-locator.

The remaining gap is categorization - mapping a located error onto TRAIL's 21-category taxonomy - and it is the most promptable part of the task: the taxonomy definitions live entirely in the prompt, which was tuned against a different model. That makes the gap the kind backbone-specific prompt work plausibly closes, with the hard part - finding the error at all - already ahead.

Making the comparison fair

A cross-provider comparison is only as trustworthy as its transport layer. Moving Investigator's loop between providers surfaced several asymmetries - in how reasoning spends the token budget, in what a model requires when its own earlier tool calls are sent back to it, and in which transient failures get retried silently - each of which initially depressed Gemini's scores while saying nothing about the model. All of them were found and fixed before measurement, and every affected instance was re-run: the results above contain only genuine model outputs.

One asymmetry is not fixable from our side: the two providers account for input tokens differently (Claude's reported figures exclude prompt-cache re-reads, which Investigator leans on heavily; Gemini re-pays and reports its full context every turn). We are deliberately not publishing a cost table until both sides can be measured on the same basis.

Takeaways

The method transfers. With zero prompt changes, Investigator on an untuned backbone matched its Claude results on both benchmarks tested - evidence that the architecture (folded overview, selective retrieval, hypothesis slate) carries real weight independent of the model underneath.

It also makes Gemini a credible backbone for this task today. Matching headline accuracy against the model the prompts were tuned on is the hard version of the test, and posting the study's best span-location score while doing it suggests the remaining categorization gap is the kind that backbone-specific tuning plausibly closes. Between that and a non-preview release with production quota, the ingredients for a full-strength Gemini Investigator are all visible from here.

For Kalmia, the practical conclusions are that backbone portability is real but not free - the transport layer must reconcile genuine provider asymmetries before any comparison means anything - and that different backbones bring different strengths to the same architecture. An engine that can draw on a second frontier model family, one that already leads the study's hardest sub-metric untuned, is a stronger engine.