Guide

Comparing agent variants

Compare different prompts, models, or tool configurations of your agent side by side. Tag each run with a variant label and Kalmia groups and diffs them for you — no manual setup.

Tag each variant

A variant is a specific configuration of your agent — a prompt, model, or set of tools. Run the same task with each configuration and tag the run with a variant label in its metadata:

# Variant A
with traced(name="agent-run", metadata={"variant": "gpt4o-with-rag"}) as span:
span.log(output=run_agent(task))
# Variant B
with traced(name="agent-run", metadata={"variant": "claude-no-rag"}) as span:
span.log(output=run_agent(task))

Runs are compared by their variant label — there is no registration call to make.

Compare in the dashboard

Open the Metrics view. Kalmia groups traces by variant and shows a comparison table. One variant is the baseline, and every other column shows its change against it; pick a different baseline from the dropdown in the table's first column. The best and worst value in each row are highlighted. Rows include:

DurationAverage execution time per variant
Tokens · costAverage tokens and estimated $/run
Tool calls · turnsHow much work each run does
Success rateShare of runs without failures
Errors · retriesTool errors and retry counts

For graded correctness (not just operational metrics), run these as an eval and read the fixed / regressed scoreboard instead.

Compare by model

You do not have to spend the variant label on the model. If your variant already means something else, such as a customer, an agent, or a prompt version, use the compare by selector next to the comparison heading to regroup the same runs:

by variantThe default: one column per variant label
by primary modelOne column per model, across every variant
by projectOne column per project

When comparing by model, a second dropdown, within, narrows the comparison to one variant. That answers "for this one agent, which model does better?" without mixing in runs from your other variants. For example, choose by primary model within checkout-agent to see only that agent's runs, one column per model.

Nothing extra to send: the model is read from your traces. The primary model of a run is the trace-level model in its metadata when you set one, otherwise the model behind most of the run's LLM calls. A gateway prefix is ignored, so openai/gpt-4o and gpt-4o are one column; dated snapshots stay separate.

Runs that mix models

A run that uses several models is counted once, under its primary model. If you route by model on purpose, set model in the trace metadata so each run lands in the column you intend.

From your coding agent

The same comparison is available over the Kalmia MCP server as the compare_metrics tool. It takes group_by (variant, model, or project), an optional within_variant, and an optional since / until window, and returns run count, duration, tokens, turns, tool calls, error rate, and estimated cost per group. Ask your agent something like "compare models for checkout-agent over the last week".

To compare variants on issues and bad behavior rather than cost and speed, use compare_variants: failure and failed-verdict rates, how often each behavior label appears, and which issues are new or gone between versions. See Issues for the details.

Tips

  • Run several traces per variant so the aggregates are meaningful.
  • Use the same inputs across variants for a fair comparison.
  • Add more runs any time — metrics recompute on every view.

See also

Variant metrics explains what each number means, and Analytics covers production trends over time.