# Comparing agent variants

Compare different prompts, models, or tool configurations of your agent side by side.
Tag each run with a `variant` label and Kalmia groups and diffs them for you — no manual setup.

## Tag each variant

A **variant** is a specific configuration of your agent — a prompt, model, or set of tools.
Run the same task with each configuration and tag the run with a variant label in its metadata:

```python
# Variant A
with traced(name="agent-run", metadata={"variant": "gpt4o-with-rag"}) as span:
    span.log(output=run_agent(task))

# Variant B
with traced(name="agent-run", metadata={"variant": "claude-no-rag"}) as span:
    span.log(output=run_agent(task))
```

```typescript
await traced(async (span) => {
  span.log({ output: await runAgent(task) });
}, { name: "agent-run", metadata: { variant: "gpt4o-with-rag" } });

await traced(async (span) => {
  span.log({ output: await runAgent(task) });
}, { name: "agent-run", metadata: { variant: "claude-no-rag" } });
```

Runs are compared by their variant label — there is no registration call to make.

## Compare in the dashboard

Open the **Metrics** view.
Kalmia groups traces by variant and shows a comparison table.
One variant is the baseline, and every other column shows its change against it; pick a different baseline from the dropdown in the table's first column.
The best and worst value in each row are highlighted.
Rows include:

- **Duration**: Average execution time per variant
- **Tokens · cost**: Average tokens and estimated $/run
- **Tool calls · turns**: How much work each run does
- **Success rate**: Share of runs without failures
- **Errors · retries**: Tool errors and retry counts

For graded correctness (not just operational metrics), run these as an [eval](https://www.kalmia.dev/docs/evaluate.md#evals) and read the fixed / regressed scoreboard instead.

## Compare by model

You do not have to spend the variant label on the model.
If your variant already means something else, such as a customer, an agent, or a prompt version, use the **compare by** selector next to the comparison heading to regroup the same runs:

- **by variant**: The default: one column per variant label
- **by primary model**: One column per model, across every variant
- **by project**: One column per project

When comparing by model, a second dropdown, **within**, narrows the comparison to one variant.
That answers "for this one agent, which model does better?" without mixing in runs from your other variants.
For example, choose `by primary model` within `checkout-agent` to see only that agent's runs, one column per model.

Nothing extra to send: the model is read from your traces.
The **primary model** of a run is the trace-level `model` in its metadata when you set one, otherwise the model behind most of the run's LLM calls.
A gateway prefix is ignored, so `openai/gpt-4o` and `gpt-4o` are one column; dated snapshots stay separate.

> **Runs that mix models:** A run that uses several models is counted once, under its primary model. If you route by model on purpose, set `model` in the trace metadata so each run lands in the column you intend.

## From your coding agent

The same comparison is available over the [Kalmia MCP server](https://www.kalmia.dev/docs/api-reference.md#mcp) as the `compare_metrics` tool.
It takes `group_by` (`variant`, `model`, or `project`), an optional `within_variant`, and an optional `since` / `until` window, and returns run count, duration, tokens, turns, tool calls, error rate, and estimated cost per group.
Ask your agent something like "compare models for checkout-agent over the last week".

To compare variants on issues and bad behavior rather than cost and speed, use `compare_variants`: failure and failed-verdict rates, how often each behavior label appears, and which issues are new or gone between versions.
See [Issues](https://www.kalmia.dev/docs/issues.md) for the details.

## Tips

- Run several traces per variant so the aggregates are meaningful.
- Use the same inputs across variants for a fair comparison.
- Add more runs any time — metrics recompute on every view.

> **See also:** [Variant metrics](https://www.kalmia.dev/docs/guides/variant-metrics.md) explains what each number means, and [Analytics](https://www.kalmia.dev/docs/analytics.md) covers production trends over time.
