Guide

Understanding variant metrics

A glossary of the metrics Kalmia computes for every run, and how to read them when comparing variants.

Core metrics

Computed for every trace and aggregated per variant.

Duration

Wall-clock time from the first span to the last — end-to-end latency of the run.

Total tokens

Sum of prompt and completion tokens across every LLM call in the trace.

Input / output tokens

Prompt vs. completion tokens. High input often means large context or long history; high output often means verbose responses.

Turns

Number of LLM conversation turns — each assistant response is one turn.

Cost

Estimated USD from token counts at list prices (see Reading cost).

Agent metrics

Extra signals for tool-using and coding agents.

Tool callsTotal tool invocations, broken down by name in the trace
Tool errorsTool calls that returned an error
RetriesRepeated attempts at the same operation — a sign the agent is stuck
BacktracksTimes the agent undid a previous action
Files read / writtenUnique files touched during the run
Subagent spawnsTimes work was delegated to a sub-agent

Failure tags

Traces are automatically tagged with failure patterns during preprocessing, so you can surface problems without reading every run:

permission_deniedPermission or access errors
file_not_foundReferenced files that don't exist
high_token_usageToken count above the normal range
excessive_retriesRetried the same operation too many times

Reading deltas

In the comparison table, one column is the baseline and every other column shows its change against it; choose the baseline from the dropdown in the first column. The best and worst value in each row are highlighted. Read the direction with the metric: lower duration, tokens, errors, and retries are better; higher success rate is better. The same table can group runs by model or project instead of variant; see Compare by model.

Watch the combinations

High retries plus high token usage is a strong signal a variant needs work. Prefer the median over the average for latency and tokens — outliers skew averages.