Understanding variant metrics
A glossary of the metrics Kalmia computes for every run, and how to read them when comparing variants.
Core metrics
Computed for every trace and aggregated per variant.
Duration
Wall-clock time from the first span to the last — end-to-end latency of the run.
Total tokens
Sum of prompt and completion tokens across every LLM call in the trace.
Input / output tokens
Prompt vs. completion tokens. High input often means large context or long history; high output often means verbose responses.
Turns
Number of LLM conversation turns — each assistant response is one turn.
Cost
Estimated USD from token counts at list prices (see Reading cost).
Agent metrics
Extra signals for tool-using and coding agents.
Reading deltas
In the comparison table, one column is the baseline and every other column shows its change against it; choose the baseline from the dropdown in the first column. The best and worst value in each row are highlighted. Read the direction with the metric: lower duration, tokens, errors, and retries are better; higher success rate is better. The same table can group runs by model or project instead of variant; see Compare by model.
Watch the combinations
High retries plus high token usage is a strong signal a variant needs work. Prefer the median over the average for latency and tokens — outliers skew averages.