Metrics & analytics
Zoom out from a single trace. Metrics compares your agent variants side-by-side on operational numbers; Analytics is a time-series surface for watching cost, latency, error rate, and quality drift over time — and every datapoint drills back into the traces behind it.
Metrics & cost
The Metrics view is a snapshot comparison of your variants over the loaded trace set. It answers “does variant B beat variant A?” and “which variant is cheaper, faster, more reliable?” across a fixed set of columns, each with an objective direction — the best and worst variant per column are highlighted.
Variant comparison
| Column | Direction |
|---|---|
| Avg duration | Lower is better |
| Avg tokens | Lower is better |
| Avg $/run · Total $ | Lower is better (unpriced excluded) |
| Avg turns · tool calls | Neutral |
| Success rate | Higher is better |
| Tool errors · retries · backtracks | Lower is better |
| Unique tools · subagents | Neutral |
Toggle variants in or out of the comparison and pick a baseline. Every other column shows its change against the baseline, in both the metrics table and the behavior prevalence table below it (how often each documented behavior appears per variant, as percentage-point deltas with a low-sample confidence flag). The same comparison can group runs by primary model or project instead of variant, and narrow a model comparison to one variant; see Compare by model. When a workspace has more runs than one rollup covers, a line above the table says how many of the newest runs the numbers describe. The metrics view also hosts the Anomaly Scan.
Reading cost
Cost is an estimate derived from token counts at list prices.
Prompt-cache reads and writes are priced at their own rates when your traces report them, so a cache-heavy agent is not overstated.
Provider billing rules that the token counts alone imply are applied too: on OpenAI GPT-5.6 and later, uncached input bills at the cache-write rate (the prompt is written to the cache by default), and a call whose input exceeds 272K tokens bills entirely at the long-context rate.
A model Kalmia doesn't have a price for shows as —, never $0, and is excluded from cost totals so it can't drag a comparison to a false low.
Third-party services
Cost also covers what your agent spends beyond the model, with no setup on your side.
A span already carries the response of the service it called, and Kalmia prices it from that.
The trace view and the summary endpoint show the total and the split (llmCostUsd, toolCostUsd, toolCostByService), so a spike names its service.
Exact, from the response. These services state the charge in every response and Kalmia reads it verbatim.
Any tool result carrying a cost_usd field (or cost with currency: "USD") is read the same way, which is how you attach a price to a service Kalmia does not know.
To price a service that is not listed, add cost_usd to the object you log as the tool span's output.
It applies per call, so put it where the call happens:
with traced(name="scrape_page", span_type="tool") as span: result = firecrawl.scrape(url) # Firecrawl reports credits; price them at your plan's rate span.log( input={"url": url}, output={**result, "cost_usd": result["creditsUsed"] * 0.0008}, )The cost lands under the tool's name in toolCostByService.
If you would rather not touch the response object, { "data": result, "cost_usd": x } works too: Kalmia looks one level down.
Estimated, from the published rate. Recognized by the tool's name or URL plus the response shape, then priced at the provider's self-serve rate. Where the rate depends on the plan, the highest self-serve rate is used so cost is never understated.
Built-in provider tools. Tools the model provider runs inside an LLM call and bills per use, counted from the call's output.
A service Kalmia doesn't recognize contributes nothing, so tool cost is a floor, not a claim of totality, and it never affects whether a run counts as priced.
For the exact figure the span must carry the raw API response: return it from your traced() tool function (or log it as the span output) rather than a trimmed-down result.
Pricing is applied retroactively, so past runs get the split too.
Reconciling with your bill
Every cost figure says what it rests on.
In toolCostByService a bare key (exa) is a charge the provider stated in its response; an -estimated key or a built-in provider tool is priced at the rate card on this page.
The summary endpoints split the total the same way: toolCostReportedUsdSum is what an invoice would confirm, toolCostRateCardUsdSum is the estimate, and toolCostBasisByService labels each key.
Model-token cost is list price, with cache reads and writes at their own list rates, so costUsdSum is an estimate, not a bill: negotiated discounts and batch pricing are not known to Kalmia.
If most of your spend lands in the estimate column, the fix is on the capture side: return the raw provider response from the tool (an Exa call through an MCP server that flattens the result to text loses costDollars) or attach cost_usd yourself.
Trends
The Analytics view watches your production SDK traffic over time — cost creep across a week, a p95 latency spike on a given day, a variant's score drifting down — without opening runs one by one. It's a curated instrument, not a chart builder: no arbitrary group-by, but every datapoint drills into the filtered trace list so the aggregate feeds the root-cause loop.
Production only
Analytics covers live SDK traffic. Eval analysis stays in the Evals view.The charts
Each card is split by variant (top variants by volume, with the rest folded into “other”):
| Chart | Shows |
|---|---|
| Throughput | Runs per bucket, stacked by variant |
| Error rate | Fraction of traces with ≥1 error, in [0,1] |
| Latency | p50 and p95 total run duration |
| LLM latency | p50/p95 model time only, excluding tools |
| Spans | Avg spans per trace, stacked LLM / tool / other |
| Spans by variant | Avg spans per trace per variant, all span types combined |
| Tokens | Avg tokens per trace, stacked input vs output |
| Tokens by variant | Avg tokens per trace per variant, input and output combined |
| Cost | Estimated $/bucket per variant (priced traces only) |
Add graph puts your own card at the top of the grid: name it after a behavior or rule you are tracking, choose the variants to compare, and it charts each variant's failed-run rate (the share of runs whose verdict is failed) over the window. These graphs are saved in your browser, per workspace, and each has its own delete button.
When your traces carry third-party spend, a Cost by service list appears under the KPI tiles: model tokens first, then each recognized service (search, scraping, and so on) largest first, each with its change versus the prior window. Services priced from a rate card rather than the provider's own response are marked as estimates.
Each per-variant chart has a ranked breakdown table beneath it; click a row or “see all runs” to drill into exactly those traces.
KPI summary
Five tiles across the top summarize the window, each with a change versus the prior equal-length window:
Reserved colors
Deltas use indigo for up and muted for down — green and red are reserved for real pass/fail verdicts, so they never lose their meaning on a dashboard. Sub-1% changes are hidden as noise.Filters & drill-through
The header filter bar controls the whole surface:
Buckets auto-size to keep charts readable at any zoom, the still-filling final bucket is drawn as incomplete rather than dropped, and each variant keeps a stable color across every chart. Clicking any datapoint jumps to the trace list filtered to that variant and time window.