Analytics

Metrics & analytics

Zoom out from a single trace. Metrics compares your agent variants side-by-side on operational numbers; Analytics is a time-series surface for watching cost, latency, error rate, and quality drift over time — and every datapoint drills back into the traces behind it.

Metrics & cost

The Metrics view is a snapshot comparison of your variants over the loaded trace set. It answers “does variant B beat variant A?” and “which variant is cheaper, faster, more reliable?” across a fixed set of columns, each with an objective direction — the best and worst variant per column are highlighted.

Variant comparison

ColumnDirection
Avg durationLower is better
Avg tokensLower is better
Avg $/run · Total $Lower is better (unpriced excluded)
Avg turns · tool callsNeutral
Success rateHigher is better
Tool errors · retries · backtracksLower is better
Unique tools · subagentsNeutral

Toggle variants in or out of the comparison and pick a baseline. Every other column shows its change against the baseline, in both the metrics table and the behavior prevalence table below it (how often each documented behavior appears per variant, as percentage-point deltas with a low-sample confidence flag). The same comparison can group runs by primary model or project instead of variant, and narrow a model comparison to one variant; see Compare by model. When a workspace has more runs than one rollup covers, a line above the table says how many of the newest runs the numbers describe. The metrics view also hosts the Anomaly Scan.

Reading cost

Cost is an estimate derived from token counts at list prices. Prompt-cache reads and writes are priced at their own rates when your traces report them, so a cache-heavy agent is not overstated. Provider billing rules that the token counts alone imply are applied too: on OpenAI GPT-5.6 and later, uncached input bills at the cache-write rate (the prompt is written to the cache by default), and a call whose input exceeds 272K tokens bills entirely at the long-context rate. A model Kalmia doesn't have a price for shows as —, never $0, and is excluded from cost totals so it can't drag a comparison to a false low.

Third-party services

Cost also covers what your agent spends beyond the model, with no setup on your side. A span already carries the response of the service it called, and Kalmia prices it from that. The trace view and the summary endpoint show the total and the split (llmCostUsd, toolCostUsd, toolCostByService), so a spike names its service.

Exact, from the response. These services state the charge in every response and Kalmia reads it verbatim. Any tool result carrying a cost_usd field (or cost with currency: "USD") is read the same way, which is how you attach a price to a service Kalmia does not know.

ExaThe `costDollars.total` field Exa returns on every response.
ValyuThe `total_deduction_dollars` field on every response.
PerplexityThe `usage.cost.total_cost` field (USD) on every response.
OpenRouterThe `usage.cost` field (USD credits) on every response.
TwilioThe `price` field when `price_unit` is USD (populated once the message is billed).

To price a service that is not listed, add cost_usd to the object you log as the tool span's output. It applies per call, so put it where the call happens:

with traced(name="scrape_page", span_type="tool") as span:
result = firecrawl.scrape(url)
# Firecrawl reports credits; price them at your plan's rate
span.log(
input={"url": url},
output={**result, "cost_usd": result["creditsUsed"] * 0.0008},
)

The cost lands under the tool's name in toolCostByService. If you would rather not touch the response object, { "data": result, "cost_usd": x } works too: Kalmia looks one level down.

Estimated, from the published rate. Recognized by the tool's name or URL plus the response shape, then priced at the provider's self-serve rate. Where the rate depends on the plan, the highest self-serve rate is used so cost is never understated.

Tavily (est.)Credit schedule at $0.008/credit: search 1 credit (advanced 2), extract 1 credit per 5 URLs (advanced 2).
Brave Search (est.)$5 per 1,000 web search requests ($0.005 per request).
Serper (est.)Credits reported on the response (1 per 10 results) at the Starter pack rate of $1 per 1,000 credits; larger packs are cheaper.
Parallel Search (est.)$0.005 per search request ($0.001 in turbo/fast mode).
Google Custom Search (est.)$5 per 1,000 queries ($0.005 per query).
Exa (est.)When the response is not logged: search $0.007 (+$0.001 per result over 10, deep $0.012) plus $0.001 per result page for text, highlights, and summaries; fetch $0.001 per URL.

Built-in provider tools. Tools the model provider runs inside an LLM call and bills per use, counted from the call's output.

Claude web search$10 per 1,000 searches, counted from the call's server_tool_use blocks or usage.
OpenAI web search$10 per 1,000 calls, counted from web_search_call output items.
OpenAI file search$2.50 per 1,000 calls, counted from file_search_call output items.

A service Kalmia doesn't recognize contributes nothing, so tool cost is a floor, not a claim of totality, and it never affects whether a run counts as priced. For the exact figure the span must carry the raw API response: return it from your traced() tool function (or log it as the span output) rather than a trimmed-down result. Pricing is applied retroactively, so past runs get the split too.

Reconciling with your bill

Every cost figure says what it rests on. In toolCostByService a bare key (exa) is a charge the provider stated in its response; an -estimated key or a built-in provider tool is priced at the rate card on this page. The summary endpoints split the total the same way: toolCostReportedUsdSum is what an invoice would confirm, toolCostRateCardUsdSum is the estimate, and toolCostBasisByService labels each key. Model-token cost is list price, with cache reads and writes at their own list rates, so costUsdSum is an estimate, not a bill: negotiated discounts and batch pricing are not known to Kalmia. If most of your spend lands in the estimate column, the fix is on the capture side: return the raw provider response from the tool (an Exa call through an MCP server that flattens the result to text loses costDollars) or attach cost_usd yourself.

The charts

Each card is split by variant (top variants by volume, with the rest folded into “other”):

ChartShows
ThroughputRuns per bucket, stacked by variant
Error rateFraction of traces with ≥1 error, in [0,1]
Latencyp50 and p95 total run duration
LLM latencyp50/p95 model time only, excluding tools
SpansAvg spans per trace, stacked LLM / tool / other
Spans by variantAvg spans per trace per variant, all span types combined
TokensAvg tokens per trace, stacked input vs output
Tokens by variantAvg tokens per trace per variant, input and output combined
CostEstimated $/bucket per variant (priced traces only)

Add graph puts your own card at the top of the grid: name it after a behavior or rule you are tracking, choose the variants to compare, and it charts each variant's failed-run rate (the share of runs whose verdict is failed) over the window. These graphs are saved in your browser, per workspace, and each has its own delete button.

When your traces carry third-party spend, a Cost by service list appears under the KPI tiles: model tokens first, then each recognized service (search, scraping, and so on) largest first, each with its change versus the prior window. Services priced from a rate card rather than the provider's own response are marked as estimates.

Each per-variant chart has a ranked breakdown table beneath it; click a row or “see all runs” to drill into exactly those traces.

KPI summary

Five tiles across the top summarize the window, each with a change versus the prior equal-length window:

RunsVolume (neutral — more traffic isn’t inherently good or bad)
Error rateFraction of runs that failed: the root span errored or the run died on its last span. Errors the agent recovered from mid-run do not count (down is good)
p95 latencyTail latency (down is good)
Cost / runPriced runs only; “unpriced” when cost is unknown
QualityA chosen primary score, or retry rate when no scores exist

Reserved colors

Deltas use indigo for up and muted for down — green and red are reserved for real pass/fail verdicts, so they never lose their meaning on a dashboard. Sub-1% changes are hidden as noise.

Filters & drill-through

The header filter bar controls the whole surface:

Time range1h to 90d presets, custom From/To, and prev/next window scrub
ModelFilter to one model in the window
VariantsRestrict aggregation to specific variants, or default to top-by-volume
ExportCSV of the current view in tidy long format

Buckets auto-size to keep charts readable at any zoom, the still-filling final bucket is drawn as incomplete rather than dropped, and each variant keeps a stable color across every chart. Clicking any datapoint jumps to the trace list filtered to that variant and time window.