# Metrics & analytics

Zoom out from a single trace.
**Metrics** compares your agent variants side-by-side on operational numbers; **Analytics** is a time-series surface for watching cost, latency, error rate, and quality drift over time — and every datapoint drills back into the traces behind it.

## Metrics & cost

The **Metrics** view is a snapshot comparison of your variants over the loaded trace set.
It answers “does variant B beat variant A?” and “which variant is cheaper, faster, more reliable?” across a fixed set of columns, each with an objective direction — the best and worst variant per column are highlighted.

## Variant comparison

| Column | Direction |
| --- | --- |
| Avg duration | Lower is better |
| Avg tokens | Lower is better |
| Avg $/run · Total $ | Lower is better (unpriced excluded) |
| Avg turns · tool calls | Neutral |
| Success rate | Higher is better |
| Tool errors · retries · backtracks | Lower is better |
| Unique tools · subagents | Neutral |

Toggle variants in or out of the comparison and pick a **baseline**.
Every other column shows its change against the baseline, in both the metrics table and the **behavior prevalence** table below it (how often each documented behavior appears per variant, as percentage-point deltas with a low-sample confidence flag).
The same comparison can group runs by primary model or project instead of variant, and narrow a model comparison to one variant; see [Compare by model](https://www.kalmia.dev/docs/guides/comparing-variants.md#by-model).
When a workspace has more runs than one rollup covers, a line above the table says how many of the newest runs the numbers describe.
The metrics view also hosts the [Anomaly Scan](https://www.kalmia.dev/docs/detect.md#anomaly).

## Reading cost

Cost is an **estimate** derived from token counts at list prices.
Prompt-cache reads and writes are priced at their own rates when your traces report them, so a cache-heavy agent is not overstated.
Provider billing rules that the token counts alone imply are applied too: on OpenAI GPT-5.6 and later, uncached input bills at the cache-write rate (the prompt is written to the cache by default), and a call whose input exceeds 272K tokens bills entirely at the long-context rate.
A model Kalmia doesn't have a price for shows as `—`, never `$0`, and is excluded from cost totals so it can't drag a comparison to a false low.

### Third-party services

Cost also covers what your agent spends beyond the model, with no setup on your side.
A span already carries the response of the service it called, and Kalmia prices it from that.
The trace view and the [summary endpoint](https://www.kalmia.dev/docs/api-reference.md) show the total and the split (`llmCostUsd`, `toolCostUsd`, `toolCostByService`), so a spike names its service.

**Exact, from the response.** These services state the charge in every response and Kalmia reads it verbatim.
Any tool result carrying a `cost_usd` field (or `cost` with `currency: "USD"`) is read the same way, which is how you attach a price to a service Kalmia does not know.

- **Exa**: The `costDollars.total` field Exa returns on every response.
- **Valyu**: The `total_deduction_dollars` field on every response.
- **Perplexity**: The `usage.cost.total_cost` field (USD) on every response.
- **OpenRouter**: The `usage.cost` field (USD credits) on every response.
- **Twilio**: The `price` field when `price_unit` is USD (populated once the message is billed).

To price a service that is not listed, add `cost_usd` to the object you log as the tool span's output.
It applies per call, so put it where the call happens:

```python
with traced(name="scrape_page", span_type="tool") as span:
    result = firecrawl.scrape(url)
    # Firecrawl reports credits; price them at your plan's rate
    span.log(
        input={"url": url},
        output={**result, "cost_usd": result["creditsUsed"] * 0.0008},
    )
```

```typescript
await span.traced("scrape_page", async (toolSpan) => {
  const result = await firecrawl.scrape(url);
  // Firecrawl reports credits; price them at your plan's rate
  toolSpan.log({
    input: { url },
    output: { ...result, cost_usd: result.creditsUsed * 0.0008 },
  });
}, { type: "tool" });
```

The cost lands under the tool's name in `toolCostByService`.
If you would rather not touch the response object, `{ "data": result, "cost_usd": x }` works too: Kalmia looks one level down.

**Estimated, from the published rate.** Recognized by the tool's name or URL plus the response shape, then priced at the provider's self-serve rate.
Where the rate depends on the plan, the highest self-serve rate is used so cost is never understated.

- **Tavily (est.)**: Credit schedule at $0.008/credit: search 1 credit (advanced 2), extract 1 credit per 5 URLs (advanced 2).
- **Brave Search (est.)**: $5 per 1,000 web search requests ($0.005 per request).
- **Serper (est.)**: Credits reported on the response (1 per 10 results) at the Starter pack rate of $1 per 1,000 credits; larger packs are cheaper.
- **Parallel Search (est.)**: $0.005 per search request ($0.001 in turbo/fast mode).
- **Google Custom Search (est.)**: $5 per 1,000 queries ($0.005 per query).
- **Exa (est.)**: When the response is not logged: search $0.007 (+$0.001 per result over 10, deep $0.012) plus $0.001 per result page for text, highlights, and summaries; fetch $0.001 per URL.

**Built-in provider tools.** Tools the model provider runs inside an LLM call and bills per use, counted from the call's output.

- **Claude web search**: $10 per 1,000 searches, counted from the call's server_tool_use blocks or usage.
- **OpenAI web search**: $10 per 1,000 calls, counted from web_search_call output items.
- **OpenAI file search**: $2.50 per 1,000 calls, counted from file_search_call output items.

A service Kalmia doesn't recognize contributes nothing, so tool cost is a floor, not a claim of totality, and it never affects whether a run counts as priced.
For the exact figure the span must carry the raw API response: return it from your `traced()` tool function (or log it as the span output) rather than a trimmed-down result.
Pricing is applied retroactively, so past runs get the split too.

### Reconciling with your bill

Every cost figure says what it rests on.
In `toolCostByService` a bare key (`exa`) is a charge the provider stated in its response; an `-estimated` key or a built-in provider tool is priced at the rate card on this page.
The summary endpoints split the total the same way: `toolCostReportedUsdSum` is what an invoice would confirm, `toolCostRateCardUsdSum` is the estimate, and `toolCostBasisByService` labels each key.
Model-token cost is list price, with cache reads and writes at their own list rates, so `costUsdSum` is an estimate, not a bill: negotiated discounts and batch pricing are not known to Kalmia.
If most of your spend lands in the estimate column, the fix is on the capture side: return the raw provider response from the tool (an Exa call through an MCP server that flattens the result to text loses `costDollars`) or attach `cost_usd` yourself.

## Trends

The **Analytics** view watches your production SDK traffic over time — cost creep across a week, a p95 latency spike on a given day, a variant's score drifting down — without opening runs one by one.
It's a curated instrument, not a chart builder: no arbitrary group-by, but every datapoint drills into the filtered trace list so the aggregate feeds the root-cause loop.

> **Production only:** Analytics covers live SDK traffic. Eval analysis stays in the Evals view.

## The charts

Each card is split by variant (top variants by volume, with the rest folded into “other”):

| Chart | Shows |
| --- | --- |
| Throughput | Runs per bucket, stacked by variant |
| Error rate | Fraction of traces with ≥1 error, in \[0,1\] |
| Latency | p50 and p95 total run duration |
| LLM latency | p50/p95 model time only, excluding tools |
| Spans | Avg spans per trace, stacked LLM / tool / other |
| Spans by variant | Avg spans per trace per variant, all span types combined |
| Tokens | Avg tokens per trace, stacked input vs output |
| Tokens by variant | Avg tokens per trace per variant, input and output combined |
| Cost | Estimated $/bucket per variant (priced traces only) |

**Add graph** puts your own card at the top of the grid: name it after a behavior or rule you are tracking, choose the variants to compare, and it charts each variant's failed-run rate (the share of runs whose verdict is failed) over the window.
These graphs are saved in your browser, per workspace, and each has its own delete button.

When your traces carry third-party spend, a **Cost by service** list appears under the KPI tiles: model tokens first, then each recognized service (search, scraping, and so on) largest first, each with its change versus the prior window.
Services priced from a rate card rather than the provider's own response are marked as estimates.

Each per-variant chart has a ranked breakdown table beneath it; click a row or “see all runs” to drill into exactly those traces.

## KPI summary

Five tiles across the top summarize the window, each with a change versus the prior equal-length window:

- **Runs**: Volume (neutral — more traffic isn’t inherently good or bad)
- **Error rate**: Fraction of runs that failed: the root span errored or the run died on its last span. Errors the agent recovered from mid-run do not count (down is good)
- **p95 latency**: Tail latency (down is good)
- **Cost / run**: Priced runs only; “unpriced” when cost is unknown
- **Quality**: A chosen primary score, or retry rate when no scores exist

> **Reserved colors:** Deltas use indigo for up and muted for down — green and red are reserved for real pass/fail verdicts, so they never lose their meaning on a dashboard. Sub-1% changes are hidden as noise.

## Filters & drill-through

The header filter bar controls the whole surface:

- **Time range**: 1h to 90d presets, custom From/To, and prev/next window scrub
- **Model**: Filter to one model in the window
- **Variants**: Restrict aggregation to specific variants, or default to top-by-volume
- **Export**: CSV of the current view in tidy long format

Buckets auto-size to keep charts readable at any zoom, the still-filling final bucket is drawn as incomplete rather than dropped, and each variant keeps a stable color across every chart.
Clicking any datapoint jumps to the trace list filtered to that variant and time window.
