# Core concepts

The vocabulary the rest of the docs assumes: what a **trace** is made of, how runs are compared as **variants**, and how **workspaces**, members, and API keys isolate and share your data.

## Traces & spans

**Trace (or run)**: One agent invocation, captured end-to-end. A trace has an `id` (unique within a workspace), a top-level `input` and `output`, free-form `metadata`, optional `tags` and `scores`, and an array of **spans**.

**Span**: One step inside a run — a model call, a tool execution, a retrieval, a sub-agent. Spans form a tree via `span_id` / `root_span_id` / `span_parents`, so the trace view can render the full call hierarchy.

Kalmia stores the full trace once, then derives a slim **summary** with computed metrics so the dashboard list stays fast without reading the multi-megabyte payload.
Only SDK- and API-ingested traces are stored; delivery is idempotent by trace `id`.

## The span model

These are the fields you can set when POSTing a trace to the ingest endpoint:

### Trace fields

- **id**: Unique within your workspace (required)
- **input / output / expected**: Top-level run I/O
- **metadata**: provider, model, variant, correlationId, …
- **tags**: String labels on the run
- **spans**: Array of spans (required; may be empty)

### Span fields

- **id / span_id / root_span_id**: Identity and tree root
- **span_parents**: Parent span ids (tree edges)
- **name / input / output / error**: What the step did
- **span_attributes.type**: llm · tool · agent · …
- **metrics**: start, end, prompt_tokens, completion_tokens

On ingest, Kalmia normalizes spans into typed steps — `agent`, `llm_call`, `tool_call`, `tool_result`, `user_message`, `assistant_message`, `error`, and a generic fallback — so the trace view and every downstream feature see a consistent shape regardless of source.

> **Traces can contain secrets:** Captured traces include system prompts, full message history, and tool inputs/results, which may contain PII or credentials. Sanitize before shipping — Kalmia does not redact. Traces are stored encrypted at rest and retained for 90 days by default; when a trace expires, its verdicts and customer feedback are deleted with it.

## Computed metrics

Every trace is preprocessed into a metrics object the moment it arrives.
These feed the trace list, [variant comparison](https://www.kalmia.dev/docs/analytics.md#metrics), and [trends](https://www.kalmia.dev/docs/analytics.md#trends):

| Metric | What it measures |
| --- | --- |
| `totalDurationMs` | Wall-clock duration of the whole run |
| `llmLatencyMs` | Model time only, excluding tools/orchestration |
| `totalTokens` | Input, output and prompt-cache tokens (reads and writes), summed across LLM spans |
| `costUsd / priced` | Estimated USD, plus whether the model was priced |
| `toolCalls / turns` | Tool executions and model turns |
| `errorCount / toolCallErrors` | Errors and failed tool calls |
| `retryCount / backtrackCount` | Retries and backtracks |
| `uniqueToolsUsed` | Distinct tools the agent reached for |

Cost is an **estimate** derived from token counts and a code-maintained price table; unpriced models show as `—`, never `$0`.
See [Metrics & cost](https://www.kalmia.dev/docs/analytics.md#metrics) for the full pricing model.

## Variants

**Variant**: A named version of your agent, carried in trace `metadata.variant` (falling back to an agent name, then `default`). Variants are how you compare two versions — RAG vs no-RAG, an old prompt vs a new one — across [metrics](https://www.kalmia.dev/docs/analytics.md#metrics) and [eval scoreboards](https://www.kalmia.dev/docs/evaluate.md#variants).

## Workspaces & members

A **workspace** is the unit of isolation and collaboration.
Traces, findings, annotations, behaviors, API keys, domain-context documents, and the audit log are all scoped to a workspace.
You can belong to several and switch the active one; each UI-created workspace is fully isolated and starts empty until you send traces.

Invite teammates by email with a role.
A workspace you are not a member of returns `404`, never `403` — so an id or header probe cannot even confirm another workspace exists.

## Roles

| Role | Can do |
| --- | --- |
| Owner | Everything a member can, plus manage members, create/rotate API keys, change settings (judge model, rate-limit override), view the audit log, and destructive workspace ops. One owner per workspace in v1. |
| Member | Full read-write, including actions that spend LLM budget: judging, detection, investigation, chat. Cannot manage membership. |
| Viewer | Read-only. Sees all workspace data but cannot create, update, delete, or trigger any budget-spending action. Budget-spending controls are hidden, not just disabled. |

Invites are for `member` or `viewer` only; acceptance is bound to the invitee's verified email.
Owner-visible actions — member changes, key lifecycle, settings changes — are recorded in an exportable audit log.

## API keys

A workspace API key authenticates the ingest and read endpoints via an `Authorization: Bearer` header; the SDKs also read it from the `KALMIA_API_KEY` env var.
Keys have the form `kal_live_…` and are **shown once** at creation — only a display prefix and a hash are stored, so a lost key must be rotated, never recovered.

- One active key per workspace. Creating a key when one already exists returns a conflict; use rotate.
- Rotation is an atomic cutover — it revokes all active keys and mints a replacement in one transaction.
- A revoked key gets a rotation-specific error, so you can tell “rotated” apart from “invalid”.
- Key lifecycle is owner-only and managed from Settings → API Keys.

### Scopes

A key is created with one of two scopes, chosen in Settings → API Keys before generating:

- **Read + write (full)**: The default. Ingests traces and reads via the API and MCP
- **Write only (ingest)**: Ingest-only - safer to embed in a shipped app

A write-only key can POST traces (SDK and OTel) and feedback, but every read or analysis surface rejects it with `403` and a message naming the scope - listing traces, MCP, verdicts, investigation, and chat all require a full key.
The scope is a property of the single active key, not a second key: rotation preserves the current scope, and swapping scopes is a rotation with the other scope selected.

## GitHub repo connect

A workspace can connect **one repository, read-only**, so confirmed [run flags](https://www.kalmia.dev/docs/watch.md#flags) can generate [fix recommendations](https://www.kalmia.dev/docs/watch.md#fix-recs) grounded in your actual code.
Connect from **Settings → GitHub**: install the Kalmia GitHub App, and if the installation grants more than one repository, pick the one this workspace's traces belong to.
Everyone sees the connection state; connecting and changing the repository are owner-only.

- Access is read-only by construction: the GitHub App holds only the `Contents: Read-only` permission. Nothing in Kalmia ever writes to your repo.
- File reads are pinned to one commit SHA per recommendation and capped (40 files, 300 KB per file), and repo content is treated as untrusted input.
- If the app is uninstalled, suspended, or loses access to the repo, the connection shows as needing reauthorization — reconnect from the same panel.
- To switch to a repository the installation does not yet grant, first edit the installation's repository access on GitHub, then use *Change repository*.

## Rate limits

Rate limiting is scoped per **(operation group, workspace)**, with two sliding windows that must both pass - a per-minute **burst** limit and a per-day **sustained** limit.
Each group has its own budget, so a bulk eval run or an ingest flood can never starve interactive judging:

- **Interactive group**: judge · embedding · agent (human-facing) - default 120/min, 5,000/day
- **Eval group**: Bulk grading and prompt audits - 60/min, 20,000/day
- **MCP group**: Read-only MCP tools - 120/min, 10,000/day
- **Ingest group**: SDK / OTel / feedback writes, one admission per request - 300/min, 100,000/day
- **Narration group**: Live-run narration ticks - 60/min, 10,000/day
- **Replay group**: Background session-replay pulls - 30/min, 5,000/day
- **Classify group**: Issue classification for the Issues tab, one small call per verdict with issues - 120/min, 10,000/day
- **Verify group**: Claim verification during scheduled sweeps, which confirms or dismisses each audited issue against the spans - 240/min, 20,000/day

On top of the request windows, each user has a **spend budget** shared by every workspace they own, kept by day and by month: every model call's actual token usage is priced and accumulated, and model-backed operations are refused once the day's or the month's cost reaches the budget.
It is separate from credits. A credit pays for an action whatever it costs to run, and the cost follows the size of your traces; the budget bounds that cost. Normal use does not meet it, and an account that does has unusually large traces and belongs on a bigger plan:

- **Free**: $5 of model usage a day, $15 a month
- **Plus**: $15 of model usage a day, $30 a month
- **Pro**: $60 of model usage a day, $300 a month
- **Enterprise**: $200 of model usage a day, no monthly limit

Creating more workspaces does not add budget.
Ingest, MCP reads, and dashboard traffic spend no model tokens and are never blocked by it - your traces keep flowing even when analysis is paused.
The daily budget resets at UTC midnight, the monthly one on the 1st (UTC).
A refusal answers `429` and says which of the two it was.

Owners can raise or lower the interactive burst in workspace settings.
When a limit is hit, the API returns `429` with a `retryAfter` and IETF `RateLimit-Policy` / `RateLimit` headers.
The ingest budget is deliberately generous - each request is one admission regardless of how many traces it carries (the per-request caps are size and trace count), so normal traffic never hits it, and the ingest gate fails open if the limiter backend is unavailable rather than dropping your traces.
Dashboard reads and the API read endpoints (the trace list and single-trace fetch) are not currently rate limited.
See the [API reference](https://www.kalmia.dev/docs/api-reference.md#rate-limits) for the response shape.

## Plans & credits

Your account has a plan and a credit balance.
Credits belong to **you**, not to a workspace: every workspace you own draws from the same balance, so creating workspaces never multiplies them.
See them in Settings → Plan & credits.

- **Free - $0/month**: 50 credits every month
- **Plus - $10/month**: 550 credits every month
- **Pro - $100/month**: 6,000 credits every month
- **Enterprise - custom price**: Unlimited credits, under a contract. Talk to us.
- **Sign-up credits**: 100 credits once, for every new account on any plan. They never expire.

- Monthly credits reset at the start of each monthly period; unused ones do not carry over. They are used before sign-up credits.
- Upgrade from Settings → Plan & credits; payment goes through Stripe Checkout, and your new plan's full monthly credits arrive as soon as the payment clears. Paid plans renew on your billing date, and each renewal payment brings that month's credits.
- Manage billing (in the same section) opens Stripe's billing portal: switch between Plus and Pro, update your card, download invoices, or cancel.
- Moving from Pro down to Plus takes effect at the end of the month you already paid for; until then you keep Pro and its credits.
- A cancelled plan stays active until the end of the paid month, then the account returns to Free.
- Ingesting, storing and browsing traces never uses credits.

### What credits pay for

- **Sweep - 2 credits per trace**: One monitoring pass over a window of traces: anomaly detection, an audit of each trace it checks, root-cause investigations, and a digest. Scheduled or run by hand. Priced by the traces it checks, which is at most the audit allowance in Settings. A sweep the balance cannot cover in full checks as many traces as it can pay for, the failed and flagged ones first. A sweep that finds no new traces is free.
- **Anomaly scan - 20 credits**: A scan you start over the newest traces: outlier detection, a small audit sample, and an investigation of each high-value cluster.
- **Trace analysis - 3 credits**: Running the Kalmia harness on one trace to find its root cause, or starting an investigation of it.
- **Question - 1 credit**: Asking Kalmia about a trace or continuing an investigation, in the app, over the API or MCP, or by messaging the Slack bot.
- **Regression investigation - 3 credits**: Investigating why one variant does worse than another.
- **Fix recommendation - 1 credit**: Regenerating the suggested fix on an error group, or asking for a fix grounded in your repository on a flagged run. The first suggestion on an error group, and any Kalmia writes on its own, are free.
- **Behavior detection - 1 credit per trace**: Checking traces for a behavior you describe, one trace or a batch.
- **Prompt audit - 1 credit per trace**: Checking a sample of traces against your agent's system prompt.
- **Eval scoring - 1 credit per case**: Scoring one eval case against its criteria, including the diagnosis of a failed case. Rescoring a case costs the same.
- **AI trace search - 1 credit per trace**: The search API that reads each trace with a model (POST /api/traces/search). Searching the trace list in the app is free.

- An action is charged when it starts, to the owner of the workspace it runs in, whoever on the team started it.
- A new workspace starts with scheduled sweeps off, so nothing spends credits until you ask: run one sweep from the Scan tab, or pick a schedule in Settings → Monitoring. Weekly is about 4 sweeps a month, daily about 30.
- A sweep is priced by the traces it checks: those that arrived since the last sweep, up to the audit allowance in Settings → Monitoring. Run a sweep states the number and the price before it starts.
- When the balance covers only part of a sweep, the sweep checks as many traces as it can pay for, the failed and flagged ones first, and its row in the history says so. It does not run only when the balance cannot cover a single trace.
- A sweep that finds no new traces is free.
- The investigations a sweep or an anomaly scan runs on the traces it flags are part of its price.
- Starting an investigation costs the same as a trace analysis. Each question after that, in the app, over MCP, or to the Slack bot, costs one question.
- Behavior detection, a prompt audit and the search API are priced by the traces they read.
- An eval case is charged when it is scored, which happens on its own when the run arrives. A case with no criteria is free.
- If an action does not deliver (a sweep that could not complete, an analysis that produced no verdict, a question that got no answer), its credits are returned. Both the charge and the refund appear under Settings → Plan & credits.
- When the balance cannot cover an action, it does not run, and the app says why. The API answers `402` with `"code": "insufficient_credits"`. A scheduled sweep that cannot pay for a single trace is recorded in the sweep history with that reason and tried again at the next interval; the traces it skipped are covered by the next sweep that runs. An eval case that could not be scored is marked "needs attention" and can be re-scored later.
- Free: sending, storing and browsing traces, searching the trace list, the first fix suggestion on an error group, live run flags, and anything else Kalmia does on its own for the traces you send.
- The Enterprise plan is not metered by credits.
