Core concepts
The vocabulary the rest of the docs assumes: what a trace is made of, how runs are compared as variants, and how workspaces, members, and API keys isolate and share your data.
Traces & spans
Trace (or run)
One agent invocation, captured end-to-end.
A trace has an id (unique within a workspace), a top-level input and output, free-form metadata, optional tags and scores, and an array of spans.
Span
One step inside a run — a model call, a tool execution, a retrieval, a sub-agent.
Spans form a tree via span_id / root_span_id / span_parents, so the trace view can render the full call hierarchy.
Kalmia stores the full trace once, then derives a slim summary with computed metrics so the dashboard list stays fast without reading the multi-megabyte payload.
Only SDK- and API-ingested traces are stored; delivery is idempotent by trace id.
The span model
These are the fields you can set when POSTing a trace to the ingest endpoint:
Trace fields
Span fields
On ingest, Kalmia normalizes spans into typed steps — agent, llm_call, tool_call, tool_result, user_message, assistant_message, error, and a generic fallback — so the trace view and every downstream feature see a consistent shape regardless of source.
Traces can contain secrets
Captured traces include system prompts, full message history, and tool inputs/results, which may contain PII or credentials. Sanitize before shipping — Kalmia does not redact. Traces are stored encrypted at rest and retained for 90 days by default; when a trace expires, its verdicts and customer feedback are deleted with it.Computed metrics
Every trace is preprocessed into a metrics object the moment it arrives. These feed the trace list, variant comparison, and trends:
| Metric | What it measures |
|---|---|
totalDurationMs | Wall-clock duration of the whole run |
llmLatencyMs | Model time only, excluding tools/orchestration |
totalTokens | Input, output and prompt-cache tokens (reads and writes), summed across LLM spans |
costUsd / priced | Estimated USD, plus whether the model was priced |
toolCalls / turns | Tool executions and model turns |
errorCount / toolCallErrors | Errors and failed tool calls |
retryCount / backtrackCount | Retries and backtracks |
uniqueToolsUsed | Distinct tools the agent reached for |
Cost is an estimate derived from token counts and a code-maintained price table; unpriced models show as —, never $0.
See Metrics & cost for the full pricing model.
Variants
Variant
A named version of your agent, carried in trace metadata.variant (falling back to an agent name, then default).
Variants are how you compare two versions — RAG vs no-RAG, an old prompt vs a new one — across metrics and eval scoreboards.
Workspaces & members
A workspace is the unit of isolation and collaboration. Traces, findings, annotations, behaviors, API keys, domain-context documents, and the audit log are all scoped to a workspace. You can belong to several and switch the active one; each UI-created workspace is fully isolated and starts empty until you send traces.
Invite teammates by email with a role.
A workspace you are not a member of returns 404, never 403 — so an id or header probe cannot even confirm another workspace exists.
Roles
| Role | Can do |
|---|---|
| Owner | Everything a member can, plus manage members, create/rotate API keys, change settings (judge model, rate-limit override), view the audit log, and destructive workspace ops. One owner per workspace in v1. |
| Member | Full read-write, including actions that spend LLM budget: judging, detection, investigation, chat. Cannot manage membership. |
| Viewer | Read-only. Sees all workspace data but cannot create, update, delete, or trigger any budget-spending action. Budget-spending controls are hidden, not just disabled. |
Invites are for member or viewer only; acceptance is bound to the invitee's verified email.
Owner-visible actions — member changes, key lifecycle, settings changes — are recorded in an exportable audit log.
API keys
A workspace API key authenticates the ingest and read endpoints via an Authorization: Bearer header; the SDKs also read it from the KALMIA_API_KEY env var.
Keys have the form kal_live_… and are shown once at creation — only a display prefix and a hash are stored, so a lost key must be rotated, never recovered.
- One active key per workspace. Creating a key when one already exists returns a conflict; use rotate.
- Rotation is an atomic cutover — it revokes all active keys and mints a replacement in one transaction.
- A revoked key gets a rotation-specific error, so you can tell “rotated” apart from “invalid”.
- Key lifecycle is owner-only and managed from Settings → API Keys.
Scopes
A key is created with one of two scopes, chosen in Settings → API Keys before generating:
A write-only key can POST traces (SDK and OTel) and feedback, but every read or analysis surface rejects it with 403 and a message naming the scope - listing traces, MCP, verdicts, investigation, and chat all require a full key.
The scope is a property of the single active key, not a second key: rotation preserves the current scope, and swapping scopes is a rotation with the other scope selected.
GitHub repo connect
A workspace can connect one repository, read-only, so confirmed run flags can generate fix recommendations grounded in your actual code. Connect from Settings → GitHub: install the Kalmia GitHub App, and if the installation grants more than one repository, pick the one this workspace's traces belong to. Everyone sees the connection state; connecting and changing the repository are owner-only.
- Access is read-only by construction: the GitHub App holds only the
Contents: Read-onlypermission. Nothing in Kalmia ever writes to your repo. - File reads are pinned to one commit SHA per recommendation and capped (40 files, 300 KB per file), and repo content is treated as untrusted input.
- If the app is uninstalled, suspended, or loses access to the repo, the connection shows as needing reauthorization — reconnect from the same panel.
- To switch to a repository the installation does not yet grant, first edit the installation's repository access on GitHub, then use Change repository.
Rate limits
Rate limiting is scoped per (operation group, workspace), with two sliding windows that must both pass - a per-minute burst limit and a per-day sustained limit. Each group has its own budget, so a bulk eval run or an ingest flood can never starve interactive judging:
On top of the request windows, each user has a spend budget shared by every workspace they own, kept by day and by month: every model call's actual token usage is priced and accumulated, and model-backed operations are refused once the day's or the month's cost reaches the budget. It is separate from credits. A credit pays for an action whatever it costs to run, and the cost follows the size of your traces; the budget bounds that cost. Normal use does not meet it, and an account that does has unusually large traces and belongs on a bigger plan:
Creating more workspaces does not add budget.
Ingest, MCP reads, and dashboard traffic spend no model tokens and are never blocked by it - your traces keep flowing even when analysis is paused.
The daily budget resets at UTC midnight, the monthly one on the 1st (UTC).
A refusal answers 429 and says which of the two it was.
Owners can raise or lower the interactive burst in workspace settings.
When a limit is hit, the API returns 429 with a retryAfter and IETF RateLimit-Policy / RateLimit headers.
The ingest budget is deliberately generous - each request is one admission regardless of how many traces it carries (the per-request caps are size and trace count), so normal traffic never hits it, and the ingest gate fails open if the limiter backend is unavailable rather than dropping your traces.
Dashboard reads and the API read endpoints (the trace list and single-trace fetch) are not currently rate limited.
See the API reference for the response shape.
Plans & credits
Your account has a plan and a credit balance. Credits belong to you, not to a workspace: every workspace you own draws from the same balance, so creating workspaces never multiplies them. See them in Settings → Plan & credits.
- Monthly credits reset at the start of each monthly period; unused ones do not carry over. They are used before sign-up credits.
- Upgrade from Settings → Plan & credits; payment goes through Stripe Checkout, and your new plan's full monthly credits arrive as soon as the payment clears. Paid plans renew on your billing date, and each renewal payment brings that month's credits.
- Manage billing (in the same section) opens Stripe's billing portal: switch between Plus and Pro, update your card, download invoices, or cancel.
- Moving from Pro down to Plus takes effect at the end of the month you already paid for; until then you keep Pro and its credits.
- A cancelled plan stays active until the end of the paid month, then the account returns to Free.
- Ingesting, storing and browsing traces never uses credits.
What credits pay for
- An action is charged when it starts, to the owner of the workspace it runs in, whoever on the team started it.
- A new workspace starts with scheduled sweeps off, so nothing spends credits until you ask: run one sweep from the Scan tab, or pick a schedule in Settings → Monitoring. Weekly is about 4 sweeps a month, daily about 30.
- A sweep is priced by the traces it checks: those that arrived since the last sweep, up to the audit allowance in Settings → Monitoring. Run a sweep states the number and the price before it starts.
- When the balance covers only part of a sweep, the sweep checks as many traces as it can pay for, the failed and flagged ones first, and its row in the history says so. It does not run only when the balance cannot cover a single trace.
- A sweep that finds no new traces is free.
- The investigations a sweep or an anomaly scan runs on the traces it flags are part of its price.
- Starting an investigation costs the same as a trace analysis. Each question after that, in the app, over MCP, or to the Slack bot, costs one question.
- Behavior detection, a prompt audit and the search API are priced by the traces they read.
- An eval case is charged when it is scored, which happens on its own when the run arrives. A case with no criteria is free.
- If an action does not deliver (a sweep that could not complete, an analysis that produced no verdict, a question that got no answer), its credits are returned. Both the charge and the refund appear under Settings → Plan & credits.
- When the balance cannot cover an action, it does not run, and the app says why. The API answers
402with"code": "insufficient_credits". A scheduled sweep that cannot pay for a single trace is recorded in the sweep history with that reason and tried again at the next interval; the traces it skipped are covered by the next sweep that runs. An eval case that could not be scored is marked "needs attention" and can be re-scored later. - Free: sending, storing and browsing traces, searching the trace list, the first fix suggestion on an error group, live run flags, and anything else Kalmia does on its own for the traces you send.
- The Enterprise plan is not metered by credits.