REST API
Everything Kalmia exposes over HTTP.
Ingestion and read endpoints authenticate with your workspace API key; dashboard endpoints (evals, members, scans) authenticate with your logged-in session.
All endpoints are served from your Kalmia instance at https://www.kalmia.dev.
Authentication
There are two authentication modes, and each endpoint accepts one or both:
API keys are workspace-scoped and shown once at creation (see API keys).
Every trace-scoped route re-verifies the trace belongs to the resolving workspace, so a key can never trigger paid analysis on another workspace's trace.
A workspace you don't belong to returns 404, never 403.
Capture is SDK-only
The legacy LLM-proxy capture endpoints (the OpenAI-compatible chat-completions path and the Anthropic-compatible messages path under/api/v1) are disabled and return 410 Gone with an endpoint_gone error body.
Capture traces with the SDK or an OTel exporter instead.OpenAPI spec
The key-authenticated API is described by a public OpenAPI 3.1 document at a stable URL, with no login and no password gate, so you can generate a client, load it into a request tool, or hand it to a coding agent instead of probing:
curl https://www.kalmia.dev/api/v1/openapi.jsonUnknown query parameters are rejected
Every read endpoint validates parameter names, not just values. A request like?since=7d or ?search=timeout gets a 400 whose body lists unknownParams and supportedParams.
A filter is never silently ignored, so an unfiltered page can never masquerade as a filtered one.
A path under /api that no endpoint owns answers 404 with the same JSON shape (error, requestId), never an HTML page.Ingestion & read
Authenticate with your workspace API key.
/api/otel/v1/tracesAPI keyIngest OpenTelemetry spans (OTLP/HTTP, protobuf or JSON). Maps GenAI + OpenInference attributes. ≤5 MB and ≤200 distinct traces per request, ≤10k spans per trace; exporter-split traces are merged by span id.
/api/v1/traces/ingestAPI keyIngest pre-built trace objects. Single trace or array (≤100 per request). Returns { ingested, traces, rejected, warnings }.
/api/v1/traces/{id}/feedbackAPI keyAttach or update end-customer feedback for a trace: { rating: 'up' | 'down', comment }. Provided fields overwrite the stored state, omitted fields keep it; no history. Accepted even while the trace is still ingesting. See Capture → Customer feedback.
/api/v1/replay/ingestAPI key (write-only allowed)Push a browser session recording (rrweb events) captured by the SDK's recordBrowser; the SDK builds the body. ≤25 MB per request (413 above); a malformed body is 400. Distilled into evidence for the investigator and, for a watchable capture, stored for the in-app player. Session replay is currently switched off in Kalmia, so nothing is linked until it returns; see Capture → Session replay.
/api/v1/tracesAPI key (full)List captured traces for the key's workspace, newest-arrival first. SQL-side filters: ?provider=, ?status=, and ?from=/?to= — an inclusive ISO window on ingested_at (server arrival time, also the sort and cursor axis), not on the caller-supplied created. ?status= takes health values (error | ok, derived from whether the run itself failed: its root span errored or it died on its last span; recovered mid-run errors do not count) or lifecycle values (in_progress | completed | stalled — the value items report in their status field, so items matched by ?status=error still show their lifecycle status). Eval traces are excluded unless ?includeEval=true. Paginated: ?limit= (default 100, max 500) and ?cursor=; the response carries nextCursor (null on the last page). count is the size of this page, not a workspace total — use /summary for totals. status, from, to, limit, and cursor are validated and unknown parameter names are rejected — a bad filter gets 400, never a silent ignore. Returns slim headers with span counts. Write-only keys get 403.
/api/v1/traces/summaryAPI key (full)One aggregate over a time window, computed SQL-side: traceCount, erroredTraceCount (runs whose root span errored or that died on their last span; recovered mid-run errors do not count), retriedTraceCount, errorRate (erroredTraceCount / traceCount), costUsdSum with pricedCount, the cost split (llmCostUsdSum, toolCostUsdSum, toolCostByService — e.g. how much Exa cost this week) and its basis (toolCostReportedUsdSum = stated by the provider, toolCostRateCardUsdSum = estimated at the published rate, toolCostBasisByService per key), duration p50/p95, LLM latency p95 with llmLatencySampleCount (how many traces it rests on; null p95 + 0 samples means LLM durations are not being recorded), usage (totalTokensSum and turnsSum with tokenSampleCount / turnSampleCount, the traces each rests on), and per-score averages. ?from=/?to= required (ISO, half-open window on ingested_at, ≤92 days); optional ?model=, repeated ?variants=. Returns { window, current, prior, priorStatus, priorWindow } — prior is the equal-length window just before, for period-over-period deltas; when its roll-up cannot be computed (priorStatus: unavailable) it is null, never zeros. Eval traces are always excluded. Answers fleet-wide questions in one call instead of paging the workspace.
/api/v1/traces/{id}API key (full)Fetch one trace in full: every span with complete input/output content, tool arguments and results, turn structure, and metrics — the same preprocessed shape the dashboard's trace view renders. 404 for an unknown id. Write-only keys get 403.
/api/traces/analyticskey / sessionTime-bucketed per-variant trend series: latency p50/p95, throughput, tokens, cost, error rate, scores. ?from=/?to= required (ISO, ≤92-day window); optional ?model=, repeated ?variants=. Siblings: /facets (distinct models/variants in the window; takes only ?from=/?to=, a ?model= or ?variants= filter is a 400), /summary (window + prior-window KPIs), /behavior-rate (per-variant failed-verdict rate over time, one SQL aggregation over the whole window; window.traceCount says how many traces it rests on).
/api/annotationskey / session / memberRead (?traceId= or ?traceIds=a,b), create ({ traceId, label, color?, note? }), or delete ({ traceId, annotationId }) behavior annotations on traces. Reads admit viewers and keys; writes need owner/member or a full-scope key.
/api/annotations/behaviorskey / session / memberManage behavior definitions ({ label, description?, critical? }). DELETE also removes the label's annotations and 409s while the behavior is attached to eval cases. Reads admit viewers and keys; writes need owner/member or a full-scope key.
/api/annotations/labelskey / sessionAll unique annotation labels in the workspace (autocomplete source).
Read traces
# Newest error traces, 25 per pagecurl "https://www.kalmia.dev/api/v1/traces?status=error&limit=25" \ -H "Authorization: Bearer kal_live_your_api_key"# → { "traces": [{ "id", "created", "ingested_at", "status", "metadata", "spanCount" }], "count", "nextCursor" }# Pass nextCursor back as ?cursor= to fetch the next (older) page.
# Full content of a single tracecurl "https://www.kalmia.dev/api/v1/traces/trace-1" \ -H "Authorization: Bearer kal_live_your_api_key"# → { "trace": { "spans": [...], "metrics": {...}, "fullInput", "fullOutput", ... } }
# Last 7 days in one call: counts, errors, cost, latency, plus the prior weekcurl "https://www.kalmia.dev/api/v1/traces/summary?from=2026-08-25T00:00:00Z&to=2026-09-01T00:00:00Z" \ -H "Authorization: Bearer kal_live_your_api_key"# → { "window", "current": { "traceCount", "erroredTraceCount", "errorRate", "costUsdSum", "pricedCount", "durationP95", ... }, "prior", "priorWindow" }
# A guessed filter name fails loudly instead of returning the full listingcurl "https://www.kalmia.dev/api/v1/traces?since=7d" \ -H "Authorization: Bearer kal_live_your_api_key"# → 400 { "error": "Unknown query parameter: "since". Supported query parameters: provider, status, from, to, limit, cursor, includeEval.", "unknownParams": ["since"], "supportedParams": [...] }Ingest a trace
curl -X POST https://www.kalmia.dev/api/v1/traces/ingest \ -H "Content-Type: application/json" \ -H "Authorization: Bearer kal_live_your_api_key" \ -d '{ "id": "trace-1", "input": [{"role": "user", "content": "Hello"}], "output": "Hi there!", "metadata": {"provider": "openai", "model": "gpt-4o"}, "spans": [{ "id": "span-1", "span_id": "span-1", "root_span_id": "span-1", "name": "openai chat", "span_attributes": {"type": "llm"}, "metrics": {"prompt_tokens": 10, "completion_tokens": 5} }] }'Analysis & agent
These spend model budget and accept either an API key or a session (owner/member). Viewers can read but not trigger analysis.
/api/v1/verdicts?traceId=key / sessionThe current verdict for a run (status, trace summary, confirmed issues). A read never triggers analysis: ingested runs are judged on arrival, so { verdict: null } means judging is still pending, or the run was examined by a monitoring sweep's audit, which records issues (GET /api/errors) rather than a verdict. Poll again or use POST /analyze. skipAnalysis=1 is accepted for compatibility (any other value is a 400) and changes nothing for an ingested run.
/api/v1/verdictskey / memberRecord a trust signal on a verdict: { traceId, response: agreed | dismissed }.
/api/v1/traces/{id}/analyzekey / member(Re-)run the judge for a trace. Async by default and durable: the run is queued as a job and 202 returns { requestId, jobId, status, statusUrl }. One active run per trace — a repeat POST while one is queued or running joins it (inFlight: true, with the original requestId) instead of spending twice. Pass ?wait=1 to block for the finished verdict. Idempotent, deduplicated within 30s.
/api/v1/traces/{id}/analyzekey / any memberThe state of the latest analysis run for a trace: status none | queued | running | done | failed, the job (attempts, queuedAt, startedAt, updatedAt, lastError — e.g. a lease expiry when a runner was killed), and the current verdict. A done run whose verdict has status null and analysisFailedAt finished but could not judge the run. ?requestId= (from the POST response) pins the lookup to that run. This is how to tell a slow run from a dead one.
/api/v1/traces/{id}/investigatekey / memberStart a multi-step investigation thread for a trace. Returns { thread } with the thread id; read it back via GET /api/agent/threads/{threadId}. 202 { inFlight: true } while a previous investigation of the trace is still running.
/api/agent/threads/{threadId}key / sessionRead an investigation thread back: status, messages, and findings. No spend, viewer-readable — the read-back counterpart of /investigate.
/api/agent/threads/{threadId}/messageskey / memberAsk a follow-up on an investigation thread: { message }. Resumes the loop and returns the updated { thread }; 202 { inFlight: true } while a previous follow-up is still running. Agent proposals can only be confirmed from the dashboard (session-only) — confirming spend-bearing actions requires a human.
/api/memory/citation-ratekey / sessionHow often investigations cite workspace memory: ?from=/?to= required (ISO, ≤92-day window). Returns { numerator, denominator, servedZero, rate } - finished threads shown at least one memory entry (denominator) vs. those that cited one (numerator); servedZero counts finished threads shown no memory, and rate is null when nothing was served. No spend, viewer-readable. The window is bounded by trace retention.
/api/v1/traces/{id}/chatkey / memberStreaming (SSE) investigator chat. GET rehydrates the thread; POST sends a message; DELETE clears it. GET is viewer-readable.
/api/v1/anomaly-findings?scanId=key / memberList persisted anomaly findings for a scan. Optional ?variant= filter; pass the literal value none to select workspace-wide findings. Regression investigations file under ?scanId=regression. Each finding carries clusterKey (the issue's identity across scans) and recursFindingId — the earlier finding under the same cluster key whose title it inherited, so one recurring issue keeps one name; group on clusterKey to count recurrences.
/api/v1/anomaly-findingskey / memberDismiss or restore a finding: { findingId, dismissed: true | false }.
/api/v1/monitoringkey / member (GET), key / owner (PATCH)Scheduled monitoring: sweeps, cadence, allowances, email digests, the Slack and Teams destinations (never the webhook URLs), and delivering - true when a connected channel receives each digest. PATCH is a partial update; chat delivery has no switch, connect a channel instead.
/api/v1/monitoring/teamskey / ownerConnect a Microsoft Teams channel from a Workflows webhook URL: { webhookUrl, label? }. Proven with a 'Kalmia connected' card before it is stored.
/api/v1/monitoring/slack-linkkey / ownerStart the Slack connect flow for a workspace owner: { ownerEmail }. Returns the 30-minute link that owner opens to pick the channel.
/api/annotations/detect-runs/{runId}key / memberPoll a batch behavior-detection run's progress and results.
/api/annotations/detectkey / memberSingle-trace AI behavior detection; auto-annotates on a match. Spend is bounded by the agent rate-limit group and the daily spend budget.
/api/annotations/detect-batchkey / memberAsync batch detection (≤500 candidates). Returns { runId, total }; poll detect-runs/{runId} for progress. Spend runs in background jobs, bounded by the rate-limit groups and the daily spend budget.
/api/v1/anomaly-scanskey / memberTrigger an anomaly scan: optional { variant?, from?, to? } scope/window (400 on a malformed window). Findings land on GET /api/v1/anomaly-findings.
/api/v1/regressionsmemberInvestigate why one variant is worse than another: { variantA, variantB, windowStart?, windowEnd? }. Pins both populations (5 to 100 runs per side), files an investigating finding under scanId=regression, and runs a durable background investigation. Returns { findingId, existing }.
/api/traces/searchkey / memberLLM-ranked semantic search over the traces you supply: { query, traces, limit }, at most 100 traces. Returns scored results with reasoning and summaries. Costs credits by the number of traces supplied (402 when they cannot be covered); the session path requires the x-workspace-id header.
/api/v1/anomaly-scanskey / memberList past anomaly scans (newest first, up to 50): scope, window, status, finding counts. Trigger a scan via POST /api/v1/anomaly-scans.
/api/v1/monitor-sweepskey / member (POST: owner)GET: scheduled-sweep history with digests: window, status, each sweep's classified digest with delivery outcome, and coverage — tracesSeen, tracesTriaged, triageBudget, auditCoverage (0..1), triagePartial, and a hint when the audit budget capped triage below the window's volume (raise auditsPerDay in Settings → Monitoring; default 250/day, max 2000). Also returns monitoring: state (on | off), offSince, coveredThrough, nextDueAt, overdue, and gaps — the stretches no completed sweep covered, e.g. a pause. POST (owner): start one sweep by hand, at the price of one sweep. With no body it covers the traces since the last completed sweep (at most the last 7 days); with { from, to } it backfills exactly that window to close a gap (clamped to retention, allowances prorated by length). Both work whether or not scheduled sweeps are on. 402 when the owner's credits cannot cover a sweep, 409 while a sweep is in flight, 422 when there are no new traces to sweep.
/api/prompt-audit/runskey / memberList prompt-audit runs, or start one: { variant, from, to, sampleSize } (sampleSize is silently clamped to 200). Returns { runId }. Viewers can read via GET.
/api/prompt-audit/runs/{runId}key / sessionThe full audit report: run status, checklists, per-trace verdicts with cited violations. POST /retry re-judges failed traces.
Issues
The grouped issue surface behind the Issues tab.
Reads are no-spend, so viewers keep access. from and to are required ISO timestamps bounding a window of at most 366 days.
/api/errors?from=&to=key / sessionOne row per issue fingerprint in the window, worst severity and most runs first: { groups, facets }. Filters: variant, severity (low | medium | high), kind, source (comma list of signal, audit, judge), confirmed=1, dismissed=1. facets lists the variants, kinds, and sources present in the window regardless of the other filters.
/api/errors/{fingerprint}?from=&to=key / session (PATCH: member)GET: { group, occurrences, fix, runs } - the aggregate (null when the window holds no occurrences), occurrences newest first (≤500), the cached fix if any, and each run's duration and cost. PATCH { dismissed: boolean } → { dismissedAt }; a dismissed group stays hidden until a newer occurrence lands.
/api/errors/{fingerprint}/fix?from=&to=[&force=1]key / memberCached fix recommendation, or one structured judge call grounded in the window's occurrences and sample span content: { fix: { summary, likelyCause, fixes[{ title, detail, target: prompt | tool | code | infra }], model, generatedAt, sampleCount }, cached }. force=1 regenerates. 404 without occurrences, 502 when generation fails, 429 on a rate limit. Interactive judge quota.
Dashboard endpoints
These back the Kalmia dashboard and authenticate with your logged-in session and workspace membership. They are not reachable with an API key.
Traces
/api/tracessessionThe dashboard trace list: one row per trace with a workspace-wide metrics rollup. A row is the trace's summary without its spans, plus spanCount and searchText (the distinct span names, tool names and error messages); the spans come with GET /api/traces/{id}. Keyset-paginated, newest first (?limit= default 100 max 500, ?cursor= → nextCursor); ?from=/?to= limit the list to a time window and ?variant= to one variant (repeat it for several, up to 50), alone or together, and both page by cursor the same way (truncated says the first page was not all of it); ?includeEval=true opts eval traces in.
/api/traces/{id}sessionOne trace in full for the active workspace (the x-workspace-id header): every span's complete content, tool arguments and results, run flags, and metrics - the heavy fields the list strips. The dashboard fetches it only for the trace you open. ?view=summary returns whole-run metrics and span previews instead, with lazySpans: true when the trace has more than 100 spans; ?spanOffset=N (a multiple of 100) then returns { spans, offset, total } with the full content of those 100 spans, 409 if the trace changed since the summary was read. 404 for an unknown trace or a workspace you are not in, never 403.
/api/traces/cited-spanssessionResolve the spans a scan finding cites: { traceIds (≤200), spanIds (≤500) } → { spans: { [spanId]: { traceId, label } } }, read from those traces in the active workspace. A span none of the traces holds is left out. Read-only, no spend.
/api/traces/{id}/replay-linksession{ replayUrl } for the trace's session recording in PostHog, or null when there is nothing to link (no completed replay row, replay off, or an unknown trace - the null shape is the common case, not an error). Requires the x-workspace-id header.
/api/traces/{id}/replay-recordingsession{ events } - the stored rrweb recording of a Kalmia watchable capture, decoded server-side so the in-app player receives plain JSON. 404 when the trace has no recording. Requires the x-workspace-id header.
Evals
The eval editor and scoreboard. All eval routes are deliberately session-only: an API-key runner submits traces and reads grades over MCP, but can never trigger judge spend or influence the scores it is graded by. Reads admit viewers; mutations require owner or member.
/api/evals/casessession / memberList cases with criteria counts, or create one manually (duplicate externalId → 409). Cap: 20 effective criteria per case.
/api/evals/cases/{caseId}session / memberFull case editor model in one fetch; rename (externalId is immutable); delete — 409 if graded results reference the case.
/api/evals/cases/{caseId}/behaviorsmemberAttach or detach a workspace behavior, with expectation occurs | does_not_occur. Idempotent; attach 422s past the cap.
/api/evals/cases/{caseId}/criteriamemberAdd, edit, and delete case criteria (≤2,000 chars each). Edits never rewrite past results.
/api/evals/cases/{caseId}/criteria/reordermemberReorder a case's criteria: { orderedIds }, which must be a permutation of the case's current criterion ids. A stale editor (a concurrent add or remove) gets 400 rather than silently corrupting positions.
/api/evals/runssessionRecent run sets, newest first, with derived outcome counts. ?limit= up to 100 (default 20), ?offset=.
/api/evals/runs/{runId}sessionThe full run report with inline evidence — readable even after the underlying traces expire.
/api/evals/compare?base=&candidate=sessionFixed / regressed / both-pass / both-fail comparison of two run ids, with a judge-model-changed badge.
/api/evals/results/{resultId}/diagnosememberOn-demand failure diagnosis for a failed result. 409 if the trace expired or the result is not failed.
/api/evals/results/{resultId}/rescorememberRe-enqueue judging for unscored criteria (202). Reviewer-touched scores are never cleared.
/api/evals/scores/{scoreId}/reviewmemberConfirm or override one criterion score: { action: 'confirm' } or { action: 'override', pass }. Latest action wins.
Search
/api/semantic-searchmemberJudge traces against a natural-language behavior query. { query, traceIds, limit ≤200 }: the server summarizes each stored trace (up to limit of them, ≤1,000 ids per request). { traceSummaries } instead supplies pre-built summaries. Spends judge budget.
Workspace & collaboration
/api/workspaces/{id}/membersownerList members.
/api/workspaces/{id}/members/{userId}ownerChange a member's role, or remove them. The last owner cannot be demoted or removed.
/api/workspaces/{id}/invitesownerList invites, or send one by email (member/viewer only).
/api/workspaces/{id}/invites/{inviteId}ownerRevoke a pending invite.
/api/invites/acceptsessionAccept pending invites matching your verified email.
/api/workspaces/{id}/settingsmember / ownerRead the workspace settings in one fetch: judge model (with the picker's options and whether it is the default), judge engine, session-replay config, scheduled-sweep config, your role, and the current rate-limit status. PATCH (owner) sends exactly one of judgeModel, judgeEngine, rateLimitOverride, replayConfig, or sweepConfig; each setting updates independently, so changing one never resets another.
/api/workspaces/{id}/githubmember / ownerRead the GitHub connection state (?include=repos for the picker), or select the connected repository (owner). Connect via Settings → GitHub.
/api/github/connect?workspace=owner (browser)Starts the GitHub App install: mints a short-lived state token bound to the workspace and your session, then redirects your browser to GitHub's install page. A navigation, not a fetch; failures redirect back to settings with an error code.
/api/github/setupbrowser callbackGitHub's post-install callback. Nothing is stored until the state verifies, the live session matches the user who started the flow, that user is still an owner, and the completing user is proven to have access to the installation.
/api/workspaces/{id}/slackmember / ownerGET: the Slack destination as any member may see it - connected, channel, team, and whether the webhook is broken; the webhook URL itself never crosses this boundary. DELETE (owner): disconnect; digests stop posting to Slack from the next sweep.
/api/workspaces/{id}/slack/connectownerStart the Slack connect flow: returns { url }, the Slack authorize page bound to this workspace and your session; the settings page navigates there and Slack's own consent screen picks the channel. 422 when Slack is not configured on the server.
/api/workspaces/{id}/teamsmember / ownerThe Microsoft Teams destination. GET: connected, label, and whether the webhook is broken; the webhook URL never crosses this boundary. PUT (owner): { webhookUrl, label? } - a Teams Workflows ('Send webhook alerts to a channel') webhook URL the owner pasted; the host is checked, a 'Kalmia connected' card is posted to prove it works, and only then is it stored encrypted (400 for a non-Workflows URL, 422 when Teams rejects or cannot be reached). DELETE (owner): disconnect; digests stop posting to Teams from the next sweep.
/api/workspaces/{id}/contextsession / memberList domain-context documents, or create one from text or markdown: { name (≤200 chars), text (≤200,000 chars), sourceType }. The document is chunked and embedded in the background and reports processing → ready | failed.
/api/workspaces/{id}/context/{docId}session / memberGET the document's editable name and raw text; PUT edits name and/or text (a text change re-embeds; a throttled re-embed returns 429); PATCH { active: boolean } switches it on or off without deleting; DELETE removes it. Viewers can read; writes need owner or member.
/api/agent/proposals/{proposalId}member (session only)Resolve a change the trace investigator proposed: { action: 'confirm' | 'decline' }. Deliberately session-only with the x-workspace-id header, so a machine can never satisfy the human-confirmation step. Exactly one of concurrent confirms applies; a lost race gets 409 with the actual resolution, and a foreign or unknown id gets 404.
/api/traces/{id}/flags/{flagId}/fix-recmemberGenerate (or retry) a repo-grounded fix recommendation for a confirmed run flag. 422 without a GitHub integration, 409 without a connected repo.
/api/workspaces/{id}/auditownerQuery and export the workspace audit log.
API key lifecycle
/api/v1/api-keysownerRead key metadata (never the key), or create a key: { name?, scope? } where scope is 'full' (default) or 'ingest' (write-only). Raw key shown once; 409 if one exists.
/api/v1/api-keys/rotateownerAtomically revoke all active keys and mint a replacement. Preserves the current scope unless a new one is passed.
Rate limits
Rate limiting is scoped per (operation group, workspace) - see Rate limits for the groups and defaults. Billable model calls (judging, detection, investigation, chat) draw from their group budgets and from the workspace owner's spend budget (actual token cost across every workspace they own, limited per day and per month); ingestion (SDK, OTel, feedback) passes a deliberately generous per-request admission gate (default 300 requests/min per workspace) that normal traffic never hits and is never blocked by the spend budget. Dashboard reads and the key-authenticated read/annotation endpoints (trace list and detail, analytics, annotations) are not currently rate limited. When a limit is exceeded, the response is:
HTTP/1.1 429 Too Many RequestsRateLimit-Policy: burst;q=120;w=60, sustained;q=5000;w=86400RateLimit: burst;r=0;t=42Retry-After: 42
{ "error": "Rate limit exceeded", "requestId": "…", "retryAfter": 42 }Under-limit responses carry no rate-limit headers. If the limiter backend is briefly unavailable, calls fail open (allowed through) rather than blocking your traffic.
MCP server
Kalmia exposes a Model Context Protocol server so your own coding agent can pull eval results, query the trace corpus, read verdicts and anomaly findings, and — with your approval — run the judge, investigate, and tag runs, all inside your repo. It's authenticated with your workspace API key and returns results as labeled data, never as instructions.
claude mcp add --transport http kalmia https://www.kalmia.dev/api/mcp \ --header "Authorization: Bearer kal_live_your_api_key"Tools
| Tool | Returns |
|---|---|
list_recent_runs | Recent eval run sets with per-status case counts |
get_run_results | Cursor-paginated case results for a run, with status counts |
get_case_diagnosis | The root-cause diagnosis for a failed case |
compare_runs | Fixed / regressed comparison of two run ids |
query_traces | Traces filtered by metric ranges, attributes (including variant), status, and time window |
aggregate_traces | Per-group counts and rates over the filtered corpus (by model, agent, variant, status, or tool) |
compare_metrics | Side-by-side metrics per variant, primary model, or project (duration, tokens, error rate, cost); can compare models within one variant |
get_trace | One run's slim summary with metrics, annotations, and customer feedback |
search_trace_content | Span ids + snippets where a keyword appears inside one trace |
get_span_detail | One span's full content: complete input/output, tool args and result (paged) |
get_verdict | A trace's current verdict: status, trace summary, confirmed issues, suggested fix |
list_anomaly_findings | Persisted detector findings: severity, type, description, trace ids (optionally only those citing one variant) |
list_issues | The Issues tab over a window, filterable by variant, kind, and severity; with compareVariant, which issues are new, shared, or gone between two variants |
get_issue | One issue: run counts, variants, its newest runs (optionally one variant's), and the cached fix |
compare_variants | 2-5 variants side by side, baseline first: failure rate, failed-verdict rate, behavior prevalence, anomaly findings, new / gone issues, and average cost, tokens, and duration, with the change against the baseline |
analyze_trace | ACTION: run the judge on one trace and return the verdict |
investigate_trace | ACTION: start or continue a multi-step investigation of one trace |
annotate_trace | ACTION: apply a behavior label to one trace (idempotent, reversible) |
get_monitoring_settings | Scheduled sweeps, email digests, the Slack and Teams destinations, and whether digests are posting to a channel |
run_sweep | ACTION: run one sweep now over the traces since the last one; costs the credits of one sweep |
configure_monitoring | ACTION: turn scheduled sweeps and digest email on or off (partial update); channel delivery follows what is connected |
connect_slack | ACTION: a 30-minute link a workspace owner opens to approve the Slack channel for digests |
connect_teams | ACTION: store a pasted Microsoft Teams Workflows webhook for digests, after a test card proves it works |
The action tools are the only ones without readOnlyHint, so MCP hosts (Claude Code and peers) ask for your approval before running them.
The three trace actions each act on exactly one trace — there is deliberately no bulk tool — and their spend rides the same rate-limit groups and daily spend budget as every other trigger; a budget refusal returns a structured rate_limited result with retryAfterSeconds.
The three monitoring actions change workspace settings only: they never touch trace data, and connect_slack cannot finish on its own — Slack requires a signed-in workspace owner to click Allow, and the callback refuses any other session.
What an agent can never do here: score, override, or change eval outcomes, agree/dismiss a verdict, or alter customer feedback — the actor being graded never holds the pen.
See Running evals → close the loop for the full workflow.