Kalmia
Kalmia is an observability and evaluation platform for LLM agents. It captures your agent runs, tells you why a run failed, grades behavior at scale, and surfaces anomalies you didn't know to look for — so you get from a failing run to the list of issues it hit in seconds, not minutes.
What is Kalmia
An agent run is hard to debug because the failure is buried in a long chain of model messages, tool calls, and retrievals. Kalmia ingests that run as a trace — a tree of spans, one per step — and then reasons over it: an automatic judge issues a verdict (healthy, needs attention, or failed) that lists every confirmed issue with its spans cited, a conversational investigator lets you ask follow-up questions grounded in the trace, and batch evaluation turns your own test set into a graded scoreboard.
You send traces from the Kalmia SDK (Python or TypeScript) or from any OpenTelemetry-instrumented library — LangChain, LlamaIndex, CrewAI, and more. Everything after ingestion — verdicts, evals, detection, analytics — works the same regardless of how the trace arrived.
Who this is for
Kalmia is built for AI/ML engineers debugging, regression-testing, and optimizing LLM agents. It reads like an instrument for an investigation, not a dashboard that buries the signal.The core loop
Everything in Kalmia serves one loop, from a single run up to a whole test suite:
Capture
Investigate
Evaluate
Detect
Analyze
Feature map
Jump straight to any area:
SDK wrappers, agent tracing, OpenTelemetry libraries, Braintrust fan-out, and direct ingestion.
Traces and spans, variants, workspaces, members, API keys, and rate limits.
Live runs as they stream in, run flags, narration, and repo-grounded fix recommendations.
Automatic verdicts that list every confirmed issue, and the streaming, grounded trace investigator.
Every issue across your runs grouped by failure, with severity, verification, and a fix recommendation.
The judge, behaviors and rules, domain context, graded evals, and variant comparison.
Plain-language behavior detection, multi-source anomaly scanning, and scheduled monitoring with digests by email, Slack, or Microsoft Teams.
Metrics compared by variant or model, estimated cost, and production trends.
Your first trace
Three lines to start collecting traces from existing code. Install the SDK, initialize the logger with your workspace API key, and wrap your model client — every call is captured automatically.
from kalmia import init_logger, wrap_anthropic, tracedimport anthropic
# Point traces at your workspace (or set KALMIA_API_KEY)init_logger(api_key="kal_live_your_api_key", base_url="https://www.kalmia.dev")
# Wrap your client — every LLM call is now auto-tracedclient = wrap_anthropic(anthropic.Anthropic())
# Group a whole agent run into one trace@traced(name="my-agent-run")def run(prompt): response = client.messages.create( model="claude-sonnet-4-20250514", max_tokens=1024, messages=[{"role": "user", "content": prompt}], ) return response.content[0].text
run("What is the capital of France?")Your API key comes from Settings → API Keys in the dashboard. Open the workspace and your traces stream in live, each preprocessed with metrics — duration, tokens, tool calls, and cost.
Where to next
- Agent setup - one prompt for Claude Code, Cursor, or Codex that instruments your repo and verifies the first trace.
- Quickstart — the full five-minute setup, including variant tagging.
- Capturing traces — every way to get runs into Kalmia.
- Investigate — verdicts and the trace investigator.
- Evaluate — graded evals, behaviors, and rules.
- API reference — every REST endpoint and the MCP server.