Documentation

Kalmia

Kalmia is an observability and evaluation platform for LLM agents. It captures your agent runs, tells you why a run failed, grades behavior at scale, and surfaces anomalies you didn't know to look for — so you get from a failing run to the list of issues it hit in seconds, not minutes.

What is Kalmia

An agent run is hard to debug because the failure is buried in a long chain of model messages, tool calls, and retrievals. Kalmia ingests that run as a trace — a tree of spans, one per step — and then reasons over it: an automatic judge issues a verdict (healthy, needs attention, or failed) that lists every confirmed issue with its spans cited, a conversational investigator lets you ask follow-up questions grounded in the trace, and batch evaluation turns your own test set into a graded scoreboard.

You send traces from the Kalmia SDK (Python or TypeScript) or from any OpenTelemetry-instrumented library — LangChain, LlamaIndex, CrewAI, and more. Everything after ingestion — verdicts, evals, detection, analytics — works the same regardless of how the trace arrived.

Who this is for

Kalmia is built for AI/ML engineers debugging, regression-testing, and optimizing LLM agents. It reads like an instrument for an investigation, not a dashboard that buries the signal.

The core loop

Everything in Kalmia serves one loop, from a single run up to a whole test suite:

1

Capture

Instrument your agent once. Every LLM call, tool execution, and turn is captured as a trace.
2

Investigate

Open a failing run. Read the automatic verdict, then ask the investigator “what went wrong?” and get a grounded, cited answer.
3

Evaluate

Turn your own eval set into a graded scoreboard — per-criterion grades, root-cause diagnosis per failure, and variant-over-variant comparison.
4

Detect

Define a behavior in plain language and let Kalmia find every trace that shows it, or run an anomaly scan to surface the failures you didn't define.
5

Analyze

Zoom out to trends — cost, latency, error rate, and quality over time — and drill from any datapoint back into the traces behind it.

Feature map

Jump straight to any area:

Your first trace

Three lines to start collecting traces from existing code. Install the SDK, initialize the logger with your workspace API key, and wrap your model client — every call is captured automatically.

from kalmia import init_logger, wrap_anthropic, traced
import anthropic
# Point traces at your workspace (or set KALMIA_API_KEY)
init_logger(api_key="kal_live_your_api_key", base_url="https://www.kalmia.dev")
# Wrap your client — every LLM call is now auto-traced
client = wrap_anthropic(anthropic.Anthropic())
# Group a whole agent run into one trace
@traced(name="my-agent-run")
def run(prompt):
response = client.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=1024,
messages=[{"role": "user", "content": prompt}],
)
return response.content[0].text
run("What is the capital of France?")

Your API key comes from Settings → API Keys in the dashboard. Open the workspace and your traces stream in live, each preprocessed with metrics — duration, tokens, tool calls, and cost.

Where to next

  • Agent setup - one prompt for Claude Code, Cursor, or Codex that instruments your repo and verifies the first trace.
  • Quickstart — the full five-minute setup, including variant tagging.
  • Capturing traces — every way to get runs into Kalmia.
  • Investigate — verdicts and the trace investigator.
  • Evaluate — graded evals, behaviors, and rules.
  • API reference — every REST endpoint and the MCP server.