# Kalmia

Kalmia is an observability and evaluation platform for LLM agents.
It captures your agent runs, tells you **why** a run failed, grades behavior at scale, and surfaces anomalies you didn't know to look for — so you get from a failing run to the list of issues it hit in seconds, not minutes.

## What is Kalmia

An agent run is hard to debug because the failure is buried in a long chain of model messages, tool calls, and retrievals.
Kalmia ingests that run as a **trace** — a tree of **spans**, one per step — and then reasons over it: an automatic judge issues a **verdict** (healthy, needs attention, or failed) that lists every confirmed issue with its spans cited, a conversational **investigator** lets you ask follow-up questions grounded in the trace, and batch **evaluation** turns your own test set into a graded scoreboard.

You send traces from the [Kalmia SDK](https://www.kalmia.dev/docs/capture.md#sdk) (Python or TypeScript) or from any [OpenTelemetry-instrumented](https://www.kalmia.dev/docs/capture.md#opentelemetry) library — LangChain, LlamaIndex, CrewAI, and more.
Everything after ingestion — verdicts, evals, detection, analytics — works the same regardless of how the trace arrived.

> **Who this is for:** Kalmia is built for AI/ML engineers debugging, regression-testing, and optimizing LLM agents. It reads like an instrument for an investigation, not a dashboard that buries the signal.

## The core loop

Everything in Kalmia serves one loop, from a single run up to a whole test suite:

1. **Capture**: Instrument your agent once. Every LLM call, tool execution, and turn is captured as a trace.

2. **Investigate**: Open a failing run. Read the automatic verdict, then ask the investigator “what went wrong?” and get a grounded, cited answer.

3. **Evaluate**: Turn your own eval set into a graded scoreboard — per-criterion grades, root-cause diagnosis per failure, and variant-over-variant comparison.

4. **Detect**: Define a behavior in plain language and let Kalmia find every trace that shows it, or run an anomaly scan to surface the failures you didn't define.

5. **Analyze**: Zoom out to trends — cost, latency, error rate, and quality over time — and drill from any datapoint back into the traces behind it.

## Feature map

Jump straight to any area:

- [Capture traces](https://www.kalmia.dev/docs/capture.md): SDK wrappers, agent tracing, OpenTelemetry libraries, Braintrust fan-out, and direct ingestion.
- [Core concepts](https://www.kalmia.dev/docs/concepts.md): Traces and spans, variants, workspaces, members, API keys, and rate limits.
- [Watch](https://www.kalmia.dev/docs/watch.md): Live runs as they stream in, run flags, narration, and repo-grounded fix recommendations.
- [Investigate](https://www.kalmia.dev/docs/investigate.md): Automatic verdicts that list every confirmed issue, and the streaming, grounded trace investigator.
- [Issues tab](https://www.kalmia.dev/docs/issues.md): Every issue across your runs grouped by failure, with severity, verification, and a fix recommendation.
- [Evaluate](https://www.kalmia.dev/docs/evaluate.md): The judge, behaviors and rules, domain context, graded evals, and variant comparison.
- [Detect](https://www.kalmia.dev/docs/detect.md): Plain-language behavior detection, multi-source anomaly scanning, and scheduled monitoring with digests by email, Slack, or Microsoft Teams.
- [Analytics](https://www.kalmia.dev/docs/analytics.md): Metrics compared by variant or model, estimated cost, and production trends.

## Your first trace

Three lines to start collecting traces from existing code.
Install the SDK, initialize the logger with your workspace API key, and wrap your model client — every call is captured automatically.

```python
from kalmia import init_logger, wrap_anthropic, traced
import anthropic

# Point traces at your workspace (or set KALMIA_API_KEY)
init_logger(api_key="kal_live_your_api_key", base_url="https://www.kalmia.dev")

# Wrap your client — every LLM call is now auto-traced
client = wrap_anthropic(anthropic.Anthropic())

# Group a whole agent run into one trace
@traced(name="my-agent-run")
def run(prompt):
    response = client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    )
    return response.content[0].text

run("What is the capital of France?")
```

```typescript
import Anthropic from "@anthropic-ai/sdk";
import { initLogger, wrapAnthropic, traced } from "kalmia-sdk";

// Point traces at your workspace (or set KALMIA_API_KEY)
initLogger({ apiKey: "kal_live_your_api_key", baseUrl: "https://www.kalmia.dev" });

// Wrap your client — every LLM call is now auto-traced
const client = wrapAnthropic(new Anthropic());

// Group a whole agent run into one trace
await traced(async () => {
  const response = await client.messages.create({
    model: "claude-sonnet-4-20250514",
    max_tokens: 1024,
    messages: [{ role: "user", content: "What is the capital of France?" }],
  });
  return response.content[0].text;
}, { name: "my-agent-run" });
```

Your API key comes from **Settings → API Keys** in the dashboard.
Open the workspace and your traces stream in live, each preprocessed with metrics — duration, tokens, tool calls, and cost.

## Where to next

- [Agent setup](https://www.kalmia.dev/setup.md) - one prompt for Claude Code, Cursor, or Codex that instruments your repo and verifies the first trace.
- [Quickstart](https://www.kalmia.dev/docs/quickstart.md) — the full five-minute setup, including variant tagging.
- [Capturing traces](https://www.kalmia.dev/docs/capture.md) — every way to get runs into Kalmia.
- [Investigate](https://www.kalmia.dev/docs/investigate.md) — verdicts and the trace investigator.
- [Evaluate](https://www.kalmia.dev/docs/evaluate.md) — graded evals, behaviors, and rules.
- [API reference](https://www.kalmia.dev/docs/api-reference.md) — every REST endpoint and the MCP server.
