# Capturing traces

Get your agent runs into Kalmia.
Use the **Kalmia SDK** to wrap your OpenAI or Anthropic client, point any **OpenTelemetry** library at the ingest endpoint, or POST pre-built traces directly.
However a run arrives, everything downstream — verdicts, evals, detection, analytics — works the same.

## Kalmia SDK

The SDK wraps your model client so every LLM call is captured automatically, with zero external dependencies on the Python side and automatic context propagation on the TypeScript side.
Two things to know before you start:

- **Python**: pip install kalmia — zero external dependencies
- **TypeScript**: npm install kalmia-sdk — uses AsyncLocalStorage
- **Auth**: API key required (or set KALMIA_API_KEY)
- **Base URL**: https://www.kalmia.dev

> **Get your API key:** Create a workspace API key from **Settings → API Keys**. The raw key is shown once at creation — store it in a secret. Keys look like `kal_live_…`.

## Python

### Install

pip:

```bash
pip install kalmia
```

### Wrap OpenAI

```python
from kalmia import init_logger, wrap_openai
import openai

init_logger(api_key="kal_live_your_api_key", base_url="https://www.kalmia.dev")
client = wrap_openai(openai.OpenAI())

# Every call is now auto-traced
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
)
```

### Wrap Anthropic

```python
from kalmia import init_logger, wrap_anthropic
import anthropic

init_logger(api_key="kal_live_your_api_key", base_url="https://www.kalmia.dev")
client = wrap_anthropic(anthropic.Anthropic())

message = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello!"}],
)
```

### API surface

- **init_logger(...)**: Configure destination: api_key (or KALMIA_API_KEY), base_url, metadata, streaming options
- **wrap_openai(client)**: Auto-trace chat completions (sync + async)
- **wrap_anthropic(client)**: Auto-trace message creation (sync + async)
- **@traced(name=..., metadata=...) / with traced()**: Group calls into one trace
- **current_span()**: Get the active span from context
- **current_trace_id()**: Trace id of the active run (None outside traced())
- **submit_feedback(trace_id, rating, comment)**: Attach end-customer feedback to a run

## TypeScript

The TypeScript SDK uses `AsyncLocalStorage` for automatic context propagation — no manual span passing.

### Install

npm:

```bash
npm install kalmia-sdk
```

### Wrap your client

OpenAI:

```typescript
import OpenAI from "openai";
import { initLogger, wrapOpenAI } from "kalmia-sdk";

initLogger({ apiKey: "kal_live_your_api_key", baseUrl: "https://www.kalmia.dev" });
const client = wrapOpenAI(new OpenAI());

const response = await client.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: "Hello!" }],
});
```

Anthropic:

```typescript
import Anthropic from "@anthropic-ai/sdk";
import { initLogger, wrapAnthropic } from "kalmia-sdk";

initLogger({ apiKey: "kal_live_your_api_key", baseUrl: "https://www.kalmia.dev" });
const client = wrapAnthropic(new Anthropic());

const message = await client.messages.create({
  model: "claude-sonnet-4-20250514",
  max_tokens: 1024,
  messages: [{ role: "user", content: "Hello!" }],
});
```

### API surface

- **initLogger({ ... })**: Configure destination: apiKey (or KALMIA_API_KEY), baseUrl, metadata, streaming options
- **wrapOpenAI(client)**: Auto-trace chat completions
- **wrapAnthropic(client)**: Auto-trace message creation
- **traced(callback, { name, type, metadata })**: Group calls into one trace
- **currentSpan()**: Get the active span from context
- **currentTraceId()**: Trace id of the active run (undefined outside traced())
- **submitFeedback(traceId, { rating, comment })**: Attach end-customer feedback to a run
- **replayMetadata(sessionId, provider?)**: Trace-level metadata linking a run to its browser session recording (PostHog by default); Python: replay_metadata
- **recordBrowser(page, { watchable, agentInsight, traceId })**: Record a Playwright or Puppeteer page for the run and ship the capture to Kalmia; TypeScript only

### Session replay

A browser-driving agent can attach a **session recording** to each run, so the investigator can cross-check what a tool claimed against what the page actually showed, and you can watch the run in the trace view.
There are two ways to link one:

- **PostHog**: You already record sessions with PostHog. Merge replayMetadata(sessionId) into the run's root metadata (initLogger metadata for a single-session process, or the traced root's metadata for a backend serving many users), and Kalmia links the recording by session id.
- **Kalmia capture**: Call recordBrowser(page, { watchable, agentInsight }) before the first navigation of a Playwright or Puppeteer page. It records the page with rrweb, tags the capture with the active trace id, and ships it on stop(). Both switches default to off, so a call with neither is inert.

> **Currently switched off:** Session replay is deactivated in Kalmia at the moment: the Settings → General row, the Replay button on a trace, and the server-side consumers are all off, and every workspace behaves as if replay were never enabled. The SDK helpers stay safe to call and never raise into your agent; recordings and replay metadata are simply not linked until the feature returns.

## Vercel AI SDK

To trace models used through the **Vercel AI SDK**, wrap them with `wrapAISDK` as `LanguageModelV2` middleware.
Tracing is observation-only: every streamed part is forwarded untouched, so it can never alter or break your model output.

```typescript
import { wrapLanguageModel } from "ai";
import { initLogger, wrapAISDK, endAISDKSession } from "kalmia-sdk";

initLogger({ apiKey: "kal_live_your_api_key" });

const model = wrapLanguageModel({
  model: baseModel,
  middleware: [wrapAISDK({ sessionID })], // omit sessionID for one trace per call
});

// ... use `model` with generateText / streamText as usual ...

await endAISDKSession(sessionID); // when the session finishes
```

Pass a `sessionID` to group all of a session's calls into one trace.
While the session is open its trace stays in progress on the server, so live analysis follows the session as it grows; `endAISDKSession(sessionID)` posts the completed trace and triggers post-run analysis exactly once, over the whole session.
A session you never end goes terminal via the server's stall timeout instead, and very long sessions are auto-evicted into a fresh trace segment.

## Agent tracing

Use `traced()` to capture an entire agent run — multiple LLM calls, tool executions, and the full conversation — as a single trace.
Without it, each LLM call creates its own standalone trace.
Pass `metadata` to tag a run, for example with the variant you're comparing:

```python
with traced(name="my-agent-run", metadata={"variant": "rag"}) as span:
    # ... run your agent variant ...
    span.log(output=result)
```

```typescript
await traced(async (span) => {
  // ... run your agent variant ...
  span.log({ output: result });
}, { name: "my-agent-run", metadata: { variant: "rag" } });
```

The `variant` metadata is what powers [variant comparison](https://www.kalmia.dev/docs/evaluate.md#variants) and [per-variant trends](https://www.kalmia.dev/docs/analytics.md#trends).
Any string works — a git SHA, a config name, `rag` vs `no-rag`.

## Tool spans

Wrap tool executions with `traced()` and `span_type="tool"` to capture them as tool spans alongside the model calls.
This is what makes a trace a full conversation view — model turns and tool calls interleaved.

```python
from kalmia import init_logger, wrap_anthropic, traced
import anthropic, json

init_logger(api_key="kal_live_your_api_key", base_url="https://www.kalmia.dev")
client = wrap_anthropic(anthropic.Anthropic())

TOOLS = [{
    "name": "get_weather",
    "description": "Get weather for a city",
    "input_schema": {
        "type": "object",
        "properties": {"city": {"type": "string"}},
        "required": ["city"],
    },
}]

@traced(name="weather-agent")
def run_agent(prompt):
    messages = [{"role": "user", "content": prompt}]
    while True:
        response = client.messages.create(
            model="claude-sonnet-4-20250514",
            max_tokens=1024, tools=TOOLS, messages=messages,
        )
        tool_uses = [b for b in response.content if b.type == "tool_use"]
        if not tool_uses:
            return response.content[0].text

        tool_results = []
        for tool_use in tool_uses:
            # Captured as a tool span
            with traced(name=tool_use.name, span_type="tool") as span:
                result = get_weather(tool_use.input["city"])
                span.log(input=tool_use.input, output=result)
            tool_results.append({
                "type": "tool_result",
                "tool_use_id": tool_use.id,
                "content": json.dumps(result),
            })

        messages.append({"role": "assistant", "content": [b.model_dump() for b in response.content]})
        messages.append({"role": "user", "content": tool_results})

run_agent("Compare weather in Tokyo, New York, and San Francisco")
```

```typescript
await traced(async (span) => {
  const response = await client.messages.create({
    model: "claude-sonnet-4-20250514",
    max_tokens: 1024,
    messages: [{ role: "user", content: "What's the weather?" }],
  });

  // Captured as a tool span
  await span.traced("get_weather", async (toolSpan) => {
    const result = await fetchWeather("Tokyo");
    toolSpan.log({ input: { city: "Tokyo" }, output: result });
  }, { type: "tool" });

  return response.content[0].text;
}, { name: "weather-agent" });
```

### What shows up in the dashboard

- **LLM spans**: Auto-captured, with token counts and model
- **Tool spans**: Your traced() tool executions, with input/output
- **Conversation**: Full message history in the trace view
- **Metrics**: Token counts, timing, cost per span
- **Tool cost**: Paid services (Exa, Tavily, Brave, Serper, and more) priced from the span's output; log the raw API response as the output to get the exact figure

For a paid service Kalmia does not recognize, add `cost_usd` to the logged output and it is priced like any other.
See [Reading cost](https://www.kalmia.dev/docs/analytics.md#cost) for the recognized services and the snippet.

A model call made through a client the SDK has no wrapper for can be recorded by hand: open a span with `type: "llm"` and log its usage with the same keys the wrappers emit (`prompt_tokens`, `completion_tokens`, `cache_read_input_tokens`, `cache_creation_input_tokens`), and Kalmia prices it like any other model call.
Report `prompt_tokens` as the uncached part of the prompt: for OpenAI that is `input_tokens` minus `input_tokens_details.cached_tokens` (and minus `cache_write_tokens`, which goes in `cache_creation_input_tokens`, when the response reports it).

```typescript
await traced(async (span) => {
  const turn = await myClient.generate(prompt);
  span.log({
    input: prompt,
    output: turn.text,
    metadata: { model: "my-model" },
    metrics: { prompt_tokens: turn.usage.input, completion_tokens: turn.usage.output },
  });
}, { name: "my-model call", type: "llm" });
```

## Live trace streaming

By default the SDK posts a trace exactly once, when its root `traced()` call ends.
Opt in to live streaming and completed spans are also posted in mid-run batches while the run is still going, so long runs appear in the dashboard as they happen instead of only after they finish.
See [Watching live runs](https://www.kalmia.dev/docs/watch.md) for what the dashboard does with a live run - status, flags, narration.

```python
init_logger(
    api_key="kal_live_your_api_key",
    streaming=True,               # or KALMIA_STREAMING=1
    streaming_batch_size=20,      # optional, this is the default
    streaming_flush_interval_s=5, # optional, this is the default
)
```

```typescript
initLogger({
  apiKey: "kal_live_your_api_key",
  streaming: true,                // or KALMIA_STREAMING=1
  streamingBatchSize: 20,         // optional, this is the default
  streamingFlushIntervalMs: 5000, // optional, this is the default
});
```

- **streaming / KALMIA_STREAMING**: Turn mid-run batch posting on. Off, behavior is identical to a single post at root end
- **streaming batch size / KALMIA_STREAMING_BATCH_SIZE**: Flush once this many completed-but-unsent spans accumulate (default 20)
- **flush interval / KALMIA_STREAMING_FLUSH_INTERVAL_MS**: Flush a smaller batch once this much time has passed since the last flush (default 5 s)

> **Durability is unchanged:** Mid-run batches are best-effort: a single send attempt, no retries. The final flush at root end is durable and posts the complete root plus every span not already delivered, with the SDK's full retry-and-buffer contract. A run whose mid-run batches all failed still arrives intact, and streaming never throws into your code.

## Customer feedback

Relay your end customer's reaction to a run, a thumbs rating and/or an open-text comment, and Kalmia attaches it to the run's trace, where it shows up alongside verdicts in the dashboard.
Capture the trace id during the run with `current_trace_id()` / `currentTraceId()`, store it with your own session or request record, then submit feedback from your backend whenever it arrives.

```python
from kalmia import traced, current_trace_id, submit_feedback

# During the run: capture the trace id and store it with your session.
with traced(name="my-agent-run"):
    trace_id = current_trace_id()
    # ... run your agent, persist trace_id with the session record ...

# Later, when the customer reacts (e.g. in your feedback endpoint):
result = submit_feedback(trace_id, rating="down", comment="It booked the wrong date")
if not result["ok"]:
    print("feedback not delivered:", result["error"])
```

```typescript
import { traced, currentTraceId, submitFeedback } from "kalmia-sdk";

// During the run: capture the trace id and store it with your session.
let traceId: string | undefined;
await traced(async () => {
  traceId = currentTraceId();
  // ... run your agent, persist traceId with the session record ...
}, { name: "my-agent-run" });

// Later, when the customer reacts (e.g. in your feedback endpoint):
const result = await submitFeedback(traceId, {
  rating: "down",
  comment: "It booked the wrong date",
});
if (!result.ok) console.warn("feedback not delivered:", result.error);
```

- **rating**: "up" or "down" - the customer’s thumbs
- **comment**: Open text, up to 10,000 characters
- **Merge semantics**: Provided fields overwrite the stored state, omitted fields keep it; one current state per trace, no history
- **Timing**: Accepted even while the trace is still ingesting - no wait-for-trace logic needed

> **Fails loudly in dev, never crashes in prod:** Invalid usage (no trace id, empty feedback, unknown rating) throws immediately. Delivery problems (network, auth, server errors) never throw: they come back as `{ ok: false, error }` after bounded retry, so a feedback relay inside your own request handler cannot crash your API. Prefer the SDK helpers, or POST to the endpoint directly:

- `POST /api/v1/traces/{id}/feedback` (API key): Attach or update end-customer feedback for a trace: { rating, comment }. Same auth and admission gate as trace ingest; write-only keys may submit.

## OpenTelemetry libraries

You aren't limited to the OpenAI and Anthropic wrappers.
Any OpenTelemetry-emitting LLM or agent library can report to Kalmia by pointing its OTLP exporter at the ingest endpoint — no Kalmia SDK required.
One adapter maps the **OpenInference family** (LangChain, LlamaIndex, CrewAI, …) and **native OTel GenAI** instrumentation onto Kalmia's trace model with full role, model, and token fidelity.

- `POST /api/otel/v1/traces` (API key): Ingest OpenTelemetry spans (OTLP/HTTP). Maps GenAI and OpenInference attributes onto the trace model. Bound to your workspace by API key.

### Point an OTLP exporter at Kalmia

```python
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

# e.g. OpenInference instrumentation for LangChain / LlamaIndex / CrewAI
from openinference.instrumentation.langchain import LangChainInstrumentor

provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(
            endpoint="https://www.kalmia.dev/api/otel/v1/traces",
            headers={"Authorization": "Bearer kal_live_your_api_key"},
        )
    )
)

LangChainInstrumentor().instrument(tracer_provider=provider)
# Run your agent as usual — every span now lands in Kalmia.
```

```typescript
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";

const provider = new NodeTracerProvider({
  spanProcessors: [
    new BatchSpanProcessor(
      new OTLPTraceExporter({
        url: "https://www.kalmia.dev/api/otel/v1/traces",
        headers: { Authorization: "Bearer kal_live_your_api_key" },
      }),
    ),
  ],
});

provider.register();
// Instrument your library against this provider — spans flow to Kalmia.
```

### Verified libraries

A library counts as supported when its spans render with the correct role, model, and token metrics — verified against real captures, not assumed from a name on a list.

- **LangChain**: Role + model + tokens (OpenInference)
- **LlamaIndex**: RAG spans — retriever/embedding as tools
- **CrewAI**: OpenInference family — handled by the adapter
- **Native OTel GenAI**: Any gen_ai.* provider instrumentation

> **Transport:** OTLP/HTTP in protobuf (the common exporter default) or JSON, selected by `Content-Type`. OTLP/gRPC is not accepted. Guardrails: 5 MB body and 200 distinct traces per request, 10,000 spans per trace. Exporters that split one trace across many POSTs are merged by span id.

> **Eval tagging:** The OpenTelemetry endpoint does not currently carry arbitrary metadata through to the trace, so eval run tagging (`eval.case_id` / `run_id` / `variant`) is only supported on the SDK ingest path. See [Running evals](https://www.kalmia.dev/docs/evaluate.md#evals).

## Use Kalmia alongside Braintrust

Already instrumented with Braintrust?
You don't have to replace anything.
Keep your existing instrumentation and add Kalmia as a **second OpenTelemetry export destination** on the same `TracerProvider` — the identical spans land in both Braintrust and Kalmia.
No re-instrumentation, no Kalmia SDK.

### Add a second span processor

```typescript
// instrumentation.ts
import { BraintrustSpanProcessor } from "@braintrust/otel";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";

const provider = new NodeTracerProvider({
  spanProcessors: [
    // Existing Braintrust destination — unchanged.
    new BraintrustSpanProcessor({ parent: process.env.BRAINTRUST_PARENT }),

    // Added: the same spans also go to Kalmia.
    new BatchSpanProcessor(
      new OTLPTraceExporter({
        url: "https://www.kalmia.dev/api/otel/v1/traces",
        headers: { Authorization: "Bearer kal_live_your_api_key" },
      }),
    ),
  ],
});

provider.register();
```

```python
# Add Kalmia to the provider your Braintrust OTel setup already uses —
# keep whatever Braintrust processor/instrumentation is on it.
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(
            endpoint="https://www.kalmia.dev/api/otel/v1/traces",
            headers={"Authorization": "Bearer kal_live_your_api_key"},
        )
    )
)
```

Braintrust receives the identical spans in parallel — nothing about your Braintrust data changes.
What lands in Kalmia:

- **Carries over**: Span tree, inputs/outputs, model + token metrics, metadata
- **Grouped by**: OTel trace id → one Kalmia trace
- **Project**: From the service.name resource attribute

Want to leave Braintrust entirely?
Drop the Braintrust processor and keep only the Kalmia exporter — your OpenTelemetry instrumentation now reports to Kalmia alone.

## Direct ingestion

Already have pre-built trace objects?
POST them directly to the ingest endpoint in Kalmia's trace format.
Accepts a single trace object or an array (up to 100 traces per request).
To send from an OpenTelemetry library instead, use the [OpenTelemetry endpoint](https://www.kalmia.dev/docs/capture.md#opentelemetry) above.

- `POST /api/v1/traces/ingest` (API key): Ingest pre-built trace objects. Accepts a single trace or an array. Returns which traces were ingested and which were rejected, with reasons.

cURL:

```bash
curl -X POST https://www.kalmia.dev/api/v1/traces/ingest \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer kal_live_your_api_key" \
  -d '[{
    "id": "trace-1",
    "input": [{"role": "user", "content": "Hello"}],
    "output": "Hi there!",
    "metadata": {"provider": "openai", "model": "gpt-4o"},
    "spans": [{
      "id": "span-1",
      "span_id": "span-1",
      "root_span_id": "span-1",
      "name": "openai chat",
      "input": [{"role": "user", "content": "Hello"}],
      "output": "Hi there!",
      "span_attributes": {"type": "llm"},
      "metrics": {"prompt_tokens": 10, "completion_tokens": 5}
    }]
  }]'
```

```typescript
const res = await fetch("https://www.kalmia.dev/api/v1/traces/ingest", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "Authorization": "Bearer kal_live_your_api_key",
  },
  body: JSON.stringify([{
    id: "trace-1",
    input: [{ role: "user", content: "Hello" }],
    output: "Hi there!",
    metadata: { provider: "openai", model: "gpt-4o" },
    spans: [{
      id: "span-1",
      span_id: "span-1",
      root_span_id: "span-1",
      name: "openai chat",
      input: [{ role: "user", content: "Hello" }],
      output: "Hi there!",
      span_attributes: { type: "llm" },
      metrics: { prompt_tokens: 10, completion_tokens: 5 },
    }],
  }]),
});
const { ingested, traces, rejected } = await res.json();
```

For the full trace and span field reference, see [Traces & spans](https://www.kalmia.dev/docs/concepts.md#traces).
Delivery is idempotent — the backend upserts by trace `id` within your workspace, so a duplicate delivery converges on one row rather than double-counting.
