Capture Traces

Capturing traces

Get your agent runs into Kalmia. Use the Kalmia SDK to wrap your OpenAI or Anthropic client, point any OpenTelemetry library at the ingest endpoint, or POST pre-built traces directly. However a run arrives, everything downstream — verdicts, evals, detection, analytics — works the same.

Kalmia SDK

The SDK wraps your model client so every LLM call is captured automatically, with zero external dependencies on the Python side and automatic context propagation on the TypeScript side. Two things to know before you start:

Pythonpip install kalmia — zero external dependencies
TypeScriptnpm install kalmia-sdk — uses AsyncLocalStorage
AuthAPI key required (or set KALMIA_API_KEY)
Base URLhttps://www.kalmia.dev

Get your API key

Create a workspace API key from Settings → API Keys. The raw key is shown once at creation — store it in a secret. Keys look like kal_live_….

Python

Install

pip install kalmia

Wrap OpenAI

from kalmia import init_logger, wrap_openai
import openai
init_logger(api_key="kal_live_your_api_key", base_url="https://www.kalmia.dev")
client = wrap_openai(openai.OpenAI())
# Every call is now auto-traced
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello!"}],
)

Wrap Anthropic

from kalmia import init_logger, wrap_anthropic
import anthropic
init_logger(api_key="kal_live_your_api_key", base_url="https://www.kalmia.dev")
client = wrap_anthropic(anthropic.Anthropic())
message = client.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=1024,
messages=[{"role": "user", "content": "Hello!"}],
)

API surface

init_logger(...)Configure destination: api_key (or KALMIA_API_KEY), base_url, metadata, streaming options
wrap_openai(client)Auto-trace chat completions (sync + async)
wrap_anthropic(client)Auto-trace message creation (sync + async)
@traced(name=..., metadata=...) / with traced()Group calls into one trace
current_span()Get the active span from context
current_trace_id()Trace id of the active run (None outside traced())
submit_feedback(trace_id, rating, comment)Attach end-customer feedback to a run

TypeScript

The TypeScript SDK uses AsyncLocalStorage for automatic context propagation — no manual span passing.

Install

npm install kalmia-sdk

Wrap your client

import OpenAI from "openai";
import { initLogger, wrapOpenAI } from "kalmia-sdk";
initLogger({ apiKey: "kal_live_your_api_key", baseUrl: "https://www.kalmia.dev" });
const client = wrapOpenAI(new OpenAI());
const response = await client.chat.completions.create({
model: "gpt-4o",
messages: [{ role: "user", content: "Hello!" }],
});

API surface

initLogger({ ... })Configure destination: apiKey (or KALMIA_API_KEY), baseUrl, metadata, streaming options
wrapOpenAI(client)Auto-trace chat completions
wrapAnthropic(client)Auto-trace message creation
traced(callback, { name, type, metadata })Group calls into one trace
currentSpan()Get the active span from context
currentTraceId()Trace id of the active run (undefined outside traced())
submitFeedback(traceId, { rating, comment })Attach end-customer feedback to a run
replayMetadata(sessionId, provider?)Trace-level metadata linking a run to its browser session recording (PostHog by default); Python: replay_metadata
recordBrowser(page, { watchable, agentInsight, traceId })Record a Playwright or Puppeteer page for the run and ship the capture to Kalmia; TypeScript only

Session replay

A browser-driving agent can attach a session recording to each run, so the investigator can cross-check what a tool claimed against what the page actually showed, and you can watch the run in the trace view. There are two ways to link one:

PostHogYou already record sessions with PostHog. Merge replayMetadata(sessionId) into the run's root metadata (initLogger metadata for a single-session process, or the traced root's metadata for a backend serving many users), and Kalmia links the recording by session id.
Kalmia captureCall recordBrowser(page, { watchable, agentInsight }) before the first navigation of a Playwright or Puppeteer page. It records the page with rrweb, tags the capture with the active trace id, and ships it on stop(). Both switches default to off, so a call with neither is inert.

Currently switched off

Session replay is deactivated in Kalmia at the moment: the Settings → General row, the Replay button on a trace, and the server-side consumers are all off, and every workspace behaves as if replay were never enabled. The SDK helpers stay safe to call and never raise into your agent; recordings and replay metadata are simply not linked until the feature returns.

Vercel AI SDK

To trace models used through the Vercel AI SDK, wrap them with wrapAISDK as LanguageModelV2 middleware. Tracing is observation-only: every streamed part is forwarded untouched, so it can never alter or break your model output.

import { wrapLanguageModel } from "ai";
import { initLogger, wrapAISDK, endAISDKSession } from "kalmia-sdk";
initLogger({ apiKey: "kal_live_your_api_key" });
const model = wrapLanguageModel({
model: baseModel,
middleware: [wrapAISDK({ sessionID })], // omit sessionID for one trace per call
});
// ... use `model` with generateText / streamText as usual ...
await endAISDKSession(sessionID); // when the session finishes

Pass a sessionID to group all of a session's calls into one trace. While the session is open its trace stays in progress on the server, so live analysis follows the session as it grows; endAISDKSession(sessionID) posts the completed trace and triggers post-run analysis exactly once, over the whole session. A session you never end goes terminal via the server's stall timeout instead, and very long sessions are auto-evicted into a fresh trace segment.

Agent tracing

Use traced() to capture an entire agent run — multiple LLM calls, tool executions, and the full conversation — as a single trace. Without it, each LLM call creates its own standalone trace. Pass metadata to tag a run, for example with the variant you're comparing:

with traced(name="my-agent-run", metadata={"variant": "rag"}) as span:
# ... run your agent variant ...
span.log(output=result)

The variant metadata is what powers variant comparison and per-variant trends. Any string works — a git SHA, a config name, rag vs no-rag.

Tool spans

Wrap tool executions with traced() and span_type="tool" to capture them as tool spans alongside the model calls. This is what makes a trace a full conversation view — model turns and tool calls interleaved.

from kalmia import init_logger, wrap_anthropic, traced
import anthropic, json
init_logger(api_key="kal_live_your_api_key", base_url="https://www.kalmia.dev")
client = wrap_anthropic(anthropic.Anthropic())
TOOLS = [{
"name": "get_weather",
"description": "Get weather for a city",
"input_schema": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
}]
@traced(name="weather-agent")
def run_agent(prompt):
messages = [{"role": "user", "content": prompt}]
while True:
response = client.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=1024, tools=TOOLS, messages=messages,
)
tool_uses = [b for b in response.content if b.type == "tool_use"]
if not tool_uses:
return response.content[0].text
tool_results = []
for tool_use in tool_uses:
# Captured as a tool span
with traced(name=tool_use.name, span_type="tool") as span:
result = get_weather(tool_use.input["city"])
span.log(input=tool_use.input, output=result)
tool_results.append({
"type": "tool_result",
"tool_use_id": tool_use.id,
"content": json.dumps(result),
})
messages.append({"role": "assistant", "content": [b.model_dump() for b in response.content]})
messages.append({"role": "user", "content": tool_results})
run_agent("Compare weather in Tokyo, New York, and San Francisco")

What shows up in the dashboard

LLM spansAuto-captured, with token counts and model
Tool spansYour traced() tool executions, with input/output
ConversationFull message history in the trace view
MetricsToken counts, timing, cost per span
Tool costPaid services (Exa, Tavily, Brave, Serper, and more) priced from the span's output; log the raw API response as the output to get the exact figure

For a paid service Kalmia does not recognize, add cost_usd to the logged output and it is priced like any other. See Reading cost for the recognized services and the snippet.

A model call made through a client the SDK has no wrapper for can be recorded by hand: open a span with type: "llm" and log its usage with the same keys the wrappers emit (prompt_tokens, completion_tokens, cache_read_input_tokens, cache_creation_input_tokens), and Kalmia prices it like any other model call. Report prompt_tokens as the uncached part of the prompt: for OpenAI that is input_tokens minus input_tokens_details.cached_tokens (and minus cache_write_tokens, which goes in cache_creation_input_tokens, when the response reports it).

await traced(async (span) => {
const turn = await myClient.generate(prompt);
span.log({
input: prompt,
output: turn.text,
metadata: { model: "my-model" },
metrics: { prompt_tokens: turn.usage.input, completion_tokens: turn.usage.output },
});
}, { name: "my-model call", type: "llm" });

Live trace streaming

By default the SDK posts a trace exactly once, when its root traced() call ends. Opt in to live streaming and completed spans are also posted in mid-run batches while the run is still going, so long runs appear in the dashboard as they happen instead of only after they finish. See Watching live runs for what the dashboard does with a live run - status, flags, narration.

init_logger(
api_key="kal_live_your_api_key",
streaming=True, # or KALMIA_STREAMING=1
streaming_batch_size=20, # optional, this is the default
streaming_flush_interval_s=5, # optional, this is the default
)
streaming / KALMIA_STREAMINGTurn mid-run batch posting on. Off, behavior is identical to a single post at root end
streaming batch size / KALMIA_STREAMING_BATCH_SIZEFlush once this many completed-but-unsent spans accumulate (default 20)
flush interval / KALMIA_STREAMING_FLUSH_INTERVAL_MSFlush a smaller batch once this much time has passed since the last flush (default 5 s)

Durability is unchanged

Mid-run batches are best-effort: a single send attempt, no retries. The final flush at root end is durable and posts the complete root plus every span not already delivered, with the SDK's full retry-and-buffer contract. A run whose mid-run batches all failed still arrives intact, and streaming never throws into your code.

Customer feedback

Relay your end customer's reaction to a run, a thumbs rating and/or an open-text comment, and Kalmia attaches it to the run's trace, where it shows up alongside verdicts in the dashboard. Capture the trace id during the run with current_trace_id() / currentTraceId(), store it with your own session or request record, then submit feedback from your backend whenever it arrives.

from kalmia import traced, current_trace_id, submit_feedback
# During the run: capture the trace id and store it with your session.
with traced(name="my-agent-run"):
trace_id = current_trace_id()
# ... run your agent, persist trace_id with the session record ...
# Later, when the customer reacts (e.g. in your feedback endpoint):
result = submit_feedback(trace_id, rating="down", comment="It booked the wrong date")
if not result["ok"]:
print("feedback not delivered:", result["error"])
rating"up" or "down" - the customer’s thumbs
commentOpen text, up to 10,000 characters
Merge semanticsProvided fields overwrite the stored state, omitted fields keep it; one current state per trace, no history
TimingAccepted even while the trace is still ingesting - no wait-for-trace logic needed

Fails loudly in dev, never crashes in prod

Invalid usage (no trace id, empty feedback, unknown rating) throws immediately. Delivery problems (network, auth, server errors) never throw: they come back as { ok: false, error } after bounded retry, so a feedback relay inside your own request handler cannot crash your API. Prefer the SDK helpers, or POST to the endpoint directly:
POST/api/v1/traces/{id}/feedbackAPI key

Attach or update end-customer feedback for a trace: { rating, comment }. Same auth and admission gate as trace ingest; write-only keys may submit.

OpenTelemetry libraries

You aren't limited to the OpenAI and Anthropic wrappers. Any OpenTelemetry-emitting LLM or agent library can report to Kalmia by pointing its OTLP exporter at the ingest endpoint — no Kalmia SDK required. One adapter maps the OpenInference family (LangChain, LlamaIndex, CrewAI, …) and native OTel GenAI instrumentation onto Kalmia's trace model with full role, model, and token fidelity.

POST/api/otel/v1/tracesAPI key

Ingest OpenTelemetry spans (OTLP/HTTP). Maps GenAI and OpenInference attributes onto the trace model. Bound to your workspace by API key.

Point an OTLP exporter at Kalmia

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
# e.g. OpenInference instrumentation for LangChain / LlamaIndex / CrewAI
from openinference.instrumentation.langchain import LangChainInstrumentor
provider = TracerProvider()
provider.add_span_processor(
BatchSpanProcessor(
OTLPSpanExporter(
endpoint="https://www.kalmia.dev/api/otel/v1/traces",
headers={"Authorization": "Bearer kal_live_your_api_key"},
)
)
)
LangChainInstrumentor().instrument(tracer_provider=provider)
# Run your agent as usual — every span now lands in Kalmia.

Verified libraries

A library counts as supported when its spans render with the correct role, model, and token metrics — verified against real captures, not assumed from a name on a list.

LangChainRole + model + tokens (OpenInference)
LlamaIndexRAG spans — retriever/embedding as tools
CrewAIOpenInference family — handled by the adapter
Native OTel GenAIAny gen_ai.* provider instrumentation

Transport

OTLP/HTTP in protobuf (the common exporter default) or JSON, selected by Content-Type. OTLP/gRPC is not accepted. Guardrails: 5 MB body and 200 distinct traces per request, 10,000 spans per trace. Exporters that split one trace across many POSTs are merged by span id.

Eval tagging

The OpenTelemetry endpoint does not currently carry arbitrary metadata through to the trace, so eval run tagging (eval.case_id / run_id / variant) is only supported on the SDK ingest path. See Running evals.

Use Kalmia alongside Braintrust

Already instrumented with Braintrust? You don't have to replace anything. Keep your existing instrumentation and add Kalmia as a second OpenTelemetry export destination on the same TracerProvider — the identical spans land in both Braintrust and Kalmia. No re-instrumentation, no Kalmia SDK.

Add a second span processor

// instrumentation.ts
import { BraintrustSpanProcessor } from "@braintrust/otel";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
const provider = new NodeTracerProvider({
spanProcessors: [
// Existing Braintrust destination — unchanged.
new BraintrustSpanProcessor({ parent: process.env.BRAINTRUST_PARENT }),
// Added: the same spans also go to Kalmia.
new BatchSpanProcessor(
new OTLPTraceExporter({
url: "https://www.kalmia.dev/api/otel/v1/traces",
headers: { Authorization: "Bearer kal_live_your_api_key" },
}),
),
],
});
provider.register();

Braintrust receives the identical spans in parallel — nothing about your Braintrust data changes. What lands in Kalmia:

Carries overSpan tree, inputs/outputs, model + token metrics, metadata
Grouped byOTel trace id → one Kalmia trace
ProjectFrom the service.name resource attribute

Want to leave Braintrust entirely? Drop the Braintrust processor and keep only the Kalmia exporter — your OpenTelemetry instrumentation now reports to Kalmia alone.

Direct ingestion

Already have pre-built trace objects? POST them directly to the ingest endpoint in Kalmia's trace format. Accepts a single trace object or an array (up to 100 traces per request). To send from an OpenTelemetry library instead, use the OpenTelemetry endpoint above.

POST/api/v1/traces/ingestAPI key

Ingest pre-built trace objects. Accepts a single trace or an array. Returns which traces were ingested and which were rejected, with reasons.

curl -X POST https://www.kalmia.dev/api/v1/traces/ingest \
-H "Content-Type: application/json" \
-H "Authorization: Bearer kal_live_your_api_key" \
-d '[{
"id": "trace-1",
"input": [{"role": "user", "content": "Hello"}],
"output": "Hi there!",
"metadata": {"provider": "openai", "model": "gpt-4o"},
"spans": [{
"id": "span-1",
"span_id": "span-1",
"root_span_id": "span-1",
"name": "openai chat",
"input": [{"role": "user", "content": "Hello"}],
"output": "Hi there!",
"span_attributes": {"type": "llm"},
"metrics": {"prompt_tokens": 10, "completion_tokens": 5}
}]
}]'

For the full trace and span field reference, see Traces & spans. Delivery is idempotent — the backend upserts by trace id within your workspace, so a duplicate delivery converges on one row rather than double-counting.