Capturing traces
Get your agent runs into Kalmia. Use the Kalmia SDK to wrap your OpenAI or Anthropic client, point any OpenTelemetry library at the ingest endpoint, or POST pre-built traces directly. However a run arrives, everything downstream — verdicts, evals, detection, analytics — works the same.
Kalmia SDK
The SDK wraps your model client so every LLM call is captured automatically, with zero external dependencies on the Python side and automatic context propagation on the TypeScript side. Two things to know before you start:
Get your API key
Create a workspace API key from Settings → API Keys. The raw key is shown once at creation — store it in a secret. Keys look likekal_live_….Python
Install
pip install kalmiaWrap OpenAI
from kalmia import init_logger, wrap_openaiimport openai
init_logger(api_key="kal_live_your_api_key", base_url="https://www.kalmia.dev")client = wrap_openai(openai.OpenAI())
# Every call is now auto-tracedresponse = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "Hello!"}],)Wrap Anthropic
from kalmia import init_logger, wrap_anthropicimport anthropic
init_logger(api_key="kal_live_your_api_key", base_url="https://www.kalmia.dev")client = wrap_anthropic(anthropic.Anthropic())
message = client.messages.create( model="claude-sonnet-4-20250514", max_tokens=1024, messages=[{"role": "user", "content": "Hello!"}],)API surface
TypeScript
The TypeScript SDK uses AsyncLocalStorage for automatic context propagation — no manual span passing.
Install
npm install kalmia-sdkWrap your client
import OpenAI from "openai";import { initLogger, wrapOpenAI } from "kalmia-sdk";
initLogger({ apiKey: "kal_live_your_api_key", baseUrl: "https://www.kalmia.dev" });const client = wrapOpenAI(new OpenAI());
const response = await client.chat.completions.create({ model: "gpt-4o", messages: [{ role: "user", content: "Hello!" }],});API surface
Session replay
A browser-driving agent can attach a session recording to each run, so the investigator can cross-check what a tool claimed against what the page actually showed, and you can watch the run in the trace view. There are two ways to link one:
Currently switched off
Session replay is deactivated in Kalmia at the moment: the Settings → General row, the Replay button on a trace, and the server-side consumers are all off, and every workspace behaves as if replay were never enabled. The SDK helpers stay safe to call and never raise into your agent; recordings and replay metadata are simply not linked until the feature returns.Vercel AI SDK
To trace models used through the Vercel AI SDK, wrap them with wrapAISDK as LanguageModelV2 middleware.
Tracing is observation-only: every streamed part is forwarded untouched, so it can never alter or break your model output.
import { wrapLanguageModel } from "ai";import { initLogger, wrapAISDK, endAISDKSession } from "kalmia-sdk";
initLogger({ apiKey: "kal_live_your_api_key" });
const model = wrapLanguageModel({ model: baseModel, middleware: [wrapAISDK({ sessionID })], // omit sessionID for one trace per call});
// ... use `model` with generateText / streamText as usual ...
await endAISDKSession(sessionID); // when the session finishesPass a sessionID to group all of a session's calls into one trace.
While the session is open its trace stays in progress on the server, so live analysis follows the session as it grows; endAISDKSession(sessionID) posts the completed trace and triggers post-run analysis exactly once, over the whole session.
A session you never end goes terminal via the server's stall timeout instead, and very long sessions are auto-evicted into a fresh trace segment.
Agent tracing
Use traced() to capture an entire agent run — multiple LLM calls, tool executions, and the full conversation — as a single trace.
Without it, each LLM call creates its own standalone trace.
Pass metadata to tag a run, for example with the variant you're comparing:
with traced(name="my-agent-run", metadata={"variant": "rag"}) as span: # ... run your agent variant ... span.log(output=result)The variant metadata is what powers variant comparison and per-variant trends.
Any string works — a git SHA, a config name, rag vs no-rag.
Tool spans
Wrap tool executions with traced() and span_type="tool" to capture them as tool spans alongside the model calls.
This is what makes a trace a full conversation view — model turns and tool calls interleaved.
from kalmia import init_logger, wrap_anthropic, tracedimport anthropic, json
init_logger(api_key="kal_live_your_api_key", base_url="https://www.kalmia.dev")client = wrap_anthropic(anthropic.Anthropic())
TOOLS = [{ "name": "get_weather", "description": "Get weather for a city", "input_schema": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"], },}]
@traced(name="weather-agent")def run_agent(prompt): messages = [{"role": "user", "content": prompt}] while True: response = client.messages.create( model="claude-sonnet-4-20250514", max_tokens=1024, tools=TOOLS, messages=messages, ) tool_uses = [b for b in response.content if b.type == "tool_use"] if not tool_uses: return response.content[0].text
tool_results = [] for tool_use in tool_uses: # Captured as a tool span with traced(name=tool_use.name, span_type="tool") as span: result = get_weather(tool_use.input["city"]) span.log(input=tool_use.input, output=result) tool_results.append({ "type": "tool_result", "tool_use_id": tool_use.id, "content": json.dumps(result), })
messages.append({"role": "assistant", "content": [b.model_dump() for b in response.content]}) messages.append({"role": "user", "content": tool_results})
run_agent("Compare weather in Tokyo, New York, and San Francisco")What shows up in the dashboard
For a paid service Kalmia does not recognize, add cost_usd to the logged output and it is priced like any other.
See Reading cost for the recognized services and the snippet.
A model call made through a client the SDK has no wrapper for can be recorded by hand: open a span with type: "llm" and log its usage with the same keys the wrappers emit (prompt_tokens, completion_tokens, cache_read_input_tokens, cache_creation_input_tokens), and Kalmia prices it like any other model call.
Report prompt_tokens as the uncached part of the prompt: for OpenAI that is input_tokens minus input_tokens_details.cached_tokens (and minus cache_write_tokens, which goes in cache_creation_input_tokens, when the response reports it).
await traced(async (span) => { const turn = await myClient.generate(prompt); span.log({ input: prompt, output: turn.text, metadata: { model: "my-model" }, metrics: { prompt_tokens: turn.usage.input, completion_tokens: turn.usage.output }, });}, { name: "my-model call", type: "llm" });Live trace streaming
By default the SDK posts a trace exactly once, when its root traced() call ends.
Opt in to live streaming and completed spans are also posted in mid-run batches while the run is still going, so long runs appear in the dashboard as they happen instead of only after they finish.
See Watching live runs for what the dashboard does with a live run - status, flags, narration.
init_logger( api_key="kal_live_your_api_key", streaming=True, # or KALMIA_STREAMING=1 streaming_batch_size=20, # optional, this is the default streaming_flush_interval_s=5, # optional, this is the default)Durability is unchanged
Mid-run batches are best-effort: a single send attempt, no retries. The final flush at root end is durable and posts the complete root plus every span not already delivered, with the SDK's full retry-and-buffer contract. A run whose mid-run batches all failed still arrives intact, and streaming never throws into your code.Customer feedback
Relay your end customer's reaction to a run, a thumbs rating and/or an open-text comment, and Kalmia attaches it to the run's trace, where it shows up alongside verdicts in the dashboard.
Capture the trace id during the run with current_trace_id() / currentTraceId(), store it with your own session or request record, then submit feedback from your backend whenever it arrives.
from kalmia import traced, current_trace_id, submit_feedback
# During the run: capture the trace id and store it with your session.with traced(name="my-agent-run"): trace_id = current_trace_id() # ... run your agent, persist trace_id with the session record ...
# Later, when the customer reacts (e.g. in your feedback endpoint):result = submit_feedback(trace_id, rating="down", comment="It booked the wrong date")if not result["ok"]: print("feedback not delivered:", result["error"])Fails loudly in dev, never crashes in prod
Invalid usage (no trace id, empty feedback, unknown rating) throws immediately. Delivery problems (network, auth, server errors) never throw: they come back as{ ok: false, error } after bounded retry, so a feedback relay inside your own request handler cannot crash your API.
Prefer the SDK helpers, or POST to the endpoint directly:/api/v1/traces/{id}/feedbackAPI keyAttach or update end-customer feedback for a trace: { rating, comment }. Same auth and admission gate as trace ingest; write-only keys may submit.
OpenTelemetry libraries
You aren't limited to the OpenAI and Anthropic wrappers. Any OpenTelemetry-emitting LLM or agent library can report to Kalmia by pointing its OTLP exporter at the ingest endpoint — no Kalmia SDK required. One adapter maps the OpenInference family (LangChain, LlamaIndex, CrewAI, …) and native OTel GenAI instrumentation onto Kalmia's trace model with full role, model, and token fidelity.
/api/otel/v1/tracesAPI keyIngest OpenTelemetry spans (OTLP/HTTP). Maps GenAI and OpenInference attributes onto the trace model. Bound to your workspace by API key.
Point an OTLP exporter at Kalmia
from opentelemetry.sdk.trace import TracerProviderfrom opentelemetry.sdk.trace.export import BatchSpanProcessorfrom opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
# e.g. OpenInference instrumentation for LangChain / LlamaIndex / CrewAIfrom openinference.instrumentation.langchain import LangChainInstrumentor
provider = TracerProvider()provider.add_span_processor( BatchSpanProcessor( OTLPSpanExporter( endpoint="https://www.kalmia.dev/api/otel/v1/traces", headers={"Authorization": "Bearer kal_live_your_api_key"}, ) ))
LangChainInstrumentor().instrument(tracer_provider=provider)# Run your agent as usual — every span now lands in Kalmia.Verified libraries
A library counts as supported when its spans render with the correct role, model, and token metrics — verified against real captures, not assumed from a name on a list.
Transport
OTLP/HTTP in protobuf (the common exporter default) or JSON, selected byContent-Type.
OTLP/gRPC is not accepted.
Guardrails: 5 MB body and 200 distinct traces per request, 10,000 spans per trace.
Exporters that split one trace across many POSTs are merged by span id.Eval tagging
The OpenTelemetry endpoint does not currently carry arbitrary metadata through to the trace, so eval run tagging (eval.case_id / run_id / variant) is only supported on the SDK ingest path.
See Running evals.Use Kalmia alongside Braintrust
Already instrumented with Braintrust?
You don't have to replace anything.
Keep your existing instrumentation and add Kalmia as a second OpenTelemetry export destination on the same TracerProvider — the identical spans land in both Braintrust and Kalmia.
No re-instrumentation, no Kalmia SDK.
Add a second span processor
// instrumentation.tsimport { BraintrustSpanProcessor } from "@braintrust/otel";import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
const provider = new NodeTracerProvider({ spanProcessors: [ // Existing Braintrust destination — unchanged. new BraintrustSpanProcessor({ parent: process.env.BRAINTRUST_PARENT }),
// Added: the same spans also go to Kalmia. new BatchSpanProcessor( new OTLPTraceExporter({ url: "https://www.kalmia.dev/api/otel/v1/traces", headers: { Authorization: "Bearer kal_live_your_api_key" }, }), ), ],});
provider.register();Braintrust receives the identical spans in parallel — nothing about your Braintrust data changes. What lands in Kalmia:
Want to leave Braintrust entirely? Drop the Braintrust processor and keep only the Kalmia exporter — your OpenTelemetry instrumentation now reports to Kalmia alone.
Direct ingestion
Already have pre-built trace objects? POST them directly to the ingest endpoint in Kalmia's trace format. Accepts a single trace object or an array (up to 100 traces per request). To send from an OpenTelemetry library instead, use the OpenTelemetry endpoint above.
/api/v1/traces/ingestAPI keyIngest pre-built trace objects. Accepts a single trace or an array. Returns which traces were ingested and which were rejected, with reasons.
curl -X POST https://www.kalmia.dev/api/v1/traces/ingest \ -H "Content-Type: application/json" \ -H "Authorization: Bearer kal_live_your_api_key" \ -d '[{ "id": "trace-1", "input": [{"role": "user", "content": "Hello"}], "output": "Hi there!", "metadata": {"provider": "openai", "model": "gpt-4o"}, "spans": [{ "id": "span-1", "span_id": "span-1", "root_span_id": "span-1", "name": "openai chat", "input": [{"role": "user", "content": "Hello"}], "output": "Hi there!", "span_attributes": {"type": "llm"}, "metrics": {"prompt_tokens": 10, "completion_tokens": 5} }] }]'For the full trace and span field reference, see Traces & spans.
Delivery is idempotent — the backend upserts by trace id within your workspace, so a duplicate delivery converges on one row rather than double-counting.