Watch

Watching live runs

When traces stream in mid-run, Kalmia watches them as they happen: a live status strip, structural run flags that catch loops and error pile-ups while the run is still going, a plain-language narration feed, and repo-grounded fix recommendations on confirmed flags.

Live runs

A run appears live when its spans arrive mid-run: opt into live trace streaming in the SDK, keep a Vercel AI SDK session open, or stream from any OpenTelemetry exporter (OTel batches arrive mid-run already, no change needed). Open the trace and the view updates in real time as new spans land.

A live run is in one of three states:

RunningSpans are arriving. The strip shows a pulsing dot and elapsed time since the first span
StalledNo spans for 5 minutes. The partial trace stays fully viewable, and the run resumes automatically if spans arrive again
CompletedThe root span ended. The live strip disappears and the trace renders exactly like any finished run

Post-run analysis (judging, embedding) triggers exactly once when a run goes terminal - whether it completed cleanly or stalled out. A stalled run that resumes and later completes is re-analyzed over the full trace, superseding the stalled snapshot. Stalled is shown in neutral gray, never red: it is not a verdict, and the run may still come back.

Run flags

While a run is in progress, every ingested batch is checked with cheap structural signals - no model calls. When a signal fires, the strip shows a low-key signal detected - investigating line and a detection agent examines the partial trace. Only confirmed flags surface, with an explanation and cited spans; dismissed signals disappear without noise. After the run ends, confirmed flags persist in a Flagged during run panel on the trace.

FlagFires when (defaults)
Repeated errors3+ errors with an identical signature in a batch
Tool loopThe same tool called 6+ times with identical arguments
StallNo spans for 5 minutes - confirmed directly, no agent needed
Cost runawayEstimated spend crosses $20 (currently off by default)
Watcher concernThe narration model spots looping, repeated errors, or off-task behavior

Flags deduplicate per run: a confirmed flag suppresses further signals with the same fingerprint, and a dismissed one only re-admits a signal if occurrences double. Eval runs never raise flags. Your own agent can read them too - the trace investigator's list_run_flags tool returns the same flags you see on screen.

Watching never intervenes

The watch layer observes only. Kalmia never pauses, stops, or steers your agent mid-run. There is no per-flag realtime alert or webhook: a flag raised during a run shows in the dashboard, and reaches you afterwards through the scheduled-sweep digest, which can post to a connected Slack or Microsoft Teams channel and emails the workspace owners unless they switch that off. A flag that was already reported live or dismissed by a person suppresses the matching sweep finding, so the digest never repeats it.

Narration

Narration is a live commentary feed on the strip: roughly once a minute, a fast model reads the spans that arrived since the last tick and appends one or two plain-language sentences about what the agent is doing now. After the run ends the feed persists as a Run narration panel, a readable timeline of the run.

CadenceAbout one tick per minute while spans keep arriving
ModelA fixed fast model (Claude Haiku) - workspace judge model settings do not affect it
BudgetCapped at $0.25 or 250k tokens per run; the feed notes when the cap is reached and stops
ConcernsThe narrator can raise a possible concern (looping, repeated errors, off-task) - it escalates through the same confirmation gate as run flags

Narration needs readable span content; if content capture is off, the feed says so once and structural signals still apply. It is off by default and enabled per deployment (KALMIA_NARRATION_ENABLED), with its own rate-limit group so many live runs can never starve interactive judging.

Fix recommendations

A confirmed flag can carry a repo-grounded fix recommendation: a read-only agent explores your connected repository, ties the flagged behavior to the code that likely produced it, and writes an advisory recommendation with cited file paths. Citations are checked against the repo's real tree, so a hallucinated path can never appear as a citation.

This requires a repository connected under Settings → GitHub. With a connection, recommendations generate automatically when a flag is confirmed; without one, the flag shows a connect a repo for a code-grounded fix link, and you can generate on demand after connecting.

Advisory only, read-only always

The fix-rec agent has exactly three tools: list the repo tree, read a file, and report its recommendation. It cannot run code, fetch URLs, or write anything - no auto-applied fixes, no PRs. The recommendation is advice for you, not an action taken for you.