Detect behaviors & anomalies
Three ways to find the runs that matter. Behavior detection finds every trace that shows a pattern you describe in plain language. The anomaly scan works the other way - it surfaces unusual or wrong runs you didn't know to look for. And the prompt audit judges runs against their own prompts to catch agents drifting from their instructions.
Behavior detection
Describe a behavior you care about — “retries the same tool 3+ times without changing approach” — and Kalmia reads through your traces, decides which ones exhibit it, and tags the matches with the evidence it found. No need to hand-label examples first; a label and a description are enough.
Behavior vs. annotation
A behavior is the definition — a label plus a description. An annotation is an instance — a specific trace tagged with that behavior.
From the Behaviors page (or the behavior panel in the metrics dashboard):
Write the behavior
Pick a scan size
Detect
Review matches
No matches?
Refine the description, raise the scan size, or tag one trace manually to give the detector a concrete example to generalize from.The Scan tab
Both AI scans live on the dashboard's Scan tab. One shared time window (presets from 1 hour to 90 days, or a custom range; default 7 days) scopes what both scans read; each scan picks its own target - scope or variant - in its dialog when you run it. Past runs of each scan are listed below its results, and reopening one re-reads its findings. Scans spend model budget, so viewers can see results but not trigger runs.
Anomaly scan
Instead of you defining what to look for, the anomaly scan surfaces unusual or wrong runs proactively. Click Scan for anomalies, pick a scope, and Kalmia runs a multi-source scan over the newest traces in the window (up to 500): metric outliers across a dozen per-run metrics, behavioral outliers by embedding distance, and correctness signals (failed verdicts, tool errors, failure tags, annotations). Outliers group into clusters, and the highest-value clusters escalate to an AI investigation whose findings fill in progressively and persist across refresh.
Unverified findings describe, they do not diagnose
Fast detectors work from metrics, labels, and short error samples, never the spans, so an unverified finding's title names the signal that recurred (what, in which span type or tool, across how many traces) and is not a confirmed cause. Numbers and ids inside a quoted sample belong to that one trace. Run an investigation on the cluster to get a verdict backed by cited spans; the badge flips to investigated when one completes.Thin scopes still scan
Your scope is honored as-is. Comparative signals (metric and behavioral outliers) need at least 20 traces to have a baseline worth comparing against and sit out below that; correctness signals are absolute and always run. A finding whose investigation is still running or hit its budget is badged as such - a partial answer is never shown as a confirmed one.Regression investigation
When one variant looks worse than another, Kalmia can investigate why. You name the two variants and, optionally, a time window. Kalmia pins a fixed set of runs from each side (5 to 100 per variant, so new traffic cannot shift the comparison mid-run), compares their metrics, reads the spans, and files one finding that names what differs, with cited evidence, or says plainly when it found no clear cause.
For a quick look before you investigate, compare the two variants side by side in the Metrics view.
Scheduled monitoring
Scheduled monitoring runs the anomaly pipeline for you, on a cadence, over exactly the traces that arrived since the last sweep - no gaps, no double-counting. Every in-window trace gets a cheap AI triage ("does this look wrong?") up to a budget you set, and only flagged clusters escalate to a full investigation. Each sweep that finds something new sends one digest, by email to the workspace owners and to any connected Slack or Microsoft Teams channel; quiet sweeps stay silent, and an issue that keeps happening is reported once, not once per sweep.
Run a sweep, or put sweeps on a schedule
Digests arrive by email, and in the app
Connect Slack or Teams
Ask Kalmia in Slack
@Kalmia anywhere in the channel.
Kalmia answers in the thread within about a minute and links the runs it cites; follow-ups in that thread need no second @mention.
In a private channel, run /invite @Kalmia once.
A channel connected before this feature shows a Reconnect button under Settings → Monitoring.Missing data is a finding
Every other detector needs runs to exist, so an agent that stops running is invisible to them. Scheduled sweeps therefore also compare each agent's run count in the window with that agent's own previous seven days (agents are told apart by variant, then agent name). An agent active on at least three of those days, averaging at least two runs a day, that produced nothing in a window where at least one run was expected is filed as went silent: high when at least 10 runs went missing in the window, medium below that (an agent that runs three times a day going quiet is a line, not a page). One that produced under 10% of the roughly 20 or more runs expected is filed as volume collapsed (medium). Agents whose traces share a project are reported together, one finding per project naming each arm, so the two arms of an A/B experiment stopping is one finding, not two. A window with no runs from any agent files one workspace-wide finding instead. Variants listed under Mute silence alerts for in Settings, Monitoring (ormutedSilenceVariants on the monitoring API) are never reported.
These findings cite no traces, repeat as still-happening rather than re-paging, and expire on their own once the agent has been quiet for a week.
Manual scans skip this check.Prompt audit
The prompt audit answers a different question: not "which runs are unusual" but "which runs disobeyed their own instructions?" Click Audit prompts, pick one variant and a sample size (default 50, max 200), and Kalmia samples matching runs from the window, extracts an instruction checklist from each distinct system prompt, and judges every sampled trace against its own prompts - both the system-prompt instructions and the user's request.
Reports store their quoted evidence at write time, so an audit stays readable even after the underlying traces age out of retention. Audits use the workspace judge model and draw from the eval rate-limit group, so a large audit can never starve interactive judging.