# Detect behaviors & anomalies

Three ways to find the runs that matter.
**Behavior detection** finds every trace that shows a pattern you describe in plain language.
The **anomaly scan** works the other way - it surfaces unusual or wrong runs you didn't know to look for.
And the **prompt audit** judges runs against their own prompts to catch agents drifting from their instructions.

## Behavior detection

Describe a behavior you care about — “retries the same tool 3+ times without changing approach” — and Kalmia reads through your traces, decides which ones exhibit it, and tags the matches with the evidence it found.
No need to hand-label examples first; a label and a description are enough.

**Behavior vs. annotation**: A **behavior** is the definition — a label plus a description. An **annotation** is an instance — a specific trace tagged with that behavior.

From the **Behaviors** page (or the behavior panel in the metrics dashboard):

1. **Write the behavior**: A label plus a description — quality scales with the description.

2. **Pick a scan size**: Bounds how many untagged traces this run scans (default 25).

3. **Detect**: Kalmia scans the sampled traces with a live progress bar. Navigating away and back resumes the run.

4. **Review matches**: Each result shows a match / no-match mark, the trace, the cited evidence, and a Tag / Untag button.

> **No matches?:** Refine the description, raise the scan size, or tag one trace manually to give the detector a concrete example to generalize from.

## The Scan tab

Both AI scans live on the dashboard's **Scan** tab.
One shared **time window** (presets from 1 hour to 90 days, or a custom range; default 7 days) scopes what both scans read; each scan picks its own target - scope or variant - in its dialog when you run it.
Past runs of each scan are listed below its results, and reopening one re-reads its findings.
Scans spend model budget, so viewers can see results but not trigger runs.

## Anomaly scan

Instead of you defining what to look for, the anomaly scan surfaces unusual or wrong runs proactively.
Click **Scan for anomalies**, pick a scope, and Kalmia runs a multi-source scan over the newest traces in the window (up to 500): metric outliers across a dozen per-run metrics, behavioral outliers by embedding distance, and correctness signals (failed verdicts, tool errors, failure tags, annotations).
Outliers group into clusters, and the highest-value clusters escalate to an AI investigation whose findings fill in progressively and persist across refresh.

- **Scope**: Whole workspace or a single variant - baselines compare only traces inside the scope
- **Severity**: Findings are ranked high · medium · low
- **Evidence**: Each finding links its traces and cited spans
- **Tiers**: Findings are badged investigated (an investigation read the spans), investigated earlier (the same issue was investigated within the last day, so this sweep reused that conclusion instead of buying another investigation), or unverified (a fast detector flagged them; no spans were read)

> **Unverified findings describe, they do not diagnose:** Fast detectors work from metrics, labels, and short error samples, never the spans, so an unverified finding's title names the signal that recurred (what, in which span type or tool, across how many traces) and is not a confirmed cause. Numbers and ids inside a quoted sample belong to that one trace. Run an investigation on the cluster to get a verdict backed by cited spans; the badge flips to investigated when one completes.

> **Thin scopes still scan:** Your scope is honored as-is. Comparative signals (metric and behavioral outliers) need at least 20 traces to have a baseline worth comparing against and sit out below that; correctness signals are absolute and always run. A finding whose investigation is still running or hit its budget is badged as such - a partial answer is never shown as a confirmed one.

## Regression investigation

When one variant looks worse than another, Kalmia can investigate why.
You name the two variants and, optionally, a time window.
Kalmia pins a fixed set of runs from each side (5 to 100 per variant, so new traffic cannot shift the comparison mid-run), compares their metrics, reads the spans, and files one finding that names what differs, with cited evidence, or says plainly when it found no clear cause.

- **Start**: POST /api/v1/regressions with { variantA, variantB, windowStart?, windowEnd? }, as a signed-in member or owner (it spends model budget, so an API key cannot start one)
- **Read**: GET /api/v1/anomaly-findings?scanId=regression, with a session or an API key
- **Duplicates**: One live investigation per pair of variants; starting it again returns the existing finding

For a quick look before you investigate, compare the two variants side by side in the [Metrics view](https://www.kalmia.dev/docs/guides/comparing-variants.md).

## Scheduled monitoring

Scheduled monitoring runs the anomaly pipeline for you, on a cadence, over exactly the traces that arrived since the last sweep - no gaps, no double-counting.
Every in-window trace gets a cheap AI triage ("does this look wrong?") up to a budget you set, and only flagged clusters escalate to a full investigation.
Each sweep that finds something new sends one digest, by email to the workspace owners and to any connected Slack or Microsoft Teams channel; quiet sweeps stay silent, and an issue that keeps happening is reported once, not once per sweep.

1. **Run a sweep, or put sweeps on a schedule**: A new workspace starts with scheduled sweeps off, because a sweep costs credits (see [Plans & credits](https://www.kalmia.dev/docs/concepts.md#credits)) and nothing should spend them before you ask. To run one sweep now, open the **Scan** tab and click **Run a sweep** (workspace owners): it covers the traces that arrived since the last sweep, up to the last 7 days, and states what it will check and cost before it starts. To have them run on their own, open **Settings → Monitoring** (workspace owners), switch scheduled sweeps on, and pick a cadence (hourly, every 3 hours, daily, or weekly). Weekly is about 4 sweeps a month and daily about 30. A sweep is priced by the traces it checks, so what a schedule costs follows how much your agent runs, up to the audit allowance below. When your credits cover only part of a sweep, it checks that part, the failed and flagged traces first, and its row in the history says so. The same page sets the two daily allowances: audits per day (default 250, max 2,000; a one-call audit per trace, failed runs first) and investigations per day (default 10; full investigations, one representative trace per finding and the rest on a random sample of unflagged traces). Both are split across the day's sweeps; a daily, weekly, or hand-run sweep gets the whole number. 0 audits means free detectors only; 0 investigations means manual only.

2. **Digests arrive by email, and in the app**: **Email digests** are on by default: each non-quiet digest goes to the workspace's owners with no channel setup needed; owners can switch them off under Settings, Monitoring. Review digests in the Scan tab's sweep history - each sweep shows its window, coverage (partial coverage is stated, never implied away), the digest it would have sent, and whether it was emailed.

3. **Connect Slack or Teams**: Connect a Slack channel (one click - Slack's own consent screen picks the channel) or a Microsoft Teams channel (needs a Microsoft 365 work account; in Teams, add the Workflows "Send webhook alerts to a channel" to the channel and paste the URL it gives you; Kalmia posts a "Kalmia connected" card to prove it works before storing it). That is all: from the next sweep, every non-quiet digest posts to every connected destination. The first digest a channel receives reports every still-live issue, so it starts with the full picture.

4. **Ask Kalmia in Slack**: Reply in a digest's thread ("why did the first one happen?"), or `@Kalmia` anywhere in the channel. Kalmia answers in the thread within about a minute and links the runs it cites; follow-ups in that thread need no second @mention. In a private channel, run `/invite @Kalmia` once. A channel connected before this feature shows a **Reconnect** button under Settings → Monitoring.

- **Noise contract**: New issues ping once; still-happening issues never re-page; issues already reported by live run flags (or dismissed by a human) are suppressed
- **Spend**: Bounded by your daily audit and investigation allowances plus the workspace's daily spend cap; recently-investigated clusters are not re-investigated for 24h
- **Digest content**: Agent health score with the sample it rests on (runs audited out of the window, and the monitoring scope), then one ranked list: issues a verification pass or an investigation confirmed against the spans, the sweep's investigated findings, and any agent that went silent - ordered by severity, then by strength of evidence, then by runs affected, each with a one-line summary. A usage line follows: runs in the window, estimated spend, turns per run, and tokens (it says so when only some runs could be priced). Unverified detector signals collapse to a single count with a link (the Scan tab shows them in full). Never raw trace content
- **Digest timing**: A sweep's investigations run before its digest is sent: the digest waits up to 45 minutes for them, so escalated issues arrive with their conclusion instead of 'investigation in progress'
- **Health score**: 0-100 per sweep window, starting at 100: up to -50 for the share of runs that failed (25% = full), up to -40 for runs carrying a confirmed issue (50% = full), -5 per investigated high-severity finding (max -10). Every deduction is stated next to the number; under 20 runs it is marked low sample, and with no runs there is no score. An agent going silent is reported as a finding, never scored
- **Asking in Slack**: Anyone in the channel can ask. Answers are read-only and grounded in the workspace's runs; unlike the digest, an answer can quote run content, because the channel is the destination an owner chose
- **Run health**: The Scan tab's sweep history answers 'did it even run' - Slack silence is the healthy default. A status line says whether monitoring is on (covered through, next due, overdue) or paused (off since), and any stretch no completed sweep covered shows as a 'Not covered' row
- **Pauses and gaps**: Switching sweeps off and on again restarts coverage at the moment you re-enabled, never a silent backfill. The paused stretch stays visible as a gap; an owner can close it with 'Sweep this window', which runs one sweep over exactly that range (allowances prorated by its length)

> **Missing data is a finding:** Every other detector needs runs to exist, so an agent that stops running is invisible to them. Scheduled sweeps therefore also compare each agent's run count in the window with that agent's own previous seven days (agents are told apart by variant, then agent name). An agent active on at least three of those days, averaging at least two runs a day, that produced nothing in a window where at least one run was expected is filed as **went silent**: high when at least 10 runs went missing in the window, medium below that (an agent that runs three times a day going quiet is a line, not a page). One that produced under 10% of the roughly 20 or more runs expected is filed as **volume collapsed** (medium). Agents whose traces share a project are reported together, one finding per project naming each arm, so the two arms of an A/B experiment stopping is one finding, not two. A window with no runs from any agent files one workspace-wide finding instead. Variants listed under **Mute silence alerts for** in Settings, Monitoring (or `mutedSilenceVariants` on the monitoring API) are never reported. These findings cite no traces, repeat as still-happening rather than re-paging, and expire on their own once the agent has been quiet for a week. Manual scans skip this check.

## Prompt audit

The prompt audit answers a different question: not "which runs are unusual" but **"which runs disobeyed their own instructions?"** Click **Audit prompts**, pick one variant and a sample size (default 50, max 200), and Kalmia samples matching runs from the window, extracts an instruction checklist from each distinct system prompt, and judges every sampled trace against its own prompts - both the system-prompt instructions and the user's request.

- **Verdicts**: Each trace lands in aligned · misaligned · could not audit · failed
- **Violations**: Each cites the instruction, the violating span, a quoted excerpt, and an explanation
- **Citations**: Span citations are validated against the trace's real spans - an invented span id is never stored
- **Retry**: Failed traces can be retried from the report without re-running the whole audit

Reports store their quoted evidence at write time, so an audit stays readable even after the underlying traces age out of retention.
Audits use the workspace [judge model](https://www.kalmia.dev/docs/evaluate.md#judges) and draw from the eval [rate-limit group](https://www.kalmia.dev/docs/concepts.md#rate-limits), so a large audit can never starve interactive judging.
