Guide

Behavior detection

Define a behavior in plain language and let Kalmia scan your traces, find the ones that show it, and tag them with the evidence it found.

Define a behavior

Behavior vs. annotation

A behavior is the definition — a label plus a plain-language description. An annotation is an instance — a trace tagged with that behavior.

Add a behavior on the Behaviors page in the dashboard. The description is what the detector matches against, so make it specific — quality scales with it.

Run detection

1

Pick a scan size

Bounds how many untagged traces this run scans (default 25).
2

Detect

Kalmia scans the sampled traces with a live progress bar. Navigating away and back resumes the run.
3

Review

Each result shows a match / no-match mark, the trace, the cited evidence, and a Tag / Untag button. Confirmed matches seed future runs.

No matches?

Refine the description, raise the scan size, or tag one trace manually to give the detector a concrete example.

From the API

A batch detection can be started from the dashboard or with your workspace API key (POST /api/annotations/detect-batch). It spends model budget, so it needs a member or owner, or a full-scope key. The run is asynchronous: the start call returns a runId, and you poll progress and results with the same key.

# Poll a batch detection run's progress + results
curl https://www.kalmia.dev/api/annotations/detect-runs/<runId> \
-H "Authorization: Bearer kal_live_your_api_key"

Example behaviors

  • Hallucination — “generates code or file references that don’t exist in the project”.
  • Excessive retries — “retries the same operation 3+ times without changing approach”.
  • Permission errors — “hits permission-denied or access errors”.
  • Backtracking — “undoes or reverts changes it previously made”.

For the proactive counterpart — surfacing runs you didn't define — see anomaly detection.