# Evaluate agent behavior

Grade agent output the way a careful reviewer would.
A **judge** scores each run against plain-language criteria and your team's standards, and **evals** turn your own test set into a graded scoreboard with per-failure diagnosis and variant-over-variant comparison.

## The judge

The judge grades every run — the automatic [verdict](https://www.kalmia.dev/docs/investigate.md#verdicts) on a production run and every criterion in an eval.
Each grade comes with a quote from the transcript as evidence, and a refusal is recorded as `unscored` rather than a silent pass or fail.

### Judge engine

The judge that analyzes a run can use one of two engines, chosen per workspace in **Settings → General → Judge engine** (owner-only):

- **Investigator (most accurate)**: Default. Reads the trace step by step and files every issue it can confirm against the spans; ~30-60s and a few cents more per analysis
- **Fast single-pass**: One model call, ~7s, cheaper, less thorough

If the investigator cannot reach a verdict (step cap, refusal), the analysis falls back to the fast engine rather than failing - a run is never left unjudged by the choice.

### Judge model

Pick the judge model in **Settings → General**.
The same model grades verdicts and evals.

- **Fable (Claude Fable 5.1)**: Most capable; slowest and about five times the cost of the Standard tier. Offered only on deployments where the operator has enabled the Fable tier
- **Standard (GPT-6.1 Sol)**: Default - finds the most issues per dollar
- **Sonnet (Claude Sonnet 5)**: Balanced quality, speed, and cost
- **Haiku (Claude Haiku 4.5)**: Fastest / cheapest

The tiers sit under **Always the latest** in the picker: each one follows the newest model of its tier as Kalmia upgrades.
A workspace that pinned a specific model id before the tiers existed keeps it under a **Pinned version** group, and can move to a tier at any time.
Two fixed exceptions: the pass that verifies audit claims against the spans and the issue classifier behind the [Issues tab](https://www.kalmia.dev/docs/issues.md) run on fixed models regardless of this setting.

**Provider fallback.** If Anthropic stops answering for a reason on its side (credit balance exhausted, key revoked, rate limited, overloaded, a 5xx or network failure), the call is retried once on the OpenAI model of the same tier and Anthropic is skipped for a few minutes, so sweeps, audits, and investigations keep running instead of silently producing nothing.
Request mistakes and refusals are never retried elsewhere.
Spend is attributed to the model that answered.
Requires an OpenAI key on the deployment;`LLM_FALLBACK=off` disables it.

## Behaviors & rules

**Behavior**: A workspace-level named standard: a short `label` plus an optional plain-language description (e.g. “escalates billing questions to a human”). Behaviors are the shared vocabulary used across evals, detection, and annotation.

**Rule (a critical behavior)**: A behavior you mark as **critical**. The judge checks every run against your rules, and a violation fails the run and names the rule.

Manage behaviors on the **Behaviors** page in the dashboard: add a behavior, then toggle *Make rule* to promote it.

## Domain context

By default the judge only knows what the agent was *told* to do — not your team's real standards.
**Domain context** is reference material you give it: policies, runbooks, tool semantics, the actual quality bar.
Upload `.txt` or `.md` documents from the **Context** tab in the dashboard, and the judge checks each run against the relevant parts, citing the document.

The Context tab lists every document with its name, when it was added, whether it is active, and a note if embedding failed.
**Add context** takes pasted text or an uploaded file (up to 200,000 characters); a document is chunked and embedded in the background, usually within seconds, and becomes searchable once ready.
Each document can be edited in place (a text change re-embeds it), switched off without deleting it, or removed.
Owners and members can change documents; viewers can read the list.
The same operations are available over the [workspace context endpoints](https://www.kalmia.dev/docs/api-reference.md#session).

> **Worked example:** A support agent issues an $80 refund on a 45-day-old order. With no context, the judge sees a polite, resolved conversation and calls it healthy. Add a refund-policy document (“refunds only within 30 days; over $50 needs approval”) and an escalation **rule**, and the same run is now flagged failed — citing the policy and naming the rule it broke.

## Running evals

Evals turn your own test set into a graded scoreboard.
You own the dataset and the runner; Kalmia grades each case, diagnoses failures, and compares variants.
Kalmia never stores your eval inputs and never runs your agent.

A run is recognized as an eval when its trace carries all three tags under the `eval` metadata namespace on the root span:

- **eval.case_id**: Which test case this run covers. Stable across variants.
- **eval.run_id**: One suite run. Unique per script invocation (a UUID).
- **eval.variant**: Which version of the agent produced this run. Any label.

```python
import os, uuid
from kalmia import init_logger, traced

init_logger(api_key="kal_live_your_api_key", base_url="https://www.kalmia.dev")

RUN_ID = os.environ.get("EVAL_RUN_ID", str(uuid.uuid4()))  # one per suite invocation
VARIANT = os.environ["EVAL_VARIANT"]                        # e.g. the git SHA

def review_case(case_id: str, diff: str):
    @traced(name="review", metadata={
        "eval": {"case_id": case_id, "run_id": RUN_ID, "variant": VARIANT},
    })
    def run():
        return review_pr(diff)   # your existing agent
    return run()
```

> **Use the SDK for evals:** All three tags are required. Eval tagging is supported on the Kalmia SDK ingest path only — the OpenTelemetry endpoint does not carry the metadata through.

## Authoring criteria

Each case is graded against its criteria, which you author in the case editor.
Criteria come from three places:

| Source | What it is |
| --- | --- |
| Case criteria | Plain-language “must occur” checks true only of this case |
| Attached behaviors | Workspace behaviors you attach to the case, as must / must-not occur |
| Rules | Every critical behavior, applied to every case automatically |

Each case reports `pass`, `fail`, `needs_attention` (a criterion couldn't be graded), or `no_criteria`.
Failed cases get a root-cause diagnosis with a suggested fix, and you can confirm or override any grade.

> **Include at least one positive check:** Give each case at least one “must occur” criterion — an agent that does nothing violates none of your prohibitions and would otherwise pass.

## Comparing variants

Once two run sets exist, Kalmia renders a **scoreboard** comparing a baseline against a candidate — the heart of regression-testing an agent:

- **Fixed**: Cases that went fail → pass
- **Regressed**: Cases that went pass → fail
- **Pass rate + delta**: Each run’s pass rate and the shift between them

> **Production variants vs. eval variants:** This scoreboard compares *graded* eval run sets. To compare *production* variants on operational metrics (duration, tokens, cost, error rate), see [Metrics & cost](https://www.kalmia.dev/docs/analytics.md#metrics).

## Close the loop (MCP)

The [MCP server](https://www.kalmia.dev/docs/api-reference.md#mcp) exposes your eval results to your own coding agent, so it can pull the scoreboard and diagnoses and fix the agent inside your repo.
Most tools only read; a few act (re-judge a trace, label it, change monitoring settings), and your agent host asks for approval before running those:

Claude Code:

```bash
claude mcp add --transport http kalmia https://www.kalmia.dev/api/mcp \
  --header "Authorization: Bearer kal_live_your_api_key"
```

For a full walkthrough, including the `kalmia-optimize` unattended fix loop, see the [Running evals guide](https://www.kalmia.dev/docs/guides/evals.md).
