# Running evals

Turn your own test set into a graded scoreboard.
You run your agent over your cases; Kalmia grades each case against plain-language criteria, diagnoses failures, and compares one variant against another.
Your coding agent can pull the results over MCP and close the fix loop.

## 1. Tag your runs

You own the dataset and the runner; Kalmia owns grading.
A run is recognized as an eval when its trace carries all three tags under the `eval` metadata namespace on the root span:

- **eval.case_id**: Which test case this run covers. Stable across variants.
- **eval.run_id**: One suite run. Unique per script invocation (a UUID).
- **eval.variant**: Which version of the agent produced this run. Any label.

```python
import os, uuid
from kalmia import init_logger, traced

init_logger(project_name="code-review-bot", api_key="kal_live_your_api_key", base_url="https://www.kalmia.dev")

RUN_ID = os.environ.get("EVAL_RUN_ID", str(uuid.uuid4()))  # one per suite invocation
VARIANT = os.environ["EVAL_VARIANT"]                        # e.g. the git SHA

def review_case(case_id: str, diff: str):
    @traced(name="review", metadata={
        "eval": {"case_id": case_id, "run_id": RUN_ID, "variant": VARIANT},
    })
    def run():
        return review_pr(diff)   # your existing agent
    return run()
```

> **Use the SDK for evals:** All three tags are required. Eval tagging is supported on the Kalmia SDK ingest path only — the OpenTelemetry endpoint does not carry the metadata through.

## 2. Author criteria

In the case editor, give each case its pass criteria.
Criteria come from three places:

- **Case criteria**: Plain-language “must occur” checks true only of this case
- **Attached behaviors**: Workspace behaviors, as must / must-not occur
- **Rules**: Every critical behavior, applied to every case automatically

> **Include at least one positive check:** Give each case at least one “must occur” criterion — an agent that does nothing violates none of your prohibitions and would otherwise pass.

## 3. Read the scoreboard

Kalmia grades each case with a quoted piece of evidence; a case passes only when all its criteria pass.
Run the suite a second time under a new variant label and the Runs tab shows the comparison:

- **Run report**: Per-case outcomes, with per-criterion evidence
- **Diagnosis**: Root cause, the surface to change, and a suggested fix per failure
- **Scoreboard**: Two variants side by side: pass rates, fixed, regressed
- **Review**: Confirm or override the judge on any case that changed outcome

## 4. Close the loop

Connect your coding agent to Kalmia once, in your agent's repo:

Claude Code:

```bash
claude mcp add --transport http kalmia https://www.kalmia.dev/api/mcp \
  --header "Authorization: Bearer kal_live_your_api_key"
```

.mcp.json:

```json
{
  "mcpServers": {
    "kalmia": {
      "type": "http",
      "url": "https://www.kalmia.dev/api/mcp",
      "headers": { "Authorization": "Bearer ${KALMIA_API_KEY:-}" }
    }
  }
}
```

Four read-only eval tools drive this loop — `list_recent_runs`, `get_run_results`, `get_case_diagnosis`, and `compare_runs` (the server also carries trace read tools and approval-gated actions — see the [API reference](https://www.kalmia.dev/docs/api-reference.md#mcp)).
With it connected, one instruction drives the whole fix loop:

Prompt:

```text
Run my eval script for a baseline, then pull the results from Kalmia
with get_run_results. For each failing case, read get_case_diagnosis,
fix the agent, and re-run the script under a new variant label. Use
compare_runs against the baseline and keep going until there are zero
regressions.
```

## Automate with kalmia-optimize

The `kalmia-optimize` CLI drives the fix loop unattended: set a pass-rate threshold, and it runs your eval command, reads the graded diagnoses, rewrites the one prompt file you point it at, and re-runs — stopping at the threshold with zero regressions, or at its iteration cap.
Your agent can be in any language; the optimizer drives your eval *command*, it does not import your code.

```bash
export KALMIA_API_KEY=kal_live_your_api_key   # your workspace API key
export ANTHROPIC_API_KEY=sk-ant-...           # for the rewriter
export KALMIA_URL=https://www.kalmia.dev

npx kalmia-optimize \
  --eval-cmd "node run-evals.mjs" \
  --prompt-file system_prompt.txt \
  --threshold 0.9 \
  --max-iterations 4
```

> **Wire up the env vars:** Your eval command must honor `EVAL_RUN_ID` and `EVAL_VARIANT` — that is how the CLI attributes each iteration's run. It commits one variant per iteration on its own branch; your branch is never touched.

New to tracing?
Instrument your agent first with the [Kalmia SDK](https://www.kalmia.dev/docs/guides/kalmia-sdk.md).
For the concepts behind grading, see [Evaluate](https://www.kalmia.dev/docs/evaluate.md).
