Guide

Running evals

Turn your own test set into a graded scoreboard. You run your agent over your cases; Kalmia grades each case against plain-language criteria, diagnoses failures, and compares one variant against another. Your coding agent can pull the results over MCP and close the fix loop.

1. Tag your runs

You own the dataset and the runner; Kalmia owns grading. A run is recognized as an eval when its trace carries all three tags under the eval metadata namespace on the root span:

eval.case_idWhich test case this run covers. Stable across variants.
eval.run_idOne suite run. Unique per script invocation (a UUID).
eval.variantWhich version of the agent produced this run. Any label.
import os, uuid
from kalmia import init_logger, traced
init_logger(project_name="code-review-bot", api_key="kal_live_your_api_key", base_url="https://www.kalmia.dev")
RUN_ID = os.environ.get("EVAL_RUN_ID", str(uuid.uuid4())) # one per suite invocation
VARIANT = os.environ["EVAL_VARIANT"] # e.g. the git SHA
def review_case(case_id: str, diff: str):
@traced(name="review", metadata={
"eval": {"case_id": case_id, "run_id": RUN_ID, "variant": VARIANT},
})
def run():
return review_pr(diff) # your existing agent
return run()

Use the SDK for evals

All three tags are required. Eval tagging is supported on the Kalmia SDK ingest path only — the OpenTelemetry endpoint does not carry the metadata through.

2. Author criteria

In the case editor, give each case its pass criteria. Criteria come from three places:

Case criteriaPlain-language “must occur” checks true only of this case
Attached behaviorsWorkspace behaviors, as must / must-not occur
RulesEvery critical behavior, applied to every case automatically

Include at least one positive check

Give each case at least one “must occur” criterion — an agent that does nothing violates none of your prohibitions and would otherwise pass.

3. Read the scoreboard

Kalmia grades each case with a quoted piece of evidence; a case passes only when all its criteria pass. Run the suite a second time under a new variant label and the Runs tab shows the comparison:

Run reportPer-case outcomes, with per-criterion evidence
DiagnosisRoot cause, the surface to change, and a suggested fix per failure
ScoreboardTwo variants side by side: pass rates, fixed, regressed
ReviewConfirm or override the judge on any case that changed outcome

4. Close the loop

Connect your coding agent to Kalmia once, in your agent's repo:

claude mcp add --transport http kalmia https://www.kalmia.dev/api/mcp \
--header "Authorization: Bearer kal_live_your_api_key"

Four read-only eval tools drive this loop — list_recent_runs, get_run_results, get_case_diagnosis, and compare_runs (the server also carries trace read tools and approval-gated actions — see the API reference). With it connected, one instruction drives the whole fix loop:

Run my eval script for a baseline, then pull the results from Kalmia
with get_run_results. For each failing case, read get_case_diagnosis,
fix the agent, and re-run the script under a new variant label. Use
compare_runs against the baseline and keep going until there are zero
regressions.

Automate with kalmia-optimize

The kalmia-optimize CLI drives the fix loop unattended: set a pass-rate threshold, and it runs your eval command, reads the graded diagnoses, rewrites the one prompt file you point it at, and re-runs — stopping at the threshold with zero regressions, or at its iteration cap. Your agent can be in any language; the optimizer drives your eval command, it does not import your code.

export KALMIA_API_KEY=kal_live_your_api_key # your workspace API key
export ANTHROPIC_API_KEY=sk-ant-... # for the rewriter
export KALMIA_URL=https://www.kalmia.dev
npx kalmia-optimize \
--eval-cmd "node run-evals.mjs" \
--prompt-file system_prompt.txt \
--threshold 0.9 \
--max-iterations 4

Wire up the env vars

Your eval command must honor EVAL_RUN_ID and EVAL_VARIANT — that is how the CLI attributes each iteration's run. It commits one variant per iteration on its own branch; your branch is never touched.

New to tracing? Instrument your agent first with the Kalmia SDK. For the concepts behind grading, see Evaluate.