Running evals
Turn your own test set into a graded scoreboard. You run your agent over your cases; Kalmia grades each case against plain-language criteria, diagnoses failures, and compares one variant against another. Your coding agent can pull the results over MCP and close the fix loop.
1. Tag your runs
You own the dataset and the runner; Kalmia owns grading.
A run is recognized as an eval when its trace carries all three tags under the eval metadata namespace on the root span:
import os, uuidfrom kalmia import init_logger, traced
init_logger(project_name="code-review-bot", api_key="kal_live_your_api_key", base_url="https://www.kalmia.dev")
RUN_ID = os.environ.get("EVAL_RUN_ID", str(uuid.uuid4())) # one per suite invocationVARIANT = os.environ["EVAL_VARIANT"] # e.g. the git SHA
def review_case(case_id: str, diff: str): @traced(name="review", metadata={ "eval": {"case_id": case_id, "run_id": RUN_ID, "variant": VARIANT}, }) def run(): return review_pr(diff) # your existing agent return run()Use the SDK for evals
All three tags are required. Eval tagging is supported on the Kalmia SDK ingest path only — the OpenTelemetry endpoint does not carry the metadata through.2. Author criteria
In the case editor, give each case its pass criteria. Criteria come from three places:
Include at least one positive check
Give each case at least one “must occur” criterion — an agent that does nothing violates none of your prohibitions and would otherwise pass.3. Read the scoreboard
Kalmia grades each case with a quoted piece of evidence; a case passes only when all its criteria pass. Run the suite a second time under a new variant label and the Runs tab shows the comparison:
4. Close the loop
Connect your coding agent to Kalmia once, in your agent's repo:
claude mcp add --transport http kalmia https://www.kalmia.dev/api/mcp \ --header "Authorization: Bearer kal_live_your_api_key"Four read-only eval tools drive this loop — list_recent_runs, get_run_results, get_case_diagnosis, and compare_runs (the server also carries trace read tools and approval-gated actions — see the API reference).
With it connected, one instruction drives the whole fix loop:
Run my eval script for a baseline, then pull the results from Kalmiawith get_run_results. For each failing case, read get_case_diagnosis,fix the agent, and re-run the script under a new variant label. Usecompare_runs against the baseline and keep going until there are zeroregressions.Automate with kalmia-optimize
The kalmia-optimize CLI drives the fix loop unattended: set a pass-rate threshold, and it runs your eval command, reads the graded diagnoses, rewrites the one prompt file you point it at, and re-runs — stopping at the threshold with zero regressions, or at its iteration cap.
Your agent can be in any language; the optimizer drives your eval command, it does not import your code.
export KALMIA_API_KEY=kal_live_your_api_key # your workspace API keyexport ANTHROPIC_API_KEY=sk-ant-... # for the rewriterexport KALMIA_URL=https://www.kalmia.dev
npx kalmia-optimize \ --eval-cmd "node run-evals.mjs" \ --prompt-file system_prompt.txt \ --threshold 0.9 \ --max-iterations 4Wire up the env vars
Your eval command must honorEVAL_RUN_ID and EVAL_VARIANT — that is how the CLI attributes each iteration's run.
It commits one variant per iteration on its own branch; your branch is never touched.New to tracing? Instrument your agent first with the Kalmia SDK. For the concepts behind grading, see Evaluate.