Watch Kalmia Catch a Silent Wrong-Order Change
Andy Xu, Calvin Huang · August 5 2026
A five-minute walkthrough of Kalmia on a production failure: a support agent resolves an ambiguous request silently, nothing crashes, and every conventional monitor stays green. Here is how Kalmia finds it, explains it, and helps ship the fix, then prove it.
The run that never errored
The demo opens in a production workspace for northwind-support-agent, a support agent for a home-goods store. Runs stream in and the Investigator reviews every one as it lands. Most are healthy. One is not, and Kalmia flagged it automatically, even though no exception was thrown, no tool call failed, and the agent wrote a confident, well-formed reply. Crash-based monitoring would never have seen it.
Why the run was flagged
Opening the run shows the developer view. A trace is long and noisy, so Kalmia distills it into its most important components and presents them as a logical flowchart a developer can read at a glance: what the customer asked, what the agent saw, what it decided, and what it changed. Each step of the chart links to the span in the trace that backs it.
The finding itself: the customer asked to redirect “my newest order”, and the agent changed order NG-4912 without an explicit reference to the order she meant. The timeline shows why that phrase was a trap. The account held two orders that could plausibly be “newest”:
- NG-4912, a $0.00 replacement order opened by store automation, the most recent by creation date,
- the order the customer actually placed herself, still in transit.
Faced with that ambiguity, the agent picked the automation-opened replacement and changed its shipping address without following up with the customer about which order she meant.
Grounded in the trace
A verdict is only useful if you can check it. Clicking a step on the chart jumps straight to the span it cites. Here, that is the tool call whose output lists both changeable orders, the exact fact the claim rests on.
The same jump works for every claim in the verdict, in particular for the mutating call where the shipping address on NG-4912 was changed. Nothing in the diagnosis is asserted without a span you can open and read for yourself.
From finding to rule to fix
Kalmia doesn't stop at the diagnosis. Alongside the verdict it suggests what should have happened: the agent should not modify an order unless it has been given an exact reference. Accepting the proposed rule teaches Kalmia to flag any future run that violates it. The one-off finding becomes a standing check.
Then the fix itself: Kalmia opens a pull request on the agent's repository that adds a confirm-first instruction to the prompt. The change also introduces a new variant of the agent, which is what makes the last act possible. The old and new versions can now run side by side in production and be compared directly.
Proving the fix in production
After the PR merges, traffic flows into both variants. Runs from the old variant that hit the same trap now fail explicitly against the rule we added, so they surface for developers to investigate instead of passing silently.
The closing beat is the comparison. In Analytics, adding a graph for the new rule across both variants, zoomed to the last twelve hours, shows the old variant still failing it while the new variant holds a flat zero. The failure was found, explained, fixed, and the fix is now being verified continuously, in production, by the same rule that caught it.