Kalmia is almost 8x faster at searching traces than conventional trace storage
Andy Xu, Calvin Huang · August 6 2026
An agent in production emits trace data faster than a general-purpose database can keep up with. A single run can be thousands of steps, each carrying a full prompt and response, and the runs never stop, so a live system accumulates them by the million. Ask a conventional observability database to search across all of it and queries crawl, and teams end up sampling their data or not querying production at all. Kalmia is a storage engine built specifically for agent traces: it runs a full-text search across the whole corpus in well under a second, and keeps ingesting new traces without slowing that search down. This note explains why that is hard, how Kalmia does it, and how it measures against ClickHouse, the fastest columnar database in common use for observability, on 1.4 million real agent traces.
Querying millions of traces
An agent run is a trace: a tree of steps, one per model call, tool call, or decision. Two operations run against that data constantly, and each is hard at the scale a production system reaches.
- Search. To find where a run went wrong, you search for an error phrase across every trace. That is a needle in millions.
- Ingestion. New traces arrive continuously, in bursts, while people are querying.
There is also an economic constraint. At billions of spans, the data has to live on object storage (cloud storage such as Amazon S3), because it is cheap and effectively unbounded; keeping everything on fast local disks would cost far too much. But object storage is reached over the network, so anything that reads a large amount of data, or reads it one item at a time, pays for it in latency. A general-purpose columnar warehouse like ClickHouse hits structural limits on exactly these operations, which is what the measurements below show.
How Kalmia is built
Kalmia is a storage engine purpose-built for agent traces, rather than a general-purpose warehouse adapted to them. Each of the two hard operations above is served by a component designed for it: a search index tuned for phrase queries, and an ingest path built for high write concurrency on object storage. Nothing falls back on a tool built for something else.
Results
We measured this configuration against ClickHouse, with both engines set up the way they run in production: durable data on object storage with a local cache, and ClickHouse given a real full-text index, so the comparison is fair. Both loaded the same corpus of 1,443,629 real agent traces on one 32-vCPU machine, and every operation was measured together in a single run.
| Operation | Kalmia | ClickHouse | Ratio |
|---|---|---|---|
| Full-text search across every trace | 626 ms | 4,628 ms | 7.4× faster |
| Single write, flush (durable) | 90 ms | 158 ms | 1.8× faster |
| Single write, visible (queryable) | 97 ms | 167 ms | 1.7× faster |
| Concurrent ingest throughput | 165 writes/s | 37 writes/s | 4.5× more |
| Search while writing (under load) | 1,441 ms | 4,770 ms | 3.3× faster |
Figure 1: The production configuration versus ClickHouse, both on object storage, measured together on one corpus of 1.4 million real traces. Read latencies are medians over repeated queries; write latencies are warm medians; throughput and reads-under-load are a single 30-second run.
Full-text search latency
An index reads little; a scan on object storage reads a lot
Search is where the gap is widest in practice. Finding an error phrase across every trace returns in under a second, against nearly five seconds on the warehouse - the difference between debugging a failure interactively and waiting on each query. And because these wins come from independent parts of the system, they hold at once: reads stay fast even as new traces pour in.
Method
Both engines ran from the same script in the same production configuration, on one loaded copy of the corpus assembled from public datasets, with ClickHouse given a real full-text index so the comparison is fair, and every operation was measured together in a single run.
Why this matters
When an agent misbehaves in production, someone has to find the run that failed, read what happened inside it, and ship a fix. Every step of that loop is a query. When each one takes seconds instead of a moment, an investigation stretches from minutes into hours, and at scale teams stop querying their production data at all.
Kalmia is built so that loop stays fast at production scale. You can search millions of traces for the error you are chasing, open the exact run that produced it, and keep ingesting new traces the whole time, without waiting on the database between steps. Finding a failure and fixing it stays a matter of minutes.