The benchmark that caught a 31× slowdown

A benchmark is an instrument, not a showcase

The LLM decides what to do; the tools do the work. In between sits the ToolExecutor — parallel-safe vs sequential classification, path-conflict detection, cascading cancellation, and the serde machinery around every ToolCall. This layer runs on every step of every worker, so its characteristics shape the latency and safety of any operation.

That is why the harness is built into the product. Fixtures are created automatically in a temp directory and removed after; each scenario repeats 5–10 times with a warm-up call, and the numbers below are one run on macOS, 10 cores, release build. No network, no LLM — the memory scenario uses an offline TF-IDF embedder, so the whole suite can run in CI.

31× from a regex cache

The first version of extract_symbols compiled two regular expressions (regex::Regex::new) per file. For code_symbols with its limit that was tolerable — the loop stops after the first ~25 files. For repo_map, which honestly walks the whole tree, it was a catastrophe: DFA compilation dominated the runtime.

The fix moved the regexes into OnceLock statics (compiled once per process) and parallelized file reads in repo_map via tokio::spawn + join_all, preserving order. An A/B run over 240 synthetic Rust files:

That is a 31× improvement on a debug build; in release the same tree takes 6.6 ms. The live reference point: before the fix, repo_map over this very project's repository (103 files) took 106 ms in release. After the fix — single-digit milliseconds.

The silent serialization

The second catch was subtler. ToolExecutor splits tools into parallel-safe and sequential; a tool missing from the classification goes sequential "just in case". When web_crawl, web_feed, code_symbols and repo_map landed, nobody added them to the classification — and any batch of several such calls silently ran one at a time, with no errors and no warnings.

The harness showed it in numbers: a spawn batch of 8 × web_feed produced 1.00× — exactly what sequential execution produces — while parse_html under the same conditions produced ~3×. After adding the tools to parallel_safe (and a classification test to keep them there):

Every new read-only tool must ship with a classification test — otherwise parallelism disappears without a trace.— Fathom runtime benchmark notes

The rest of the scoreboard

The same run measures the machinery around the tools. Serializing a ToolCall's arguments costs ~750 ns; the executor's overhead over raw registry dispatch is ~2.5 ms for a single-call batch and amortizes to 753 µs per call in an 8-call batch. The execution layer disappears against the work the tools do.

For CPU-bound batches the difference between execution modes is dramatic: 8 × parse_html of a ~1 MB table runs 3.78× faster under execute_batch_spawn (tokio tasks spread across cores) than sequentially, while join_all — which polls futures on one thread — only helps for I/O waits. HTML selector and RSS parser throughput are measured in isolated fixtures, while the executor remains responsible for preserving ordering and concurrency semantics.

A realistic mixed turn — four reads, three writes and a grep in one batch — is partitioned automatically: five calls run concurrently, three serialize, and the result vector still matches the original call order. Total wall time: 70.9 ms.

Run it in CI

Everything above reproduces with one command — fathom bench --scenario all — with fixtures that create and clean themselves. The scenarios: dispatch, parallel-io, parallel-cpu, mixed, parse-scale, extract-json, feed-parse, code-map and memory. Because nothing touches the network and nothing calls an LLM, the suite runs in CI on every build — which is exactly how the next silent regression gets caught.

And because synthetic numbers are only half the story, fathom stats reads the SQLite tracing of a real session and reports per-tool p50/p95 durations and the batching coefficient from production runs.