Making xgo Fast: 4x Faster CI and a 16x Smaller Test Cache
Ditto’s Go backend has 323 packages and about 3,700 Go files. Its tests mock functions with xgo, a tool that instruments Go source at build time so tests can patch any function, including ones from the standard library. xgo is a great tool, and it was also the slowest and largest part of our test pipeline.
Before this work, the CI test job took about 650 seconds. Its build cache archive was 16 GB compressed (70 GiB on disk), and restoring it alone took up to two minutes. On laptops, xgo caches grew past 70 GB until someone deleted them by hand.
We maintain a fork at omniaura/xgo. Over two days, a coding agent set out to halve the cache footprint and double performance. This post covers how we set that up, what we changed and what we measured.
CI numbers come from Depot CI runs on the same runner class. xgo numbers come from a local harness that runs each build cold, warm, after an edit and warm again, with isolated caches.
Why We Ran This on a VM
We wanted xgo to be faster and to use less RAM and disk. My laptop could not do this work. A cold build of Ditto’s backend under xgo needs several GB of cache per toolchain, and the old xgo held 3.4 GB of memory while the compiler ran next to it. Each experiment needs clean, isolated caches to measure fairly, so a few runs fill a laptop disk.
I had been trying out cmux VMs and wanted a real problem to give one. This one fit: a long-running task that needs room to build, measure, throw the caches away and build again. The VM had 8 vCPUs, 15 GB of RAM and 126 GB of disk. Even so, the agent had to clear old caches during the session to keep working, which is the problem we set out to fix.
Setup took two sign-ins. The VM came with the coding agents preinstalled, so I logged into GitHub and into Ditto Router. Both sign-ins use a one-time code: the terminal shows the code, a web page opens, and you enter it. Ditto Router’s login follows the same flow as gh auth login. After that, the agent could push branches and open pull requests.
Ditto Router is our inference endpoint. It gives you one OpenAI-compatible URL with memory, tools and traces. One command starts Claude Code through it:
npx @heyditto/cli claudeIf you haven’t signed in yet, the command prompts you with the same browser code flow, then starts Claude Code on your default endpoint. If you have no default yet, it lets you pick one. The endpoint records each request of the session as a trace.
Two Prompts
I knew this would be a long-running task, so I did not plan out the steps. I gave the agent requirements and let it iterate and measure.
The first prompt asked for performance: make xgo 2 times faster, and reduce its RAM use and disk cache size. The agent built a benchmark harness first, then worked through the slowest parts of xgo one pull request at a time. That covers most of this post.
The second prompt pointed it at our pipeline: now specifically work on CI and make CI way better as well. That produced the cache workflow changes and the largest time savings.
During the CI work, I added one more request: one subagent per open issue on the fork, with the lead agent reviewing and merging their work.
Where The Time Went
Every xgo test run does work before the Go compiler starts. It loads and parses the packages, finds the functions to instrument and rewrites them. Then it writes the rewritten files to an overlay. We call this setup. On Ditto’s backend it took 16.6 seconds for ./..., before any test compiled.
Profiling showed four problems:
- xgo instrumented the whole module on every command. Testing one package rewrote all 3,696 files of the module, although that package and everything it compiles amount to 257.
- Most of the pipeline ran on one core. Parsing, traversal, instrumentation and overlay writes ran one package at a time.
- Paths were canonicalized on every lookup. Each overlay access called
filepath.EvalSymlinks, onelstatper path component, which cost about 8% of setup CPU. - xgo kept its heap. After instrumenting, the xgo process held 3.4 GB of memory for the rest of the run, while the Go compiler needed its own. On an 8 GB machine, that combination got killed for running out of memory.
The fixes, in the order they landed:
- Targeted instrumentation (#17): xgo now instruments only the main-module packages the command builds. A single-package test went from about 13s to 5.5s warm.
- Return memory before running go (#15): the xgo process drops from 3.4 GB to 110 MB once the Go command starts.
- Cheaper resolution (#21, #25): a per-file expression cache, and skipping selectors that cannot reach a package variable. Traversal went from 2.47s to 1.73s.
- Symlinks resolved per directory (#23, #39): one resolution per directory instead of one per file access.
- Parallel everything (#27, #35, #36, #37): traversal, parsing, instrumentation, registration and overlay writes now use all cores. For each change, we checked that the overlay stayed byte-identical and that the race detector stayed quiet.
Setup is the time from starting xgo test until the go command starts compiling.
Faster Test Binaries
xgo inserts a small trap at the start of every instrumented function. The trap checks whether a mock is registered and, if so, calls it. In the original version, every trap unwound the stack and allocated. It did so even when no test had registered a mock.
Two changes fixed it:
- A fast path that skips unwinding when nothing can intercept the call (#9).
- Skipping the trap and its allocations entirely while nothing is registered (#13), then enabling the trap per function when a mock targets it (#19).
We found a related bug late in the work. xgo test added a few standard library packages (time, os, net/http, io and others) to its package list so it could load them. As a side effect, xgo treated them as packages the user asked to test, so every function in them got a trap. Calls like time.Time.Sub started to allocate, which broke allocation tests in Ditto’s backend. #46 keeps those packages on their normal allowlist. time.Now is still mockable, and a regression test checks that the other calls allocate nothing.
Halving The Disk
xgo builds a patched copy of the Go toolchain for each Go version, plus a separate build cache for instrumented builds. Three things made those large:
- A full toolchain copy. The patched GOROOT copied all 297 MB of Go, although only about 20 files change. It now hard-links the unchanged files, leaving 67 MB of unique data (#10). The same change stopped writing 476 MB of setup builds into the user’s default Go cache, where nothing read them again.
- Nothing collected old caches. Go trims a build cache only while something uses it, so every xgo or Go upgrade orphaned the previous cache for good. xgo now marks what it uses and removes toolchains and caches left unused past their max age (#11). It keeps at most three build caches per toolchain (#47).
- Removal had to be safe. Two xgo runs can share a toolchain. #45 keys toolchains by revision, builds them in a staging directory and renames them into place atomically. File leases stop one run from deleting a cache another run is using.
For gin, xgo’s total disk use dropped from about 1,032 MB to 326 MB.
The CI Cache
The biggest CI win came from how the workflow saved its cache, more than from xgo itself.
The old workflow saved its build cache only when every test passed. Our main branch had failing tests at the time, so the cache never updated and every run started close to cold. The archive it did restore had grown to 146,000 entries, most of them from old xgo versions that no current run would ever read.
The new workflow changes three things:
- It saves whenever the tests ran to completion, pass or fail. A build or setup failure still skips the save, so a broken run cannot poison the cache.
- It prunes before saving. After restoring, it marks every cache entry as old. Go refreshes the timestamp of each entry a build uses. After the tests, a small script deletes every entry that is still old, but keeps any output that a used entry points to. The saved archive holds only what this commit needed: 1 GB instead of 16 GB.
- Its cache keys fall back across branches. A new branch starts from the latest cache with the same
go.sum, then from any recent cache. Depot shares its cache across branches, so pull requests start warm.
We also build tests without DWARF debug info (-gcflags=all=-dwarf=false -ldflags=-w). Locally, that cut the build cache by 17% and peak memory by 23%. Tests don’t need debug info, and a stack trace still has file names and line numbers.
To separate the effects, we ran the old xgo with the new workflow:
The workflow change halves the step. The xgo upgrade cuts the rest by two thirds. In the final setup, 232 of 239 packages come from cache.
How The Work Got Done
The agent did the work in pull requests against the fork. The setup:
- A benchmark harness ran each build cold, warm, after an edit and warm again, with isolated caches and memory sampling, so every PR had numbers before and after.
- Ditto Review reviewed every PR. For changes to instrumentation logic, we waited for its verdict before merging. Some of its findings were false positives, and those got a reply with a test or a measurement as evidence.
- One subagent took each open issue, and the lead agent reviewed and merged their PRs. That review caught a 230ms regression in warm setup in the concurrency PR, which its author then cut to 8ms before it merged.
- Each release is one command,
go run ./script/release. It tags the release, publishes it and warms the Go module proxy, so Ditto’s backend could pick up a fix minutes after it merged.
To measure CI, the agent pushed throwaway branches with a copy of the test job that only runs on demand. It triggered each run through Depot. That kept the main CI configuration unchanged and let us compare cold and warm runs of the same commit.
Update: One Allocation per Trapped Call
The next slice went after the trap code itself. Each instrumented function used to carry its own code to copy its inputs and outputs. It also built two slices of arguments for the trap. So a call that a mock or interceptor could see allocated up to six times.
#48 moves that work into helpers that the functions of a file with the same signature share. It also packs the trap’s arguments into one block of memory, so a trapped call allocates once. On gin, the helpers dropped from 796 to 375. The instrumented function still calls the trap itself, because the runtime finds which function it is from the caller’s address.
Median of three runs. Calls take this path once any mock is registered in the test binary.
We also measured what this does not change. On Ditto’s backend, test times and the build cache stayed the same within noise. xgo’s cache there is now only about 10% larger than a native one, and the test time comes from the tests themselves. The shared helpers trim xgo’s added code in gin by 7.5%.
Update: No Defer in Every Function
The biggest remaining cost was a defer. Every instrumented function carried one, in case an interceptor wanted to run code after the call. Each defer adds a small wrapper function and extra code at every return.
#49 removes it. Mocks don’t need code after the call, so with only mocks registered the function now runs straight through. When an interceptor does need to run after the call, as tracing and recorders do, the function hands over to a helper. The helper defers that post step and calls the function again. A flag on the goroutine makes the second call skip the trap. xgo marks the helper as a wrapper, so it never shows up in stack traces, and t.Helper() still points at the right line.
Machine code in the gin package beyond what a native build has. Native: 478 KB.
On Ditto’s backend, the test binary of its pkg/api/v2 package shrank from 314.6 MB to 306.2 MB. That brings xgo’s code overhead in the backend’s own packages down 14%. The build cache stayed the same, since dependencies dominate it.
There is one trade-off, and only while a post step is active. The function’s body runs one call deeper, so code that counts stack frames from inside it sees the function as its own caller. With only mocks registered, nothing changes.
What’s Next
- Go 1.28. The fork already supports Go 1.28 development builds. Once a release candidate ships, we will verify it against Ditto’s backend.
- Upstream. We offered all of this work back to xgo in one pull request, xhd2015/xgo#399: our 50 commits squashed onto upstream, minus the fork’s own release tooling. Thanks to xhd2015 for xgo itself: instrumenting Go source is hard, and the design held up well under this much change.
- Binary size. An instrumented test binary is still larger than a native one. The largest remaining cost is now the helpers that copy each function’s inputs and outputs for the trap.
The fork is public at github.com/omniaura/xgo. Until the upstream pull request lands, you can try it: replace github.com/xhd2015/xgo with github.com/omniaura/xgo at the latest release in your go.mod.