Posts
Agent evals: measuring the noise before comparing agents
In our previous post we replaced a team of agents with one agent loop. It diagnosed 1.5x as many faults without the Kubernetes tools, at 60% less cost. But there is a problem with evaluating agents. Running the same system twice can give very different results, and in our runs the same build, run twice, moved between 0.65x and 1.38x. So how much of the gain was real and how much was noise? That is what we set out to measure.
How Brig could have interrupted the Binance attack
Binance's initial incident report names the path a malicious interview repository overwrote on an engineer's laptop. That write is a boundary a microVM sandbox can enforce before anyone knows the code is malicious. This post walks through the Brig v0.5.0 setup that contains it: a disposable guest home, no credentials, an egress allowlist, and a credential broker for the agents that do need an API.
We deleted most of our agents. Our evals told us to.
Earlier this year, we ran NOFire’s root cause investigations on a multi-agent architecture. Daily evals against SREGym and our own test cases showed us what actually improved diagnostic accuracy. A single agent loop delivered the best results: it diagnosed 50% more faults with 49% fewer tool calls, cut the cost per investigation by 60%, and reached a diagnosis 2.5× faster.
Brig on hvi is immune to the PwnEscape guest-to-host escape
A guest-controlled checksum can corrupt an IPv4 header and leak Linux host-kernel memory. hvi drops that metadata before it reaches the host. Here's the boundary, the regression test, and what Brig users on Linux need to update.
Why AI Agents Need a Control System Before Touching Production
An agent needs a versioned, inspectable account of production before it can investigate an incident, and a control system outside itself before it can act on one. This is what each of those has to do.
The Sandboxing Manifesto
A sandbox is a falsifiable claim. "Sandbox" has been diluted until it guarantees nothing. Ours is a definition sharp enough to be proven wrong: seven checkable properties, enforced by hardware-level microVM isolation, not by configuration.
Tight enough to move fast. Deep enough to matter.
A tight, multidisciplinary, fully distributed team spanning a thirty year range in age and experience: systems researchers, software engineers, and production veterans built into a lean execution and research machine, not a headcount to grow past.
Design for breach: the agent in production is untrusted by default
The industry just agreed that assume-breach is the operating model. For AI agents acting in production, that means hardware-level isolation and hard controls at the execution boundary, not trust in the model.
The Context & Control Model: The missing piece of the production AI stack
AI agents can't act autonomously in production without a model of how it behaves, and governance that constrains every action they take.