Why we wrote our own microVMM

We are shipping an open-source sandbox for AI agents. brig runs a coding agent inside a sandbox, while hull provides the microVM runtime that brig uses on macOS. Both live under the brig-sh organisation. The model is deliberately simple: each agent session gets its own hardware-isolated microVM; the agent sees a private home directory and the project you specify; and brig rm discards the sandbox while leaving your host files intact. We have already written about what a sandbox must guarantee for the term to mean anything, and about why an agent should be treated as untrusted by default.

Production systems run on Linux, while many developers work on macOS. For brig to provide the same isolation guarantees in both environments, it needs the same execution layer, device model and observable behaviour on both hosts. Linux has several mature options for building that layer. macOS has fewer.

We therefore implemented a microVMM called hvi, licensed under Apache-2.0 and available at brig-sh/hvi-vmm. The main architectural requirement was a common hypervisor interface. Without one, supporting another host means duplicating the VMM state and device-integration layers.

Figure 1: two approaches to supporting a second hypervisor. Without an explicit interface, each platform has separate VMM state, MMIO and interrupt-controller implementations; the diagram uses actual libkrun file names. With an interface, the boot protocols, device models and guest-RAM accessor are shared, while each host backend is implemented in one file containing roughly 800 to 1,100 lines of product code.

Evaluating a Firecracker port

We first considered porting Firecracker to Apple's Hypervisor.framework. Firecracker is a reference microVMM written in Rust, licensed under Apache-2.0 and capable of booting Linux guests in milliseconds.

We analysed that option in detail. Unless otherwise noted, every Firecracker measurement below comes from commit 753a817, dated 6 August 2026.

Firecracker does not define a common hypervisor interface: grep -rn 'trait Hypervisor' src/ returns nothing. Its VMM state layer is vstate/kvm.rs, which is built directly around kvm_ioctls::Kvm.

Under src/vmm/src, 42 files totalling 19,592 lines reference kvm_ioctls or kvm_bindings. Some contain portable logic that merely happens to mention KVM; much of the code is intrinsically KVM-specific. The arch/aarch64/gic/ subtree consists of 2,007 lines of KVM VGIC device-attribute plumbing, while cpu_config/ contains 6,327 lines of CPU templates expressed as KVM CPUID and MSR modifications.

The vCPU loop itself is portable. Its structure—run, exit, dispatch—is common to both hypervisors. The exit representation is not. KVM returns an already decoded MmioRead(addr, data). On arm64, Hypervisor.framework returns HV_EXIT_REASON_EXCEPTION, leaving the VMM to decode ESR_EL2. That decoder is 51 lines of product Rust in our tree; Firecracker has no corresponding concept or obvious place to put one.

Snapshots expose the same dependency at the persistence layer. This is Firecracker's saved per-vCPU state on x86, verbatim:

pub struct VcpuState {
    pub cpuid: CpuId,
    pub saved_msrs: Vec<Msrs>,
    pub debug_regs: kvm_debugregs,
    pub lapic: kvm_lapic_state,
    pub mp_state: kvm_mp_state,
    pub regs: kvm_regs,
    pub sregs: kvm_sregs,
    pub vcpu_events: kvm_vcpu_events,
    pub xcrs: kvm_xcrs,
    pub xsave: Xsave,
    pub tsc_khz: Option<u32>,
}

The snapshot file format is the KVM ABI. Hypervisor.framework can neither produce nor consume these structures. Supporting snapshots on macOS would therefore require a second, incompatible format rather than an adapter—precisely the kind of divergence Firecracker's snapshot version policy is designed to prevent.

Much of the platform coupling is independent of KVM. EventFd appears in 36 files under src/vmm/src, epoll in 17 and io_uring in 14. The seccomp-filter JSON spans 2,855 lines, while the jailer contains 3,170 lines built around cgroups, chroot, pivot_root and namespaces. macOS has mechanisms that pursue similar goals, and we implement both forms of confinement, but they share no implementation. Moreover, event-manager, on which every Firecracker virtio device is built, does not compile on macOS because it depends directly on vmm_sys_util::epoll.

Firecracker does not target macOS

Firecracker's maintainers have addressed macOS support directly.

Issue #2845 was closed as not_planned in April 2023:

We do not plan to support MacOS. Firecracker is highly dependent on KVM. To change this would require significant effort required and we do not currently see a significant use case.

Issue #5017, a "Running Firecracker on macOS with Apple Silicon" proof of concept, was opened and closed on the same day in January 2025, with further discussion redirected elsewhere. That decision is consistent with Firecracker's scope: running Linux workloads at AWS scale.

The prototype is sometimes cited as evidence that the port is straightforward, but its implementation shows otherwise. The branch changes 71 files (+2,563/-245), and its notes acknowledge that substantial amounts of code were disabled with #[cfg(target_os = "linux")] to produce a minimal working version. The new backend itself is only 92 lines because it uses VirtualMachineConfiguration, LinuxBootLoader, VirtioBlockDeviceConfiguration and NATNetworkDeviceAttachment. These belong to Apple's Virtualization.framework, which supplies an entire VMM rather than a low-level hypervisor API: it owns the vCPUs, memory, virtio-PCI devices and bootloader. The prototype therefore uses none of Firecracker's device models, transport, boot path or vCPU loop. What remains is primarily the CLI and JSON schema.

Virtualization.framework remains useful, and we ship it as a selectable hull backend alongside QEMU. It is fast, maintained by Apple and can boot a Linux guest with very little code. Its trade-off is control: clients configure a virtual machine but do not implement its internals. We cannot guarantee the same device model across hosts when the framework owns the devices on one of them. hvi is the third hull backend and the only one that talks directly to Hypervisor.framework.

libkrun's Firecracker-derived port

libkrun provides a useful reference because it has already implemented the port: KVM on Linux and Hypervisor.framework on macOS/arm64, starting from Firecracker-derived code.

At commit e6cdb55, dated 16 September 2026, its hand-written Hypervisor.framework integration spans 2,767 lines on top of 4,712 lines of generated bindings. The platform adaptation also includes 484 lines that use kqueue in place of epoll and 114 lines that use pipes in place of eventfd.

The file layout shows how the port is structured:

libkrun could not reuse a common hypervisor abstraction from Firecracker. It split the VMM state and MMIO layers by platform and now maintains those implementations side by side. Windows support later added a third variant of the same layer. The port also omitted snapshots, MMDS, the rate limiter, io_uring, the jailer and the HTTP API. macOS support is arm64-only because Hypervisor.framework provides no in-kernel interrupt controller on x86; a VMM would need to emulate the LAPIC, IOAPIC and PIT entirely in userspace.

In practical terms, the port would mean inheriting Firecracker's full codebase, replacing its Linux-specific layers, and maintaining a long-lived fork of a project that has twice declined macOS support.

The hvi architecture

hvi contains about 11,500 lines of product Rust under src/, excluding tests. We measured it at 19a764c, 14 September 2026, by running tokei on a clean archive and separating code inside #[cfg(test)] modules. The exact split is 11,462 product-code lines and 5,284 test-code lines across 246 test functions. Unless noted otherwise, the hvi source counts below exclude blank lines, comments and tests. Three host backends sit behind one CLI:

Guest	Host	Hypervisor API	Backend product code
aarch64	macOS, Apple silicon	Hypervisor.framework	1,057 lines
aarch64	Linux	KVM	812 lines
x86-64	Linux	KVM	956 lines

These figures are not a feature-parity comparison with Firecracker, which includes snapshots, an HTTP API, MMDS, rate limiting and years of hardening at a scale where we do not operate. They show the size of hvi's hypervisor-specific surface: each backend is implemented in one file containing between 812 and 1,057 lines of product code.

Everything else is host-neutral: boot-image parsing, the guest-memory layout, the virtio device models and the event ledger. These components depend only on a guest-RAM accessor. The hypervisor interface has been part of the design from the beginning.

We expected the hypervisor implementations to account for most of the platform-specific work. In practice, boot protocols differ much more than the hypervisors do.

On arm64, there is no firmware or bootloader. The VMM loads the kernel Image at the offset specified by its header, builds a devicetree describing PSCI, the GIC, the timer and one virtio-mmio node per device, then enters the kernel directly at EL1. On x86-64, there is no devicetree. A bzImage requires a boot_params zero page, an e820 map, an Intel MP table for CPU discovery without ACPI, identity-mapped page tables, a flat GDT and a hand-written transition into long mode.

Figure 2: booting the same Linux guest on two architectures. On arm64, the VMM loads the kernel and devicetree before entering EL1. On x86-64, it constructs the zero page, memory and CPU tables, page tables and GDT before entering long mode. The hypervisor-independent device and memory layers are shared.

Three boot requirements are particularly easy to miss because their failure modes produce no diagnostic output:

VMX still requires real-mode state for a direct long-mode boot. set_tss_address and set_identity_map_address must be configured even though the guest never executes in real mode. Omitting them causes a triple fault.

Early boot requires an entropy source. If the vCPU's CPUID does not advertise RDRAND, the kernel can stall while waiting for randomness before the console is available.

The x86 boot path requires a working CMOS RTC. read_persistent_clock64() polls the update-in-progress bit with interrupts disabled. An unimplemented I/O port returns 0xff, so the bit never clears and the guest loops indefinitely before console output begins.

Guest-memory layout introduces a separate design constraint. RAM cannot occupy one contiguous range starting at address zero. If it overlaps the virtio-mmio window, it shadows the device registers: KVM satisfies accesses from memory without a VM exit, and the devices silently stop responding. If RAM overlaps the in-kernel LAPIC page, KVM rejects the region with EEXIST. hvi therefore ends low memory at the device window and resumes it above 4 GiB. Tools that inspect guest-physical memory must account for this hole.

Adding Linux backends

We started with macOS because that was the unsupported platform. Linux parity was still a requirement: production runs on Linux, and records produced at the sandbox boundary must have the same meaning on both hosts.

The Linux implementation validated the hypervisor interface. The arm64 KVM, x86-64 KVM and macOS backends each occupy one file and share every other module. All three boot Linux guests with virtio-blk, virtio-net and virtio-vsock, and CI exercises each configuration.

The guest interrupt-controller version follows the host: a GICv3 host exposes vGICv3, while a GIC-400 host exposes vGICv2. The latter limits the guest to eight vCPUs; this is an architectural constraint, not an implementation bug.

We also changed where the boot tests run. Hosted arm64 runners do not expose /dev/kvm, so arm64 testing always required self-hosted hardware. Hosted x86 runners do expose KVM, and we initially boot-tested the x86 backend there on every pull request. However, the guest-kernel installation step hung five times in one afternoon, including one 55-minute run for a step that normally took two minutes. A persistent runner installs the kernel once and reuses it, removing that dependency from the test path and running the guest directly on bare metal. All three backends now boot-test on self-hosted machines. Those jobs do not run for pull requests from forks because fork branches contain untrusted code and the runners are persistent. Hosted runners continue to perform builds, lints, unit tests, the dependency audit and the macOS sandbox self-test.

Instrumentation at the VMM boundary

Portability was not the only reason to own the VMM. The VMM maps guest memory, can stop vCPUs between guest entries and processes every virtio request from the guest. Debuggers, tracers, profilers and crash dumpers need access to some combination of those facilities, but most VMMs do not expose them. The VMM is also outside the agent's control, so the agent cannot rewrite records captured there. That property motivates who audits the AI agent.

Instrumentation code does not belong directly in the VM-exit loop, so hvi exposes a separate four-trait interface from that loop. A tool receives a handle when it attaches, runs at a safepoint on CPU 0 between guest entries, may pause the other vCPUs to obtain a consistent view, and may write records to the VMM event stream. hvi includes two tools built on this interface: a memory dumper and an I/O tracer. The examples/watch_guest.rs example counts device I/O and samples the boot vCPU once per second, and is intended to be used as a plugin template.

Implementing these tools exposed three important rules:

Keep the safepoint path minimal. A tool with no pending work should detect that state with one atomic load and return.

Every successful pause requires exactly one resume. If pause() returns true, all other vCPUs are parked. Every exit path, including early returns, must call resume() exactly once or the VM remains stopped.

Publish work before waking the vCPU. An idle guest may remain in WFI on arm64 or HLT on x86 and will not reach a safepoint by itself. A timer-driven tool must first set its pending-work flag and then kick the vCPU. Without the kick, it runs under load but fails when the guest is idle.

A read-only tool should create its own read-only mapping of guest RAM from the descriptor and region list supplied by the interface, rather than borrowing the VMM's writable mapping. This requires one mmap call when the tool attaches and prevents the tool from corrupting guest memory even if the rest of its implementation is faulty.

Confinement and rust-vmm reuse

Before servicing any guest I/O, the VMM applies its own confinement by default: a Seatbelt profile on macOS and seccomp-bpf allowlists on Linux. The virtio backends parse guest-controlled data on the same threads that run the vCPUs. A vulnerability in a backend therefore compromises a process that would otherwise retain the host's full system-call surface. Filtering at this layer has practical value: in ITScape, the first public guest-to-host escape on KVM/arm64, urunc's seccomp filter was the only default runtime control that blocked the relevant attack surface.

This confinement operates at a different layer from a jailer. Firecracker's jailer is a launcher: it configures a chroot, cgroups and namespaces, then drops privileges before executing the VMM. It establishes an outer boundary around the process. When hvi runs under a container runtime, that runtime provides the corresponding outer boundary. Neither mechanism restricts which kernel services the VMM process may request after startup; seccomp-bpf does.

The macOS and Linux mechanisms also have different semantics. seccomp-bpf filters system calls. Seatbelt applies policy to named operations on resources, so it does not filter a system call that touches no governed resource. The macOS profile can therefore be short and deny-by-default: hvi acquires everything it needs before starting the guest and retains those resources as file descriptors, while Seatbelt primarily governs resource acquisition rather than I/O through descriptors already held. If confinement occurred one step later in startup, the profile would need to grant filesystem and socket operations that it currently denies. Linux instead installs two filters: a restrictive vcpu filter and a vmm filter for the main thread and host-side I/O. Each backend reports the policy it installed, making a confinement failure visible. The self-tests install the production policies and verify both permitted and denied operations in CI.

We reused the implementation but not its policy. seccompiler, the rust-vmm crate extracted from Firecracker, provides the system-call tables and BPF code generation. We did not vendor Firecracker's filter lists, although the licence permits it. Firecracker's lists target musl and devices driven by epoll and io_uring; hvi uses glibc and blocking reads on dedicated threads. Its filters would allow open, stat, io_uring_* and epoll_*, which hvi never calls, while omitting openat, statx, rseq, set_robust_list, sched_getaffinity and clone3, which a glibc-linked Rust binary may need before reaching main. Reusing the lists would make the policy both unnecessarily permissive and unable to start hvi. We did incorporate Firecracker's coverage of uncommon execution paths: entries not observed in our own traces are labelled "safety net" in hvi's pinned aarch64 and x86-64 policies.

We applied the same approach to the device layer after opening the repository. Virtio feature bits, device IDs, MMIO register offsets and the virtio_net_hdr_v1 length were previously hand-transcribed constants spread across three files. They now come from virtio-bindings, generated by bindgen from the kernel headers, eliminating a class of transcription errors during feature negotiation. The custom x86 COM1 implementation is now a thin wrapper around vm-superio's Serial type.

We later made the same change to the boot paths. We had initially parsed both image formats ourselves because linux-loader would have introduced a second vm-memory abstraction alongside our custom guest-RAM accessor. Once guest RAM was represented as a vm-memory collection behind the existing device and plugin APIs, that objection no longer applied. PE and load_dtb now load the arm64 Image; BzImage, Cmdline and LinuxBootConfigurator handle x86. Adopting the crate also added ELF support, so hvi can boot an uncompressed vmlinux without running the kernel decompressor. CI tests this path on every run.

Some components remain local because the available crates do not match the required hardware model. hvi retains its PL011 and MC146818 CMOS implementations because vm-superio has no PL011 and implements the arm64 PL031 RTC rather than the CMOS device expected by an x86 guest. hvi also constructs the devicetree, e820 map, page tables and GDT, and MP table. linux-loader loads the image and writes the zero page, but it does not define the machine presented to the guest. The general rule is to reuse a crate where its abstraction fits, while deriving configuration from the system that will actually run it.

Current limitations

hvi remains a work in progress. The main missing features, in approximate implementation order, are:

rust-vmm virtqueue. Our split-virtqueue implementation is written and reviewed in-house, while rust-vmm provides virtio-queue. This is the highest-priority replacement because the virtqueue handles more guest-controlled input than any other component in the tree, and the shared implementation receives broader review. The ecosystem is divided: Cloud Hypervisor uses the crate, while Firecracker retains an in-tree queue. Since the boot loaders moved to linux-loader, the virtqueue is the last major rust-vmm component that hvi still implements itself.

Snapshots. hvi currently has no snapshot support. Unlike Firecracker, its format would not need to mirror KVM structures; it could be defined in terms of hvi's hypervisor-neutral vCPU state. The trade-off is that there is no existing format with which to maintain compatibility.

Network egress. With --net, a userspace stack inside the VMM handles ARP, ICMP, DNS and DHCP. It recognises TCP traffic but does not yet forward it. Egress requires either a gateway socket or, on Linux, a tap device. Some required operations are not yet permitted by the confinement policy; the remaining work is tracked in the open issues.

virtio-fs on Linux. Directory sharing works on macOS but is not yet connected on Linux.

Other limitations include support for only one disk and one NIC, with no hotplug, PCI, HTTP API, MMDS or rate limiter. hvi also lacks the operational hardening that comes from running a VMM at very large scale. On macOS, it requires the hypervisor entitlement and an interactive session. The binary that actually runs must be signed in place because copying a signed Mach-O invalidates its signature.

The decision would be different for a different workload. For snapshot-based, low-latency cold starts on Linux at scale behind a stable public API, with macOS as an optional target, adopting Firecracker as-is would be the better choice. Those were not our requirements.

Takeaways

Our decision not to adopt Firecracker was based on architecture rather than a preference for in-house code. A macOS port requires a long-lived fork; libkrun demonstrates its maintenance cost; and the resulting design still would not expose the instrumentation interface we needed.

VMM portability depends on an explicit boundary between host-neutral code and the hypervisor backend. Firecracker has no such boundary: its closest equivalent is vstate/kvm.rs, and its snapshot format directly embeds the KVM ABI. That is a reasonable design for Firecracker's purpose—running Linux workloads at AWS scale—but it does not fit a runtime that needs equivalent behaviour on macOS and Linux. Owning the VMM also gives us a stable point from which to inspect guest memory, vCPU state and device I/O.

hvi is small, early and open source. Its four-trait instrumentation interface and examples/watch_guest.rs provide a compact starting point for tools that need access at the VMM boundary. Issues and technical discussion are welcome.

You can try hvi through brig, which runs a coding agent in its own microVM on macOS or Linux. It is licensed under Apache-2.0 and requires no account. hull is included:

curl -fsSL https://brig.sh/install | sh
mkdir -p ~/code/demo
brig run claude ~/code/demo

To verify the isolation boundary, run uname -r inside the sandbox: it reports the guest kernel rather than the host kernel. brig.sh pairs each security claim with a command that tests it and documents six things brig does not do.