bhyve sandbox substrate

The jail backend is the default sandbox substrate; bhyve is the second substrate for the hardware-isolated path. Session A built the pool-control plane and the guest image; this appendix is the substrate-verification receipt. Pool warm → checkout → SSH-resume → return → drain all gate-pass on honor.

Pieces

• /vms/templates/<tpl>.img — raw guest disk image. Built by benchmarks/rigs/bhyve-sandbox-image-build.sh: mdconfig+mount the FreeBSD-15.0-RELEASE base, grow p4 to 10 GiB, drop /etc/rc.d/coppice-net to pick up per-entry /etc/coppice-instance (IP, hostname), install python311
  • py311-ipykernel via pkg-in-chroot, bake the pool SSH pubkey into /root/.ssh/authorized_keys, set PerSourcePenalties no in sshd_config. Wall time on honor: ~7 min (dominated by UFS-in-vnode write throughput during python311 extraction).
• /vms/templates/<tpl>.conf — optional bhyve sidecar config. Lets one template override boot mode, guest memory wiring, SSH user, shell program, disk device, network device, mutation strategy, and extra bhyve -s slots without forking the pool controller. The legacy FreeBSD path works unchanged when the sidecar is absent.
• /usr/local/sbin/coppice-bhyve-pool-ctl — shell lifecycle controller. Commands: init, warm <tpl> —count N, checkout <tpl>, return <id>, drain <tpl>, list [—json], plus the host-console pair console-start <tpl> <sandbox-id> and console-stop <sandbox-id>. State at /var/db/coppice/bhyve-pool/<tpl>.json.
• /var/db/coppice/bhyve-pool/console/<id>.json — host-console state. Tracks the bhyve framebuffer port, VM name, tap, supervisor pid, and cloned disk for a live console guest such as windows-server-2025.
• coppbhyve0 — dedicated bridge at 10.77.0.1/24 (disjoint from the jail bridge coppicenet0/10.78.0.0/24). Pool entries get .50–.249.
• /root/coppice-signing/pool-key{,.pub} — ed25519 keypair. The image-build script generates it on first run; the public half is baked into every pool image.

Lifecycle (hot path)

1. warm copies the template image per-entry, applies the template’s mutation mode (/etc/coppice-instance on UFS for the FreeBSD default, an msdosfs seed disk for the Windows path, or a NoCloud CIDATA-labeled FAT seed disk with a single partition attached as ahci-hd for Linux cloud images, carrying the exact meta-data / user-data / network-config filenames cloud-init expects plus a simple Coppice env/key fallback for Linux guests), starts bhyve with the template’s boot profile (bhyveload or UEFI), polls SSH as the template’s configured login user, then kill -STOPs the bhyve process.
2. checkout picks the first available entry, flips it to in-use, and kill -CONTs. SSH on the guest answers within ~150 ms on honor — dominated by TCP-probe latency plus sshd wake-up.
3. return destroys the in-use VM and respawns a fresh one from the template (v1 policy; per-entry ZFS snapshots are a future optimization).
4. drain tears every entry down, removes /vms/bhyve-pool/<id>.img, and deletes the state file. list afterwards is empty.

The Windows host-console path is parallel to this, not a replacement:

1. console-start clones the template image, reserves a host VNC port under a lockf-guarded critical section, attaches a tap, and boots bhyve with a framebuffer on 127.0.0.1:<port>.
2. bhyve is launched under a tiny supervisor script. If bhyve exits 0 (“guest rebooted”), the supervisor starts it again on the same disk, tap, and VNC port. That is what lets the portal ride through Windows first-boot reboots instead of treating them as fatal.
3. console-stop kills the supervisor, destroys the VM, tears down the tap, removes the cloned disk, and deletes the console-state file.

Smoke-rig receipt

benchmarks/rigs/bhyve-sandbox-pool-smoke.sh drives the full lifecycle. Sample transcript on honor 2026-04-23:

smoke: warm wall time: 15s
smoke: checkout: id=pool-0 ip=10.77.0.50 pid=22149
smoke: checkout → ssh-ready: 147 ms
smoke: python3 1+1 = 2
smoke: checkout #2 (while #1 still in-use): id=pool-1 (!= pool-0, concurrency-safe)
smoke: return: pool-0 back to available
smoke: drain python-bhyve
template        pool_size       warm_s  checkout_ms     py_result
python-bhyve    2               15      147             2
smoke: ALL GATES PASSED

Full sample at benchmarks/rigs/bhyve-sandbox-pool-smoke.sample.txt.

Numbers, honestly

• Image build: 430 s wall on honor. Most of that is pkg install python311 extracting onto the mdconfig-backed UFS slice — dirty buffers flush at ~3 MiB/s sustained on a 5900HX laptop. Not fast, but it runs once.
• Warm 2 entries: 15 s wall. Two parallel bhyveloads + two FreeBSD boots to sshd. Dominated by guest boot, not bhyve.
• Checkout → SSH-ready: 147 ms wall for the current FreeBSD pool image. This is the user-visible “create latency” — what an SDK client would observe from checkout to being able to ssh <template-user>@<ip>.
• Kernel-level resume (bhyvectl —resume → vCPU runtime advances): 17 ms, measured separately in snapshot-cloning. The two numbers measure different things: 17 ms is the VMM latency; 147 ms is the full REST-ish path including SSH handshake and a few poll iterations.

Gotchas we hit

• PerSourcePenalties in OpenSSH 9.8+ bans the source IP after a few failed auth attempts and returns “Not allowed at this time”. The warm path SIGSTOPs mid-SSH and the probe loop hammers connects — enough to trip the penalty counter. Disabled in the image’s sshd_config. Without this fix, the second smoke-rig run fails with a spurious “connect: Permission denied” at the TCP layer.
• First-available ordering: checkout uses grep -m1 available — a sequential return+checkout gets the same entry back. The smoke rig’s concurrency check needs to run while the first entry is still in-use, not after return. The rig was reshaped accordingly; the pool-ctl behavior is correct.
• No startup reconciliation: if honor reboots with live pool entries, the state file and the IpAllocator forget the bhyve processes. Handled the same way the jail path does it — rare in practice, drain-then-warm recovers. TODO shared with the jail backend.
• Host-console VNC ports also needed a reservation gate. The first cut only wrote console state after bhyve was already booting, which let two concurrent Windows launches grab the same port. The current path writes a placeholder state file under the lock before boot so a second launch sees the reservation.

What this does not cover

• The Rust BhyveBackend in e2b-compat — Session B’s job, separate. That work is now wired for both the original SSH guest path and the newer host-console VNC guest path; what remains is richer in-guest bootstrap, not the basic substrate hand-off.
• Per-entry ZFS snapshot rollback (instead of destroy+respawn on return) — Session C.
• A bhyve-pool mise task and a CI gate for the image-build step. Left for a follow-up.