linux bhyve template

The bhyve path is no longer “FreeBSD guest plus theory note for Linux”. Coppice now has a repo-native Linux guest recipe: debian-12-bhyve, built from Debian’s official generic cloud image and configured through the same bhyve template sidecar system the Windows and FreeBSD guests already use.

What landed

• benchmarks/rigs/debian12-cloud-bhyve-template.sh downloads Debian’s Bookworm generic cloud image, converts it to a raw bhyve disk image, and writes /vms/templates/debian-12-bhyve.{img,conf,img.DESCRIPTION}.
• The bhyve substrate now supports TPL_PATCH_MODE=nocloud-seed, which writes a small MBR disk with a CIDATA-labeled FAT partition and the exact meta-data, user-data, and network-config.
• That seed disk gives each warmed clone:
  • a unique static 10.77.0.X/24 address
  • a hostname matching the pool entry or sandbox id
  • the host’s pool SSH key on the guest login user
• The builder also writes a small /etc/cloud/cloud.cfg.d/90_coppice_nocloud.cfg override into the guest image so cloud-init prefers the local CIDATA datasource under bhyve instead of falling back to a plain localhost boot.
• The image also carries a tiny coppice-seed.service fallback that reads coppice-instance.env and coppice-authorized-key.pub from the same seed disk and applies hostname, static IPv4, DNS, and SSH authorization even if cloud-init stays conservative on local media.
• The sidecar also gains TPL_EXTRA_BHYVE_SLOTS, a generic “append these -s slots to bhyve” hook. The immediate use is GPU/PCI passthrough, but it is intentionally broader than that.

Template shape

The Debian template sidecar is straightforward:

TPL_GATEWAY_MODE=ssh
TPL_BOOT_MODE=uefi
TPL_PATCH_MODE=nocloud-seed
TPL_DISK_DEVICE=virtio-blk
TPL_NET_DEVICE=virtio-net
TPL_SSH_USER=root
TPL_SHELL=/bin/bash
TPL_SHELL_ARGS="-i"

That means:

1. coppice-bhyve-pool-ctl.sh warm debian-12-bhyve —count N clones the raw disk image.
2. The controller writes a NoCloud seed disk for the clone.
3. bhyve boots the Debian image under UEFI with a virtio-blk root disk and a virtio-net NIC.
4. The guest consumes the NoCloud seed, and the baked coppice-seed.service fallback still applies the same static IP, hostname, DNS, and SSH key if cloud-init stays cautious about local media.
5. The normal checkout path returns a guest IP and the gateway reaches it over SSH as root, exactly like the other bhyve-backed templates.

Smoke receipt

The Linux receipt is:

• benchmarks/rigs/linux-bhyve-smoke.sh
• latest captured honor output: benchmarks/results/linux-bhyve/latest.txt

It is a thin wrapper around the generic bhyve pool smoke, but it sets:

• SSH_USER=root
• PROBE_CMD=‘python3 -c “print(1+1)“‘
• PROBE_EXPECT=2

So the Linux-specific gates are still the real ones:

1. warm the pool
2. checkout a clone
3. wait for SSH on the Linux guest
4. run a guest command through the SSH path
5. return and drain cleanly

Why NoCloud

For FreeBSD guests, the pool controller mutates the guest disk directly and drops /etc/coppice-instance. That is a good fit for a UFS image we own.

Linux cloud images already have a standard first-boot control plane: cloud-init. The correct analogue is therefore not “mount ext4 and rewrite random files from the host”, it is “attach a NoCloud seed disk and let the guest consume it the way the distro expects”.

That is what nocloud-seed implements.

GPU hook

The same sidecar layer now also exposes:

TPL_EXTRA_BHYVE_SLOTS='20:0,passthru,ppt0;21:0,passthru,ppt1'

The controller appends each declared slot directly to the bhyve command line. On a host with an NVIDIA card already bound to ppt(4), that is enough to express GPU passthrough without forking the launcher per template.

Honor now has that proof path measured. The debian-12-bhyve-gpu template uses the same sidecar hook, the host applies patches/freebsd-src/bhyve-nvidia-linux-passthrough.diff for the NVIDIA/Linux guest workaround, and benchmarks/results/gpu-passthrough/latest.txt captures a Debian guest checkout ending in GPU_OK. The smoke does more than list PCI visibility: it requires nvidia-smi -L to return an actual GPU N: row, queries name, driver, memory, and GPU utilization through NVML, and captures one nvidia-smi pmon sample from inside the guest before printing GPU_OK. Honor currently runs that receipt with a 2 GiB guest memory window; 4 GiB triggered host AMD-Vi IO_PAGE_FAULT diagnostics and guest RmInitAdapter failures. The passthrough smoke drains the entry after the probe instead of treating same-device destroy/respawn as part of the pass condition; reset/recycle behavior is tracked as the next GPU hardening receipt. CUDA kernel execution and GPU memory-snapshot receipts remain separate hardening steps.