There are three kinds of “FreeBSD can’t do that”: things FreeBSD is genuinely behind on, things the Linux/CubeSandbox side is quietly excluding from its claims, and things where both sides are playing different games and a bar chart misleads. Keep them separate.
Where FreeBSD is genuinely behind
eBPF ecosystem
FreeBSD has a kernel BPF subsystem — bpf(4) — but it’s the classic-BPF
filter language used for tcpdump and dhclient, not the eBPF VM that
Linux added in 3.18 and has been extending ever since. Notably absent:
- BPF maps as a kernel-resident state store for userspace-mutable policy. Netgraph has per-node state, but the programming model is very different.
- TC clsact / tcx ingress + egress attach points driven by a BPF
program.
pf+dummynet+netgraphcover the programmability surface, but none match the “compile-and-attach” ergonomics. - XDP-equivalent pre-stack ingress path. DTrace’s FBT probes can observe packets, but can’t redirect them.
- Tooling.
bpftrace,bcc,bpftool,libbpf— none have FreeBSD analogs beyond DTrace (which is powerful for observability but orthogonal).
There has been intermittent work to bring eBPF to FreeBSD (generic-ebpf
by Hayakawa et al., Matt Macy’s branch, Netflix internal work), but
nothing in base as of 15.0-RELEASE-p4. This remains the largest
single capability gap, but for the agent-sandbox policy-update
workload specifically, pf + VNET + dummynet + netgraph reach parity
— see /appendix/ebpf-on-freebsd for
the full ecosystem survey and
/appendix/ebpf-to-pf for measured numbers.
microVM snapshot/restore (partially closed, 2026-04-22)
Cloud Hypervisor has a production snapshot/restore API that Cube uses
for the pool-resume path. FreeBSD bhyve has BHYVE_SNAPSHOT — still
marked experimental upstream, off in GENERIC kernels. We build it
anyway: options BHYVE_SNAPSHOT in the honor SNAPSHOT kernel, paired
with bhyvectl --suspend for durable checkpoints. The
bhyve-durable-prewarm-pool config in
/essays/freebsd-bhyve uses this in production
shape — durable cold tier, SIGSTOP’d hot tier, rebuild on reboot. 17 ms
resume. So the original framing of this caveat — “that’s fine for an
agent pool; not the same capability” — is now closer to “same
capability, built on an experimental kernel knob we had to flip on
ourselves.”
Cloud Hypervisor / rust-vmm crate ecosystem
rust-vmm crates (kvm-ioctls, vm-memory, virtio-queue,
vhost-user-*, seccompiler) are Linux-specific by design — KVM and
seccomp-bpf are Linux kernel APIs. Porting these to call into bhyve’s
libvmmapi and capsicum(4) would be a significant rewrite of a large
crate family. In practice, a FreeBSD microVM story uses libvmmapi
directly (the way vm-bhyve and bhyvectl do) and forgoes rust-vmm.
virtio-fs in base
CubeSandbox relies on virtio-fs for efficient host-to-guest filesystem
sharing. FreeBSD 15.0 does not ship a virtio-fs implementation in base.
9p-over-virtio is the workable substitute (and is in the tree), but it is
a different guest contract — the Cube agent would need a 9p mount code
path where today it has a virtio-fs mount.
Where CubeSandbox is quietly excluding something
”Cold start < 60ms” excludes the cold-cold path
Per the anatomy notes, the sub-60ms number is the pool-hit path: pick an already-booted paused VM, resume its memory snapshot, reconfigure the network attachment, return to the caller. The pool must be pre-warmed.
What’s excluded from the measurement:
- Host reboot recovery. After a reboot the pool is empty; the first N creates are cold boots. Cloud Hypervisor cold-boot with a minimal Linux guest is tens to hundreds of ms (not sub-60ms).
- Template switch. The pool is presumably keyed by template; a request for a template with no pool entry is a cold boot.
- Cloud Hypervisor process startup. The VMM itself is a binary costing ms of fork/exec overhead. “Pool-hit” measurements that start the clock after the VMM is running aren’t measuring this.
None of these are wrong ways to benchmark, but the quoted headline
metric is the best-case path. An apples-to-apples comparison needs to
match “same pool-hit path” against “same pool-hit path”
— which is what we do in /essays/freebsd-bhyve
with bhyve-prewarm-pool.
”<5MB memory overhead” excludes shared pages
The <5MB/instance number depends on host-level CoW page sharing: many microVMs booting the same kernel image and the same rootfs, with the host’s page cache de-duplicating. This is real and works, but:
- It’s an ensemble property: the 1,000th sandbox costs <5MB, but the first sandbox costs the full RSS of the VMM + the unique pages of the guest.
- It’s fragile under KASLR / ASLR at the guest kernel level if those vary per-VM (which Cube’s pool approach sidesteps by cloning snapshots).
- It’s a property of rooms full of identical sandboxes. Heterogeneous fleets (different templates, different guest-userspace versions) share less.
FreeBSD doesn’t ship KSM, but the same ensemble property is
reproducible on bhyve via
patches/vmm-memseg-vnode.diff — back
the memseg directly with the checkpoint file’s vnode and let MAP_PRIVATE
semantics handle the CoW. 1000 × 256 MiB microVMs from one ckp in
9.1 GiB of host RAM on honor.
”E2B drop-in replacement”
Nine of seventeen E2B endpoints are implemented. If your agent code calls
Sandbox.create → run_code → close, it’s truly drop-in. If your agent
code uses /metrics, /logs, persistent /snapshots, /timeout, or
per-sandbox network policy updates, it isn’t. The /anatomy
page quotes the CubeAPI/README.md support matrix verbatim.
Where both sides are playing different games
Jails vs microVMs: not two points on a spectrum
A bar chart comparing jail cold-start to microVM cold-start implies they differ by a factor, not by a category. They differ by category.
- Jail: shared host kernel, jail partitioning at
cred(9)andprison(9)level. A kernel vulnerability exposed to a jailed process (CPU microarchitecture, syscall surface, filesystem driver parsing) potentially reaches every jail on the host and the host itself. - MicroVM: separate guest kernel. A kernel vulnerability in the
guest compromises that guest; the host kernel is protected by the
hypervisor boundary. The hypervisor itself is a different attack
surface (KVM +
cube-hypervisor) but that surface is smaller than a full kernel.
For an agent workload running LLM-generated Python, the operative question is whether Python (via syscalls) can exploit the host kernel from inside a jail. Often the answer is “yes in principle, no in practice” — which is exactly where the threat model gets fuzzy and the engineering choice hard. Jails are not a worse choice; they are a different choice.
”Concurrency” means different things
CubeSandbox’s 50-concurrency p95 is “50 concurrent creates against a
pool sized ≥ 50”. A FreeBSD jail-raw-cc50 is “50 concurrent
jail -c invocations against a host with no pool”. These measure
different things, and it’s tempting to put the numbers next to each
other and call it a comparison. The comparable Cube measurement would be
“50 concurrent creates against an empty pool” — which Cube
doesn’t publish.
Network policy expressiveness
CubeNet’s eBPF programs can implement arbitrary L2-L4 filtering with
per-flow state and userspace-mutable maps. pf + tables can implement
most of that statically, but cannot express a policy that depends on
runtime state the same way (“allow if this flow was permitted in
the last 30 seconds by the policy engine”). Whether this is a real
expressiveness gap or a false one depends on what CubeNet policies
actually are — open question for /appendix/ebpf-to-pf.
What this site is not
- A bakeoff that declares a winner. Cube vs. jails vs. bhyve is three different answers to overlapping questions.
- A migration guide. We’re not shipping a FreeBSD port; we’re estimating what one would cost.
- A security claim. Where we describe isolation models, we describe what they can protect against in principle, not what any specific deployment actually delivers.
The most useful thing to carry out of here is a sharper read on the “drop-in replacement” framing: CubeSandbox is real, thoughtful engineering; the framing is marketing; and the answer to “can we do this on FreeBSD?” depends heavily on which threat model you bought into before reading the charts.