Every Coppice claim on the rest of the site traces back to one row
on this page. Every row traces back to a script under
benchmarks/rigs/. Every number traces back to a file
under benchmarks/results/. If you don’t see the number
here, we haven’t run it. If you see it here with a dash, the rig
exists but hasn’t captured yet. Updated as benchmarks land.
Host
All FreeBSD numbers on this page measured on honor:
| CPU | AMD Ryzen 9 5900HX (8c/16t Zen 3, laptop APU) |
| RAM | 32 GB DDR4-3200 non-ECC |
| Storage | single NVMe, GELI-encrypted, ZFS |
| OS | FreeBSD 15.0-RELEASE-p4 |
| Kernel | custom SNAPSHOT: GENERIC + options BHYVE_SNAPSHOT + patches/vmm-memseg-vnode.diff + patches/bhyve-vnode-restore.diff |
| pf | enabled, net.link.bridge.pfil_member=1, set limit anchors 4096 |
Not bare-metal parity with Cube’s undisclosed benchmark box — that’s the point of the homepage’s “stricter clock, smaller box” framing.
Cold-start latency — bhyve configurations
Chart · cc=1 resume / boot latency — log scale
▸ reproduce · mise run bench:bhyve-full ·mise run bench:bhyve-durable-pool ·mise run bench:bhyve-durable-prewarm-pool ·mise run bench:bhyve-prewarm-pool · methodology
| config | cc=1 mean | cc=10 mean | cc=50 p95 | rig |
|---|---|---|---|---|
| bhyve-full (cold boot) | 3 906 ms | 5 846 ms | 100 194 ms | Full GENERIC guest. Upper bound; no one ships this. |
| bhyve-prewarm-pool | 10 ms | 10 ms | 41 ms | SIGSTOP’d live VMs. Process-level suspend/resume; not durable across host reboot. |
| bhyve-durable-pool | 271 ms | 1 565 ms | 3 103 ms | bhyvectl —suspend → disk → bhyve -r. Durable across reboot. |
| bhyve-durable-prewarm-pool | 17 ms | 39 ms | 1 290 ms ‡ | Two-tier: durable on-disk ckps as cold, SIGSTOP’d hot tier in front. Production shape. ‡ cc=50 is dominated by 50 guests sharing 16 physical threads — SIGCONT delivers immediately, but each vCPU waits for a time-slice. Latency degrades smoothly from cc=10 (105 ms mean) → cc=20 (333 ms) → cc=50 (995 ms). On a 32-thread host this falls back into the cc=10 band. Early measurement reported 2 903 ms p95; that was the poll-loop competing with itself (fixed by adding a 1 ms delay between bhyvectl probes). See parity-gaps. |
All four are resume-from-ready; cc is concurrent creates. Full rig recipe + SNAPSHOT kernel build at /appendix/bench-rig.
Density — N microVMs from one checkpoint, with vmm-vnode patch
| concurrent VMs | host Δ | per-VM effective | notes |
|---|---|---|---|
| 8 × 256 MiB | 103 MiB | 12.9 MiB | Template in page cache once; per-VM is bhyve state. |
| 50 × 256 MiB | 939 MiB | 18 MiB | Original KSM-gap sample; now ensemble-matched. |
| 200 × 256 MiB | 5 366 MiB | 26 MiB | 16 threads saturating; cp cost visible. |
| 400 × 256 MiB | 6 492 MiB | 16 MiB | Fixed cost amortizing. |
| 1 000 × 256 MiB | 9 117 MiB | 9 MiB | Naive: ~250 GiB. Laptop fit. |
bhyve-fanout-rss.sh.
Mechanism: /appendix/vmm-vnode-patch.
Network isolation — cubenet on pf
| metric | cubenet (honor) | Cube/eBPF (typical) | rig |
|---|---|---|---|
| sandbox↔sandbox p50 RTT | 7 µs | ~5–10 µs | netperf TCP_RR; run-net-bench.sh |
| sandbox↔sandbox p99 RTT | 8 µs | ~10–15 µs | stddev 0.51 µs |
| TCP throughput, 1 stream | 14.6 Gbit/s | ~15–20 Gbit/s | iperf3 intra-host, memory-bandwidth-bound |
| Policy update, single add | 4.2 ms wall | 1–5 ms (bpftool) | dominated by pfctl spawn; kernel-side is µs |
| Policy update, 1000 IPs batched | 4 ms total | Cilium bulk: similar | pfctl -T replace; 250k ops/sec effective |
| Policy mutation under 14 Gbit/s | +43 µs p99 vs idle | Cilium: ~similar | no visible contention; throughput stable. policy-churn-under-load.sh |
| Per-sandbox anchors, N=1000 | 1.5 ms p95 load | Cilium: flat | requires set limit anchors 4096; see /appendix/ebpf-to-pf. policy-anchor-churn.sh |
| External rdr/DNAT add-latency | +0.24 µs p50 | Cilium: similar | 9.87 vs 9.63 µs bare s2s TCP_RR. ext-to-sandbox.sh |
| IPv6 sandbox↔sandbox p50 RTT | 8 µs | ~5–10 µs | netperf -6 TCP_RR; tied with v4 (8 µs). run-net-bench-v6.sh |
| IPv6 TCP throughput, 1 stream | 6.19 Gbit/s | ~15–20 Gbit/s | 84% of v4 on same host state; v6 header + extra rule-block fixed cost. Dual-stack via fd77::/64 ULA; NAT66 on re0 for egress. |
| IPv6 external egress (NAT66) | 23.8 ms | n/a | sbx-a → 2606:4700:4700::1111 vs 23.5 ms host-direct; NAT66 add-latency below sample noise. |
| Multi-stream TCP scaling | pending T2 | linear | 1-stream 14.6 Gbit/s → 16-stream 9 Gbit/s observed; attribution TBD |
| Per-sandbox rate limit (dummynet) | pending T2 | Cilium: bandwidth manager | rate-limit-dummynet.sh |
Two VNET jails on cubenet0 + cube_policy
anchor. Full methodology + gotchas at /appendix/ebpf-to-pf.
Lifecycle — per-sandbox provisioning
| operation | mean | p95 | notes |
|---|---|---|---|
| checkout (IP + tap + anchor + pool entry) | 21.5 ms | 23 ms | Per-sandbox anchor load dominates. |
| release (tap destroy + anchor flush + state kill) | 308 ms | 610 ms | ifconfig tap destroy bound; pf is µs. |
| pf states leaked after release-all | 0 / 10 | — | End-to-end correctness, not just latency. |
N=10 e2e run of pool-cubenet-e2e.sh
using coppice-pool-ctl.
E2B compatibility surface
| endpoint class | status | notes |
|---|---|---|
| API server (CRUD + pause/resume + metrics + timeout) | 10/10 SDK calls pass | Python e2b-code-interpreter SDK against our Rust/Axum gateway. |
envd /execute (run_code) | NDJSON streams | jexec python3 backend; print(1+1) returns 2. See /appendix/run-code-protocol. |
<port>-<id>.domain routing | verified | pf rdr + dnsmasq + Go cubeproxy; LAN-peer curl returns 200. |
Filesystem API (sandbox.files.*) | pending | Upstream envd REST; ~1 day of work. |
| Persistent kernel (ipykernel state across calls) | 7/7 SDK checks pass | ipykernel in jail + in-jail bridge translates iopub → NDJSON. x = 42 persists; pandas → text/html; matplotlib → image/png; errors → name/value/traceback. Rig: benchmarks/rigs/jupyter-e2e.sh. Minimal reproductions under examples/ (01–04, 07). See /appendix/run-code-protocol. |
Kernel patches
| patch | state | receipt |
|---|---|---|
vmm-memseg-vnode.diff | working, measured | N=1000 density above; full audit at upstream-review.md |
bhyve-vnode-restore.diff | working, measured | bhyve userland integration; -o snapshot.vnode_restore=true |
What isn’t on this page
- Linux+Cilium head-to-head — we don’t have a second honor-class box. The “Cube/eBPF typical” columns are published figures, not our measurements. Closes when we can rent one.
- cc ≥ 50 on
bhyve-full— saturates 16 threads. Captured as the upper-bound row above, not a practical target. - L7 policy — Cilium does per-HTTP-request; FreeBSD answer is Envoy or VPP sidecar. Acknowledged gap; see /appendix/parity-gaps.
- IPv6 parity — IPv4 lab only today.
See /appendix/parity-gaps for everything still open.