OCI templates

Cube’s cubemastercli tpl create-from-image ingests an OCI reference and turns it into a sandbox template. The equivalent on Coppice is coppice tpl import-oci <name> <oci-ref>: the gateway shells out to tools/coppice-import-oci.sh, which runs the standard FreeBSD-on-OCI pipeline — buildah pull, buildah mount, rsync -a onto a fresh ZFS dataset, zfs snapshot @base — and the TemplateRegistry hot-reloads so the new template is addressable on the next GET /templates without a gateway restart.

Why OCI, and why this specific pipeline

The honor box’s jail templates have, until now, been baked by hand: zfs create zroot/jails/python-template, pkg -r /jails/python-template install python311, zfs snapshot zroot/jails/python-template@base. That’s fine for the house templates we ship with the gateway, but it’s a dead end for the “bring your own image” use case. The Cube version consumes container images pulled from a registry; we want the same shape so an operator who’s already building OCI images for Kubernetes can point us at the same registry and get a jail template back.

The FreeBSD Foundation has been putting real work into OCI on FreeBSD — buildah, skopeo, and ocijail all work natively, and images like quay.io/dougrabson/freebsd14-minimal exist specifically so FreeBSD sandboxes aren’t limited to Linux-compat images. That ecosystem was the missing piece; the other direction (using an OCI runtime as the jail runtime) is ocijail’s territory and we deliberately didn’t take it. Rationale in the “alternative we didn’t take” subsection below.

The pipeline

# tools/coppice-import-oci.sh <name> <oci-ref>

zfs create zroot/jails/<name>-template

buildah pull <oci-ref>
container=$(buildah from <oci-ref>)
src=$(buildah mount $container)

rsync -a --numeric-ids \
  --exclude=dev/* --exclude=proc/* \
  $src/ /jails/<name>-template/

buildah umount $container
buildah rm     $container

# DNS — match the convention established by
# tools/coppice-jail-templates-dns.sh. Fresh jails inherit this.
cat > /jails/<name>-template/etc/resolv.conf <<EOF
# coppice: DNS via local_unbound on the bridge gateway
nameserver 10.78.0.1
EOF

zfs snapshot zroot/jails/<name>-template@base

Four things matter about this shape:

  1. rsync, not buildah unshare cp. —numeric-ids preserves the container’s uid/gid numbers without remapping against the host’s /etc/passwd; —exclude=dev/* drops the image’s device nodes, which the jail kernel will populate on its own. The fast path is a read(2) from the buildah overlay storage to a write(2) on the new ZFS dataset — no tarball intermediate.
  2. ZFS dataset per template. Not a directory under a shared dataset — a real zfs create. That’s the invariant the rest of the gateway depends on: create_with_limits does zfs clone on @base, which requires a snapshot, which requires a separate dataset. Templates that end up as directories instead of datasets break the sandbox-create path silently.
  3. DNS at import time, not at create time. /etc/resolv.conf gets planted during import so every clone inherits it. This matches what tools/coppice-jail-templates-dns.sh does for the hand-baked templates; skipping it means every VNET jail cloned from the new template has DNS pointing at 127.0.0.1 (its own empty loopback), which is a silent failure mode — packets egress, name lookups don’t.
  4. Atomic + idempotent. A cleanup trap destroys the partial dataset and buildah container on any failure, so a network blip during buildah pull doesn’t leave half a dataset sitting on disk. If the template already exists the script refuses unless —force is passed.

The REST surface

POST /templates is the sole entry point for template creation (pre-#oci-import it returned 501 everywhere). Request:

{
"name": "fbsd14-min",
"from": "oci:quay.io/dougrabson/freebsd14-minimal"
}

Response (201 CREATED):

{
"name": "fbsd14-min",
"snapshot": "zroot/jails/fbsd14-min-template@base",
"sizeBytes": 164822016
}

The from: field is a URI-ish discriminator so future schemes slot in without a wire change. Today only oci: is honoured; dockerfile: (buildah-build), git:, and zfs-send: are on the radar. The handler blocks until the import finishes — on the honor box a first pull of freebsd14-minimal runs ~25 s, a warm re-pull ~3 s. Callers needing fire-and-forget can wrap this in a background job externally; there’s no ?async=true mode in v1.

After a successful import the gateway hot-reloads its TemplateRegistry (the same path POST /templates/reload uses), so GET /templates sees the new entry on the next request without a restart.

Alternative we didn’t take: ocijail

ocijail is the obvious thing to reach for — it speaks the OCI runtime spec and starts jails directly from an image manifest. Replacing freebsd_jail.rs’s backend with ocijail is a real option and may be the right long-term move, but it would have meant rewriting the network story at the same time: ocijail has its own ideas about jail lifecycle, VNET wiring, and per-jail IP allocation that don’t compose cleanly with the coppicenet0 bridge + IpAllocator + per-sandbox pf anchor machinery we landed in #69. Every browser-sandbox and per-sandbox-IP feature on the audit page would need to be re-measured against a new runtime.

The import pipeline above is the conservative bet: keep the jail runtime we know, just grow the template source. One 170-line shell script + one Rust handler instead of a backend swap.

Supported from: schemes

schemestatusnotes
oci:<ref>closedAny OCI ref buildah can pull. Includes quay.io/…, docker.io/…, registry.local/….
dockerfile:<path>stubbedCLI exits 3. Planned: buildah build -f <path> -t localhost/<name> then fall through to the OCI path.
git:<url>#<rev>not plannedCovered by the Dockerfile path once that lands.
zfs-send:<path>not plannedAir-gapped deploys can zfs receive directly without going through this endpoint.

Caveats

Linux-only OCI images won’t run under FreeBSD without linuxulator. The import succeeds (it’s just rsync), but coppice sandbox create —template <name> will fail at exec time because the binaries in the rootfs are ELF amd64/Linux and the jail kernel is FreeBSD. Two routes around it: (a) pick a FreeBSD-native image — quay.io/dougrabson/* is the standard starter set — or (b) enable the linuxulator (kldload linux64) and accept the performance hit on the syscall boundary. We use (a); the audit row is closed against the FreeBSD-native case and parks the linuxulator route as a future bench rig.

Signing lives downstream. The signify(1) gate from #74-sign kicks in at zfs clone time, not at import time, so a freshly imported template is unsigned and will 403 on sandbox create if COPPICE_REQUIRE_SIGNED_TEMPLATES=1 is set. Operator workflow: import, sign, launch.

coppice tpl import-oci fbsd14-min oci:quay.io/dougrabson/freebsd14-minimal
# → template=fbsd14-min snapshot=zroot/jails/fbsd14-min-template@base size=157.2M

sudo COPPICE_SIGN_PRIVKEY=/root/coppice-signing/coppice.sec \
  coppice tpl sign fbsd14-min

coppice sandbox create --template fbsd14-min

buildah cache is keyed by digest, not tag. Re-importing the same tag after a registry push will silently use the cached layers unless the digest changed. Pass —force to the script (which destroys the existing dataset first) to force a re-pull against the current tag.

Receipts

Credit

The FreeBSD Foundation’s OCI-on-FreeBSD programme is the reason this works at all. buildah and skopeo both run as first-class FreeBSD packages, and the quay.io/dougrabson/freebsd14-minimal image is the canonical minimal FreeBSD-in-OCI starter — the work backing those three lines is the actual primitive here. Our script is a 170-line shim around it.

Cross-refs