Per-sandbox logs

Cube’s SDK calls GET /sandboxes/:id/logs and expects a paginated list of recent log lines. Until this release our route returned []. The missing piece wasn’t a FreeBSD primitive — the jail’s own stdout + stderr are right there on the jexec’d child’s pipes — it was the host-side plumbing to keep a bounded recent-history window per sandbox. This page describes the ring buffer, the source/stream attribution, the jexec tail fallback, and the CLI’s —follow poll.

The ring buffer

Every Sandbox owns an Arc<LogBuffer> (see e2b-compat/src/state.rs). The buffer is a bounded VecDeque<LogLine> with drop-oldest overflow; the default cap is 1024 entries (~1 MiB upper bound on a chatty Python process, which is fine next to the 512 MiB default sandbox memory budget). A latching truncated flag flips to true the first time the buffer evicts, so callers can tell I have the full history from this is a recent window.

Each entry carries four fields:

pub struct LogLine {
    pub ts:     DateTime<Utc>,   // RFC3339 on the wire
    pub source: String,           // "kernel" | "exec" | "jail" | "syslog"
    pub stream: String,           // "stdout" | "stderr"
    pub text:   String,           // trailing newline stripped
}

source names the producer. Today the one wired path is “kernel” — kernel::spawn_kernel now pipes the ipykernel’s stdout and stderr (previously /dev/null) and spawns two tokio line readers that push into the buffer as each line arrives. “exec” and “jail” are reserved for the equivalent plumbing on the shell-command and jail- init paths; “syslog” is the fallback described below.

The endpoint

GET /sandboxes/:id/logs and GET /v2/sandboxes/:id/logs share one response shape:

{
  "logs": [
    {
      "ts": "2026-04-22T23:45:01.203Z",
      "source": "kernel",
      "stream": "stderr",
      "text": "NOTE: /tmp/ipykernel.lock already held"
    },
    {
      "ts": "2026-04-22T23:45:01.301Z",
      "source": "kernel",
      "stream": "stdout",
      "text": "42"
    }
  ],
  "truncated": false
}

Two query params shape the window:

An unknown sandbox id returns 404 with the same envelope the rest of the lifecycle handlers use.

The syslog fallback

If the in-memory buffer is empty and the caller didn’t pass ?since=, the handler runs jexec e2b-<id> tail -n <limit|200> /var/log/messages and wraps each line as source=“syslog”, stream=“stdout”, ts=now. We don’t try to parse the FreeBSD syslog timestamp format — the handler contract is what’s in the buffer, not authoritative history. The fallback is best-effort: any failure (jail gone, file missing) just returns an empty Vec rather than an HTTP 500.

The fallback only fires on the empty + no-since case deliberately. A tail-polling client that sends ?since=<last_ts> every two seconds would otherwise receive the same /var/log/messages chunk on every tick, which defeats the point of the filter.

The counter

Each push() bumps coppice_sandbox_log_lines_total{sandbox,template}, a per-sandbox counter rendered in the existing per-sandbox block on /metrics. Cardinality is bounded — the counter is removed on sandbox teardown, same path that drops the sampler reading. rate(coppice_sandbox_log_lines_total[1m]) tells you how chatty a sandbox is without having to pull the buffer itself.

The CLI

coppice sandbox logs <id> formats each line as [ts] source/stream: text and exits. —json emits the raw gateway envelope. —limit N defaults to 100; —since 5m accepts s/m/h/d suffixes and resolves to an RFC3339 cutoff relative to now.

—follow polls every 2 s, carrying the newest ts it saw across iterations and re-querying with ?since=<that> so the same line doesn’t print twice. Exits on Ctrl-C (no installed signal handler — blocking stdio lets SIGINT kill the process). Transient HTTP errors warn on stderr and retry; 404 terminates cleanly.

This is the same shape kubectl logs -f uses. It isn’t a streaming endpoint — that would want a WebSocket or SSE — but for tail my sandbox while I iterate it’s the right ergonomic cost for the implementation effort: zero extra gateway code, one poll loop in the CLI.

Cost

The ring buffer is one std::sync::RwLock<VecDeque> plus two atomic counters per sandbox. push() is O(1); snapshot() copies into a new Vec for the response, also O(n) in the number of buffered lines. At the 1024-cap default a full snapshot is under 100 microseconds on a warm host.

The tokio line readers survive until the child pipe EOFs (kernel exited), so the long-running cost is two blocked tokio tasks per live sandbox — which, at tokio’s scheduling model, is free until one of them has bytes to read.