docs: trim SPEC.md to the contracts the code cannot carry

The code is the source of truth now; what stays is the part code
cannot carry — the founding rule and confinement budget (fixed: the
budget said no child process, the machine child is one), the run
contracts (exit codes rewritten to match the engine-failure and
reserved-code semantics), the trust model, the falsification tests,
and the roadmap.

Cut: the engine call list, the RunOptions listing, the CLI flags
table, the cache layout tree, the workspace/module map, the ecr
porting table, and the open questions — all answered or all mirrors
of code that answers them faster.
This commit is contained in:
vhaudiquet committed 2026-10-30 13:27:51 +01:00
1 parent b0ff40dd5d
commit 4aadc5d28a
1 file changed
+147 -344
+147 -344
View File
@@ -1,83 +1,48 @@
# ecrou - implementation specification # ecrou - contracts and invariants
## Identity The code is the source of truth for how ecrou works. This document
holds what the code cannot carry: the founding rule and the
confinement budget, the run contracts, the trust model, and the
falsification tests that keep them honest. Anything descriptive of the
current code lives with the code.
ecrou is a microVM runtime for disposable Linux environments. It descends ## What ecrou is
from ecr, the ephemeral chroot: the name keeps the ancestor, and the
answer to "chroot ou VM" is now VM. As a bonus, an écrou is a nut: the
small part that keeps a bolt from walking out of the wall.
One sentence: resolve an image reference, boot it as a one-shot microVM Resolve an image reference, boot it as a one-shot microVM on libkrun,
on libkrun, run one command with live stdio, propagate its exit code, run one command with live stdio, propagate its exit code, keep
keep whatever crossed the workspace share, throw the machine away. whatever crossed a share, throw the machine away.
The project is library-first like ecr: `crates/ecrou` is the library The split is library-first and enforced by the dependency edges: the
(package `ecrou`), `crates/ecrou-cli` is the thin front-end (binary library links no clap and no rendering crate, and never writes to the
`ecrou`) that parses flags, maps them onto the library API, and terminal. Diagnostics go through the `log` facade, filtered by
propagates exit codes. The split is enforced by the dependency edges: `RunOptions::log_level`; long operations report through the `Progress`
the library links no clap and no rendering crate. It never writes to sink; in-process failures come back as typed errors. The one exception
the terminal: diagnostics go through the `log` facade, filtered by is the forked machine child's last-gasp diagnostics, which land in the
`RunOptions::log_level` (an engine-owned enum, not UI booleans), long console file the parent streams. The CLI owns clap, rendering, log
operations report through the `Progress` sink (sequential output, and exit codes. A consumer embeds the library exactly the way
`begin`/`advance`/`finish` operations plus uncounted `message` the CLI does: a `RootfsCache` to prepare a rootfs, then `run` to boot
status lines), all output is the CLI's. In-process failures come back as typed errors; the one it. The first consumer shells out to the binary the way it shells out
exception is the forked machine child's last-gasp diagnostics, which to `docker`; the API must let a second consumer arrive without forking
land in the console file the parent streams. The CLI owns clap, the the runtime.
indicatif rendering of progress events, the log sink, and exit codes.
A consumer embeds the library exactly the way the CLI does: a
`RootfsCache` to prepare a rootfs, then `run` to boot it.
The first consumer is rhema's sandbox seam, which shells out to the ## The engine
binary the way it shells out to `docker` and `lxc` today. The library
API is designed so a second consumer can arrive without forking the
runtime.
## What carries over from ecr One engine, libkrun, driven through its stable C ABI (the
`stable-1.19.x` headers; the main branch is mid-rewrite to a vtable
| module | disposition | API and is not the target). ecrou `dlopen`s `libkrun.so.1` at runtime
|---|---| and fails with a remediation message naming the install when it is
| `distro`, `download`, `extract`, `rootfs`, `utils`, `progress` | ported; the image machinery is confinement-neutral plain fs and network | missing; nothing links at build time, and the snap bundles the library
| `kernel` | dropped; the kernel is libkrun's, bundled in libkrunfw |
| `qemu_vm` | dropped from v1; its KVM probe logic (open `/dev/kvm`, `KVM_GET_API_VERSION` == 12) moves into the `krun` module |
| `namespace`, `chroot`, `mount`, `exec` | dropped; they stay in ecr, their ancestor. Their error variants (`UserNamespacesUnavailable`, `BinfmtNotRegistered`, `MissingSubidRange`) leave with them |
No initramfs and no cpio builder: the rootfs crosses as a virtiofs
share, never as a packed archive.
No cache compatibility with ecr: the formats may match but the directory
is `~/.cache/ecrou` and never shared.
## The engine: libkrun
v1 has exactly one engine, libkrun, driven through its stable C ABI
(the `stable-1.19.x` headers; the main branch is mid-rewrite to a
vtable API and is not the target). ecrou `dlopen`s `libkrun.so.1` at
runtime and fails with a remediation message naming the install when it
is missing; nothing links at build time, and a snap bundles the library
plus libkrunfw next to itself. plus libkrunfw next to itself.
The VMM runs inside the ecrou process: `krun_start_enter` enters the KVM is required; there is no TCG fallback. A machine without working
VM loop on the calling thread. There is no child process to shepherd, KVM is `Error::KvmUnavailable` with the remediation text, and `ecrou
which makes the kill contract below trivial. probe` reports it before any run. The VMM runs inside the machine
child, not on the caller's thread (see Cancellation).
The FFI lives in `crates/ecrou/src/krun.rs`. ecrou carries unsafe libc libkrun's mapped-volumes entry point is explicitly unsupported on
code; rhema, its first consumer, keeps `#![forbid(unsafe_code)]` and Linux — its header asks for user namespaces and bind mounts, precisely
drives the CLI. what ecrou cannot do — so persistence crosses a virtiofs share mounted
by the guest init instead.
The v1 API surface is eleven calls: `krun_set_log_level`,
`krun_create_ctx`, `krun_set_vm_config`, `krun_set_root`,
`krun_add_virtiofs`, `krun_set_workdir`, `krun_set_exec`,
`krun_set_env`, `krun_set_console_output`, `krun_start_enter`,
`krun_check_nested_virt`. `krun_set_mapped_volumes` exists but is
explicitly unsupported on Linux (its header says to use user namespaces
and bind-mounts instead, which is precisely what ecrou cannot do); the
workspace is a tagged `krun_add_virtiofs` share mounted by the guest
init instead.
KVM is required. There is no TCG fallback; a machine without working
KVM is an `Error::KvmUnavailable` with the remediation text, and
`ecrou probe` reports it before any run. QEMU as a fallback engine is a
Later row, and its arrival would not change the API.
## The founding rule ## The founding rule
@@ -91,9 +56,9 @@ The rule is enforced by audit, not by trust: test T6 straces a full run
and fails on a single hit from the denied set. It runs in CI. and fails on a single hit from the denied set. It runs in CI.
Guest-side mounts are not host mounts. Confinement binds the host Guest-side mounts are not host mounts. Confinement binds the host
process; the guest kernel may mount proc, devtmpfs, and the workspace process; the guest kernel may mount proc, devtmpfs, and the shares
share inside the machine, and nothing a guest does lands on the host inside the machine, and nothing a guest does lands on the host beyond
beyond what the shares expose. what the shares expose.
### Confinement budget ### Confinement budget
@@ -106,14 +71,15 @@ Everything the host process may do, and nothing else:
- `/dev/kvm`: open, ioctl, mmap - `/dev/kvm`: open, ioctl, mmap
- net: TCP and UDP client sockets; with TSI these carry the guest's - net: TCP and UDP client sockets; with TSI these carry the guest's
connections too, proxied host-side connections too, proxied host-side
- process: threads, futexes, eventfds, timers; no children in v1 - process: threads, futexes, eventfds, timers; one forked child, the
machine, carrying PR_SET_PDEATHSIG
Denied by design: `mount`, `umount2`, `unshare`, `setns`, `pivot_root`, Denied by design: `mount`, `umount2`, `unshare`, `setns`, `pivot_root`,
`open_tree`, `move_mount`, `mount_setattr`, `fsopen`, `fsmount`, `open_tree`, `move_mount`, `mount_setattr`, `fsopen`, `fsmount`,
`ptrace`, kernel module loading, keyring calls. All absent from snapd's `ptrace`, kernel module loading, keyring calls. All absent from snapd's
seccomp allowlist, which is where the allowed list above is drawn from. seccomp allowlist, which is where the allowed list above is drawn from.
Two absences need a spike-time check because VMMs plausibly touch them: Two absences need a spike-time check because VMMs plausibly touch them:
`sched_setaffinity` and `openat2`. The spike straces for EPERM on `sched_setaffinity` and `openat2` (T1). The spike straces for EPERM on
allowed-looking calls, and anything libkrun genuinely needs that the allowed-looking calls, and anything libkrun genuinely needs that the
allowlist denies goes on a snapd issue, never into a workaround. allowlist denies goes on a snapd issue, never into a workaround.
@@ -122,272 +88,105 @@ allowlist denies goes on a snapd issue, never into a workaround.
One-shot machines. `ecrou run IMAGE -- CMD` prepares a rootfs, boots One-shot machines. `ecrou run IMAGE -- CMD` prepares a rootfs, boots
one VM whose payload is CMD, streams its stdio, and exits with CMD's one VM whose payload is CMD, streams its stdio, and exits with CMD's
code when the machine powers off. There is no exec into a running code when the machine powers off. There is no exec into a running
machine. No engine in the family has that API: `krun_set_exec` is machine: no engine in the family has that API, and per-command boots
per-context before start, and crun's krun handler boots a fresh VM for are the shape. A guest agent and long-lived machines are a Later row.
every `crun exec`. A guest agent and long-lived machines are a Later
row; per-command boots are the v1 shape, and the boot budget below is
what keeps them honest.
Ephemeral rootfs, durable workspace. The base system is a scratch copy Ephemeral rootfs, durable workspace. The base system is a scratch copy
of the image (a `PreparedRootfs`, TempDir-backed): it exists only for of the image; it exists only for the run and dies with it. Persistence
the run and dies with it. Persistence crosses a workspace share or it crosses a share or it does not happen. No initramfs and no cpio
does not happen. builder: the rootfs crosses as a virtiofs share, never as a packed
archive.
**Workspace shares.** `-v HOST:GUEST[:ro]` (`--volume`), one virtiofs **Workspace shares.** `-v HOST:GUEST[:ro]`, one virtiofs share per
share per entry (`krun_add_virtiofs`, `krun_set_root` keeps the rootfs entry. The host side is plain fs; the guest init mounts each share at
itself). its guest path. There is no implicit share: an empty machine is the
The host side is plain fs; the guest init mounts each share at its default, and the consumer chooses what crosses — a sandbox runtime
guest path. There is no implicit share: an empty sandbox is the does not silently attach the caller's cwd.
default, and the consumer chooses what crosses. This is deliberate; a
sandbox runtime does not silently attach the caller's cwd.
Ownership needs no mapping layer. The guest runs as root inside its own Ownership needs no mapping layer. The guest runs as root inside its
kernel, and libkrun's simplified virtiofs semantics report the own kernel, and libkrun's simplified virtiofs semantics report the
requesting guest uid for ownership while storing permission bits requesting guest uid for ownership while storing permission bits
host-side; writes through a share land owned by the host user. The host-side; writes through a share land owned by the host user.
rootless subuid problem the podman shape has does not exist here.
**The guest init.** A small script provisioned into the scratch rootfs **The guest init.** A small script provisioned into the scratch rootfs
before boot, for every run: it mounts each share, cds to the workdir, for every run: it mounts each share, cds to the workdir, runs the
runs the payload, and writes the exit code to the control file. The payload, and writes the exit code to the control file. It never execs
hostname and workdir ride `krun_set_exec`'s config natively (the init — the init must outlive the payload to report it — and has no
blob applies both); the script's own hostname and cd lines stay as share-less special case: the same contract covers a bare shell run.
fallbacks. There is no exec (the init must outlive the payload to The payload is an argv vector taken literally; a single argument
report it) and no share-less special case: the same contract covers a containing whitespace is a quoted shell line (`apk add x && y`) and
bare shell run. The payload itself is an argv vector taken runs through the image's shell as `sh -c`. Ending the machine is not
literally; the one rewrite is a single argument containing whitespace, the script's job: when it exits, the init blob reports the code and
which is a quoted shell line ("apk add x && y") and runs through the the VMM powers the machine off.
image's shell as `sh -c`. Ending the machine is not the script's job
either:
when the script exits, the init blob reports the code and the VMM
powers the machine off (the libkrunfw kernel has no sysrq).
**Exit codes.** On a normal payload exit the VMM terminates its own **Exit codes.** Only the guest init can mint a guest exit code: it
process with the blob-reported code, so `krun_start_enter` never reports the payload's code through the control file and the VMM exits
returns: run() forks, the child enters the VM loop, and the parent the machine child with it. A signal on the machine child means the VMM
reaps it and reads the code from the exit status. Signal deaths map to or the engine died — the payload never reported — and `run` fails with
128+signal. The control file remains the diagnostic when the boot dies an engine-failure error, never a 128+signal guess at a guest code.
without the init reporting. The mechanism is one hop shorter than 125 and 126 are reserved by the machinery itself: the child bails
designed here, and the contract for consumers is unchanged. with 125 on a boot failure and the init refuses with 126 when a share
would not mount or the workdir is missing. Ending on either without a
control-file code is an error naming the cause; a payload that
genuinely exits either code reported through the file first, and the
file is the arbiter. The control file remains the boot diagnostic in
every other death.
**Cancellation is a kill.** The VMM lives in a child of the ecrou **Cancellation is a kill.** The VMM lives in a child of the ecrou
process carrying PR_SET_PDEATHSIG, so a killed consumer kills the process carrying PR_SET_PDEATHSIG, so a killed consumer kills the
machine with it. No orphan survives. This is stronger than `docker machine with it. No orphan survives. This is stronger than `docker
exec`'s cancel, which kills only the local client; the tests keep it exec`'s cancel, which kills only the local client (T4).
true (T4).
**Console.** Three wirings, chosen by the run's flags. **Console.** Three wirings, chosen by the run's flags.
Headless (the default): guest output reaches real stdout through a Headless (the default): guest output reaches real stdout through a
console file the parent streams while the machine runs, with a final console file the parent streams while the machine runs, with a final
drain after the reap. The direct stdio wiring does not survive contact drain after the reap. The child unwires fd 0 to `/dev/null`: with a
with reality: with a tty on fd 0 libkrun's terminal setup stops a tty on fd 0 libkrun's terminal setup stops a background caller with
background caller with SIGTTOU, and without one its console bytes go SIGTTOU, and /dev/null keeps the console plain while still forwarding
to the log sink, not the terminal. The child therefore unwires fd 0 to a Ctrl-C sent to the process group into the guest.
/dev/null (no guest input; a Ctrl-C to the process group is still
forwarded into the guest).
`-t`: the machine owns the terminal. With no console output file set, `-t`: the machine owns the terminal. The guest console rides the
libkrun wires the guest console to the process's own stdin and stdout, process's own stdin and stdout, the tty goes raw, and the parent saves
puts the tty in raw mode, and restores it through its exit observers. and restores the termios around the run — libkrun's exit observers do
The parent additionally saves and restores the termios around the run: not run in a SIGKILLed machine. Input is the typed keyboard; Ctrl-C
the observers do not run in a SIGKILLed machine. Input is the typed reaches the guest as a byte (raw mode has no ISIG) and the guest's
keyboard; Ctrl-C reaches the guest as a byte (raw mode has no ISIG) line discipline signals its own foreground job. The run refuses to
and the guest's line discipline signals its own foreground job. The start without a terminal on stdin and stdout, and `--capture` is
run refuses to start without a terminal on stdin and stdout, and refused with `-t`: the payload's streams cannot land in files and on
`--capture` is refused with `-t` (the payload's streams cannot land in the terminal at once.
files and on the terminal at once).
`-i` without `-t`: pipes. libkrun still owns the process stdio, and `-i` without `-t`: pipes. libkrun exposes the run's non-tty stdio
its non-tty streams surface in the guest as virtio console ports streams as virtio console ports (`krun-stdin`, `krun-stdout`,
(`krun-stdin`, `krun-stdout`, `krun-stderr` under `/dev/virtio-ports`); `krun-stderr` under `/dev/virtio-ports`), and the guest init points
the guest init re-points the payload's stdio onto whichever port nodes the payload's stdio onto whichever port nodes exist, so
exist, so `echo hi | ecrou run -i alpine -- cat` crosses the machine. `echo hi | ecrou run -i alpine -- cat` crosses the machine. A stream
A stream the host left on a tty has no port node and stays on the the host left on a tty has no port node and stays on the console.
console. There is no QEMU monitor to mux; `--verbose` maps to
`krun_set_log_level`.
**Networking.** TSI, libkrun's default: the guest has no interface, and
its socket calls are proxied on the host's stack by the VMM process.
ecrou pins it explicitly (a vsock with AF_INET hijacked) instead of
leaving the heuristics mode to sniff. The guest resolver is the host's
uplink config with loopback nameservers stripped (loopback never
crosses the proxy; verified by hand). DNS is proxied with the rest.
Known upstream limitation: per-stream throughput in this generation of
the ln driver is tens of KB/s against a CDN, TLS included; workable,
not fast, and worth re-measuring on every engine update. Port
publishing (`krun_set_port_map`) is how a later
sandboxd-in-guest serves the host. No tap, no bridges, ever: they need
host setup the budget cannot afford.
**Memory and CPUs.** `krun_set_vm_config`; default 1G and
all-logical-cpus, `-m` to override. A per-command machine is
provisioned per command, and agent consumers run several concurrently.
## Workspace layout
Cargo workspace, two crates, the ecr split:
- `crates/ecrou` - the library. Modules: `config`, `distro`,
`download`, `error`, `extract`, `guest` (the init script), `krun`
(the FFI), `progress`, `rootfs`, `utils`, `vm`.
- `crates/ecrou-cli` - the CLI (binary `ecrou`).
The library must not depend on CLI types; `clap` is a CLI-only
dependency. Everything else from ecr's rules stands: anyhow internal,
`ecrou::Error` semantic variants at the sites that detect them, `log`
for narration, `Progress` for long operations.
## Library API
### Rootfs lifecycle (ported)
`RootfsCache::new(dir)` / `default_dir()`, `prepare`,
`prepare_provisioned`, `pull`, `persist`, `PrepareRequest`,
`PreparedRootfs`, `Pulled`: `prepare` is the boot path — a present
entry is trusted, the registry is never contacted on a hit, a cold
cache downloads. `pull` is the explicit freshness check (unpinned
tags re-verified, moved digests re-downloaded, nothing extracted).
Plus the `.provisioned` hot-cell marker. The per-run scratch copy is
`PreparedRootfs` as ecr already builds it. See ecr's SPEC for the
reference description; differences are the cache directory and the
absent kernel entries only.
### `vm::run`
```rust
pub struct Share { pub host: PathBuf, pub guest: String, pub read_only: bool }
pub struct RunOptions {
pub image: String, // distro ref, OCI image ref, or local
// rootfs archive path (./x.tar.gz)
pub arch: String, // empty = host arch
pub memory: String, // default "1G"
pub workspace: Option<Share>, // the one rw share consumers name
pub shares: Vec<Share>, // additional named shares, ro allowed
pub command: Vec<String>, // empty = the image's default shell
pub workdir: String, // guest cwd; empty = /
pub env: Vec<(String, String)>, // composed envp; host env never inherited
pub hostname: String, // empty = generated ecrou-vm-XXXXXX
pub interactive: bool, // host stdin feeds the guest
pub tty: bool, // console on the terminal (implies interactive)
pub log_level: LogLevel, // Quiet | Normal | Verbose
pub capture: Option<PathBuf>, // payload stdout/stderr into files
}
pub fn run(options: &RunOptions, progress: &dyn Progress) -> Result<i32>
```
Guest output reaches real stdout through the streamed console; the
return value is the guest exit code. A run
without working KVM is `Error::KvmUnavailable` with the remediation
text. Everything else is `Error::Other` over the anyhow chain, ecr's
shape. There is no long-lived machine; consumers pump the child's
stdio themselves when they drive the CLI.
**Capture.** The console is one stream, kernel and guest mixed, and a **Capture.** The console is one stream, kernel and guest mixed, and a
consumer that needs the payload's stdout and stderr apart - rhema's consumer that needs the payload's streams apart cannot read it. With
sandbox seam reports them as separate fields - cannot read it. With `--capture DIR` the run adds the directory as an internal rw share at
`capture` set, the run adds the directory as an internal rw share the reserved guest path `/.ecrou-io`, and the init wraps the payload
(guest path `/.ecrou-io`, reserved), and the init wraps the payload in so its stdout and stderr land host-side as `<DIR>/stdout` and
a redirection group pointing at it: the two streams land host-side as `<DIR>/stderr`, written live and simply there when `run` returns. The
`<dir>/stdout` and `<dir>/stderr`, written live, owned by the host console stays on stdout for diagnostics, which is what makes capture
user, and simply there when `run` returns. The console stays on stdout and a nulled stdout composable. A boot failure leaves the files
for diagnostics, which is what makes capture and a nulled stdout absent. No caller share may take the reserved guest paths.
composable. A boot failure leaves the files absent.
## CLI **Networking.** TSI, libkrun's default, pinned explicitly instead of
the heuristics mode: the guest has no interface, and its socket calls
are proxied on the host's stack by the VMM process. The guest resolver
is the host's uplink config with loopback nameservers stripped (loopback
never crosses the proxy); DNS is proxied with the rest. Known upstream
limitation: per-stream throughput in this generation of the ln driver
is tens of KB/s against a CDN, TLS included; workable, not fast, and
worth re-measuring on every engine update. No tap, no bridges, ever:
they need host setup the budget cannot afford. Port publishing is how
a later in-guest agent serves the host (Later).
``` **Memory and CPUs.** Default 1G and all logical CPUs; `-m` overrides.
ecrou run [OPTIONS] <IMAGE> [-- COMMAND...] A per-command machine is provisioned per command, and agent consumers
ecrou provision [OPTIONS] <IMAGE> [-- COMMAND...] run several concurrently.
ecrou pull [OPTIONS] <IMAGE>
ecrou probe
```
`pull` fetches or refreshes the image's cache entry without booting
it: it re-resolves floating references (distro aliases like
`latest`/`lts`/`edge`, unpinned OCI tags against the registry
digest) and downloads only when the entry is absent or has moved.
It is the freshness operation — see the trust model below for how
`run` and `pull` divide the work. Local archives are rejected:
a file on disk has nothing to fetch.
The image is a distro name (`alpine`, `ubuntu:noble`), an OCI
reference (`ghcr.io/org/image:tag`, `ubuntu@sha256:…`,
`repo:tag@sha256:…` — the digest wins over a co-present tag, as in
docker), or a local rootfs archive named with a path
(`./rootfs.tar.gz`, `/tmp/rootfs.tar`): a path reference is read in
place on every run, never downloaded and never cached as a pristine
entry.
| Flag | Default | Description |
|------|---------|-------------|
| `-i, --interactive` | false | Keep the machine's stdin open: the host's stdin feeds the guest |
| `-t, --tty` | false | Console on the terminal: raw mode, the guest sees a tty (implies `-i`; needs a terminal on stdin and stdout) |
| `-v, --volume <HOST:GUEST[:ro]>` | none | Share a directory into the machine; the first rw one is the workspace; repeatable |
| `-m, --memory <size>` | 1G | Guest memory |
| `-w, --workdir <guest path>` | `/` | Guest cwd |
| `-e, --env <KEY=VALUE>` | none | Guest env; repeatable |
| `--quiet` | false | Suppress kernel log on the console |
| `--capture <DIR>` | none | Payload stdout/stderr into `<DIR>/stdout` and `<DIR>/stderr`; incompatible with `-t` |
| `--no-cache` | false | Fresh download, ignore the cache (a local archive is always read fresh) |
| `--verbose` | false | Diagnostics; raises the libkrun log level |
`ecrou provision IMAGE -- COMMAND` bakes a command into the image's
cache entry: the command runs in a machine over a freshly prepared
rootfs and the result is persisted, so every later boot of that image
carries what the command installed — packages, users, whatever a
one-shot boot cannot keep. A marked entry is reused as-is (the
command is skipped), `--no-cache` re-downloads and re-provisions, and
a failing command leaves the entry unprovisioned for the next attempt.
The marker dies with the entry: a digest refresh drops it, so a moved
upstream image is provisioned again, never silently served pristine
under an old marker.
`ecrou probe` reports what a run would find: KVM working (open, ioctl
version, nested-virt state), `libkrun.so.1` loadable, the cache
directory writable. It is the confinement diagnostic and the support
tool.
## File layout
```
~/.cache/ecrou/
├── ubuntu-noble-amd64.tar.gz
├── alpine-edge-x86_64.tar.gz
├── <...>.digest # manifest digest sidecar
└── <...>.provisioned # hot-cell marker
```
## Snap packaging
Confinement: strict. Plugs: `network`, `home`, `kvm`. Manual
`snap connect` for kvm is accepted for now; auto-connect is a store
request and a later product decision.
The snap stages `libkrun.so.1` and libkrunfw next to the binary; at
roughly 6 MB combined they cost less than any qemu-system package, and
the kernel ships inside libkrunfw, so there is nothing else to fetch or
stage.
Where the libraries come from, as of now: no Ubuntu release publishes
a `libkrun.so.1` package at all, and libkrunfw5 exists only in the
current development series' universe, not in any LTS the snap bases
ship from. Debian sid has both (source package `libkrun`), so the gap
is sync lag and it should close by the next LTS. Until then the snap
carries both halves itself: a part that builds libkrun from source
(the recipe is small: rust, clang for the C shim, `make build-release`
with the net feature for TSI) and stages `libkrun.so.1`; and a part
that fetches the `libkrunfw5` .deb from the Ubuntu pool and unpacks it
into the staging tree. snapcraft puts staged libraries on
`LD_LIBRARY_PATH` at runtime, which is exactly the resolution path the
`dlopen` in krun.rs takes; there is no ldconfig and no host dependency.
When an LTS base archive ships both packages, both parts shrink to
`stage-packages` entries and the from-source build is deleted.
The first milestone is the spike snap, before any porting: a minimal
libkrun binary behind strict confinement running a VM. T1 is the gate
that proves the shape under real AppArmor and seccomp, and everything
else in this document is design until it passes.
## Trust model ## Trust model
@@ -436,8 +235,9 @@ stop:
test exists for, and libkrun's virtiofs DAX windows are exactly where test exists for, and libkrun's virtiofs DAX windows are exactly where
it bites. it bites.
- **T3, exit codes.** `ecrou run alpine -- sh -c "exit 7"` exits 7. - **T3, exit codes.** `ecrou run alpine -- sh -c "exit 7"` exits 7.
A guest signal death exits 128+signal. A failed VMM reports an A guest signal death exits 128+signal (the shell convention crosses
engine failure, never a guest code. the control file). A failed VMM reports an engine failure, never a
guest code.
- **T4, kill chain.** SIGKILL ecrou-cli mid-run; the VM is gone with - **T4, kill chain.** SIGKILL ecrou-cli mid-run; the VM is gone with
the process, no stray threads or KVM VMs, verified by scanning the the process, no stray threads or KVM VMs, verified by scanning the
process table and `/dev/kvm` users. process table and `/dev/kvm` users.
@@ -450,6 +250,26 @@ stop:
- **T7, guest network.** An https fetch from the guest succeeds, DNS - **T7, guest network.** An https fetch from the guest succeeds, DNS
included; a published port is reachable from the host. included; a published port is reachable from the host.
## Snap packaging
Confinement: strict. Plugs: `network`, `home`, `kvm`. The manual
`snap connect` for kvm is accepted until auto-connect is a store
request and a later product decision.
The snap stages `libkrun.so.1` and libkrunfw next to the binary; the
kernel ships inside libkrunfw, so there is nothing else to fetch or
stage. No Ubuntu release publishes a `libkrun.so.1` package today, and
libkrunfw5 exists only in the current development series' universe,
not in any LTS the snap bases ship from — so one part builds libkrun
from source and one fetches the libkrunfw5 .deb from the Ubuntu
archives, per architecture (the main archive for amd64 and arm64, the
ports archive for riscv64), each pinned by digest. snapcraft puts
staged libraries on `LD_LIBRARY_PATH` at runtime, which is exactly the
resolution path the `dlopen` in krun.rs takes; there is no ldconfig
and no host dependency. When an LTS base archive ships both packages,
both parts shrink to `stage-packages` entries and the from-source
build is deleted.
## Later ## Later
- **QEMU engine.** The TCG fallback for machines without KVM. The - **QEMU engine.** The TCG fallback for machines without KVM. The
@@ -461,29 +281,12 @@ stop:
to grow into when per-command VMs stop being right. to grow into when per-command VMs stop being right.
- **Provisioned layers, keyed.** `provision` today bakes one entry - **Provisioned layers, keyed.** `provision` today bakes one entry
per image; the next step keys entries by (base digest, hash of the per image; the next step keys entries by (base digest, hash of the
provisioning spec) so consumers — rhema's projects — each get their provisioning spec) so consumers each get their own layer over a
own layer over a shared base, the spec file living in the repo and shared base, the spec file living in the repo and the artifact
the artifact reproducible from it. Marker-hit semantics then skip reproducible from it. Marker-hit semantics then skip the command
the command only for the identical spec, and eviction (LRU or an only for the identical spec, and eviction (LRU or an explicit
explicit prune) keeps derived entries bounded. prune) keeps derived entries bounded.
- **passt** (`krun_set_passt_fd`) if TSI throughput or UDP listening - **passt** (`krun_set_passt_fd`) if TSI throughput or UDP listening
ever matters. ever matters.
- **sandboxd-in-guest** over `krun_set_port_map`: rhema's http backend - **An agent in the guest** over port publishing: an http backend
pointed into a machine. pointed into a machine.
- **Rootfs persistence** through `prepare_provisioned` hot cells,
already in the ported code.
## Open questions
- Does `krun_start_enter` return the guest exit code natively on
poweroff? The control file works regardless; if the return value
carries it, the mechanism shrinks to a read.
- Console wiring: does the default virtio-console land on the VMM's
stdio, and is stdin carried? Decided by a first-run experiment, with
`krun_set_console_output` as the fallback.
- TSI DNS: which resolver file the guest needs, and whether lookups
just work. T7 decides.
- dlopen versus link-time linking for the snap; dlopen is the default
for the remediation error it enables.
- Which distros package libkrun (Fedora does; Debian and Ubuntu do not
yet) and what the dev-install story says meanwhile.