docs: make libkrun the v1 engine

QEMU was the committed v1 engine with libkrun later; that order is
flipped. libkrun deletes the initramfs pack, the kernel fetch, slirp
config, and the uid mapping problem, and its VMM runs in-process, which
collapses the kill chain. Verified against the stable-1.19.x headers:
eleven-call v1 surface, krun_set_workdir native, mapped volumes dead on
Linux (tagged virtiofs plus guest init mount instead), KVM required
with QEMU as the later TCG fallback. Snap spike now stages libkrun
plus libkrunfw instead of qemu-system.

SPEC.md

@@ -7,16 +7,16 @@ from ecr, the ephemeral chroot: the name keeps the ancestor, and the
 answer to "chroot ou VM" is now VM. As a bonus, an écrou is a nut: the
 small part that keeps a bolt from walking out of the wall.
 
-One sentence: resolve an image reference, boot it as a one-shot microVM,
+One sentence: resolve an image reference, boot it as a one-shot microVM
-run one command with live stdio, propagate its exit code, keep whatever
+on libkrun, run one command with live stdio, propagate its exit code,
-crossed the workspace share, throw the machine away.
+keep whatever crossed the workspace share, throw the machine away.
 
 The project is library-first like ecr: `crates/ecrou` is the library
-(package `ecrou`), `crates/ecr-cli` becomes `crates/ecrou-cli` (binary
+(package `ecrou`), `crates/ecrou-cli` is the thin front-end (binary
-`ecrou`), a thin front-end that parses flags, maps them onto the library
+`ecrou`) that parses flags, maps them onto the library API, and
-API, and propagates exit codes. The library never writes to the
+propagates exit codes. The library never writes to the terminal:
-terminal: diagnostics go through the `log` facade, long operations
+diagnostics go through the `log` facade, long operations report through
-report through the `Progress` sink, all output is the CLI's.
+the `Progress` sink, all output is the CLI's.
 
 The first consumer is rhema's sandbox seam, which shells out to the
 binary the way it shells out to `docker` and `lxc` today. The library
@@ -28,13 +28,48 @@ runtime.
 | module | disposition |
 |---|---|
 | `distro`, `download`, `extract`, `rootfs`, `utils`, `progress` | ported; the image machinery is confinement-neutral plain fs and network |
-| `qemu_vm` (cpio builder, KVM probe, launch) | ported and grown; becomes `initramfs` + `qemu` + `vm` |
+| `kernel` | dropped; the kernel is libkrun's, bundled in libkrunfw |
-| `kernel` | ported; Alpine `linux-virt` fetch, cache entry renamed |
+| `qemu_vm` | dropped from v1; its KVM probe logic (open `/dev/kvm`, `KVM_GET_API_VERSION` == 12) moves into the `krun` module |
 | `namespace`, `chroot`, `mount`, `exec` | dropped; they stay in ecr, their ancestor. Their error variants (`UserNamespacesUnavailable`, `BinfmtNotRegistered`, `MissingSubidRange`) leave with them |
 
+No initramfs and no cpio builder: the rootfs crosses as a virtiofs
+share, never as a packed archive.
+
 No cache compatibility with ecr: the formats may match but the directory
 is `~/.cache/ecrou` and never shared.
 
+## The engine: libkrun
+
+v1 has exactly one engine, libkrun, driven through its stable C ABI
+(the `stable-1.19.x` headers; the main branch is mid-rewrite to a
+vtable API and is not the target). ecrou `dlopen`s `libkrun.so.1` at
+runtime and fails with a remediation message naming the install when it
+is missing; nothing links at build time, and a snap bundles the library
+plus libkrunfw next to itself.
+
+The VMM runs inside the ecrou process: `krun_start_enter` enters the
+VM loop on the calling thread. There is no child process to shepherd,
+which makes the kill contract below trivial.
+
+The FFI lives in `crates/ecrou/src/krun.rs`. ecrou carries unsafe libc
+code; rhema, its first consumer, keeps `#![forbid(unsafe_code)]` and
+drives the CLI.
+
+The v1 API surface is eleven calls: `krun_set_log_level`,
+`krun_create_ctx`, `krun_set_vm_config`, `krun_set_root`,
+`krun_add_virtiofs`, `krun_set_workdir`, `krun_set_exec`,
+`krun_set_env`, `krun_set_console_output`, `krun_start_enter`,
+`krun_check_nested_virt`. `krun_set_mapped_volumes` exists but is
+explicitly unsupported on Linux (its header says to use user namespaces
+and bind-mounts instead, which is precisely what ecrou cannot do); the
+workspace is a tagged `krun_add_virtiofs` share mounted by the guest
+init instead.
+
+KVM is required. There is no TCG fallback; a machine without working
+KVM is an `Error::KvmUnavailable` with the remediation text, and
+`ecrou probe` reports it before any run. QEMU as a fallback engine is a
+Later row, and its arrival would not change the API.
+
 ## The founding rule
 
 ecrou's host process never mounts a filesystem and never creates or
@@ -55,101 +90,96 @@ beyond what the shares expose.
 
 Everything the host process may do, and nothing else:
 
-- fs: read, write, stat, rename, unlink, mkdir under the cache directory
+- fs: read, write, stat, rename, unlink, mkdir, fallocate under the
-  and the share roots it was given, plus its own private tmpdir
+  cache directory and the share roots it was given, plus the scratch
-- exec: the kernel it boots and `qemu-system-*` (later: libkrun), found
+  rootfs TempDir
-  in PATH or staged in the snap
+- exec: itself; libraries via `dlopen` (`libkrun.so.1`, libkrunfw)
-- `/dev/kvm`: open, ioctl, mmap. Optional; TCG is the fallback when the
+- `/dev/kvm`: open, ioctl, mmap
-  probe finds no working KVM (`KVM_GET_API_VERSION` != 12)
+- net: TCP and UDP client sockets; with TSI these carry the guest's
-- net: TCP and UDP client sockets for registry fetches, slirp's host
+  connections too, proxied host-side
-  side, and localhost listeners only through explicit `hostfwd`
+- process: threads, futexes, eventfds, timers; no children in v1
-- process: fork, exec, waitpid, and signals for its own children
 
 Denied by design: `mount`, `umount2`, `unshare`, `setns`, `pivot_root`,
 `open_tree`, `move_mount`, `mount_setattr`, `fsopen`, `fsmount`,
 `ptrace`, kernel module loading, keyring calls. All absent from snapd's
 seccomp allowlist, which is where the allowed list above is drawn from.
-Host `mknod` is allowlisted but AppArmor mediates what it can create;
+Two absences need a spike-time check because VMMs plausibly touch them:
-ecrou has no use for it either way.
+`sched_setaffinity` and `openat2`. The spike straces for EPERM on
+allowed-looking calls, and anything libkrun genuinely needs that the
+allowlist denies goes on a snapd issue, never into a workaround.
 
 ## Execution model
 
-One-shot machines. `ecrou run IMAGE -- CMD` boots one VM whose init
+One-shot machines. `ecrou run IMAGE -- CMD` prepares a rootfs, boots
-execs CMD as the only workload; when CMD exits, the machine powers off.
+one VM whose payload is CMD, streams its stdio, and exits with CMD's
-There is no exec into a running machine. No engine in the family has
+code when the machine powers off. There is no exec into a running
-that API: libkrun's `krun_set_exec` is per-context before start, and
+machine. No engine in the family has that API: `krun_set_exec` is
-crun's krun handler boots a fresh VM for every `crun exec`. A guest
+per-context before start, and crun's krun handler boots a fresh VM for
-agent and long-lived machines are a later row; per-command boots are the
+every `crun exec`. A guest agent and long-lived machines are a Later
-v1 shape, and the boot budget below is what keeps them honest.
+row; per-command boots are the v1 shape, and the boot budget below is
+what keeps them honest.
 
-Ephemeral rootfs, durable workspace. The base system is an initramfs:
+Ephemeral rootfs, durable workspace. The base system is a scratch copy
-it exists only in RAM and dies with the machine. Persistence crosses a
+of the image (a `PreparedRootfs`, TempDir-backed): it exists only for
-workspace share or it does not happen.
+the run and dies with it. Persistence crosses a workspace share or it
+does not happen.
 
-**Workspace shares.** `--share HOST:GUEST[:ro]`, served by 9p
+**Workspace shares.** `--share HOST:GUEST[:ro]`, one virtiofs share per
-(`qemu -virtfs local`). The host side is plain fs under one more
+entry (`krun_add_virtiofs`, `krun_set_root` keeps the rootfs itself).
-directory the budget covers; the guest init mounts it. There is no
+The host side is plain fs; the guest init mounts each share at its
-implicit share: an empty sandbox is the default, and the consumer
+guest path. There is no implicit share: an empty sandbox is the
-chooses what crosses. This is deliberate; a sandbox runtime does not
+default, and the consumer chooses what crosses. This is deliberate; a
-silently attach the caller's cwd.
+sandbox runtime does not silently attach the caller's cwd.
 
-**Exit codes.** The machine has a private control share: a host tmpdir
+Ownership needs no mapping layer. The guest runs as root inside its own
-the API does not surface, mounted read-write at a fixed guest path. The
+kernel, and libkrun's simplified virtiofs semantics report the
-init writes CMD's exit code to a file there, then powers off; ecrou
+requesting guest uid for ownership while storing permission bits
-reads it back and exits with it. Signal deaths map to 128+signal, the
+host-side; writes through a share land owned by the host user. The
-shell convention the init already produces. Alternatives considered and
+rootless subuid problem the podman shape has does not exist here.
-rejected for v1: isa-debug-exit (x86-only, awkward from a shell init)
-and semihosting exit (arch-dependent). If the exit file is absent and
-QEMU itself failed, that is an engine failure, reported as one, never as
-a guest code.
 
-**Cancellation is a kill.** ecrou-cli spawns QEMU with
+**The guest init.** When shares exist, the payload is a small script
-`PR_SET_PDEATHSIG` and a kill-on-drop guard, so a killed consumer kills
+provisioned into the scratch rootfs before boot: it mounts each share,
-the machine. No orphan QEMU survives the CLI. This is stronger than
+`cd`s nowhere (the workdir rides `krun_set_workdir` natively), execs
-`docker exec`'s cancel, which kills only the local client; the tests
+the real payload, and writes its exit code to the control file. With no
-keep it true (T4).
+shares, the payload execs directly.
 
-**Console.** `-serial stdio`, no monitor mux. The QEMU monitor is a
+**Exit codes.** The scratch rootfs doubles as the control channel: the
-host-level control hatch over the machine; a sandbox runtime does not
+guest writes its exit code to a file there, and ecrou reads it back
-mux it into the workload's stdio by accident. `--monitor` restores
+host-side after the machine powers off. Signal deaths map to
-`mon:stdio` for development.
+128+signal, the shell convention the init already produces. If the
+file is absent and the VMM reports a failure, that is an engine error,
+reported as one, never as a guest code. Native propagation through
+`krun_start_enter`'s return value is on the verify list; the file is
+the contract either way, and the mechanism can only get simpler.
 
-**Networking.** slirp user-mode only. The init configures the guest
+**Cancellation is a kill.** The VMM lives in the ecrou process, so a
-statically (10.0.2.15/24, gateway 10.0.2.2, DNS 10.0.2.3); no DHCP
+killed consumer kills the machine with it. No orphan survives. This is
-client is assumed. No tap, no bridges, ever: they need host setup the
+stronger than `docker exec`'s cancel, which kills only the local
-budget cannot afford. `hostfwd` stays available for a later
+client; the tests keep it true (T4).
-sandboxd-in-guest.
 
-**Env and argv** ride the kernel command line base64-encoded, ecr's
+**Console.** virtio-console wired to the process's stdio, live lines
-trick, extended to the environment. Sane size is a few KB; oversized
+out, stdin through. `krun_set_console_output` is the fallback if the
-payloads are a caller bug and fail loudly.
+default wiring needs redirecting. There is no QEMU monitor to mux;
+`--verbose` maps to `krun_set_log_level`.
 
-**Memory.** Default 1G, `-m` to override. ecr's CLI defaulted 2G; a
+**Networking.** TSI, libkrun's default: the guest has no interface, and
-per-command machine is provisioned per command, and agent consumers run
+its socket calls are proxied on the host's stack by the VMM process.
-several concurrently.
+DNS is proxied with them (verified in T7, including the resolver file
+the guest needs). Port publishing (`krun_set_port_map`) is how a later
+sandboxd-in-guest serves the host. No tap, no bridges, ever: they need
+host setup the budget cannot afford.
 
-## The guest side
+**Memory and CPUs.** `krun_set_vm_config`; default 1G and
-
+all-logical-cpus, `-m` to override. A per-command machine is
-The init is ours, a generated shell script packed into the initramfs and
+provisioned per command, and agent consumers run several concurrently.
-selected with `rdinit=`. It grows from ecr's (hostname, shell detection,
-argv decode, sysrq poweroff fallback) with static net config, share
-mounts, the workdir `cd`, and the exit-code handshake added. It uses
-only what the image ships; `busybox` is not assumed.
-
-Kernel: Alpine `linux-virt` per target arch (the `linux-lts` fallback
-when no virt flavor exists), fetched and cached as today. The pack step
-adds the booted kernel's `/lib/modules` tree into the initramfs: 9p and
-virtio_net are modules in `linux-virt`, and the distro image's module
-tree does not match the kernel we boot. The size cost is measured in
-T5, and gzip compression of the initramfs is the knob if it hurts.
 
 ## Workspace layout
 
 Cargo workspace, two crates, the ecr split:
 
 - `crates/ecrou` - the library. Modules: `config`, `distro`,
-  `download`, `error`, `extract`, `initramfs`, `kernel`, `progress`,
+  `download`, `error`, `extract`, `guest` (the init script), `krun`
-  `qemu`, `rootfs`, `utils`, `vm`.
+  (the FFI), `progress`, `rootfs`, `utils`, `vm`.
 - `crates/ecrou-cli` - the CLI (binary `ecrou`).
 
 The library must not depend on CLI types; `clap` is a CLI-only
@@ -161,11 +191,13 @@ for narration, `Progress` for long operations.
 
 ### Rootfs lifecycle (ported)
 
-`RootfsCache::new(dir)` / `default_dir()`, `prepare`, `prepare_provisioned`,
+`RootfsCache::new(dir)` / `default_dir()`, `prepare`,
-`persist`, `PrepareRequest`, `PreparedRootfs`: ecr's semantics verbatim,
+`prepare_provisioned`, `persist`, `PrepareRequest`, `PreparedRootfs`:
-including digest freshness on `:latest` and the `.provisioned` hot-cell
+ecr's semantics verbatim, including digest freshness on `:latest` and
-marker. See ecr's SPEC for the reference description; differences are
+the `.provisioned` hot-cell marker. The per-run scratch copy is
-the cache directory and the kernel entry names only.
+`PreparedRootfs` as ecr already builds it. See ecr's SPEC for the
+reference description; differences are the cache directory and the
+absent kernel entries only.
 
 ### `vm::run`
 
@@ -178,21 +210,20 @@ pub struct RunOptions {
     pub workspace: Option<Share>,   // the one rw share consumers name
     pub shares: Vec<Share>,     // additional named shares, ro allowed
     pub command: Vec<String>,   // empty = the image's default shell
-    pub workdir: String,        // guest cwd; empty = first share's guest path, else /
+    pub workdir: String,        // guest cwd; empty = /
     pub env: Vec<(String, String)>, // composed envp; host env never inherited
     pub hostname: String,       // empty = generated ecrou-vm-XXXXXX
     pub quiet: bool,
-    pub require_kvm: bool,
 }
 pub fn run(options: &RunOptions, progress: &dyn Progress) -> Result<i32>
 ```
 
-stdio is inherited; the return value is the guest exit code. A
+stdio is inherited; the return value is the guest exit code. A run
-`require_kvm` run with no working KVM is `Error::KvmUnavailable` with
+without working KVM is `Error::KvmUnavailable` with the remediation
-the remediation text. Everything else is `Error::Other` over the anyhow
+text. Everything else is `Error::Other` over the anyhow chain, ecr's
-chain, ecr's shape. The v1 API has no captured-output variant and no
+shape. The v1 API has no captured-output variant and no long-lived
-long-lived machine; consumers pump the child's stdio themselves when
+machine; consumers pump the child's stdio themselves when they drive
-they drive the CLI.
+the CLI.
 
 ## CLI
 
@@ -206,17 +237,16 @@ ecrou probe
 | `-a, --arch <arch>` | host arch | Target architecture |
 | `--share <HOST:GUEST[:ro]>` | none | Workspace share; repeatable |
 | `-m, --memory <size>` | 1G | Guest memory |
-| `--workdir <guest path>` | first share, else `/` | Guest cwd |
+| `--workdir <guest path>` | `/` | Guest cwd |
 | `-e, --env <KEY=VALUE>` | none | Guest env; repeatable |
-| `--require-kvm` | false | Fail instead of falling back to TCG |
-| `--monitor` | false | Dev: mux the QEMU monitor onto stdio |
 | `--quiet` | false | Suppress kernel log on the console |
 | `--no-cache` | false | Fresh download, ignore the cache |
-| `-v, --verbose` | false | Diagnostics |
+| `-v, --verbose` | false | Diagnostics; raises the libkrun log level |
 
-`ecrou probe` reports what a run would find: KVM working or the TCG
+`ecrou probe` reports what a run would find: KVM working (open, ioctl
-fallback, the QEMU binary, the kernel in cache, the cache directory
+version, nested-virt state), `libkrun.so.1` loadable, the cache
-writable. It is the confinement diagnostic and the support tool.
+directory writable. It is the confinement diagnostic and the support
+tool.
 
 ## File layout
 
@@ -225,8 +255,7 @@ writable. It is the confinement diagnostic and the support tool.
 ├── ubuntu-noble-amd64.tar.gz
 ├── alpine-edge-x86_64.tar.gz
 ├── <...>.digest                        # manifest digest sidecar
-├── <...>.provisioned                   # hot-cell marker
+└── <...>.provisioned                   # hot-cell marker
-└── ecrou-default-kernel-x86_64.vmlinuz
 ```
 
 `~/.config/ecrou.yaml` carries over (DNS overrides today).
@@ -237,57 +266,71 @@ Confinement: strict. Plugs: `network`, `home`, `kvm`. Manual
 `snap connect` for kvm is accepted for now; auto-connect is a store
 request and a later product decision.
 
-The first milestone is the spike snap, ecr's VM mode behind strict
+The snap stages `libkrun.so.1` and libkrunfw next to the binary; at
-confinement, before any porting: stage `qemu-system-x86_64` and the
+roughly 6 MB combined they cost less than any qemu-system package, and
-kernel in the snap, or fetch both through the cache in $HOME. Both fit
+the kernel ships inside libkrunfw, so there is nothing else to fetch or
-the budget; T1 is the gate that proves the shape under real AppArmor
+stage.
-and seccomp, and everything else in this document is design until it
+
-passes.
+The first milestone is the spike snap, before any porting: a minimal
+libkrun binary behind strict confinement running a VM. T1 is the gate
+that proves the shape under real AppArmor and seccomp, and everything
+else in this document is design until it passes.
 
 ## Falsification tests
 
 - **T1, confinement.** The spike snap runs `ecrou run alpine -- uname
   -a` under strict confinement. The guest kernel version differing from
   the host is the microVM proof, the same check crun's krun man page
-  prescribes.
+  prescribes. The same run straces for EPERM on plausible VMM calls
+  (`sched_setaffinity`, `openat2`) and reports findings to the snapd
+  issue list.
 - **T2, workspace coherence.** In a shared workspace, the guest runs
   git init, add, commit, status; the host sees the commits; a second
   machine sees the first's writes. Git's mmap'd index is the trap this
-  test exists for.
+  test exists for, and libkrun's virtiofs DAX windows are exactly where
+  it bites.
 - **T3, exit codes.** `ecrou run alpine -- sh -c "exit 7"` exits 7.
-  A guest signal death exits 128+signal. A killed QEMU reports an
+  A guest signal death exits 128+signal. A failed VMM reports an
   engine failure, never a guest code.
-- **T4, kill chain.** SIGKILL ecrou-cli mid-run; QEMU is gone within a
+- **T4, kill chain.** SIGKILL ecrou-cli mid-run; the VM is gone with
-  second, no orphan, verified by scanning the process table.
+  the process, no stray threads or KVM VMs, verified by scanning the
-- **T5, boot economics.** KVM boot-to-init for alpine with
+  process table and `/dev/kvm` users.
-  `linux-virt` lands under 1.5 s including the module-tree pack cost.
+- **T5, boot economics.** Alpine, cache-warm: extract plus boot lands
-  TCG numbers are recorded, not gated.
+  under 1.5 s to first guest output. Larger images report their numbers
+  unblocked; the shared-ro-rootfs row in Later is the fix if they hurt.
 - **T6, no-mount audit.** `strace -f -e trace=mount,umount2,unshare,
   setns,pivot_root,open_tree,move_mount,fsopen,fsmount` over a full run
   with shares: zero hits. This test is the founding rule.
-- **T7, guest network.** An https fetch from the guest succeeds.
+- **T7, guest network.** An https fetch from the guest succeeds, DNS
+  included; a published port is reachable from the host.
 
 ## Later
 
-- **libkrun engine.** The FFI lives here; ecrou already carries unsafe
+- **QEMU engine.** The TCG fallback for machines without KVM. The
-  libc, and rhema keeps its `forbid(unsafe_code)`. Sub-second boots,
+  `RunOptions` API does not change; the engine choice is the probe's.
-  the kernel bundled in libkrunfw, TSI networking instead of slirp.
+- **Shared read-only rootfs.** Serve the cache rootfs directly, ro,
-  Same budget, one more engine.
+  with a guest-side overlay for writes: no extraction per run. The
-- **Tiny initramfs.** A fixed small init with the rootfs itself served
+  correct-and-slow v1 posture graduates into this.
-  over 9p, killing the per-run pack cost and the RAM floor.
 - **Guest agent, long-lived machines.** Exec without a boot; the shape
   to grow into when per-command VMs stop being right.
-- **passt** if slirp throughput ever matters.
+- **passt** (`krun_set_passt_fd`) if TSI throughput or UDP listening
-- **sandboxd-in-guest** over hostfwd: rhema's http backend pointed
+  ever matters.
-  into a machine.
+- **sandboxd-in-guest** over `krun_set_port_map`: rhema's http backend
+  pointed into a machine.
 - **Rootfs persistence** through `prepare_provisioned` hot cells,
   already in the ported code.
 
 ## Open questions
 
-- 9p versus virtiofsd as the share transport. 9p needs no extra daemon;
+- Does `krun_start_enter` return the guest exit code natively on
-  virtiofsd is the modern path but is another staged binary. Decide
+  poweroff? The control file works regardless; if the return value
-  when T2 runs; the `Share` API does not change either way.
+  carries it, the mechanism shrinks to a read.
-- Initramfs compression default: uncompressed boots fastest, gzip
+- Console wiring: does the default virtio-console land on the VMM's
-  halves the RAM floor. Measure in T5.
+  stdio, and is stdin carried? Decided by a first-run experiment, with
-- Whether `probe` belongs in the library or stays CLI-only.
+  `krun_set_console_output` as the fallback.
+- TSI DNS: which resolver file the guest needs, and whether lookups
+  just work. T7 decides.
+- dlopen versus link-time linking for the snap; dlopen is the default
+  for the remediation error it enables.
+- Which distros package libkrun (Fedora does; Debian and Ubuntu do not
+  yet) and what the dev-install story says meanwhile.