From af06264e60b89ade9ab9b27480434259b492fe8e Mon Sep 17 00:00:00 2001 From: Valentin Haudiquet Date: Mon, 21 Sep 2026 00:08:49 +0200 Subject: [PATCH] docs: document library crate, exec and rootfs cache apis README gains a Library section covering prepare, the exec options (envp, bind targets, arch) and the provisioned-rootfs hot-cell flow. SPEC documents the workspace layout, the library API, the cache sidecars (.digest, .provisioned) and the updated execution flow. AGENTS.md scope paths follow the new crates/ layout. --- AGENTS.md | 33 +++++++++++-------- README.md | 52 ++++++++++++++++++++++++++++- SPEC.md | 97 +++++++++++++++++++++++++++++++++++++++++++++++-------- 3 files changed, 154 insertions(+), 28 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index c9f04a8..19a0dfa 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -30,19 +30,26 @@ Rules: - Types: `feat`, `fix`, `refactor`, `perf`, `test`, `docs`, `build`, `ci`, `chore`. -- Scopes match the component touched — the module name under `src/`: - - `cli` — entry point and flags (`src/main.rs`, `src/cli.rs`) - - `config` — config file (`src/config.rs`) - - `distro` — distro definitions and mirrors (`src/distro.rs`) - - `download` — image download (`src/download.rs`) - - `extract` — rootfs extraction (`src/extract.rs`) - - `chroot` — chroot setup (`src/chroot.rs`) - - `namespace` — Linux namespaces (`src/namespace.rs`) - - `mount` — bind mounts and mount table (`src/mount.rs`) - - `kernel` — kernel/initramfs handling for `--kernel` (`src/kernel.rs`) - - `qemu` — QEMU VM mode (`src/qemu.rs`, `src/qemu_vm.rs`) - - `utils`, `verbose` — shared helpers (`src/utils.rs`, - `src/verbose.rs`) +- Scopes match the component touched — the module name under + `crates/ecr/src/` (library) or `crates/ecr-cli/src/` (CLI): + - `cli` — entry point and flags (`crates/ecr-cli/src/main.rs`, + `crates/ecr-cli/src/cli.rs`) + - `rootfs` — cache-aware preparation and persistence + (`crates/ecr/src/rootfs.rs`) + - `exec` — namespace-mode execution API (`crates/ecr/src/exec.rs`) + - `config` — config file (`crates/ecr/src/config.rs`) + - `distro` — distro definitions and mirrors (`crates/ecr/src/distro.rs`) + - `download` — image download (`crates/ecr/src/download.rs`) + - `extract` — rootfs extraction (`crates/ecr/src/extract.rs`) + - `chroot` — chroot setup (`crates/ecr/src/chroot.rs`) + - `namespace` — Linux namespaces (`crates/ecr/src/namespace.rs`) + - `mount` — bind mounts and mount table (`crates/ecr/src/mount.rs`) + - `kernel` — kernel/initramfs handling for `--kernel` + (`crates/ecr/src/kernel.rs`) + - `qemu` — QEMU VM mode (`crates/ecr/src/qemu.rs`, + `crates/ecr/src/qemu_vm.rs`) + - `utils`, `verbose` — shared helpers (`crates/ecr/src/utils.rs`, + `crates/ecr/src/verbose.rs`) - `deps` — dependency additions/bumps (manifests, lockfile) - Omit the scope entirely for repo-wide changes that do not belong to a single component (README.md, SPEC.md, root config). diff --git a/README.md b/README.md index 48de759..9439039 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,56 @@ ecr fedora # any Docker Hub image `ecr` pulls a root filesystem (Alpine/Ubuntu direct from their CDNs; everything else from Docker Hub), extracts it into a temporary directory, and execs a shell inside a user + mount + PID + UTS namespace. The process tree is isolated, the rootfs is discarded on exit, and your host is never touched. +## Library + +`ecr` is a cargo workspace: `crates/ecr` is the library crate (package `ecr`), `crates/ecr-cli` the thin command-line front-end. Rust consumers can use the library directly — resolve an image, download it through the content cache, and run commands with a fully caller-controlled environment, bind targets and architecture: + +```toml +[dependencies] +ecr = { git = "https://github.com/…" } # or a path / registry source +``` + +```rust +use ecr::{chroot, BindTarget, ExecOptions, PrepareRequest, RootfsCache}; + +// Download alpine through the cache (~/.cache/ecr) and extract to a scratch dir +let cache = RootfsCache::new(RootfsCache::default_dir()?); +let rootfs = cache.prepare(&PrepareRequest::new("alpine:3.23"))?; + +// Caller-composed envp, explicit bind targets, target architecture +let code = rootfs.exec(&ExecOptions { + arch: "amd64".into(), // empty = host arch + binds: vec![BindTarget { // host dir -> in-rootfs mount + source: "./data".into(), + target: "/mnt/data".into(), + read_only: false, // true = overlay (ro) + }], + env: chroot::default_env(rootfs.dir()), // or your own envp + command: vec!["cat".into(), "/etc/os-release".into()], + ..ExecOptions::default() +})?; + +rootfs.persist()?; // write the rootfs back into the cache (see below) +``` + +### Provisioned-rootfs caching (hot cells) + +Prepare once, provision once, amortize every later run — including runs from other processes sharing the same cache directory: + +```rust +let rootfs = cache.prepare_provisioned(&PrepareRequest::new("alpine:3.23"), |rootfs| { + // runs at most once per cache entry + rootfs.exec(&ExecOptions { + command: vec!["apk".into(), "add".into(), "curl".into()], + env: chroot::default_env(rootfs.dir()), + ..ExecOptions::default() + })?; + Ok(()) +})?; +``` + +On the first call the pristine image is downloaded, the closure provisions it, and the result is persisted into the cache with a `.provisioned` marker. On every subsequent call the entry is hit directly: no download, no provisioning. Note a provisioned entry no longer tracks the upstream image; delete the entry (or pass `no_cache: true`) to re-provision. + ## Usage ``` @@ -105,7 +155,7 @@ Requirements: ## Cache -Downloaded images are cached in `~/.cache/ecr/`. For `latest` OCI tags the registry manifest digest is checked on each run — the image is only re-downloaded when it has actually changed. Use `--no-cache` to force a fresh pull regardless. +Downloaded images are cached in `~/.cache/ecr/`. For `latest` OCI tags the registry manifest digest is checked on each run — the image is only re-downloaded when it has actually changed. Use `--no-cache` to force a fresh pull regardless. The library's `PreparedRootfs::persist` can replace a cache entry with a provisioned rootfs (see *Provisioned-rootfs caching* above). ## Requirements diff --git a/SPEC.md b/SPEC.md index 30bc4aa..7548c14 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1,5 +1,63 @@ # ecr - implementation specification +## Workspace Layout + +Cargo workspace with two crates: + +- `crates/ecr` — the library (package `ecr`). All functionality lives here: + image resolution, download, extraction, namespaces, mounts, chroot exec, + QEMU VM mode. +- `crates/ecr-cli` — the CLI front-end (package `ecr-cli`, binary `ecr`). + Parses flags, maps them onto the library API, propagates exit codes. + +The library must not depend on CLI types (`clap` is a CLI-only dependency). + +## Library API + +### Rootfs lifecycle (`ecr::rootfs`) + +- `RootfsCache::new(dir)` / `RootfsCache::default_dir()` — the image tarball + cache (default `~/.cache/ecr`). +- `cache.prepare(&PrepareRequest)` — parse the image reference, resolve + floating version aliases ("latest", "lts", "edge") to a concrete version + before computing the cache key, run the OCI `:latest` digest freshness + check, download through the cache when needed, and extract into a scratch + directory. Returns a `PreparedRootfs` (TempDir-backed; dropped on drop). +- `cache.prepare_provisioned(&req, provision)` — hot-cell amortization: on a + cache hit with a `.provisioned` sidecar the entry is extracted as-is (no + freshness check, no provisioning); otherwise the rootfs is prepared, + `provision(&PreparedRootfs)` runs once, and `persist` writes the result + back. `no_cache` forces re-download + re-provision. +- `PreparedRootfs::persist()` — packs the current rootfs into its cache + entry (compression chosen from the entry's filename extension: gz, xz, + zstd, plain tar; symlinks stored as links; permissions preserved) and + writes the `.provisioned` sidecar. The entry's contents are replaced: + plain `prepare` also returns the provisioned contents from then on. + +### Execution (`ecr::exec`) + +- `exec(rootfs, &ExecOptions)` — run a command inside a prepared rootfs in + fresh user/PID/mount/UTS namespaces; returns the exit code (128+signal on + kill). Empty option fields select defaults: + - `arch`: host architecture; a foreign arch requires binfmt_misc. + - `env`: caller-composed envp as `(key, value)` pairs; empty selects + `chroot::default_env`. The host environment is never inherited. + - `binds`: explicit `BindTarget { source, target, read_only }` — host + directory mounted at an absolute in-rootfs mount point; read-only via + overlay, read-write via bind. `..` targets are rejected. + - `dns`: nameservers written to /etc/resolv.conf; empty copies the host + resolver. + - `command`: argv; empty runs the rootfs default shell. + - `working_dir`: explicit in-rootfs cwd; empty picks the first read-write + bind target that exists, then `/root`, then `/`. + +### Mount plumbing (`ecr::mount`) + +`setup_mounts(rootfs, &[BindTarget])` mounts proc/dev/devpts/sys and applies +the bind targets; `in_rootfs(rootfs, target)` maps in-chroot paths and +rejects escaping targets. Overlay upper/work directories are temp dirs the +caller must keep alive (returned by `setup_mounts`). + ## Synopsis ``` @@ -36,11 +94,21 @@ ecr [OPTIONS] -- [COMMAND]... ~/.cache/ecr/ ├── ubuntu-noble-amd64.tar.gz ├── alpine-latest-x86_64.tar.gz -├── debian-bookworm-amd64.tar.gz +├── oci-docker_io-library_archlinux-latest-amd64.tar.gz +├── oci-docker_io-library_archlinux-latest-amd64.tar.gz.digest └── ... ``` -No metadata files. Tarballs are downloaded once and never redownloaded. Users can delete files manually or use `--no-cache` to fetch fresh. +Sidecar files, never counted as image entries: + +- `.digest` — manifest digest of the last OCI download, used by the + `:latest` freshness check. +- `.provisioned` — marker written by `PreparedRootfs::persist`; + `prepare_provisioned` treats an entry with this marker as provisioned. + +Tarballs are downloaded once and never redownloaded (unless the digest +moves, `--no-cache` is passed, or a provisioned entry is deleted). Users +can delete files manually. ### Config File @@ -111,20 +179,20 @@ Error: No manifest found for architecture 'riscv64'. Available: amd64, arm64, pp ## Execution Flow -1. Parse CLI arguments -2. Resolve distro/version/arch to image source +The CLI delegates to the library; the namespace-mode flow is: + +1. Parse CLI arguments, map flags onto library requests +2. `cache.prepare`: resolve distro/version/arch to image source 3. Check cache for existing tarball 4. If not cached, download tarball (direct or OCI) -5. Create temp directory for extraction -6. Extract tarball to temp directory -7. Create namespaces: user, pid, mount, uts -8. Set up mounts: /proc, /sys (ro), /dev, /dev/pts +5. Extract tarball to a temporary directory +6. `ecr::exec`: create namespaces: user, pid, mount, uts +7. Set up mounts: /proc, /sys (ro), /dev, /dev/pts +8. Apply bind targets: overlays (ro) and bind mounts (rw) 9. Write /etc/resolv.conf with DNS servers -10. Set up overlay mounts for bind paths -11. Set up read-write bind mounts -12. Set environment variables -13. Exec shell or command in chroot -14. On exit, clean up temp directory +10. Set the working directory +11. Exec shell or command in chroot with the composed envp +12. On exit, clean up the temporary directory; propagate the exit code ## Namespace Setup @@ -300,7 +368,8 @@ dns: ## Environment Variables -Default environment inside chroot: +Default environment inside chroot (`chroot::default_env`, used by the CLI; +library callers compose their own envp): - HOME=/root - USER=root