# `pkh deb` Incremental Builds — Spec ## Problem `pkh deb` has no incremental story. Every invocation — including a rerun seconds after a failed build — pays the full pipeline again: 1. mmdebstrap tarball **extraction** into a fresh `/tmp/pkh-` chroot (the tarball itself is cached, but extraction of a kernel-sized chroot is minutes), 2. device nodes + `/proc` bind mount, 3. `apt-get update`, 4. essentials install, 5. `quilt push -a`, 6. native build-dependency resolution + install, 7. `debian/rules build` **from scratch** in a freshly staged copy of the tree, 8. `fakeroot debian/rules binary`. The failures this hurts most are the late ones. A kernel package that dies in `debian/rules build` after 25 minutes is restarted from zero; fixing one compile error and rerunning recompiles everything, and a cycle of "tweak, rebuild, fail again" costs a full pipeline each time. The raw material for resumption already exists but is unreachable: - **Failed chroots are kept.** `EphemeralContextGuard::drop` (`src/deb/ephemeral.rs`) deliberately keeps the chroot when the build did not succeed — but the next run creates a new `pkh-` directory and cannot find or reuse the old one. The kept chroot is a leak until `pkh prune`, not a cache. - **The staged tree is an overlayfs mount** (`src/context/unshare.rs`): host tree = lowerdir (read live), build writes = upperdir inside the chroot. If the upperdir survives between runs, object files survive too, and host-side edits propagate automatically — exactly the semantics incremental builds need. - **Ctrl+C deletes everything** (`sigint_cleanup_chroot`), which is the worst behavior for a long build interrupted on purpose. ## Goal `pkh deb --resume` after a failed build resumes from the point of failure (plain `pkh deb` always rebuilds from scratch — see §5): - same environment, source untouched → picks up at `debian/rules build`, recompiling nothing (make sees a warm tree); - source modified since the failure → still reuses chroot, apt state and installed build-deps; `debian/rules build` recompiles only what the modification affects; - `debian/control` or patch series changed → environment-level state (build-deps, quilt) is redone; the chroot and object files still carry over when possible. Non-goal: distributing or sharing sessions between machines; resuming `pkh build` (the native source pipeline — it is seconds-fast already); resuming inside ssh/schroot contexts (sessions are a local-unshare feature; other drivers keep today's behavior). ## Design ### 1. Build sessions: named, journaled, discoverable A **session** is one (package, series, arch, cross) build environment plus its journal. Replace the anonymous `pkh-` chroot directory with a session root: ``` /var/tmp/pkh/sessions// session.json # manifest + phase journal (see below) chroot/ # the chroot tree (was /tmp/pkh-) tree-upper/ # overlayfs upperdir for the staged package tree pkh-overlay/ # overlay workdirs (as today) ``` `` is derived from the session identity (package, series, arch, cross) plus a short content hash, so a rerun can find a previous session by identity without guessing timestamps. One live session per identity — the newest attempt *replaces* the previous one (same chroot environment, fresher state; keeping per-attempt history would mean one full chroot per attempt, which is a cache nobody wants). A locked session (concurrent build) is never replaced (§6). What the session list shows for a tree is therefore one entry per identity (series/arch/cross combination), each carrying its latest attempt's id. Moving off `/tmp` is deliberate: sessions must survive a reboot to be worth keeping, and `/tmp` is often tmpfs (a kernel chroot on tmpfs is RAM). `/var/tmp` is the conventional persistent-scratch location; the base directory is overridable (`PKH_SESSIONS_DIR` env, then config). Sessions are a cache, and `pkh prune` is their primary GC — if systemd-tmpfiles sweeps `/var/tmp` on some setup, that is an acceptable, if blunt, secondary cleanup. `session.json` records: ```jsonc { "id": "20260926T143505", // build-start timestamp, UTC; changes on every attempt "host_tree": "/home/me/linux", // tree the session was built from (list scoping) "identity": { "package": "linux", "series": "stonking", "arch": "arm64", "cross": true }, "created": "...", "last_used": "...", "chroot": { "tarball": "stonking-arm64-buildd.tar.xz", "tarball_sha256": "...", // reuse only if the cached tarball is the same "ready": true // device nodes + /proc done }, "phases": { "apt_update": { "at": "..." }, // journal-only: apt update always reruns on resume "essentials": { "at": "..." }, "patches": { "stamp": "", "applied": true }, "build_deps": { "stamp": "", "at": "..." }, "build": { "at": "...", "result": "failed" } }, "tree": { "version": "7.3.0-5.6~local2", // changelog version of the last attempt "host_snapshot": "", // for deletion propagation } } ``` The journal is written **after** each phase completes (and after a phase fails, recording the failure), so a crash or Ctrl+C at any point leaves a truthful journal. Discovery is: scan `$PKH_SESSIONS_DIR/*/session.json`, match identity. ### 2. Two layers of resume The journal separates **environment resume** (expensive, insensitive to source edits) from **build resume** (sensitive to every edit): | Phase | Reused when | Redone when | |---|---|---| | Chroot extract, nodes, /proc | chroot present, `ready`, same tarball hash | never for the same identity (otherwise the session is unusable) | | `apt-get update` | — (always rerun on resume) | — | | Essentials | done (verify cheaply via `dpkg-query` at most) | chroot redone | | quilt push | `debian/patches` hash unchanged | hash changed → `quilt pop -a` + re-push | | Build-deps | control/arch/cross/ppa/inject/pocket stamp unchanged | stamp changed → the resolver runs again (it is idempotent; apt installs only what is missing) | | `debian/rules build` | always *attempted* in the warm tree | — | | `fakeroot debian/rules binary` | always rerun | — | Key consequences: - **`apt-get update` is always rerun on resume.** It is the cheapest phase, and stale package lists fail confusingly ("candidate version not found") on fast-moving suites — there is nothing worth caching here. The journal entry exists for observability only. - **A changelog version bump** (`pkh chlog` between attempts) invalidates build resume but not environment resume: the chroot, apt lists and build-deps carry over. The staged tree is re-populated for the new version (see §3); the session's `tree.version` tracks which attempt the upperdir belongs to, and a version change discards the upperdir (object files reference paths/flags of the old version). - **The build phase is never skipped**, only made incremental — make decides. pkh's job is to hand `debian/rules build` a tree whose previously produced objects are still there and whose sources match the host. - Before rerunning the binary phase on a resume, clear `debian/files` (and `debian/*.log`, `debian/substvars`) from the staged tree so artifact collection (`collect_binary_artifacts`) only sees the new attempt's outputs — today's "no globbing stale files" contract is preserved. ### 3. Tree propagation on resume Today each build stages a fresh copy/overlay of the host tree. On resume the staged tree already exists inside the session; the problem is syncing it with a possibly-modified host tree. - **Overlay contexts (the default when overlayfs is available):** remount the overlay with the *same* upperdir. Host modifications are visible immediately (lowerdir is read live); objects written in the upperdir during the previous attempt are still on top. Host-side *deletions* are the one gap — overlayfs only hides a lower file via a whiteout in the upper, and nothing creates whiteouts for files the user deleted on the host. Fix: the session stores a snapshot of the host tree's file list at first staging; on resume, files present in the upper (or in the snapshot) but gone from the host get an explicit whiteout (`mknod c 0 0` inside the upper). This is a bounded, manifest-driven operation, not a tree walk of the whole chroot. - **Copy fallback (no overlayfs):** `rsync -a --delete` of the host tree over the staged tree, *excluding* the build artifacts make produced (this needs per-package exclusion knowledge and is best effort). Deletions propagate naturally here. Sessions on the copy path get build resume only for trees where the exclusion set is sane; the honest fallback is: copy path → environment resume only. The overlay logic stays inside the unshare driver, matching the principle established in `plans/overlayfs-integration.md`: callers stage a tree through `ensure_available()` and never learn how. ### 4. Interrupt behavior Ctrl+C currently removes the chroot through the interrupt hook. Change: **an interrupted build keeps the session**, journaling the interrupted phase as incomplete — an intentional Ctrl+C on a 30-minute build is a "pause", and deleting the environment on pause defeats the feature. The cleanup hook (`sigint_cleanup_chroot`) becomes session-aware: it still unmounts the overlays and `/proc` (leaving mounted state in a session is what makes stale sessions dangerous), but keeps the tree and leaves a `interrupted: true` marker in the journal so the next run knows the session needs no special recovery (phases after the last completed one simply rerun). Users who want the old hard-discard behavior on interrupt get it via `pkh prune`. The failure path is unchanged in spirit: failed builds already keep their chroot; they now keep a *usable* one. ### 5. CLI surface and policy **Recording is always on; reuse is opt-in.** The two are deliberately separated, because the first attempt cannot know it will fail: a plain `pkh deb` writes and maintains the session (journal, kept chroot on failure or interrupt — today's keep-on-failure, made usable) but never reads one. Only `pkh deb --resume` adopts an existing session. The default run therefore always rechecks everything from scratch — no stale-build risk unless the user asks for one. - `pkh deb` — today's behavior, plus session recording: any existing session of the same identity is ignored and replaced by this run's outcome. On failure or interrupt the session is kept (as today); on success it is **removed**, unless `--keep` is given. - `pkh deb --keep` — also keep the session after a successful build, so a later `--resume` can iterate on it (the edit–rebuild loop). Composes with `--resume`: `--resume --keep` chains sessions across iterations; a resumed build without `--keep` consumes the session on success — the natural end of the loop. - `pkh deb --resume []` — adopt a session and continue it, announcing what is being skipped ("Resuming session 20260926T143505 for linux/stonking-arm64: chroot, package lists and build-deps reused"). - Without an id: the newest session recorded from this tree (manifest `host_tree` matches the tree being built), whatever its last outcome — failure, interrupt, or a `--keep` success. - With an id (a build-start timestamp as shown by `pkh deb list`): that exact session. A unique unambiguous prefix is accepted; an ambiguous one is an error listing the candidates. - Selector conflicts are errors, not surprises: if `-a`/`-s`/ `--cross`/`--ppa`/`--inject` are given explicitly and disagree with the adopted session, refuse and point at `pkh deb list`. A session for another series/arch is a *different* build environment; silently building it under the requested selectors would be exactly the stale-build risk this design avoids. - With no session to adopt (none recorded, pruned, or the tree moved): say so and build from scratch — never silently. - `pkh deb list` — a subcommand, not a flag: `pkh deb` with flags always means "start a build", and listing is a different action. It prints the sessions recorded from the current tree, one row per identity (§1), newest attempt first: ``` $ pkh deb list Sessions for /home/me/linux: ID PACKAGE VERSION TARGET LAST RUN AGE 20260926T143505 linux 7.3.0-5.6~local2 stonking/arm64 failed 2h 20260925T090012 linux 7.3.0-5.5~local1 stonking/arm64* success 1d 20260924T171100 linux 7.3.0-5.6~local2 stonking/riscv64* failed 2d * cross build. Resume with: pkh deb --resume [] ``` `pkh deb list` never touches build machinery: it reads the session manifests under the sessions directory and renders. As an optional subcommand of `deb` (clap allows subcommands alongside args), the build flags stay on `deb` itself and `list` takes none of them. - `pkh prune` learns sessions: by default removes sessions untouched for longer than a retention window (7 days) and any session whose journal is missing/corrupt; `--all` removes every session. Prune's report distinguishes "session (resumable build)" from the other residual categories, since sessions are large and users must be able to see and reclaim them. ### 6. Correctness guards A session is only reused when **all** of these hold; otherwise it is discarded (replaced) with a logged reason: - identity matches (package, series, arch, cross); - the cached chroot tarball hash matches the one the chroot was built from; - `chroot/` passes an integrity probe (marker file, `chroot/bin/sh` exists and executes); - pkh version compatibility: `session.json` records the pkh version and manifest schema version; a mismatched schema discards the session (the chroot is generic enough that only schema-relevant changes matter, but being conservative is cheap). - no other live session for the same identity (lock file in the session root, in the spirit of the existing tarball lockfile). Concurrent builds of the same identity take the lock and fall back to a fresh ephemeral session (current behavior) rather than waiting — correctness over caching. One more guard on the resume path, at the CLI layer: explicit selectors (`-a`, `-s`, `--cross`, `--ppa`, `--inject`) that disagree with the adopted session refuse the build (§5) — the session is only ever reused under the selectors it was created with. ## Implementation sketch New `src/deb/session.rs` — session identity, manifest (serde) read/write, discovery, locking, slug generation. Pure logic, fully unit-testable. - `src/deb/ephemeral.rs` — `EphemeralContextGuard` learns a `resume` path: when a session is adopted, skip download/extract and reuse the existing chroot (still verifying `ready` and remounting `/proc` if a previous run unmounted it); on teardown, hand the tree to the session instead of `rm -rf`-ing it. Chroot dir moves from `create_temp_dir()` to the session layout — the local driver's temp naming stays for non-session uses. - `src/deb/local.rs` — journal writes around each phase; the skip-or-run decisions per the table in §2; `debian/files` cleanup before the binary phase on resume. - `src/context/unshare.rs` — upperdir reuse + whiteout propagation for host deletions (§3); `ensure_available()` gains a "stage into existing session tree" mode. - `src/interrupt.rs` + `src/deb/ephemeral.rs` — the interrupt hook keeps the session (§4). - `src/prune.rs` — session discovery, retention, reporting. - `src/main.rs` — `--keep`, `--resume []` argument wiring, the `pkh deb list` subcommand and selector-conflict checks; `README.md` roadmap update (this retires part of the "deb: asynchronous build, detachable and monitorable" itch: a persistent session is also the natural attach point for detached builds later). Staged rollout, each step independently shippable: 1. Sessions with manifest + `pkh deb list` + adoption of failed-build chroots via `--resume` (environment resume only — biggest win, no overlay changes). 2. Build resume: upperdir reuse + whiteout propagation. 3. Interrupt keeps session; `--keep` after success; prune integration. ## Testing - Unit: manifest round-trip, slug/identity matching, whiteout plan computation (host snapshot vs current host listing), skip-decision table given journals with various stamps. - Integration (small fixture package, `#[cfg(test)]` style already used by `deb` tests): fail `debian/rules build` on file B after compiling file A; resume and assert A is *not* recompiled (make log via the tee log); modify A; resume and assert only A recompiles; modify `debian/control`; assert build-deps rerun and objects survive; `pkh prune` GC behavior; a plain `pkh deb` ignoring an existing session (fresh rebuild) while `--resume` adopts it. - Unit/CLI: `pkh deb list` rendering against fixture manifests (scoping by `host_tree`, cross marker, ordering); `--resume ` exact and prefix resolution, ambiguity error; selector-conflict refusals; `--keep` keeping the session on success while a plain build removes it. - Chroot reuse tests need the unshare/mount machinery — follow the existing pattern of `#[ignore]` tests for deliberate ad-hoc runs where root/unshare is required. ## Decisions - **Resume is opt-in (`--resume`), never the silent default.** A plain `pkh deb` always rechecks everything from scratch, so the no-flag behavior can never build anything stale; recording stays on by default so the follow-up `--resume` after a failure has a session to adopt. - **Keeping a session after success is opt-in (`--keep`).** Today's disk profile — success cleans up — stays the default; the edit–rebuild iteration loop is served by `pkh deb --resume --keep`. Failures and interrupts keep their session unconditionally, as today. - **Sessions are addressed by id (build-start timestamp), listed via the `pkh deb list` subcommand.** Listing is a subcommand, not a mode-switching flag: `pkh deb` with flags always starts a build. One live session per identity keeps the cache bounded (a per-attempt history would be one full chroot per attempt); the list therefore shows one row per series/arch/cross combination for the current tree, each carrying its latest attempt's id. Explicit selectors that disagree with the adopted session are an error, never a silent environment switch. - **Ctrl+C keeps the session**, same as a failed build: an interrupt on a long build is a pause, and there is nothing inherent in "stop now" that means "discard half an hour of environment setup". The hard-discard behavior stays reachable through `pkh prune`. - **`apt-get update` always reruns on resume** (no stamp, no TTL) — see §2. - **Sessions live under `/var/tmp/pkh/sessions`** (overridable via `PKH_SESSIONS_DIR` then config), and are cleaned by `pkh prune` (retention window by default, `--all` for everything). Cleanup by systemd-tmpfiles from time to time is accepted; sessions are a cache. --- # Implementation Notes (as built) Implemented across `src/deb/session.rs` (new), `src/deb/{mod,local, ephemeral}.rs`, `src/context/{api,unshare}.rs`, `src/prune.rs` and `src/main.rs`. Deviations from the design above, all deliberate: - **Phase stamps gate less than specced.** The journal records every phase (with the patch-tree and control stamps), but on resume only the expensive non-idempotent work is actually skipped: the chroot bootstrap (extraction, device nodes, `/proc`) and, via the reused overlay upperdir, the compiled objects. `apt-get update`, the essentials install, the quilt push and the build-dep resolution always rerun — each is idempotent and seconds-cheap, and rerunning them removes a whole class of stale-state bugs. The stamps still drive the quilt `pop -a` decision (patch tree changed since the recorded attempt) and the version-change handling. - **The advisory lock lives outside the session root** (`/.lock`): a teardown deletes the root while still holding the lock, and a lock file inside it would be removed under the holder — a concurrent opener would then create and lock a fresh inode, and mutual exclusion silently dies (observed with two concurrent same-identity e2e builds before the move). - **`pkh prune` session GC is confined to the production `prune()`.** `prune_in()` (the testable core) deliberately does not scan the real sessions root: the test suite runs prune tests and e2e builds concurrently, and the prune tests would delete live sessions mid-build (observed). `prune_in_roots()` takes the sessions root as an explicit opt-in parameter. - **Fallbacks, in order:** no local base context → the historical anonymous temp chroot (no session); session locked by a concurrent build → same; overlay mount unsupported/failed on resume → fresh copy staging (environment resume only, artifacts discarded); no host-tree snapshot → upper wiped and re-snapshotted.