diff --git a/README.md b/README.md index 87aa67d..a86aa37 100644 --- a/README.md +++ b/README.md @@ -111,6 +111,28 @@ pkh put --ppa user/hello_xxx git push xxx user-fork ``` +### Incremental builds (build sessions) + +Every `pkh deb` build records a **session** under `/var/tmp/pkh/sessions`: +the bootstrapped chroot, the installed build dependencies and the build +artifacts of the staged tree. When a build fails (or is interrupted), the +session is kept and can be resumed: + +``` +pkh deb # fails after 25 minutes +pkh deb --resume # reuses the chroot, build deps and objects; + # only what changed is recompiled +pkh deb list # the sessions of this tree, with their ids +pkh deb --resume # resume a specific session +pkh deb --keep # keep the session even after a successful build + # (iterate: edit, `pkh deb --resume --keep`, ...) +pkh prune # garbage-collect old sessions (7-day retention) +``` + +A plain `pkh deb` never reuses a session — everything is rechecked from +scratch — and it replaces the session of its target. `pkh deb --resume` +refuses to adopt a session built for a different series/architecture. + ## Future improvement ideas - pull: try to fetch the correct git branch for series on Debian diff --git a/plans/deb-incremental-builds.md b/plans/deb-incremental-builds.md new file mode 100644 index 0000000..bcc4ed4 --- /dev/null +++ b/plans/deb-incremental-builds.md @@ -0,0 +1,429 @@ +# `pkh deb` Incremental Builds — Spec + +## Problem + +`pkh deb` has no incremental story. Every invocation — including a rerun +seconds after a failed build — pays the full pipeline again: + +1. mmdebstrap tarball **extraction** into a fresh `/tmp/pkh-` chroot + (the tarball itself is cached, but extraction of a kernel-sized chroot + is minutes), +2. device nodes + `/proc` bind mount, +3. `apt-get update`, +4. essentials install, +5. `quilt push -a`, +6. native build-dependency resolution + install, +7. `debian/rules build` **from scratch** in a freshly staged copy of the + tree, +8. `fakeroot debian/rules binary`. + +The failures this hurts most are the late ones. A kernel package that +dies in `debian/rules build` after 25 minutes is restarted from zero; +fixing one compile error and rerunning recompiles everything, and a +cycle of "tweak, rebuild, fail again" costs a full pipeline each time. + +The raw material for resumption already exists but is unreachable: + +- **Failed chroots are kept.** `EphemeralContextGuard::drop` + (`src/deb/ephemeral.rs`) deliberately keeps the chroot when the build + did not succeed — but the next run creates a new `pkh-` directory + and cannot find or reuse the old one. The kept chroot is a leak until + `pkh prune`, not a cache. +- **The staged tree is an overlayfs mount** (`src/context/unshare.rs`): + host tree = lowerdir (read live), build writes = upperdir inside the + chroot. If the upperdir survives between runs, object files survive + too, and host-side edits propagate automatically — exactly the + semantics incremental builds need. +- **Ctrl+C deletes everything** (`sigint_cleanup_chroot`), which is the + worst behavior for a long build interrupted on purpose. + +## Goal + +`pkh deb --resume` after a failed build resumes from the point of +failure (plain `pkh deb` always rebuilds from scratch — see §5): + +- same environment, source untouched → picks up at + `debian/rules build`, recompiling nothing (make sees a warm tree); +- source modified since the failure → still reuses chroot, apt state + and installed build-deps; `debian/rules build` recompiles only what + the modification affects; +- `debian/control` or patch series changed → environment-level state + (build-deps, quilt) is redone; the chroot and object files still + carry over when possible. + +Non-goal: distributing or sharing sessions between machines; resuming +`pkh build` (the native source pipeline — it is seconds-fast already); +resuming inside ssh/schroot contexts (sessions are a local-unshare +feature; other drivers keep today's behavior). + +## Design + +### 1. Build sessions: named, journaled, discoverable + +A **session** is one (package, series, arch, cross) build environment +plus its journal. Replace the anonymous `pkh-` chroot +directory with a session root: + +``` +/var/tmp/pkh/sessions// + session.json # manifest + phase journal (see below) + chroot/ # the chroot tree (was /tmp/pkh-) + tree-upper/ # overlayfs upperdir for the staged package tree + pkh-overlay/ # overlay workdirs (as today) +``` + +`` is derived from the session identity (package, series, arch, +cross) plus a short content hash, so a rerun can find a previous +session by identity without guessing timestamps. One live session per +identity — the newest attempt *replaces* the previous one (same +chroot environment, fresher state; keeping per-attempt history would +mean one full chroot per attempt, which is a cache nobody wants). A +locked session (concurrent build) is never replaced (§6). What the +session list shows for a tree is therefore one entry per identity +(series/arch/cross combination), each carrying its latest attempt's +id. + +Moving off `/tmp` is deliberate: sessions must survive a reboot to be +worth keeping, and `/tmp` is often tmpfs (a kernel chroot on tmpfs is +RAM). `/var/tmp` is the conventional persistent-scratch location; the +base directory is overridable (`PKH_SESSIONS_DIR` env, then config). +Sessions are a cache, and `pkh prune` is their primary GC — if +systemd-tmpfiles sweeps `/var/tmp` on some setup, that is an +acceptable, if blunt, secondary cleanup. + +`session.json` records: + +```jsonc +{ + "id": "20260926T143505", // build-start timestamp, UTC; changes on every attempt + "host_tree": "/home/me/linux", // tree the session was built from (list scoping) + "identity": { + "package": "linux", "series": "stonking", + "arch": "arm64", "cross": true + }, + "created": "...", "last_used": "...", + "chroot": { + "tarball": "stonking-arm64-buildd.tar.xz", + "tarball_sha256": "...", // reuse only if the cached tarball is the same + "ready": true // device nodes + /proc done + }, + "phases": { + "apt_update": { "at": "..." }, // journal-only: apt update always reruns on resume + "essentials": { "at": "..." }, + "patches": { "stamp": "", "applied": true }, + "build_deps": { "stamp": "", "at": "..." }, + "build": { "at": "...", "result": "failed" } + }, + "tree": { + "version": "7.3.0-5.6~local2", // changelog version of the last attempt + "host_snapshot": "", // for deletion propagation + } +} +``` + +The journal is written **after** each phase completes (and after a +phase fails, recording the failure), so a crash or Ctrl+C at any point +leaves a truthful journal. Discovery is: scan +`$PKH_SESSIONS_DIR/*/session.json`, match identity. + +### 2. Two layers of resume + +The journal separates **environment resume** (expensive, insensitive to +source edits) from **build resume** (sensitive to every edit): + +| Phase | Reused when | Redone when | +|---|---|---| +| Chroot extract, nodes, /proc | chroot present, `ready`, same tarball hash | never for the same identity (otherwise the session is unusable) | +| `apt-get update` | — (always rerun on resume) | — | +| Essentials | done (verify cheaply via `dpkg-query` at most) | chroot redone | +| quilt push | `debian/patches` hash unchanged | hash changed → `quilt pop -a` + re-push | +| Build-deps | control/arch/cross/ppa/inject/pocket stamp unchanged | stamp changed → the resolver runs again (it is idempotent; apt installs only what is missing) | +| `debian/rules build` | always *attempted* in the warm tree | — | +| `fakeroot debian/rules binary` | always rerun | — | + +Key consequences: + +- **`apt-get update` is always rerun on resume.** It is the cheapest + phase, and stale package lists fail confusingly ("candidate version + not found") on fast-moving suites — there is nothing worth caching + here. The journal entry exists for observability only. + +- **A changelog version bump** (`pkh chlog` between attempts) invalidates + build resume but not environment resume: the chroot, apt lists and + build-deps carry over. The staged tree is re-populated for the new + version (see §3); the session's `tree.version` tracks which attempt + the upperdir belongs to, and a version change discards the upperdir + (object files reference paths/flags of the old version). +- **The build phase is never skipped**, only made incremental — make + decides. pkh's job is to hand `debian/rules build` a tree whose + previously produced objects are still there and whose sources match + the host. +- Before rerunning the binary phase on a resume, clear + `debian/files` (and `debian/*.log`, `debian/substvars`) from the + staged tree so artifact collection (`collect_binary_artifacts`) only + sees the new attempt's outputs — today's "no globbing stale files" + contract is preserved. + +### 3. Tree propagation on resume + +Today each build stages a fresh copy/overlay of the host tree. On +resume the staged tree already exists inside the session; the problem +is syncing it with a possibly-modified host tree. + +- **Overlay contexts (the default when overlayfs is available):** remount + the overlay with the *same* upperdir. Host modifications are visible + immediately (lowerdir is read live); objects written in the upperdir + during the previous attempt are still on top. Host-side *deletions* + are the one gap — overlayfs only hides a lower file via a whiteout in + the upper, and nothing creates whiteouts for files the user deleted + on the host. Fix: the session stores a snapshot of the host tree's + file list at first staging; on resume, files present in the upper + (or in the snapshot) but gone from the host get an explicit whiteout + (`mknod c 0 0` inside the upper). This is a bounded, + manifest-driven operation, not a tree walk of the whole chroot. +- **Copy fallback (no overlayfs):** `rsync -a --delete` of the host + tree over the staged tree, *excluding* the build artifacts make + produced (this needs per-package exclusion knowledge and is best + effort). Deletions propagate naturally here. Sessions on the copy + path get build resume only for trees where the exclusion set is + sane; the honest fallback is: copy path → environment resume only. + +The overlay logic stays inside the unshare driver, matching the +principle established in `plans/overlayfs-integration.md`: callers +stage a tree through `ensure_available()` and never learn how. + +### 4. Interrupt behavior + +Ctrl+C currently removes the chroot through the interrupt hook. Change: +**an interrupted build keeps the session**, journaling the interrupted +phase as incomplete — an intentional Ctrl+C on a 30-minute build is a +"pause", and deleting the environment on pause defeats the feature. + +The cleanup hook (`sigint_cleanup_chroot`) becomes session-aware: it +still unmounts the overlays and `/proc` (leaving mounted state in a +session is what makes stale sessions dangerous), but keeps the tree +and leaves a `interrupted: true` marker in the journal so the next run +knows the session needs no special recovery (phases after the last +completed one simply rerun). + +Users who want the old hard-discard behavior on interrupt get it via +`pkh prune`. The failure path is unchanged in spirit: failed builds +already keep their chroot; they now keep a *usable* one. + +### 5. CLI surface and policy + +**Recording is always on; reuse is opt-in.** The two are deliberately +separated, because the first attempt cannot know it will fail: a +plain `pkh deb` writes and maintains the session (journal, kept +chroot on failure or interrupt — today's keep-on-failure, made +usable) but never reads one. Only `pkh deb --resume` adopts an +existing session. The default run therefore always rechecks +everything from scratch — no stale-build risk unless the user asks +for one. + +- `pkh deb` — today's behavior, plus session recording: any existing + session of the same identity is ignored and replaced by this run's + outcome. On failure or interrupt the session is kept (as today); + on success it is **removed**, unless `--keep` is given. +- `pkh deb --keep` — also keep the session after a successful build, + so a later `--resume` can iterate on it (the edit–rebuild loop). + Composes with `--resume`: `--resume --keep` chains sessions across + iterations; a resumed build without `--keep` consumes the session + on success — the natural end of the loop. +- `pkh deb --resume []` — adopt a session and continue it, + announcing what is being skipped ("Resuming session + 20260926T143505 for linux/stonking-arm64: chroot, package lists + and build-deps reused"). + - Without an id: the newest session recorded from this tree + (manifest `host_tree` matches the tree being built), whatever its + last outcome — failure, interrupt, or a `--keep` success. + - With an id (a build-start timestamp as shown by `pkh deb list`): + that exact session. A unique unambiguous prefix is accepted; an + ambiguous one is an error listing the candidates. + - Selector conflicts are errors, not surprises: if `-a`/`-s`/ + `--cross`/`--ppa`/`--inject` are given explicitly and disagree + with the adopted session, refuse and point at `pkh deb list`. A + session for another series/arch is a *different* build + environment; silently building it under the requested selectors + would be exactly the stale-build risk this design avoids. + - With no session to adopt (none recorded, pruned, or the tree + moved): say so and build from scratch — never silently. +- `pkh deb list` — a subcommand, not a flag: `pkh deb` with flags + always means "start a build", and listing is a different action. + It prints the sessions recorded from the current tree, one row per + identity (§1), newest attempt first: + + ``` + $ pkh deb list + Sessions for /home/me/linux: + ID PACKAGE VERSION TARGET LAST RUN AGE + 20260926T143505 linux 7.3.0-5.6~local2 stonking/arm64 failed 2h + 20260925T090012 linux 7.3.0-5.5~local1 stonking/arm64* success 1d + 20260924T171100 linux 7.3.0-5.6~local2 stonking/riscv64* failed 2d + + * cross build. Resume with: pkh deb --resume [] + ``` + + `pkh deb list` never touches build machinery: it reads the session + manifests under the sessions directory and renders. As an optional + subcommand of `deb` (clap allows subcommands alongside args), the + build flags stay on `deb` itself and `list` takes none of them. +- `pkh prune` learns sessions: by default removes + sessions untouched for longer than a retention window (7 days) and + any session whose journal is missing/corrupt; `--all` removes every + session. Prune's report distinguishes "session (resumable build)" + from the other residual categories, since sessions are large and + users must be able to see and reclaim them. + +### 6. Correctness guards + +A session is only reused when **all** of these hold; otherwise it is +discarded (replaced) with a logged reason: + +- identity matches (package, series, arch, cross); +- the cached chroot tarball hash matches the one the chroot was built + from; +- `chroot/` passes an integrity probe (marker file, `chroot/bin/sh` + exists and executes); +- pkh version compatibility: `session.json` records the pkh version + and manifest schema version; a mismatched schema discards the + session (the chroot is generic enough that only schema-relevant + changes matter, but being conservative is cheap). +- no other live session for the same identity (lock file in the + session root, in the spirit of the existing tarball lockfile). + +Concurrent builds of the same identity take the lock and fall back to +a fresh ephemeral session (current behavior) rather than waiting — +correctness over caching. + +One more guard on the resume path, at the CLI layer: explicit +selectors (`-a`, `-s`, `--cross`, `--ppa`, `--inject`) that disagree +with the adopted session refuse the build (§5) — the session is only +ever reused under the selectors it was created with. + +## Implementation sketch + +New `src/deb/session.rs` — session identity, manifest +(serde) read/write, discovery, locking, slug generation. Pure logic, +fully unit-testable. + +- `src/deb/ephemeral.rs` — `EphemeralContextGuard` learns a `resume` + path: when a session is adopted, skip download/extract and reuse the + existing chroot (still verifying `ready` and remounting `/proc` if a + previous run unmounted it); on teardown, hand the tree to the + session instead of `rm -rf`-ing it. Chroot dir moves from + `create_temp_dir()` to the session layout — the local driver's temp + naming stays for non-session uses. +- `src/deb/local.rs` — journal writes around each phase; the + skip-or-run decisions per the table in §2; `debian/files` cleanup + before the binary phase on resume. +- `src/context/unshare.rs` — upperdir reuse + whiteout propagation for + host deletions (§3); `ensure_available()` gains a + "stage into existing session tree" mode. +- `src/interrupt.rs` + `src/deb/ephemeral.rs` — the interrupt hook + keeps the session (§4). +- `src/prune.rs` — session discovery, retention, reporting. +- `src/main.rs` — `--keep`, `--resume []` argument wiring, + the `pkh deb list` subcommand and selector-conflict checks; + `README.md` roadmap + update (this retires part of the "deb: asynchronous build, + detachable and monitorable" itch: a persistent session is also the + natural attach point for detached builds later). + +Staged rollout, each step independently shippable: + +1. Sessions with manifest + `pkh deb list` + adoption of failed-build + chroots via `--resume` (environment resume only — biggest win, no + overlay changes). +2. Build resume: upperdir reuse + whiteout propagation. +3. Interrupt keeps session; `--keep` after success; prune integration. + +## Testing + +- Unit: manifest round-trip, slug/identity matching, whiteout plan + computation (host snapshot vs current host listing), skip-decision + table given journals with various stamps. +- Integration (small fixture package, `#[cfg(test)]` style already + used by `deb` tests): fail `debian/rules build` on file B after + compiling file A; resume and assert A is *not* recompiled (make log + via the tee log); modify A; resume and assert only A recompiles; + modify `debian/control`; assert build-deps rerun and objects + survive; `pkh prune` GC behavior; a plain `pkh deb` ignoring an + existing session (fresh rebuild) while `--resume` adopts it. +- Unit/CLI: `pkh deb list` rendering against fixture manifests (scoping by + `host_tree`, cross marker, ordering); `--resume ` exact and + prefix resolution, ambiguity error; selector-conflict refusals; + `--keep` keeping the session on success while a plain build removes + it. +- Chroot reuse tests need the unshare/mount machinery — follow the + existing pattern of `#[ignore]` tests for deliberate ad-hoc runs + where root/unshare is required. + +## Decisions + +- **Resume is opt-in (`--resume`), never the silent default.** A + plain `pkh deb` always rechecks everything from scratch, so the + no-flag behavior can never build anything stale; recording stays on + by default so the follow-up `--resume` after a failure has a + session to adopt. +- **Keeping a session after success is opt-in (`--keep`).** Today's + disk profile — success cleans up — stays the default; the + edit–rebuild iteration loop is served by `pkh deb --resume --keep`. + Failures and interrupts keep their session unconditionally, as + today. +- **Sessions are addressed by id (build-start timestamp), listed via + the `pkh deb list` subcommand.** Listing is a subcommand, not a + mode-switching flag: `pkh deb` with flags always starts a build. + One live session per identity keeps the cache + bounded (a per-attempt history would be one full chroot per + attempt); the list therefore shows one row per series/arch/cross + combination for the current tree, each carrying its latest attempt's + id. Explicit selectors that disagree with the adopted session are + an error, never a silent environment switch. +- **Ctrl+C keeps the session**, same as a failed build: an interrupt + on a long build is a pause, and there is nothing inherent in "stop + now" that means "discard half an hour of environment setup". The + hard-discard behavior stays reachable through `pkh prune`. +- **`apt-get update` always reruns on resume** (no stamp, no TTL) — + see §2. +- **Sessions live under `/var/tmp/pkh/sessions`** (overridable via + `PKH_SESSIONS_DIR` then config), and are cleaned by `pkh prune` + (retention window by default, `--all` for everything). Cleanup by + systemd-tmpfiles from time to time is accepted; sessions are a + cache. + +--- + +# Implementation Notes (as built) + +Implemented across `src/deb/session.rs` (new), `src/deb/{mod,local, +ephemeral}.rs`, `src/context/{api,unshare}.rs`, `src/prune.rs` and +`src/main.rs`. Deviations from the design above, all deliberate: + +- **Phase stamps gate less than specced.** The journal records every + phase (with the patch-tree and control stamps), but on resume only + the expensive non-idempotent work is actually skipped: the chroot + bootstrap (extraction, device nodes, `/proc`) and, via the reused + overlay upperdir, the compiled objects. `apt-get update`, the + essentials install, the quilt push and the build-dep resolution + always rerun — each is idempotent and seconds-cheap, and rerunning + them removes a whole class of stale-state bugs. The stamps still + drive the quilt `pop -a` decision (patch tree changed since the + recorded attempt) and the version-change handling. +- **The advisory lock lives outside the session root** + (`/.lock`): a teardown deletes the root while + still holding the lock, and a lock file inside it would be removed + under the holder — a concurrent opener would then create and lock a + fresh inode, and mutual exclusion silently dies (observed with two + concurrent same-identity e2e builds before the move). +- **`pkh prune` session GC is confined to the production `prune()`.** + `prune_in()` (the testable core) deliberately does not scan the real + sessions root: the test suite runs prune tests and e2e builds + concurrently, and the prune tests would delete live sessions + mid-build (observed). `prune_in_roots()` takes the sessions root as + an explicit opt-in parameter. +- **Fallbacks, in order:** no local base context → the historical + anonymous temp chroot (no session); session locked by a concurrent + build → same; overlay mount unsupported/failed on resume → fresh + copy staging (environment resume only, artifacts discarded); no + host-tree snapshot → upper wiped and re-snapshotted.