From f1e8244e7310f090adcf1a2f4d7d573bd8f12905 Mon Sep 17 00:00:00 2001 From: Valentin Haudiquet Date: Fri, 2 Oct 2026 21:31:27 +0200 Subject: [PATCH] docs: drop the plans directory MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The specs describe shipped features — incremental builds, the live view, overlayfs staging, pkh new, pkh test, pkh lint phase 1 — and the satellite-tools plan is implemented in full (no shell-outs to dpkg-architecture, dpkg-checkbuilddeps or dpkg-parsechangelog remain). The two unshipped proposals (the lint porting track and the data/configuration architecture) stay in whatever notes the project keeps outside the tree; the README remains the user-facing documentation. --- plans/deb-incremental-builds.md | 455 -------------------------------- src/lint/mod.rs | 2 +- src/new/mod.rs | 3 +- 3 files changed, 2 insertions(+), 458 deletions(-) delete mode 100644 plans/deb-incremental-builds.md diff --git a/plans/deb-incremental-builds.md b/plans/deb-incremental-builds.md deleted file mode 100644 index 1544750..0000000 --- a/plans/deb-incremental-builds.md +++ /dev/null @@ -1,455 +0,0 @@ -# `pkh deb` Incremental Builds — Spec - -## Problem - -`pkh deb` has no incremental story. Every invocation — including a rerun -seconds after a failed build — pays the full pipeline again: - -1. mmdebstrap tarball **extraction** into a fresh `/tmp/pkh-` chroot - (the tarball itself is cached, but extraction of a kernel-sized chroot - is minutes), -2. device nodes + `/proc` bind mount, -3. `apt-get update`, -4. essentials install, -5. `quilt push -a`, -6. native build-dependency resolution + install, -7. `debian/rules build` **from scratch** in a freshly staged copy of the - tree, -8. `fakeroot debian/rules binary`. - -The failures this hurts most are the late ones. A kernel package that -dies in `debian/rules build` after 25 minutes is restarted from zero; -fixing one compile error and rerunning recompiles everything, and a -cycle of "tweak, rebuild, fail again" costs a full pipeline each time. - -The raw material for resumption already exists but is unreachable: - -- **Failed chroots are kept.** `EphemeralContextGuard::drop` - (`src/deb/ephemeral.rs`) deliberately keeps the chroot when the build - did not succeed — but the next run creates a new `pkh-` directory - and cannot find or reuse the old one. The kept chroot is a leak until - `pkh prune`, not a cache. -- **The staged tree is an overlayfs mount** (`src/context/unshare.rs`): - host tree = lowerdir (read live), build writes = upperdir inside the - chroot. If the upperdir survives between runs, object files survive - too, and host-side edits propagate automatically — exactly the - semantics incremental builds need. -- **Ctrl+C deletes everything** (`sigint_cleanup_chroot`), which is the - worst behavior for a long build interrupted on purpose. - -## Goal - -`pkh deb --resume` after a failed build resumes from the point of -failure (plain `pkh deb` always rebuilds from scratch — see §5): - -- same environment, source untouched → picks up at - `debian/rules build`, recompiling nothing (make sees a warm tree); -- source modified since the failure → still reuses chroot, apt state - and installed build-deps; `debian/rules build` recompiles only what - the modification affects; -- `debian/control` or patch series changed → environment-level state - (build-deps, quilt) is redone; the chroot and object files still - carry over when possible. - -Non-goal: distributing or sharing sessions between machines; resuming -`pkh build` (the native source pipeline — it is seconds-fast already); -resuming inside ssh/schroot contexts (sessions are a local-unshare -feature; other drivers keep today's behavior). - -## Design - -### 1. Build sessions: named, journaled, discoverable - -A **session** is one (package, series, arch, cross) build environment -plus its journal. Replace the anonymous `pkh-` chroot -directory with a session root: - -``` -/var/tmp/pkh/sessions// - session.json # manifest + phase journal (see below) - chroot/ # the chroot tree (was /tmp/pkh-) - tree-upper/ # overlayfs upperdir for the staged package tree - pkh-overlay/ # overlay workdirs (as today) -``` - -`` is derived from the session identity (package, series, arch, -cross) plus a short content hash, so a rerun can find a previous -session by identity without guessing timestamps. One live session per -identity — the newest attempt *replaces* the previous one (same -chroot environment, fresher state; keeping per-attempt history would -mean one full chroot per attempt, which is a cache nobody wants). A -locked session (concurrent build) is never replaced (§6). What the -session list shows for a tree is therefore one entry per identity -(series/arch/cross combination), each carrying its latest attempt's -id. - -Moving off `/tmp` is deliberate: sessions must survive a reboot to be -worth keeping, and `/tmp` is often tmpfs (a kernel chroot on tmpfs is -RAM). `/var/tmp` is the conventional persistent-scratch location; the -base directory is overridable (`PKH_SESSIONS_DIR` env, then config). -Sessions are a cache, and `pkh prune` is their primary GC — if -systemd-tmpfiles sweeps `/var/tmp` on some setup, that is an -acceptable, if blunt, secondary cleanup. - -`session.json` records: - -```jsonc -{ - "id": "20260926T143505", // build-start timestamp, UTC; changes on every attempt - "host_tree": "/home/me/linux", // tree the session was built from (list scoping) - "identity": { - "package": "linux", "series": "stonking", - "arch": "arm64", "cross": true - }, - "created": "...", "last_used": "...", - "chroot": { - "tarball": "stonking-arm64-buildd.tar.xz", - "tarball_sha256": "...", // reuse only if the cached tarball is the same - "ready": true // device nodes + /proc done - }, - "phases": { - "apt_update": { "at": "..." }, // journal-only: apt update always reruns on resume - "essentials": { "at": "..." }, - "patches": { "stamp": "", "applied": true }, - "build_deps": { "stamp": "", "at": "..." }, - "build": { "at": "...", "result": "failed" } - }, - "tree": { - "version": "7.3.0-5.6~local2", // changelog version of the last attempt - "host_snapshot": "", // for deletion propagation - } -} -``` - -The journal is written **after** each phase completes (and after a -phase fails, recording the failure), so a crash or Ctrl+C at any point -leaves a truthful journal. Discovery is: scan -`$PKH_SESSIONS_DIR/*/session.json`, match identity. - -### 2. Two layers of resume - -The journal separates **environment resume** (expensive, insensitive to -source edits) from **build resume** (sensitive to every edit): - -| Phase | Reused when | Redone when | -|---|---|---| -| Chroot extract, nodes, /proc | chroot present, `ready`, same tarball hash | never for the same identity (otherwise the session is unusable) | -| `apt-get update` | — (always rerun on resume) | — | -| Essentials | done (verify cheaply via `dpkg-query` at most) | chroot redone | -| quilt push | `debian/patches` hash unchanged | hash changed → `quilt pop -a` + re-push | -| Build-deps | control/arch/cross/ppa/inject/pocket stamp unchanged | stamp changed → the resolver runs again (it is idempotent; apt installs only what is missing) | -| `debian/rules build` | always *attempted* in the warm tree | — | -| `fakeroot debian/rules binary` | always rerun | — | - -Key consequences: - -- **`apt-get update` is always rerun on resume.** It is the cheapest - phase, and stale package lists fail confusingly ("candidate version - not found") on fast-moving suites — there is nothing worth caching - here. The journal entry exists for observability only. - -- **A changelog version bump** (`pkh chlog` between attempts) invalidates - build resume but not environment resume: the chroot, apt lists and - build-deps carry over. The staged tree is re-populated for the new - version (see §3); the session's `tree.version` tracks which attempt - the upperdir belongs to, and a version change discards the upperdir - (object files reference paths/flags of the old version). -- **The build phase is never skipped**, only made incremental — make - decides. pkh's job is to hand `debian/rules build` a tree whose - previously produced objects are still there and whose sources match - the host. -- Before rerunning the binary phase on a resume, clear - `debian/files` (and `debian/*.log`, `debian/substvars`) from the - staged tree so artifact collection (`collect_binary_artifacts`) only - sees the new attempt's outputs — today's "no globbing stale files" - contract is preserved. - -### 3. Tree propagation on resume - -Today each build stages a fresh copy/overlay of the host tree. On -resume the staged tree already exists inside the session; the problem -is syncing it with a possibly-modified host tree. - -- **Overlay contexts (the default when overlayfs is available):** remount - the overlay with the *same* upperdir. Host modifications are visible - immediately (lowerdir is read live); objects written in the upperdir - during the previous attempt are still on top. Host-side *deletions* - are the one gap — overlayfs only hides a lower file via a whiteout in - the upper, and nothing creates whiteouts for files the user deleted - on the host. Fix: the session stores a snapshot of the host tree's - file list at first staging; on resume, files present in the upper - (or in the snapshot) but gone from the host get an explicit whiteout - (`mknod c 0 0` inside the upper). This is a bounded, - manifest-driven operation, not a tree walk of the whole chroot. -- **Copy fallback (no overlayfs):** `rsync -a --delete` of the host - tree over the staged tree, *excluding* the build artifacts make - produced (this needs per-package exclusion knowledge and is best - effort). Deletions propagate naturally here. Sessions on the copy - path get build resume only for trees where the exclusion set is - sane; the honest fallback is: copy path → environment resume only. - -The overlay logic stays inside the unshare driver, matching the -principle established in `plans/overlayfs-integration.md`: callers -stage a tree through `ensure_available()` and never learn how. - -### 4. Interrupt behavior - -Ctrl+C currently removes the chroot through the interrupt hook. Change: -**an interrupted build keeps the session**, journaling the interrupted -phase as incomplete — an intentional Ctrl+C on a 30-minute build is a -"pause", and deleting the environment on pause defeats the feature. - -The cleanup hook (`sigint_cleanup_chroot`) becomes session-aware: it -still unmounts the overlays and `/proc` (leaving mounted state in a -session is what makes stale sessions dangerous), but keeps the tree -and leaves a `interrupted: true` marker in the journal so the next run -knows the session needs no special recovery (phases after the last -completed one simply rerun). - -Users who want the old hard-discard behavior on interrupt get it via -`pkh prune`. The failure path is unchanged in spirit: failed builds -already keep their chroot; they now keep a *usable* one. - -### 5. CLI surface and policy - -**Recording is always on; reuse is opt-in.** The two are deliberately -separated, because the first attempt cannot know it will fail: a -plain `pkh deb` writes and maintains the session (journal, kept -chroot on failure or interrupt — today's keep-on-failure, made -usable) but never reads one. Only `pkh deb --resume` adopts an -existing session. The default run therefore always rechecks -everything from scratch — no stale-build risk unless the user asks -for one. - -- `pkh deb` — today's behavior, plus session recording: any existing - session of the same identity is ignored and replaced by this run's - outcome. On failure or interrupt the session is kept (as today); - on success it is **removed**, unless `--keep` is given. -- `pkh deb --keep` — also keep the session after a successful build, - so a later `--resume` can iterate on it (the edit–rebuild loop). - Composes with `--resume`: `--resume --keep` chains sessions across - iterations; a resumed build without `--keep` consumes the session - on success — the natural end of the loop. -- `pkh deb --resume []` — adopt a session and continue it, - announcing what is being skipped ("Resuming session - 20260926T143505 for linux/stonking-arm64: chroot, package lists - and build-deps reused"). - - Without an id: the newest session recorded from this tree - (manifest `host_tree` matches the tree being built), whatever its - last outcome — failure, interrupt, or a `--keep` success. - - With an id (a build-start timestamp as shown by `pkh deb list`): - that exact session. A unique unambiguous prefix is accepted; an - ambiguous one is an error listing the candidates. - - Selector conflicts are errors, not surprises: if `-a`/`-s`/ - `--cross`/`--ppa`/`--inject` are given explicitly and disagree - with the adopted session, refuse and point at `pkh deb list`. A - session for another series/arch is a *different* build - environment; silently building it under the requested selectors - would be exactly the stale-build risk this design avoids. - - With no session to adopt (none recorded, pruned, or the tree - moved): say so and build from scratch — never silently. -- `pkh deb list` — a subcommand, not a flag: `pkh deb` with flags - always means "start a build", and listing is a different action. - It prints the sessions recorded from the current tree, one row per - identity (§1), newest attempt first: - - ``` - $ pkh deb list - Sessions for /home/me/linux: - ID PACKAGE VERSION TARGET LAST RUN AGE - 20260926T143505 linux 7.3.0-5.6~local2 stonking/arm64 failed 2h - 20260925T090012 linux 7.3.0-5.5~local1 stonking/arm64* success 1d - 20260924T171100 linux 7.3.0-5.6~local2 stonking/riscv64* failed 2d - - * cross build. Resume with: pkh deb --resume [] - ``` - - `pkh deb list` never touches build machinery: it reads the session - manifests under the sessions directory and renders. As an optional - subcommand of `deb` (clap allows subcommands alongside args), the - build flags stay on `deb` itself and `list` takes none of them. -- `pkh prune` learns sessions: by default removes - sessions untouched for longer than a retention window (7 days) and - any session whose journal is missing/corrupt; `--all` removes every - session. Prune's report distinguishes "session (resumable build)" - from the other residual categories, since sessions are large and - users must be able to see and reclaim them. - -### 6. Correctness guards - -A session is only reused when **all** of these hold; otherwise it is -discarded (replaced) with a logged reason: - -- identity matches (package, series, arch, cross); -- the cached chroot tarball hash matches the one the chroot was built - from; -- `chroot/` passes an integrity probe (marker file, `chroot/bin/sh` - exists and executes); -- pkh version compatibility: `session.json` records the pkh version - and manifest schema version; a mismatched schema discards the - session (the chroot is generic enough that only schema-relevant - changes matter, but being conservative is cheap). -- no other live session for the same identity (lock file in the - session root, in the spirit of the existing tarball lockfile). - -Concurrent builds of the same identity take the lock and fall back to -a fresh ephemeral session (current behavior) rather than waiting — -correctness over caching. - -One more guard on the resume path, at the CLI layer: explicit -selectors (`-a`, `-s`, `--cross`, `--ppa`, `--inject`) that disagree -with the adopted session refuse the build (§5) — the session is only -ever reused under the selectors it was created with. - -## Implementation sketch - -New `src/deb/session.rs` — session identity, manifest -(serde) read/write, discovery, locking, slug generation. Pure logic, -fully unit-testable. - -- `src/deb/ephemeral.rs` — `EphemeralContextGuard` learns a `resume` - path: when a session is adopted, skip download/extract and reuse the - existing chroot (still verifying `ready` and remounting `/proc` if a - previous run unmounted it); on teardown, hand the tree to the - session instead of `rm -rf`-ing it. Chroot dir moves from - `create_temp_dir()` to the session layout — the local driver's temp - naming stays for non-session uses. -- `src/deb/local.rs` — journal writes around each phase; the - skip-or-run decisions per the table in §2; `debian/files` cleanup - before the binary phase on resume. -- `src/context/unshare.rs` — upperdir reuse + whiteout propagation for - host deletions (§3); `ensure_available()` gains a - "stage into existing session tree" mode. -- `src/interrupt.rs` + `src/deb/ephemeral.rs` — the interrupt hook - keeps the session (§4). -- `src/prune.rs` — session discovery, retention, reporting. -- `src/main.rs` — `--keep`, `--resume []` argument wiring, - the `pkh deb list` subcommand and selector-conflict checks; - `README.md` roadmap - update (this retires part of the "deb: asynchronous build, - detachable and monitorable" itch: a persistent session is also the - natural attach point for detached builds later). - -Staged rollout, each step independently shippable: - -1. Sessions with manifest + `pkh deb list` + adoption of failed-build - chroots via `--resume` (environment resume only — biggest win, no - overlay changes). -2. Build resume: upperdir reuse + whiteout propagation. -3. Interrupt keeps session; `--keep` after success; prune integration. - -## Testing - -- Unit: manifest round-trip, slug/identity matching, whiteout plan - computation (host snapshot vs current host listing), skip-decision - table given journals with various stamps. -- Integration (small fixture package, `#[cfg(test)]` style already - used by `deb` tests): fail `debian/rules build` on file B after - compiling file A; resume and assert A is *not* recompiled (make log - via the tee log); modify A; resume and assert only A recompiles; - modify `debian/control`; assert build-deps rerun and objects - survive; `pkh prune` GC behavior; a plain `pkh deb` ignoring an - existing session (fresh rebuild) while `--resume` adopts it. -- Unit/CLI: `pkh deb list` rendering against fixture manifests (scoping by - `host_tree`, cross marker, ordering); `--resume ` exact and - prefix resolution, ambiguity error; selector-conflict refusals; - `--keep` keeping the session on success while a plain build removes - it. -- Chroot reuse tests need the unshare/mount machinery — follow the - existing pattern of `#[ignore]` tests for deliberate ad-hoc runs - where root/unshare is required. - -## Decisions - -- **Resume is opt-in (`--resume`), never the silent default.** A - plain `pkh deb` always rechecks everything from scratch, so the - no-flag behavior can never build anything stale; recording stays on - by default so the follow-up `--resume` after a failure has a - session to adopt. -- **Keeping a session after success is opt-in (`--keep`).** Today's - disk profile — success cleans up — stays the default; the - edit–rebuild iteration loop is served by `pkh deb --resume --keep`. - Failures and interrupts keep their session unconditionally, as - today. -- **Sessions are addressed by id (build-start timestamp), listed via - the `pkh deb list` subcommand.** Listing is a subcommand, not a - mode-switching flag: `pkh deb` with flags always starts a build. - One live session per identity keeps the cache - bounded (a per-attempt history would be one full chroot per - attempt); the list therefore shows one row per series/arch/cross - combination for the current tree, each carrying its latest attempt's - id. Explicit selectors that disagree with the adopted session are - an error, never a silent environment switch. -- **Ctrl+C keeps the session**, same as a failed build: an interrupt - on a long build is a pause, and there is nothing inherent in "stop - now" that means "discard half an hour of environment setup". The - hard-discard behavior stays reachable through `pkh prune`. -- **`apt-get update` always reruns on resume** (no stamp, no TTL) — - see §2. -- **Sessions live under `/var/tmp/pkh/sessions`** (overridable via - `PKH_SESSIONS_DIR` then config), and are cleaned by `pkh prune` - (retention window by default, `--all` for everything). Cleanup by - systemd-tmpfiles from time to time is accepted; sessions are a - cache. - ---- - -# Implementation Notes (as built) - -Implemented across `src/deb/session.rs` (new), `src/deb/{mod,local, -ephemeral}.rs`, `src/context/{api,unshare}.rs`, `src/prune.rs` and -`src/main.rs`. Deviations from the design above, all deliberate: - -- **Phase stamps gate less than specced.** The journal records every - phase (with the patch-tree and control stamps), but on resume only - the expensive non-idempotent work is actually skipped: the chroot - bootstrap (extraction, device nodes, `/proc`) and, via the reused - overlay upperdir, the compiled objects. `apt-get update`, the - essentials install, the quilt push and the build-dep resolution - always rerun — each is idempotent and seconds-cheap, and rerunning - them removes a whole class of stale-state bugs. The stamps still - drive the quilt `pop -a` decision (patch tree changed since the - recorded attempt) and the version-change handling. -- **The advisory lock lives outside the session root** - (`/.lock`): a teardown deletes the root while - still holding the lock, and a lock file inside it would be removed - under the holder — a concurrent opener would then create and lock a - fresh inode, and mutual exclusion silently dies (observed with two - concurrent same-identity e2e builds before the move). -- **`pkh prune` session GC is confined to the production `prune()`.** - `prune_in()` (the testable core) deliberately does not scan the real - sessions root: the test suite runs prune tests and e2e builds - concurrently, and the prune tests would delete live sessions - mid-build (observed). `prune_in_roots()` takes the sessions root as - an explicit opt-in parameter. -- **Fallbacks, in order:** no local base context → the historical - anonymous temp chroot (no session); session locked by a concurrent - build → same; overlay mount unsupported/failed on resume → fresh - copy staging (environment resume only, artifacts discarded); no - host-tree snapshot → upper wiped and re-snapshotted. - -# Addendum: resume correctness (post-merge findings) - -Two real-world failures against a kept kernel session, both fixed: - -- **The kernel's config chain is stamp-blind** — `stamp-prepare-%` - (`debian/rules.d/2-binary-arch.mk`) exports `.config` from the - annotations with no config prerequisite, so a resumed build silently - kept the previous attempt's configuration; the config edit never - reached the .deb even though `debian/rules build` ran. Dropping the - stamp cache unconditionally would be a package-specific decision, so - it is data-driven: the kernels declare `resume_clear: [debian/stamps]` - in `data/quirks.yml`, and resumed builds remove those tree-relative - paths before `debian/rules build`. The inner kbuild keeps its own - state, so the re-run wrappers stay cheap and only config-affected - objects recompile. -- **A SIGKILLed build (OOM) poisons the next one.** The crashed attempt - leaves its overlay mounts and /proc bind mount behind, and overlayfs - creates root-owned `work` state inside the workdir; the next PLAIN - build (recording is always on: it replaces the session of its - identity) failed to clear the leftovers with a permission error. - `clear_dir()` now unmounts everything under each entry at depth - (re-reading /proc/mounts, tolerating mount stacks from consecutive - crashes) and escalates through sudo; if the tree still cannot be - cleared, the plain build reports it and continues WITHOUT a session - instead of failing or layering over a half-cleared tree. diff --git a/src/lint/mod.rs b/src/lint/mod.rs index 75b150b..4e90fe7 100644 --- a/src/lint/mod.rs +++ b/src/lint/mod.rs @@ -1,6 +1,6 @@ //! `pkh lint`: lint a Debian source tree, with lintian feature parity. //! -//! The strategy is *wrap first, port second* (see `plans/pkh-lint.md`): the +//! The strategy is *wrap first, port second*: the //! [wrapper](wrapper) runs the installed lintian over an ephemeral source //! package for day-one parity with every lintian check, while the native //! engine (check registry, collectors, emitter) hosts pkh-specific workflow diff --git a/src/new/mod.rs b/src/new/mod.rs index 1cd29a0..40677ac 100644 --- a/src/new/mod.rs +++ b/src/new/mod.rs @@ -1,5 +1,4 @@ -//! `pkh new`: interactive-first package scaffolding (see -//! `plans/pkh-new.md`). +//! `pkh new`: interactive-first package scaffolding. //! //! This module orchestrates a scaffold run: target directory checks, project //! detection, in-memory rendering of every file (all-or-nothing write), the