pkh/plans/deb-incremental-builds.md

pkh deb Incremental Builds — Spec

Problem

pkh deb has no incremental story. Every invocation — including a rerun seconds after a failed build — pays the full pipeline again:

1. mmdebstrap tarball extraction into a fresh /tmp/pkh-<ts> chroot (the tarball itself is cached, but extraction of a kernel-sized chroot is minutes),
2. device nodes + /proc bind mount,
3. apt-get update,
4. essentials install,
5. quilt push -a,
6. native build-dependency resolution + install,
7. debian/rules build from scratch in a freshly staged copy of the tree,
8. fakeroot debian/rules binary.

The failures this hurts most are the late ones. A kernel package that dies in debian/rules build after 25 minutes is restarted from zero; fixing one compile error and rerunning recompiles everything, and a cycle of "tweak, rebuild, fail again" costs a full pipeline each time.

The raw material for resumption already exists but is unreachable:

• Failed chroots are kept. EphemeralContextGuard::drop (src/deb/ephemeral.rs) deliberately keeps the chroot when the build did not succeed — but the next run creates a new pkh-<ts> directory and cannot find or reuse the old one. The kept chroot is a leak until pkh prune, not a cache.
• The staged tree is an overlayfs mount (src/context/unshare.rs): host tree = lowerdir (read live), build writes = upperdir inside the chroot. If the upperdir survives between runs, object files survive too, and host-side edits propagate automatically — exactly the semantics incremental builds need.
• Ctrl+C deletes everything (sigint_cleanup_chroot), which is the worst behavior for a long build interrupted on purpose.

Goal

pkh deb --resume after a failed build resumes from the point of failure (plain pkh deb always rebuilds from scratch — see §5):

• same environment, source untouched → picks up at debian/rules build, recompiling nothing (make sees a warm tree);
• source modified since the failure → still reuses chroot, apt state and installed build-deps; debian/rules build recompiles only what the modification affects;
• debian/control or patch series changed → environment-level state (build-deps, quilt) is redone; the chroot and object files still carry over when possible.

Non-goal: distributing or sharing sessions between machines; resuming pkh build (the native source pipeline — it is seconds-fast already); resuming inside ssh/schroot contexts (sessions are a local-unshare feature; other drivers keep today's behavior).

Design

1. Build sessions: named, journaled, discoverable

A session is one (package, series, arch, cross) build environment plus its journal. Replace the anonymous pkh-<timestamp> chroot directory with a session root:

/var/tmp/pkh/sessions/<slug>/
  session.json        # manifest + phase journal (see below)
  chroot/             # the chroot tree (was /tmp/pkh-<ts>)
  tree-upper/         # overlayfs upperdir for the staged package tree
  pkh-overlay/        # overlay workdirs (as today)

<slug> is derived from the session identity (package, series, arch, cross) plus a short content hash, so a rerun can find a previous session by identity without guessing timestamps. One live session per identity — the newest attempt replaces the previous one (same chroot environment, fresher state; keeping per-attempt history would mean one full chroot per attempt, which is a cache nobody wants). A locked session (concurrent build) is never replaced (§6). What the session list shows for a tree is therefore one entry per identity (series/arch/cross combination), each carrying its latest attempt's id.

Moving off /tmp is deliberate: sessions must survive a reboot to be worth keeping, and /tmp is often tmpfs (a kernel chroot on tmpfs is RAM). /var/tmp is the conventional persistent-scratch location; the base directory is overridable (PKH_SESSIONS_DIR env, then config). Sessions are a cache, and pkh prune is their primary GC — if systemd-tmpfiles sweeps /var/tmp on some setup, that is an acceptable, if blunt, secondary cleanup.

session.json records:

{
  "id": "20260926T143505",   // build-start timestamp, UTC; changes on every attempt
  "host_tree": "/home/me/linux",  // tree the session was built from (list scoping)
  "identity": {
    "package": "linux", "series": "stonking",
    "arch": "arm64", "cross": true
  },
  "created": "...", "last_used": "...",
  "chroot": {
    "tarball": "stonking-arm64-buildd.tar.xz",
    "tarball_sha256": "...",      // reuse only if the cached tarball is the same
    "ready": true                  // device nodes + /proc done
  },
  "phases": {
    "apt_update": { "at": "..." },  // journal-only: apt update always reruns on resume
    "essentials": { "at": "..." },
    "patches":    { "stamp": "<hash of debian/patches>", "applied": true },
    "build_deps": { "stamp": "<hash of control+arch+cross+ppa+inject+pocket>", "at": "..." },
    "build":      { "at": "...", "result": "failed" }
  },
  "tree": {
    "version": "7.3.0-5.6~local2",  // changelog version of the last attempt
    "host_snapshot": "<path or inline list>",  // for deletion propagation
  }
}

The journal is written after each phase completes (and after a phase fails, recording the failure), so a crash or Ctrl+C at any point leaves a truthful journal. Discovery is: scan $PKH_SESSIONS_DIR/*/session.json, match identity.

2. Two layers of resume

The journal separates environment resume (expensive, insensitive to source edits) from build resume (sensitive to every edit):

Phase	Reused when	Redone when
Chroot extract, nodes, /proc	chroot present, ready, same tarball hash	never for the same identity (otherwise the session is unusable)
apt-get update	— (always rerun on resume)	—
Essentials	done (verify cheaply via dpkg-query at most)	chroot redone
quilt push	debian/patches hash unchanged	hash changed → quilt pop -a + re-push
Build-deps	control/arch/cross/ppa/inject/pocket stamp unchanged	stamp changed → the resolver runs again (it is idempotent; apt installs only what is missing)
debian/rules build	always attempted in the warm tree	—
fakeroot debian/rules binary	always rerun	—

Key consequences:

• apt-get update is always rerun on resume. It is the cheapest phase, and stale package lists fail confusingly ("candidate version not found") on fast-moving suites — there is nothing worth caching here. The journal entry exists for observability only.

• A changelog version bump (pkh chlog between attempts) invalidates build resume but not environment resume: the chroot, apt lists and build-deps carry over. The staged tree is re-populated for the new version (see §3); the session's tree.version tracks which attempt the upperdir belongs to, and a version change discards the upperdir (object files reference paths/flags of the old version).

• The build phase is never skipped, only made incremental — make decides. pkh's job is to hand debian/rules build a tree whose previously produced objects are still there and whose sources match the host.

• Before rerunning the binary phase on a resume, clear debian/files (and debian/*.log, debian/substvars) from the staged tree so artifact collection (collect_binary_artifacts) only sees the new attempt's outputs — today's "no globbing stale files" contract is preserved.

3. Tree propagation on resume

Today each build stages a fresh copy/overlay of the host tree. On resume the staged tree already exists inside the session; the problem is syncing it with a possibly-modified host tree.

• Overlay contexts (the default when overlayfs is available): remount the overlay with the same upperdir. Host modifications are visible immediately (lowerdir is read live); objects written in the upperdir during the previous attempt are still on top. Host-side deletions are the one gap — overlayfs only hides a lower file via a whiteout in the upper, and nothing creates whiteouts for files the user deleted on the host. Fix: the session stores a snapshot of the host tree's file list at first staging; on resume, files present in the upper (or in the snapshot) but gone from the host get an explicit whiteout (mknod <path> c 0 0 inside the upper). This is a bounded, manifest-driven operation, not a tree walk of the whole chroot.
• Copy fallback (no overlayfs): rsync -a --delete of the host tree over the staged tree, excluding the build artifacts make produced (this needs per-package exclusion knowledge and is best effort). Deletions propagate naturally here. Sessions on the copy path get build resume only for trees where the exclusion set is sane; the honest fallback is: copy path → environment resume only.

The overlay logic stays inside the unshare driver, matching the principle established in plans/overlayfs-integration.md: callers stage a tree through ensure_available() and never learn how.

4. Interrupt behavior

Ctrl+C currently removes the chroot through the interrupt hook. Change: an interrupted build keeps the session, journaling the interrupted phase as incomplete — an intentional Ctrl+C on a 30-minute build is a "pause", and deleting the environment on pause defeats the feature.

The cleanup hook (sigint_cleanup_chroot) becomes session-aware: it still unmounts the overlays and /proc (leaving mounted state in a session is what makes stale sessions dangerous), but keeps the tree and leaves a interrupted: true marker in the journal so the next run knows the session needs no special recovery (phases after the last completed one simply rerun).

Users who want the old hard-discard behavior on interrupt get it via pkh prune. The failure path is unchanged in spirit: failed builds already keep their chroot; they now keep a usable one.

5. CLI surface and policy

Recording is always on; reuse is opt-in. The two are deliberately separated, because the first attempt cannot know it will fail: a plain pkh deb writes and maintains the session (journal, kept chroot on failure or interrupt — today's keep-on-failure, made usable) but never reads one. Only pkh deb --resume adopts an existing session. The default run therefore always rechecks everything from scratch — no stale-build risk unless the user asks for one.

• pkh deb — today's behavior, plus session recording: any existing session of the same identity is ignored and replaced by this run's outcome. On failure or interrupt the session is kept (as today); on success it is removed, unless --keep is given.

• pkh deb --keep — also keep the session after a successful build, so a later --resume can iterate on it (the edit–rebuild loop). Composes with --resume: --resume --keep chains sessions across iterations; a resumed build without --keep consumes the session on success — the natural end of the loop.

• pkh deb --resume [<id>] — adopt a session and continue it, announcing what is being skipped ("Resuming session 20260926T143505 for linux/stonking-arm64: chroot, package lists and build-deps reused").

  • Without an id: the newest session recorded from this tree (manifest host_tree matches the tree being built), whatever its last outcome — failure, interrupt, or a --keep success.
  • With an id (a build-start timestamp as shown by pkh deb list): that exact session. A unique unambiguous prefix is accepted; an ambiguous one is an error listing the candidates.
  • Selector conflicts are errors, not surprises: if -a/-s/ --cross/--ppa/--inject are given explicitly and disagree with the adopted session, refuse and point at pkh deb list. A session for another series/arch is a different build environment; silently building it under the requested selectors would be exactly the stale-build risk this design avoids.
  • With no session to adopt (none recorded, pruned, or the tree moved): say so and build from scratch — never silently.

• pkh deb list — a subcommand, not a flag: pkh deb with flags always means "start a build", and listing is a different action. It prints the sessions recorded from the current tree, one row per identity (§1), newest attempt first:

  $ pkh deb list
  Sessions for /home/me/linux:
  ID               PACKAGE  VERSION           TARGET             LAST RUN  AGE
  20260926T143505  linux    7.3.0-5.6~local2  stonking/arm64     failed    2h
  20260925T090012  linux    7.3.0-5.5~local1  stonking/arm64*    success   1d
  20260924T171100  linux    7.3.0-5.6~local2  stonking/riscv64*  failed    2d
  
  * cross build. Resume with: pkh deb --resume [<id>]
  

  pkh deb list never touches build machinery: it reads the session manifests under the sessions directory and renders. As an optional subcommand of deb (clap allows subcommands alongside args), the build flags stay on deb itself and list takes none of them.

• pkh prune learns sessions: by default removes sessions untouched for longer than a retention window (7 days) and any session whose journal is missing/corrupt; --all removes every session. Prune's report distinguishes "session (resumable build)" from the other residual categories, since sessions are large and users must be able to see and reclaim them.

6. Correctness guards

A session is only reused when all of these hold; otherwise it is discarded (replaced) with a logged reason:

• identity matches (package, series, arch, cross);
• the cached chroot tarball hash matches the one the chroot was built from;
• chroot/ passes an integrity probe (marker file, chroot/bin/sh exists and executes);
• pkh version compatibility: session.json records the pkh version and manifest schema version; a mismatched schema discards the session (the chroot is generic enough that only schema-relevant changes matter, but being conservative is cheap).
• no other live session for the same identity (lock file in the session root, in the spirit of the existing tarball lockfile).

Concurrent builds of the same identity take the lock and fall back to a fresh ephemeral session (current behavior) rather than waiting — correctness over caching.

One more guard on the resume path, at the CLI layer: explicit selectors (-a, -s, --cross, --ppa, --inject) that disagree with the adopted session refuse the build (§5) — the session is only ever reused under the selectors it was created with.

Implementation sketch

New src/deb/session.rs — session identity, manifest (serde) read/write, discovery, locking, slug generation. Pure logic, fully unit-testable.

• src/deb/ephemeral.rs — EphemeralContextGuard learns a resume path: when a session is adopted, skip download/extract and reuse the existing chroot (still verifying ready and remounting /proc if a previous run unmounted it); on teardown, hand the tree to the session instead of rm -rf-ing it. Chroot dir moves from create_temp_dir() to the session layout — the local driver's temp naming stays for non-session uses.
• src/deb/local.rs — journal writes around each phase; the skip-or-run decisions per the table in §2; debian/files cleanup before the binary phase on resume.
• src/context/unshare.rs — upperdir reuse + whiteout propagation for host deletions (§3); ensure_available() gains a "stage into existing session tree" mode.
• src/interrupt.rs + src/deb/ephemeral.rs — the interrupt hook keeps the session (§4).
• src/prune.rs — session discovery, retention, reporting.
• src/main.rs — --keep, --resume [<id>] argument wiring, the pkh deb list subcommand and selector-conflict checks; README.md roadmap update (this retires part of the "deb: asynchronous build, detachable and monitorable" itch: a persistent session is also the natural attach point for detached builds later).

Staged rollout, each step independently shippable:

1. Sessions with manifest + pkh deb list + adoption of failed-build chroots via --resume (environment resume only — biggest win, no overlay changes).
2. Build resume: upperdir reuse + whiteout propagation.
3. Interrupt keeps session; --keep after success; prune integration.

Testing

• Unit: manifest round-trip, slug/identity matching, whiteout plan computation (host snapshot vs current host listing), skip-decision table given journals with various stamps.
• Integration (small fixture package, #[cfg(test)] style already used by deb tests): fail debian/rules build on file B after compiling file A; resume and assert A is not recompiled (make log via the tee log); modify A; resume and assert only A recompiles; modify debian/control; assert build-deps rerun and objects survive; pkh prune GC behavior; a plain pkh deb ignoring an existing session (fresh rebuild) while --resume adopts it.
• Unit/CLI: pkh deb list rendering against fixture manifests (scoping by host_tree, cross marker, ordering); --resume <id> exact and prefix resolution, ambiguity error; selector-conflict refusals; --keep keeping the session on success while a plain build removes it.
• Chroot reuse tests need the unshare/mount machinery — follow the existing pattern of #[ignore] tests for deliberate ad-hoc runs where root/unshare is required.

Decisions

• Resume is opt-in (--resume), never the silent default. A plain pkh deb always rechecks everything from scratch, so the no-flag behavior can never build anything stale; recording stays on by default so the follow-up --resume after a failure has a session to adopt.
• Keeping a session after success is opt-in (--keep). Today's disk profile — success cleans up — stays the default; the edit–rebuild iteration loop is served by pkh deb --resume --keep. Failures and interrupts keep their session unconditionally, as today.
• Sessions are addressed by id (build-start timestamp), listed via the pkh deb list subcommand. Listing is a subcommand, not a mode-switching flag: pkh deb with flags always starts a build. One live session per identity keeps the cache bounded (a per-attempt history would be one full chroot per attempt); the list therefore shows one row per series/arch/cross combination for the current tree, each carrying its latest attempt's id. Explicit selectors that disagree with the adopted session are an error, never a silent environment switch.
• Ctrl+C keeps the session, same as a failed build: an interrupt on a long build is a pause, and there is nothing inherent in "stop now" that means "discard half an hour of environment setup". The hard-discard behavior stays reachable through pkh prune.
• apt-get update always reruns on resume (no stamp, no TTL) — see §2.
• Sessions live under /var/tmp/pkh/sessions (overridable via PKH_SESSIONS_DIR then config), and are cleaned by pkh prune (retention window by default, --all for everything). Cleanup by systemd-tmpfiles from time to time is accepted; sessions are a cache.

---

Implementation Notes (as built)

Implemented across src/deb/session.rs (new), src/deb/{mod,local, ephemeral}.rs, src/context/{api,unshare}.rs, src/prune.rs and src/main.rs. Deviations from the design above, all deliberate:

• Phase stamps gate less than specced. The journal records every phase (with the patch-tree and control stamps), but on resume only the expensive non-idempotent work is actually skipped: the chroot bootstrap (extraction, device nodes, /proc) and, via the reused overlay upperdir, the compiled objects. apt-get update, the essentials install, the quilt push and the build-dep resolution always rerun — each is idempotent and seconds-cheap, and rerunning them removes a whole class of stale-state bugs. The stamps still drive the quilt pop -a decision (patch tree changed since the recorded attempt) and the version-change handling.
• The advisory lock lives outside the session root (<sessions root>/<slug>.lock): a teardown deletes the root while still holding the lock, and a lock file inside it would be removed under the holder — a concurrent opener would then create and lock a fresh inode, and mutual exclusion silently dies (observed with two concurrent same-identity e2e builds before the move).
• pkh prune session GC is confined to the production prune(). prune_in() (the testable core) deliberately does not scan the real sessions root: the test suite runs prune tests and e2e builds concurrently, and the prune tests would delete live sessions mid-build (observed). prune_in_roots() takes the sessions root as an explicit opt-in parameter.
• Fallbacks, in order: no local base context → the historical anonymous temp chroot (no session); session locked by a concurrent build → same; overlay mount unsupported/failed on resume → fresh copy staging (environment resume only, artifacts discarded); no host-tree snapshot → upper wiped and re-snapshotted.