docs: document incremental builds and the session design
CI / build (push) Successful in 3m6s
CI / test (push) Skipped
CI / publish (push) Skipped
CI / snap (push) Successful in 6m25s

README: the build-session workflow (auto-recording, --resume,
'pkh deb list', --keep, prune retention) under a new section.
plans/: the design spec with the decisions taken along the way
(opt-in resume, --keep for the iteration loop, /var/tmp/pkh/
sessions, Ctrl+C keeping the session) and the implementation notes
recording the as-built deviations.
This commit is contained in:
2026-09-26 11:52:07 +02:00
parent 1b71a6814c
commit 1c8f4cf35b
2 changed files with 451 additions and 0 deletions
+22
View File
@@ -111,6 +111,28 @@ pkh put --ppa user/hello_xxx
git push xxx user-fork git push xxx user-fork
``` ```
### Incremental builds (build sessions)
Every `pkh deb` build records a **session** under `/var/tmp/pkh/sessions`:
the bootstrapped chroot, the installed build dependencies and the build
artifacts of the staged tree. When a build fails (or is interrupted), the
session is kept and can be resumed:
```
pkh deb # fails after 25 minutes
pkh deb --resume # reuses the chroot, build deps and objects;
# only what changed is recompiled
pkh deb list # the sessions of this tree, with their ids
pkh deb --resume <id> # resume a specific session
pkh deb --keep # keep the session even after a successful build
# (iterate: edit, `pkh deb --resume --keep`, ...)
pkh prune # garbage-collect old sessions (7-day retention)
```
A plain `pkh deb` never reuses a session — everything is rechecked from
scratch — and it replaces the session of its target. `pkh deb --resume`
refuses to adopt a session built for a different series/architecture.
## Future improvement ideas ## Future improvement ideas
- pull: try to fetch the correct git branch for series on Debian - pull: try to fetch the correct git branch for series on Debian
+429
View File
@@ -0,0 +1,429 @@
# `pkh deb` Incremental Builds — Spec
## Problem
`pkh deb` has no incremental story. Every invocation — including a rerun
seconds after a failed build — pays the full pipeline again:
1. mmdebstrap tarball **extraction** into a fresh `/tmp/pkh-<ts>` chroot
(the tarball itself is cached, but extraction of a kernel-sized chroot
is minutes),
2. device nodes + `/proc` bind mount,
3. `apt-get update`,
4. essentials install,
5. `quilt push -a`,
6. native build-dependency resolution + install,
7. `debian/rules build` **from scratch** in a freshly staged copy of the
tree,
8. `fakeroot debian/rules binary`.
The failures this hurts most are the late ones. A kernel package that
dies in `debian/rules build` after 25 minutes is restarted from zero;
fixing one compile error and rerunning recompiles everything, and a
cycle of "tweak, rebuild, fail again" costs a full pipeline each time.
The raw material for resumption already exists but is unreachable:
- **Failed chroots are kept.** `EphemeralContextGuard::drop`
(`src/deb/ephemeral.rs`) deliberately keeps the chroot when the build
did not succeed — but the next run creates a new `pkh-<ts>` directory
and cannot find or reuse the old one. The kept chroot is a leak until
`pkh prune`, not a cache.
- **The staged tree is an overlayfs mount** (`src/context/unshare.rs`):
host tree = lowerdir (read live), build writes = upperdir inside the
chroot. If the upperdir survives between runs, object files survive
too, and host-side edits propagate automatically — exactly the
semantics incremental builds need.
- **Ctrl+C deletes everything** (`sigint_cleanup_chroot`), which is the
worst behavior for a long build interrupted on purpose.
## Goal
`pkh deb --resume` after a failed build resumes from the point of
failure (plain `pkh deb` always rebuilds from scratch — see §5):
- same environment, source untouched → picks up at
`debian/rules build`, recompiling nothing (make sees a warm tree);
- source modified since the failure → still reuses chroot, apt state
and installed build-deps; `debian/rules build` recompiles only what
the modification affects;
- `debian/control` or patch series changed → environment-level state
(build-deps, quilt) is redone; the chroot and object files still
carry over when possible.
Non-goal: distributing or sharing sessions between machines; resuming
`pkh build` (the native source pipeline — it is seconds-fast already);
resuming inside ssh/schroot contexts (sessions are a local-unshare
feature; other drivers keep today's behavior).
## Design
### 1. Build sessions: named, journaled, discoverable
A **session** is one (package, series, arch, cross) build environment
plus its journal. Replace the anonymous `pkh-<timestamp>` chroot
directory with a session root:
```
/var/tmp/pkh/sessions/<slug>/
session.json # manifest + phase journal (see below)
chroot/ # the chroot tree (was /tmp/pkh-<ts>)
tree-upper/ # overlayfs upperdir for the staged package tree
pkh-overlay/ # overlay workdirs (as today)
```
`<slug>` is derived from the session identity (package, series, arch,
cross) plus a short content hash, so a rerun can find a previous
session by identity without guessing timestamps. One live session per
identity — the newest attempt *replaces* the previous one (same
chroot environment, fresher state; keeping per-attempt history would
mean one full chroot per attempt, which is a cache nobody wants). A
locked session (concurrent build) is never replaced (§6). What the
session list shows for a tree is therefore one entry per identity
(series/arch/cross combination), each carrying its latest attempt's
id.
Moving off `/tmp` is deliberate: sessions must survive a reboot to be
worth keeping, and `/tmp` is often tmpfs (a kernel chroot on tmpfs is
RAM). `/var/tmp` is the conventional persistent-scratch location; the
base directory is overridable (`PKH_SESSIONS_DIR` env, then config).
Sessions are a cache, and `pkh prune` is their primary GC — if
systemd-tmpfiles sweeps `/var/tmp` on some setup, that is an
acceptable, if blunt, secondary cleanup.
`session.json` records:
```jsonc
{
"id": "20260926T143505", // build-start timestamp, UTC; changes on every attempt
"host_tree": "/home/me/linux", // tree the session was built from (list scoping)
"identity": {
"package": "linux", "series": "stonking",
"arch": "arm64", "cross": true
},
"created": "...", "last_used": "...",
"chroot": {
"tarball": "stonking-arm64-buildd.tar.xz",
"tarball_sha256": "...", // reuse only if the cached tarball is the same
"ready": true // device nodes + /proc done
},
"phases": {
"apt_update": { "at": "..." }, // journal-only: apt update always reruns on resume
"essentials": { "at": "..." },
"patches": { "stamp": "<hash of debian/patches>", "applied": true },
"build_deps": { "stamp": "<hash of control+arch+cross+ppa+inject+pocket>", "at": "..." },
"build": { "at": "...", "result": "failed" }
},
"tree": {
"version": "7.3.0-5.6~local2", // changelog version of the last attempt
"host_snapshot": "<path or inline list>", // for deletion propagation
}
}
```
The journal is written **after** each phase completes (and after a
phase fails, recording the failure), so a crash or Ctrl+C at any point
leaves a truthful journal. Discovery is: scan
`$PKH_SESSIONS_DIR/*/session.json`, match identity.
### 2. Two layers of resume
The journal separates **environment resume** (expensive, insensitive to
source edits) from **build resume** (sensitive to every edit):
| Phase | Reused when | Redone when |
|---|---|---|
| Chroot extract, nodes, /proc | chroot present, `ready`, same tarball hash | never for the same identity (otherwise the session is unusable) |
| `apt-get update` | — (always rerun on resume) | — |
| Essentials | done (verify cheaply via `dpkg-query` at most) | chroot redone |
| quilt push | `debian/patches` hash unchanged | hash changed → `quilt pop -a` + re-push |
| Build-deps | control/arch/cross/ppa/inject/pocket stamp unchanged | stamp changed → the resolver runs again (it is idempotent; apt installs only what is missing) |
| `debian/rules build` | always *attempted* in the warm tree | — |
| `fakeroot debian/rules binary` | always rerun | — |
Key consequences:
- **`apt-get update` is always rerun on resume.** It is the cheapest
phase, and stale package lists fail confusingly ("candidate version
not found") on fast-moving suites — there is nothing worth caching
here. The journal entry exists for observability only.
- **A changelog version bump** (`pkh chlog` between attempts) invalidates
build resume but not environment resume: the chroot, apt lists and
build-deps carry over. The staged tree is re-populated for the new
version (see §3); the session's `tree.version` tracks which attempt
the upperdir belongs to, and a version change discards the upperdir
(object files reference paths/flags of the old version).
- **The build phase is never skipped**, only made incremental — make
decides. pkh's job is to hand `debian/rules build` a tree whose
previously produced objects are still there and whose sources match
the host.
- Before rerunning the binary phase on a resume, clear
`debian/files` (and `debian/*.log`, `debian/substvars`) from the
staged tree so artifact collection (`collect_binary_artifacts`) only
sees the new attempt's outputs — today's "no globbing stale files"
contract is preserved.
### 3. Tree propagation on resume
Today each build stages a fresh copy/overlay of the host tree. On
resume the staged tree already exists inside the session; the problem
is syncing it with a possibly-modified host tree.
- **Overlay contexts (the default when overlayfs is available):** remount
the overlay with the *same* upperdir. Host modifications are visible
immediately (lowerdir is read live); objects written in the upperdir
during the previous attempt are still on top. Host-side *deletions*
are the one gap — overlayfs only hides a lower file via a whiteout in
the upper, and nothing creates whiteouts for files the user deleted
on the host. Fix: the session stores a snapshot of the host tree's
file list at first staging; on resume, files present in the upper
(or in the snapshot) but gone from the host get an explicit whiteout
(`mknod <path> c 0 0` inside the upper). This is a bounded,
manifest-driven operation, not a tree walk of the whole chroot.
- **Copy fallback (no overlayfs):** `rsync -a --delete` of the host
tree over the staged tree, *excluding* the build artifacts make
produced (this needs per-package exclusion knowledge and is best
effort). Deletions propagate naturally here. Sessions on the copy
path get build resume only for trees where the exclusion set is
sane; the honest fallback is: copy path → environment resume only.
The overlay logic stays inside the unshare driver, matching the
principle established in `plans/overlayfs-integration.md`: callers
stage a tree through `ensure_available()` and never learn how.
### 4. Interrupt behavior
Ctrl+C currently removes the chroot through the interrupt hook. Change:
**an interrupted build keeps the session**, journaling the interrupted
phase as incomplete — an intentional Ctrl+C on a 30-minute build is a
"pause", and deleting the environment on pause defeats the feature.
The cleanup hook (`sigint_cleanup_chroot`) becomes session-aware: it
still unmounts the overlays and `/proc` (leaving mounted state in a
session is what makes stale sessions dangerous), but keeps the tree
and leaves a `interrupted: true` marker in the journal so the next run
knows the session needs no special recovery (phases after the last
completed one simply rerun).
Users who want the old hard-discard behavior on interrupt get it via
`pkh prune`. The failure path is unchanged in spirit: failed builds
already keep their chroot; they now keep a *usable* one.
### 5. CLI surface and policy
**Recording is always on; reuse is opt-in.** The two are deliberately
separated, because the first attempt cannot know it will fail: a
plain `pkh deb` writes and maintains the session (journal, kept
chroot on failure or interrupt — today's keep-on-failure, made
usable) but never reads one. Only `pkh deb --resume` adopts an
existing session. The default run therefore always rechecks
everything from scratch — no stale-build risk unless the user asks
for one.
- `pkh deb` — today's behavior, plus session recording: any existing
session of the same identity is ignored and replaced by this run's
outcome. On failure or interrupt the session is kept (as today);
on success it is **removed**, unless `--keep` is given.
- `pkh deb --keep` — also keep the session after a successful build,
so a later `--resume` can iterate on it (the edit–rebuild loop).
Composes with `--resume`: `--resume --keep` chains sessions across
iterations; a resumed build without `--keep` consumes the session
on success — the natural end of the loop.
- `pkh deb --resume [<id>]` — adopt a session and continue it,
announcing what is being skipped ("Resuming session
20260926T143505 for linux/stonking-arm64: chroot, package lists
and build-deps reused").
- Without an id: the newest session recorded from this tree
(manifest `host_tree` matches the tree being built), whatever its
last outcome — failure, interrupt, or a `--keep` success.
- With an id (a build-start timestamp as shown by `pkh deb list`):
that exact session. A unique unambiguous prefix is accepted; an
ambiguous one is an error listing the candidates.
- Selector conflicts are errors, not surprises: if `-a`/`-s`/
`--cross`/`--ppa`/`--inject` are given explicitly and disagree
with the adopted session, refuse and point at `pkh deb list`. A
session for another series/arch is a *different* build
environment; silently building it under the requested selectors
would be exactly the stale-build risk this design avoids.
- With no session to adopt (none recorded, pruned, or the tree
moved): say so and build from scratch — never silently.
- `pkh deb list` — a subcommand, not a flag: `pkh deb` with flags
always means "start a build", and listing is a different action.
It prints the sessions recorded from the current tree, one row per
identity (§1), newest attempt first:
```
$ pkh deb list
Sessions for /home/me/linux:
ID PACKAGE VERSION TARGET LAST RUN AGE
20260926T143505 linux 7.3.0-5.6~local2 stonking/arm64 failed 2h
20260925T090012 linux 7.3.0-5.5~local1 stonking/arm64* success 1d
20260924T171100 linux 7.3.0-5.6~local2 stonking/riscv64* failed 2d
* cross build. Resume with: pkh deb --resume [<id>]
```
`pkh deb list` never touches build machinery: it reads the session
manifests under the sessions directory and renders. As an optional
subcommand of `deb` (clap allows subcommands alongside args), the
build flags stay on `deb` itself and `list` takes none of them.
- `pkh prune` learns sessions: by default removes
sessions untouched for longer than a retention window (7 days) and
any session whose journal is missing/corrupt; `--all` removes every
session. Prune's report distinguishes "session (resumable build)"
from the other residual categories, since sessions are large and
users must be able to see and reclaim them.
### 6. Correctness guards
A session is only reused when **all** of these hold; otherwise it is
discarded (replaced) with a logged reason:
- identity matches (package, series, arch, cross);
- the cached chroot tarball hash matches the one the chroot was built
from;
- `chroot/` passes an integrity probe (marker file, `chroot/bin/sh`
exists and executes);
- pkh version compatibility: `session.json` records the pkh version
and manifest schema version; a mismatched schema discards the
session (the chroot is generic enough that only schema-relevant
changes matter, but being conservative is cheap).
- no other live session for the same identity (lock file in the
session root, in the spirit of the existing tarball lockfile).
Concurrent builds of the same identity take the lock and fall back to
a fresh ephemeral session (current behavior) rather than waiting —
correctness over caching.
One more guard on the resume path, at the CLI layer: explicit
selectors (`-a`, `-s`, `--cross`, `--ppa`, `--inject`) that disagree
with the adopted session refuse the build (§5) — the session is only
ever reused under the selectors it was created with.
## Implementation sketch
New `src/deb/session.rs` — session identity, manifest
(serde) read/write, discovery, locking, slug generation. Pure logic,
fully unit-testable.
- `src/deb/ephemeral.rs` — `EphemeralContextGuard` learns a `resume`
path: when a session is adopted, skip download/extract and reuse the
existing chroot (still verifying `ready` and remounting `/proc` if a
previous run unmounted it); on teardown, hand the tree to the
session instead of `rm -rf`-ing it. Chroot dir moves from
`create_temp_dir()` to the session layout — the local driver's temp
naming stays for non-session uses.
- `src/deb/local.rs` — journal writes around each phase; the
skip-or-run decisions per the table in §2; `debian/files` cleanup
before the binary phase on resume.
- `src/context/unshare.rs` — upperdir reuse + whiteout propagation for
host deletions (§3); `ensure_available()` gains a
"stage into existing session tree" mode.
- `src/interrupt.rs` + `src/deb/ephemeral.rs` — the interrupt hook
keeps the session (§4).
- `src/prune.rs` — session discovery, retention, reporting.
- `src/main.rs` — `--keep`, `--resume [<id>]` argument wiring,
the `pkh deb list` subcommand and selector-conflict checks;
`README.md` roadmap
update (this retires part of the "deb: asynchronous build,
detachable and monitorable" itch: a persistent session is also the
natural attach point for detached builds later).
Staged rollout, each step independently shippable:
1. Sessions with manifest + `pkh deb list` + adoption of failed-build
chroots via `--resume` (environment resume only — biggest win, no
overlay changes).
2. Build resume: upperdir reuse + whiteout propagation.
3. Interrupt keeps session; `--keep` after success; prune integration.
## Testing
- Unit: manifest round-trip, slug/identity matching, whiteout plan
computation (host snapshot vs current host listing), skip-decision
table given journals with various stamps.
- Integration (small fixture package, `#[cfg(test)]` style already
used by `deb` tests): fail `debian/rules build` on file B after
compiling file A; resume and assert A is *not* recompiled (make log
via the tee log); modify A; resume and assert only A recompiles;
modify `debian/control`; assert build-deps rerun and objects
survive; `pkh prune` GC behavior; a plain `pkh deb` ignoring an
existing session (fresh rebuild) while `--resume` adopts it.
- Unit/CLI: `pkh deb list` rendering against fixture manifests (scoping by
`host_tree`, cross marker, ordering); `--resume <id>` exact and
prefix resolution, ambiguity error; selector-conflict refusals;
`--keep` keeping the session on success while a plain build removes
it.
- Chroot reuse tests need the unshare/mount machinery — follow the
existing pattern of `#[ignore]` tests for deliberate ad-hoc runs
where root/unshare is required.
## Decisions
- **Resume is opt-in (`--resume`), never the silent default.** A
plain `pkh deb` always rechecks everything from scratch, so the
no-flag behavior can never build anything stale; recording stays on
by default so the follow-up `--resume` after a failure has a
session to adopt.
- **Keeping a session after success is opt-in (`--keep`).** Today's
disk profile — success cleans up — stays the default; the
edit–rebuild iteration loop is served by `pkh deb --resume --keep`.
Failures and interrupts keep their session unconditionally, as
today.
- **Sessions are addressed by id (build-start timestamp), listed via
the `pkh deb list` subcommand.** Listing is a subcommand, not a
mode-switching flag: `pkh deb` with flags always starts a build.
One live session per identity keeps the cache
bounded (a per-attempt history would be one full chroot per
attempt); the list therefore shows one row per series/arch/cross
combination for the current tree, each carrying its latest attempt's
id. Explicit selectors that disagree with the adopted session are
an error, never a silent environment switch.
- **Ctrl+C keeps the session**, same as a failed build: an interrupt
on a long build is a pause, and there is nothing inherent in "stop
now" that means "discard half an hour of environment setup". The
hard-discard behavior stays reachable through `pkh prune`.
- **`apt-get update` always reruns on resume** (no stamp, no TTL) —
see §2.
- **Sessions live under `/var/tmp/pkh/sessions`** (overridable via
`PKH_SESSIONS_DIR` then config), and are cleaned by `pkh prune`
(retention window by default, `--all` for everything). Cleanup by
systemd-tmpfiles from time to time is accepted; sessions are a
cache.
---
# Implementation Notes (as built)
Implemented across `src/deb/session.rs` (new), `src/deb/{mod,local,
ephemeral}.rs`, `src/context/{api,unshare}.rs`, `src/prune.rs` and
`src/main.rs`. Deviations from the design above, all deliberate:
- **Phase stamps gate less than specced.** The journal records every
phase (with the patch-tree and control stamps), but on resume only
the expensive non-idempotent work is actually skipped: the chroot
bootstrap (extraction, device nodes, `/proc`) and, via the reused
overlay upperdir, the compiled objects. `apt-get update`, the
essentials install, the quilt push and the build-dep resolution
always rerun — each is idempotent and seconds-cheap, and rerunning
them removes a whole class of stale-state bugs. The stamps still
drive the quilt `pop -a` decision (patch tree changed since the
recorded attempt) and the version-change handling.
- **The advisory lock lives outside the session root**
(`<sessions root>/<slug>.lock`): a teardown deletes the root while
still holding the lock, and a lock file inside it would be removed
under the holder — a concurrent opener would then create and lock a
fresh inode, and mutual exclusion silently dies (observed with two
concurrent same-identity e2e builds before the move).
- **`pkh prune` session GC is confined to the production `prune()`.**
`prune_in()` (the testable core) deliberately does not scan the real
sessions root: the test suite runs prune tests and e2e builds
concurrently, and the prune tests would delete live sessions
mid-build (observed). `prune_in_roots()` takes the sessions root as
an explicit opt-in parameter.
- **Fallbacks, in order:** no local base context → the historical
anonymous temp chroot (no session); session locked by a concurrent
build → same; overlay mount unsupported/failed on resume → fresh
copy staging (environment resume only, artifacts discarded); no
host-tree snapshot → upper wiped and re-snapshotted.