The size cap was the wrong shape: an --incremental user asked for a
carried tree, and the invariant that matters is which trees survive —
the newest successful one (the next run's automatic fork base) and the
newest one (what a failed run is re-forked from explicitly). The cap
is gone, with the tree measurement and manifest field it needed; the
retention planner now works on the records' outcomes alone and fills a
budget: staged_trees staged trees per target, the newer ones first,
the pair always among them. When the budget cannot hold the pair (one
staged tree configured, the newest build failed and an older
successful tree present) only one of the two is successful and that
one is kept; with no successful tree at all, the most recent one is.
The budget is a setting of the nested config file:
deb:
incremental:
staged_trees: 2
(default 2; 0 clamps to 1 with a warning — the newest successful tree
always survives). The fork-base selection no longer truncates the
candidate list to the budget before ranking it: with a budget of one
and a failed newest tree, that truncation hid the successful record
and forked the failed one instead.
The retention kept the newest five staged trees per target, so the
staged trees of a kernel-scale target alone could hold three-digit
GiB. The policy is now the pair that earns its disk: the newest
staged tree (what a failed run is re-forked from explicitly) and the
newest successful one (what the next --incremental run forks
automatically) — one record when they coincide; every other staged
tree degrades to metadata-only at the next build's retention. The
retention decision moved into a pure planner over (success, size) so
the policy is table-testable.
The size cap no longer drops a tree outright: an --incremental run's
own tree is always kept by its own run — the user asked for a carried
tree, and at least one always exists — and the cap only thins the kept
pair while another member covers the resume, the newest successful
tree being the last one standing. Oversized trees left by older pkh
versions or interrupted builds come down at the next build's
retention, once their lock is free.
The cap moves to the first user settings file, config.yml in
~/.config/pkh (YAML, like the data files; missing file or keys mean
defaults, a corrupt file is backed up, PKH_STAGED_MAXSIZE overrides),
replacing the JSON sketch from the previous commit. Default 20G, 0
disables the size rule. Every retained tree's size is measured once at
retention and recorded in the manifest, so sweeps never re-walk the
trees.
The per-target retention kept the newest five staged trees however
large they are: five carried kernel trees are three-digit GiB, most of
it compiled objects whose resume value does not scale with their size.
A staged tree larger than the retention cap (a user setting, see
below) is now dropped at the end of its own build — the record
degrades to metadata-only and the next build re-stages from scratch —
and the retention sweep also degrades older oversized trees, covering
records kept by older pkh versions and interrupted builds. The cap of
0 disables the size rule; the count cap is unchanged.
The cap is the first setting of a user configuration file,
~/.config/pkh/config.json (the directory already holds contexts.json):
missing file and missing keys mean defaults, a corrupt file is backed
up and the defaults apply, and PKH_STAGED_MAXSIZE overrides the file.
The default is 20G, the scale where a carried kernel tree stops
earning its disk.
Every retained tree's size is measured once at retention (the same
concurrent, mount-crossing-safe walker the prune report uses) and
stored in the record manifest, so sweeps reuse it instead of re-walking
the trees on every build.
Rootless builds handed the staged tree to the build user with a
recursive chown; through the overlay staging every lower-layer file
the chown touched copy-ups — the full data movement of the tree, 19%
of a kernel build's wall clock.
The rules steps are now flagged and the unshare driver enters the
chroot by pivot_root instead of chroot(2), then creates a nested user
namespace mapping the context's root onto a fixed build id: the
chroot's files are the invoking user's host-side and the context maps
that user to uid 0, so the staged tree is owned by the build identity
by construction — no chown, no data movement. The kernel refuses user
namespace creation to chrooted processes, hence the pivot, and the
bind is recursive and taken inside the namespace: a plain bind of the
chroot's host-made submounts fails EINVAL on locked children, while
the rbind carries them across the pivot, and a fresh /proc is mounted
before the old root is shed (umount resolves through /proc/mounts).
fakeroot's chown wrappers attempt the real chown and swallow only
EPERM; as the mapped identity, root targets are unmapped and fail
EINVAL, so mapped builds export FAKEROOTDONTTRYCHOWN — fakeroot's own
knob — and the fake ownership is still recorded. Files the build
writes come out as the invoking user host-side, which also fixes the
silently missing .buildinfo/.changes of rootless builds: the host-side
rewrite of debian/files hit a subuid-owned file with EACCES.
The wrapper is probed per build; when nested namespaces are
unavailable (old util-linux, userns disabled), the previous setpriv
plus selective parallel chown handover runs unchanged. A setpriv-era
staged tree is handed to the invoking user once, host-side and
metadata-only, before the overlay mounts it — afterwards overlayfs
serves stale owners from its attribute cache.
Forking an incremental base copies its whole staged tree — bounded,
and without reflinks a matter of minutes — but the copy emitted no
build events: the live view showed the target line with nothing below
it, and an interrupt there printed raw cp errors with no hint of what
was running (the fork happens before the first bootstrap phase owns
the view).
Announce the copy through a new BuildView::byte_progress port. The
fork sizes the source up front (the copy's total), re-measures the
growing destination from a scoped thread while cp runs (cp has no
progress output to tap), and reports the completed copy when it
returns; unmeasurable or empty sources copy unreported. The terminal
view renders it like the other determinate bars, with byte-unit
counters.
A rootless build's upperdir entries are copy-up'd by whoever wrote
them during the build: the unprivileged build user, which host-side
is a subuid of the invoking user. The host-side sync of a forked
record removes the stale upper copy of every host-modified file, and
the plain fs removals failed with EPERM on those build-user-owned
entries: the kernel incremental run died with a bare
'Permission denied (os error 13)' and no context, ten minutes into
the run.
Route the sync removals through the privileged path (the same sudo
fallback the record teardowns use) and wrap the sync error at the
call site, so a failure names the operation instead of surfacing a
bare errno string.
The staged tree was the parent of the tree the caller pointed at,
so building a tree sitting in a busy directory dragged every
sibling into the chroot: a /tmp checkout copied the whole host /tmp
(snap sockets, other jobs' scratch), and the build user's chown -R
then failed per foreign-owned file, or aborted the build outright.
Stage the tree itself: its content is the merge point of the build
root, the overlay lower is the tree and the record's upper stacks
on top. The package resolution treats the staged root as the
package when it carries debian/, keeping the pattern search for
nested layouts. The .debs, the upload metadata and the dose3
scratch follow dh_builddeb's .. (the package directory's parent)
instead of assuming the build root, and the host-tree snapshot
covers the tree only. Records staged the old way self-heal: their
snapshots carry parent-relative paths, so the next incremental
run's sync wipes the stale upper entries and rebuilds once.
Siblings are no longer visible to debian/rules — dpkg packaging
never crosses that boundary, and pkh test already staged the tree
alone.
Time every Phase the build flow announces through enter_phase — the
chroot bootstrap (tarball download, extraction, finalization), the
apt phases, patch application, the build and binary steps and the
artifact retrieval — and land the ordered profile in the record's
manifest of every run, failed builds included: their profile ends
at the phase the error propagated out of.
Timing rides the enter_phase choke point instead of new measurement
sites, so a phase added later profiles itself. The manifest field
is additive (serde default, skipped when absent), so older manifests
keep loading without a schema bump. The coarse timings stay the
source of the deb list columns: they measure the apt transactions
themselves, where the profile measures whole phases.
pkh deb profile renders a record's profile through the same
read-only record discovery as pkh deb log — the newest record of
the tree, or an id prefix as shown by pkh deb list — so
record_for_log becomes the neutral find_record. Phase keys are
storage names, distinct from the display labels; Phase::from_key
turns them back and a test keeps both in sync.
Every binary build measures its dependency installation (apt-get
update, the essential and injected packages, each build-dependency
pass) and its package build (debian/rules build plus binary) into a
timings accumulator; the record manifest stores both as an optional
`timings` field when the run ends, whatever the outcome — a failed
build keeps its partial durations, and a run that died before any
timed command records nothing.
`pkh deb list` shows the two durations as DEPS and BUILD columns,
formatted like the age ("45s", "2m41s", "1h02m"); records predating
the field show dashes. The field is serde-optional, so the manifest
schema stays at 1 and existing records remain discoverable and
forkable.
Pinned runners (a caching proxy, a chosen mirror) must resolve every
package from one base URL. 'pkh deb --mirror <url>' passes the mirror
to mmdebstrap as its mirror positional, so the chroot tarball's apt
sources carry it from the bootstrap on and every apt inside the build
follows - no per-build source rewriting. The mirror must serve every
suite and architecture the build resolves (the security pocket and,
for cross builds, the target archive included); that contract is
documented, not enforced.
The tarball cache is keyed per mirror (a short sha prefix before the
'-buildd.tar.xz' suffix prune matches): a --mirror run over a stale
distro-default tarball would silently resolve from the distro, and
'--incremental' records must never fork across a mirror change.
The runtime source passes recognize the pinned URL as an archive
source (it carries no official host for the host-based checks): the
pocket pass and the universe pass treat it as the archive, and the
cross setup gives it every component and suite, no arch scoping (the
mirror serves every architecture by contract) and no ports entry.
'pkh test' passes no mirror: its tarballs stay distro-default.
The record-log wrapper provides a sink for every command, which removed
the runner's subprocess-inherit fallback -- and with it the raw output
of --verbose runs (and of piped stdout, where the live view disables
itself): Quiet's None sink got wrapped inside, every line went to
build.log, and the terminal showed nothing but pkh's own logs, breaking
what --help still promises. When the caller's view has no sink of its
own, a raw terminal sink (stdout/stderr writers, one per stream) now
takes its place inside the record-log wrapper: the terminal shows the
raw lines again while build.log keeps the full tee. A view that
consumes lines itself (the live view on a tty) is wrapped exactly as
before.
Every --incremental run claimed "kept its staged tree" and set
kept_upper=true at teardown — even when the run died before staging
created tree-upper, leaving a manifest that lied about the record being
a fork candidate and a message contradicting the disk. The retention
now follows the disk through upper_retention(requested, has_upper()):
a keep with a staged tree keeps it as before, a plain run drops it, and
a keep with nothing staged keeps the record as metadata + log with
kept_upper=false and says so ("no staged tree was created — the run
ended before staging"). The interrupt path records the same honest
state — failed outcome, kept_upper only if a tree exists — and the
SIGINT hook's message no longer claims a tree that is not there.
The snapshot walk runs over the whole staged PARENT, which contains
trees unrelated to the build; one unreadable path anywhere under it —
a root-owned sysroot left there by the user's own setup script — made
fs::read_dir or symlink_metadata fail and aborted the whole build
seven seconds in, before any staging could happen. The walk is now
best-effort below its root: unreadable directories and entries lost to
a mid-walk race are warned about and skipped, and only the root itself
being unreadable propagates (the package tree being invisible is a
real error). Absent entries are already the handled case for the
whiteout sync: they are simply untracked.
Fresh chroots also mean fresh compiles: nothing survived to make the
recompile-heavy middle of a run cheap. Every deb build now wires a
shared ccache — /var/tmp/pkh/ccache/<slug>/, bind-mounted at the
chroot's /var/cache/pkh-ccache — through a PATH shim rather than CC
environment variables: gcc/g++/cc/c++ symlinks in /usr/lib/pkh-ccache-
shim resolve to the ccache the essentials install, so rules that call
the plain compiler names are cached while explicit exotic compilers
stay untouched. CCACHE_COMPILERCHECK=content is critical — the default
is mtime-based and would never hit across fresh chroots, where every
toolchain's mtime is the extraction's — with CCACHE_BASEDIR=/tmp/pkh-
build and CCACHE_NOHASHDIR=true keying the staged tree's relative
paths, and CCACHE_MAXSIZE=10G (PKH_CCACHE_MAXSIZE overrides).
No mtime clamping anywhere: the tarball extraction preserves mtimes,
so the toolchain and libc headers stay older than carried objects and
make keeps them, while build-dependency reinstalls are newer and make
rebuilds their dependents — ccache turns those into hits, and a real
archive library change is a miss that rebuilds them, exactly the
signal clamping would silence. `pkh deb --no-ccache` skips the shim,
the env and the mount entirely: an escape hatch for rules that
mishandle compiler wrappers (kbuild-style flavour configs).
Records carried a full chroot each, and space turned out to be the hard
constraint: five records per target multiplied by a multi-GB chroot does
not fit. The chroot is now per-run scratch — every build extracts a
fresh one from the sha-verified mmdebstrap tarball cache into its
record's chroot/ dir and removes it at the end (success, failure and
interrupt included); resume_chroot, the .pkh-ready probe and
chroot_is_usable are gone, and crashed-run leftovers are swept when the
next run of that identity starts.
What a record keeps instead: manifest.json (renamed, now with the
tarball sha it used, a kept_upper flag and a tree-only journal — the
environment phases rerun in every fresh chroot), a log/build.log full
tee of the build output (kept for EVERY build; plain runs become
metadata-only records, so keep-iff-incremental now applies to the
staged tree alone), and — for --incremental runs — tree-upper/ with its
host-files.list. The fork gate carries a base's staged tree only while
its recorded tarball sha still matches the cache and the changelog
version is unchanged; a fork candidate is any record with a staged tree.
The keep-last-5 cap now applies to staged trees: a sixth retention
degrades the oldest record to metadata-only instead of deleting it.
New `pkh deb log [-f] [<id>]` prints (or follows) a record's build log,
and `pkh deb list` shows the staged-tree state and plain-run records
too. README rewritten around the shrunk model.
The single mutable session per identity loses the last-good state: a
failed run after a good one adopts the session in place and leaves only
the failed state behind, so "continue from the last success" was
impossible. Each build now records an immutable record under
/var/tmp/pkh/sessions/<slug>/<id>/ (chroot, tree-upper, journal,
manifest), and --incremental forks a NEW record — a full copy of the
base's chroot and, at the same changelog version, its staged artifacts
— under the base's flock, never mutating the base itself.
Selection picks the newest successful record of the identity among the
last five usable, falling back to the newest when none succeeded, and
--incremental <id> pins an exact record (failed ones allowed: their
environment may be fine). The lifecycle is strict keep-iff-incremental:
plain runs build in a throwaway record discarded on success, failure
and interrupt alike (the SIGINT cleanup hook follows the same policy),
while --incremental records are always kept. Each identity keeps its
newest five records; pkh prune's 7-day sweep and orphaned-lock cleanup
follow the nested layout.
--resume and --keep are removed (no aliases): their adoption-in-place
and keep-on-success semantics are exactly what the record model
replaces. pkh deb list shows id, outcome, tree version, fork lineage
and age. README rewritten around the record workflow.
Two failures from resumed kernel sessions:
A resumed build re-runs debian/rules build, but the kernels keep
their wrapper-step stamps in debian/stamps and the flavour config
rule there exports .config from the annotations with no config
prerequisite: the resumed build silently kept the previous
attempt's configuration and a config edit never reached the .deb.
Dropping the stamp cache unconditionally would impose the kernel
packaging's shape on every package, so it is data-driven instead:
packages declare the caches under a resume_clear quirk, and
resumed builds remove those tree-relative paths before the build.
The inner kbuild keeps its own incremental state, so only the
cheap wrapper passes re-run and config-affected objects recompile.
A SIGKILLed build (OOM) leaves its overlay mounts and /proc bind
mount behind, and overlayfs creates root-owned work state inside
the workdir: the next plain build (recording is always on, it
replaces the session of its identity) failed to clear the
leftovers with a permission error. Session clearing now unmounts
everything under each entry at depth (re-reading /proc/mounts,
tolerating mount stacks from consecutive crashes) and escalates
through sudo; when the tree still cannot be cleared, the build
reports it and continues without a session instead of layering
over a half-cleared one.
On a resumed session whose overlay mount fails (no overlayfs, mount
error, non-overlay driver), the fallback used to copy the host tree
into the staged tree of the previous attempt: a merge that keeps the
old build stamps and resurrects files deleted on the host — a
silently half-stale build, exactly what resume must never produce.
The driver now reports the failure instead of copying, and the
caller wipes the staged build root and re-stages a fresh copy
(environment resumable, build artifacts discarded).
Every local build records a session under /var/tmp/pkh/sessions:
the bootstrapped chroot, the installed build dependencies, the
phase journal and the staged tree with a persistent overlay
upperdir. Rebuilding after a failure currently redoes the tarball
extraction, apt update, build-dep resolution and the whole compile;
a kernel-sized package loses half an hour per iteration.
Recording is always on, reuse is opt-in (--resume [<id>]): the
newest session of the tree, or the one matching an id as shown by
'pkh deb list' (one session per series/arch/cross identity, ids
are build-start timestamps). Resume skips the chroot bootstrap
(integrity marker + tarball check), re-mounts the previous
upperdir so make recompiles only what changed, pops the quilt
series first when debian/patches changed, and clears debian/files
before re-packaging so artifact collection stays exact. A version
bump keeps the environment and discards the build artifacts.
Explicit selectors disagreeing with the adopted session are an
error, never a silent environment switch. Failures and Ctrl+C keep
the session (the interrupt hook unmounts but preserves the tree); a
success consumes it unless --keep. Concurrent same-identity builds
serialize on a lock file kept OUTSIDE the session root: teardown
removes the root while holding the lock, and a lock inside it would
be deleted under the holder, letting the next opener lock a fresh
inode. The apt phases rerun on resume (idempotent, seconds-cheap)
rather than being stamp-gated.
vhaudiquet