The per-target retention kept the newest five staged trees however
large they are: five carried kernel trees are three-digit GiB, most of
it compiled objects whose resume value does not scale with their size.
A staged tree larger than the retention cap (a user setting, see
below) is now dropped at the end of its own build — the record
degrades to metadata-only and the next build re-stages from scratch —
and the retention sweep also degrades older oversized trees, covering
records kept by older pkh versions and interrupted builds. The cap of
0 disables the size rule; the count cap is unchanged.
The cap is the first setting of a user configuration file,
~/.config/pkh/config.json (the directory already holds contexts.json):
missing file and missing keys mean defaults, a corrupt file is backed
up and the defaults apply, and PKH_STAGED_MAXSIZE overrides the file.
The default is 20G, the scale where a carried kernel tree stops
earning its disk.
Every retained tree's size is measured once at retention (the same
concurrent, mount-crossing-safe walker the prune report uses) and
stored in the record manifest, so sweeps reuse it instead of re-walking
the trees on every build.
- -f/--force skips the size pass and the confirmation prompt: remove
immediately, reporting counts but no sizes. The fastest cleanup
path; --dry-run still previews.
- The default path shows the grouped digest and asks before removing
(the raw-mode yes/no prompt; non-interactive sessions keep their
headless path, so scripts reach for --force).
- Sizes are computed under a spinner ("Computing sizes of
directories to prune..."), and the digest colors the counts (bold)
and the sizes (cyan) on terminals only.
The prune output spammed one line per path. The report now groups
entries by artifact kind with their total size, and the size is
measured before removal whatever the mode, so --dry-run prints a
digest ("24 residual build directories for 1.2 GiB") and both modes
end with the freed total. The size walk never crosses a mount point:
residual chroots hold bind-mounted /proc trees whose virtual files
(/proc/kcore alone reports 128 TiB) would poison the total.
--all is also honest now: it removes recent build records and staged
trees (that is where the space is), and the flag description says so.
One behavior fix rides along: a record or lock a live build holds was
previously prunable with --all, which would wreck a running build;
held locks are now out of bounds whatever the policy.
Plain-speech pass: drop the em-dash asides and mid-sentence colons,
reword the boldface digest constructions ("What --incremental buys
is ..."), and replace the fluffy intro ("aims at wrapping ... while
being smarter at integrating all workflows") with what pkh actually
does. No content changes.
The profile's build-dependency phase spanned everything between the
apt passes and the build step, and for a kernel incremental build
that hid ~13 minutes of housekeeping: the recursive chown handing
the staged tree to the build user, and the resume_clear quirk
dropping the previous attempt's stamp caches. Both now enter the
profiler (ClearingResumeCaches only when the quirk has paths and a
tree is carried, PreparingBuildUser only for rootless builds), so
the status bar and pkh deb profile show where the time actually
goes — on a large carried tree the chown dominates, which is the
honest answer to a 6m30s DEPS column next to a 19m19s phase.
Time every Phase the build flow announces through enter_phase — the
chroot bootstrap (tarball download, extraction, finalization), the
apt phases, patch application, the build and binary steps and the
artifact retrieval — and land the ordered profile in the record's
manifest of every run, failed builds included: their profile ends
at the phase the error propagated out of.
Timing rides the enter_phase choke point instead of new measurement
sites, so a phase added later profiles itself. The manifest field
is additive (serde default, skipped when absent), so older manifests
keep loading without a schema bump. The coarse timings stay the
source of the deb list columns: they measure the apt transactions
themselves, where the profile measures whole phases.
pkh deb profile renders a record's profile through the same
read-only record discovery as pkh deb log — the newest record of
the tree, or an id prefix as shown by pkh deb list — so
record_for_log becomes the neutral find_record. Phase keys are
storage names, distinct from the display labels; Phase::from_key
turns them back and a test keeps both in sync.
Every binary build measures its dependency installation (apt-get
update, the essential and injected packages, each build-dependency
pass) and its package build (debian/rules build plus binary) into a
timings accumulator; the record manifest stores both as an optional
`timings` field when the run ends, whatever the outcome — a failed
build keeps its partial durations, and a run that died before any
timed command records nothing.
`pkh deb list` shows the two durations as DEPS and BUILD columns,
formatted like the age ("45s", "2m41s", "1h02m"); records predating
the field show dashes. The field is serde-optional, so the manifest
schema stays at 1 and existing records remain discoverable and
forkable.
Pinned runners (a caching proxy, a chosen mirror) must resolve every
package from one base URL. 'pkh deb --mirror <url>' passes the mirror
to mmdebstrap as its mirror positional, so the chroot tarball's apt
sources carry it from the bootstrap on and every apt inside the build
follows - no per-build source rewriting. The mirror must serve every
suite and architecture the build resolves (the security pocket and,
for cross builds, the target archive included); that contract is
documented, not enforced.
The tarball cache is keyed per mirror (a short sha prefix before the
'-buildd.tar.xz' suffix prune matches): a --mirror run over a stale
distro-default tarball would silently resolve from the distro, and
'--incremental' records must never fork across a mirror change.
The runtime source passes recognize the pinned URL as an archive
source (it carries no official host for the host-based checks): the
pocket pass and the universe pass treat it as the archive, and the
cross setup gives it every component and suite, no arch scoping (the
mirror serves every architecture by contract) and no ports entry.
'pkh test' passes no mirror: its tarballs stay distro-default.
Fresh chroots also mean fresh compiles: nothing survived to make the
recompile-heavy middle of a run cheap. Every deb build now wires a
shared ccache — /var/tmp/pkh/ccache/<slug>/, bind-mounted at the
chroot's /var/cache/pkh-ccache — through a PATH shim rather than CC
environment variables: gcc/g++/cc/c++ symlinks in /usr/lib/pkh-ccache-
shim resolve to the ccache the essentials install, so rules that call
the plain compiler names are cached while explicit exotic compilers
stay untouched. CCACHE_COMPILERCHECK=content is critical — the default
is mtime-based and would never hit across fresh chroots, where every
toolchain's mtime is the extraction's — with CCACHE_BASEDIR=/tmp/pkh-
build and CCACHE_NOHASHDIR=true keying the staged tree's relative
paths, and CCACHE_MAXSIZE=10G (PKH_CCACHE_MAXSIZE overrides).
No mtime clamping anywhere: the tarball extraction preserves mtimes,
so the toolchain and libc headers stay older than carried objects and
make keeps them, while build-dependency reinstalls are newer and make
rebuilds their dependents — ccache turns those into hits, and a real
archive library change is a miss that rebuilds them, exactly the
signal clamping would silence. `pkh deb --no-ccache` skips the shim,
the env and the mount entirely: an escape hatch for rules that
mishandle compiler wrappers (kbuild-style flavour configs).
With the chroot now per-run scratch, every build re-downloaded the
packages its dependency install needed — the very cost the chroot reuse
used to hide. Every chroot's /var/cache/apt/archives is now a bind
mount over a shared cache at /var/tmp/pkh/apt-cache/<series>-<arch>/
(the chroot's architecture, like the tarball cache), which one run's
installs fill and the next runs reuse: apt keeps downloaded packages
through a dropped conf (APT::Keep-Downloaded-Packages), and an
apt-get autoclean at the end of each run drops the debs the archive no
longer serves. pkh prune removes caches untouched for a week.
The run holds a host-side flock on the cache for its whole lifetime:
apt's own archives lock is acquired inside the chroot's pid namespace,
where a concurrent build's apt shows the holder as "process 0" — an
invisible process it refuses to wait for — so pkh serializes the
traffic itself; concurrent builds of the same target wait for each
other. Any bind-mount failure falls back to the chroot's own cache
with a warning (correct, just slower), and the temp-chroot teardown now
unmounts everything under the chroot before rm -rf, so the removal can
never reach through the cache bind mount and delete the shared cache
itself.
Records carried a full chroot each, and space turned out to be the hard
constraint: five records per target multiplied by a multi-GB chroot does
not fit. The chroot is now per-run scratch — every build extracts a
fresh one from the sha-verified mmdebstrap tarball cache into its
record's chroot/ dir and removes it at the end (success, failure and
interrupt included); resume_chroot, the .pkh-ready probe and
chroot_is_usable are gone, and crashed-run leftovers are swept when the
next run of that identity starts.
What a record keeps instead: manifest.json (renamed, now with the
tarball sha it used, a kept_upper flag and a tree-only journal — the
environment phases rerun in every fresh chroot), a log/build.log full
tee of the build output (kept for EVERY build; plain runs become
metadata-only records, so keep-iff-incremental now applies to the
staged tree alone), and — for --incremental runs — tree-upper/ with its
host-files.list. The fork gate carries a base's staged tree only while
its recorded tarball sha still matches the cache and the changelog
version is unchanged; a fork candidate is any record with a staged tree.
The keep-last-5 cap now applies to staged trees: a sixth retention
degrades the oldest record to metadata-only instead of deleting it.
New `pkh deb log [-f] [<id>]` prints (or follows) a record's build log,
and `pkh deb list` shows the staged-tree state and plain-run records
too. README rewritten around the shrunk model.
The single mutable session per identity loses the last-good state: a
failed run after a good one adopts the session in place and leaves only
the failed state behind, so "continue from the last success" was
impossible. Each build now records an immutable record under
/var/tmp/pkh/sessions/<slug>/<id>/ (chroot, tree-upper, journal,
manifest), and --incremental forks a NEW record — a full copy of the
base's chroot and, at the same changelog version, its staged artifacts
— under the base's flock, never mutating the base itself.
Selection picks the newest successful record of the identity among the
last five usable, falling back to the newest when none succeeded, and
--incremental <id> pins an exact record (failed ones allowed: their
environment may be fine). The lifecycle is strict keep-iff-incremental:
plain runs build in a throwaway record discarded on success, failure
and interrupt alike (the SIGINT cleanup hook follows the same policy),
while --incremental records are always kept. Each identity keeps its
newest five records; pkh prune's 7-day sweep and orphaned-lock cleanup
follow the nested layout.
--resume and --keep are removed (no aliases): their adoption-in-place
and keep-on-success semantics are exactly what the record model
replaces. pkh deb list shows id, outcome, tree version, fork lineage
and age. README rewritten around the record workflow.
Iterating on a package currently needs two flags (--resume --keep):
without --keep a successful build consumes the session, and without
--resume nothing is reused, so incremental iteration never "just
works" opt-in.
--incremental is the single-flag form: with no --resume it adopts the
newest usable session of this tree (like `pkh deb --resume`), and it
always keeps the session on success, so the loop (edit, `pkh deb
--incremental`, edit, ...) reuses the chroot, build deps and objects
and only recompiles what changed. With an explicit `--resume <id>` it
only adds keep.
A plain `pkh deb` is unchanged: fresh session, consumed on success.
The changelog path was hardcoded to debian/changelog in the series
detection, the entry generation and the editor invocation. Packages
keeping the changelog elsewhere (e.g. a staging copy) could not use
chlog on it.
Add -f/--file defaulting to debian/changelog, threaded through all
three uses. The library side was already parameterized on the file,
so no changelog.rs behavior changes beyond a test pinning the
non-default path.
Without a version tag, the boundary of a new entry was found by walking
the whole history and diffing each commit's tree against the changelog
path. On large repositories that per-commit tree diff takes 10+ minutes.
Add --limit (default 500, 0 = unlimited) capping that walk: when no
changelog commit is found within the last N commits, the oldest visited
commit becomes the boundary — the version tag probe is skipped there,
since it could jump the boundary arbitrarily far back and defeat the
cap — and a warning explains the fallback. A found version tag still
takes precedence and is unaffected by the limit.
Two failures from resumed kernel sessions:
A resumed build re-runs debian/rules build, but the kernels keep
their wrapper-step stamps in debian/stamps and the flavour config
rule there exports .config from the annotations with no config
prerequisite: the resumed build silently kept the previous
attempt's configuration and a config edit never reached the .deb.
Dropping the stamp cache unconditionally would impose the kernel
packaging's shape on every package, so it is data-driven instead:
packages declare the caches under a resume_clear quirk, and
resumed builds remove those tree-relative paths before the build.
The inner kbuild keeps its own incremental state, so only the
cheap wrapper passes re-run and config-affected objects recompile.
A SIGKILLed build (OOM) leaves its overlay mounts and /proc bind
mount behind, and overlayfs creates root-owned work state inside
the workdir: the next plain build (recording is always on, it
replaces the session of its identity) failed to clear the
leftovers with a permission error. Session clearing now unmounts
everything under each entry at depth (re-reading /proc/mounts,
tolerating mount stacks from consecutive crashes) and escalates
through sudo; when the tree still cannot be cleared, the build
reports it and continues without a session instead of layering
over a half-cleared one.
Add the DEP-8 runner to the command list and a usage section: default
chroot testbed sharing the deb tarball cache, binary reuse from pkh deb
with in-process rebuilds, the local and vm modes, and the 0/1/2 exit
contract. autopkgtest joins the shell-out list (vm mode) and the
roadmap bullet comes off — the feature exists.
README: the build-session workflow (auto-recording, --resume,
'pkh deb list', --keep, prune retention) under a new section.
plans/: the design spec with the decisions taken along the way
(opt-in resume, --keep for the iteration loop, /var/tmp/pkh/
sessions, Ctrl+C keeping the session) and the implementation notes
recording the as-built deviations.
pkh v0.1.0 is published to crates.io, so cargo install pkh is now the
primary installation path; building from source stays as the
alternative. Drop the 'no distribution channel' notice.
pkh put only spoke SFTP to the PPA queue, so a failure of the SSH
transport itself (TCP, banner exchange) failed the upload even though
dput happily pushes the same files: its plain ppa: profile goes over
the anonymous FTP queue of ppa.launchpad.net, the same destination
over another port.
Classify the SSH connection failures: Transport (the connection never
came up: resolution, TCP, banner or key exchange) degrades to that FTP
queue — the upload order (payload first, .changes last), the
reverse-order DELE cleanup of a failed upload and the per-chunk
progress reporting all mirror the SFTP path, sharing cleanup_list.
Refused failures (host key not accepted, no matching authentication)
stay errors: silently switching transport would bypass the refusal.
The FTP client is suppaftp's blocking stream, with the time bounds it
does not carry by itself: the control channel's reads and writes, the
data channel's writes and connect (through a custom passive stream
builder), and the NAT workaround for PASV replies announcing an
unroutable address. The queue endpoints (host, port) join
data/launchpad.yml next to the SFTP ones, and the FTP transport is
covered by unit tests against an in-process fake queue plus a live
control-channel handshake with the real server (ignored, network).
The chlog series selector only appeared when the changelog's current
distribution resolved to a known series; a Debian package targeting
'unstable' (or 'stable', 'testing', ...) fell through to keeping the
current series, silently, with no menu.
Resolve the changelog distribution through the suite aliases first
(unstable identifies the same series as sid, which resolves to the
Debian series list). The selector offers an aliased series as
'<suite> (<series>)' — 'unstable (sid)' — preselected, but selects
the suite name: what a changelog distribution field expects, instead
of the codename. Every other label and free-typed input selects
itself, unchanged.
Reflect the selector in the README roadmap checklist.
The command block is now the actual pkh --help output (new, lint and
prune were missing). The example workflow used pkh commit, a
subcommand that does not exist; commits go through git until
chlog/pkh commit land. The roadmap now reflects what is implemented
(pull -v, deb --mode local, lint, prune, new) and an installation
section documents the source build and its system dependencies.
The subcommand is gone from the CLI; context survives only as the
internal build backend, so there is no user-facing surface left to
track on the roadmap.
The version flags were never reachable: --backport was declared but not
read, and compute_new_version's NMU/rebuild numbering sat behind a TODO
asking for CLI wiring (the old positional-bool signature always received
false). generate_entry now takes an EntryKind selected by three mutually
exclusive flags, and is async because the backport numbering derives the
Debian release number of the target series from distro-info:
- --backport: 1.0-1 becomes 1.0-1~bpo12+1 (12 = release number of the
target series, backport suite names accepted too). Re-running on an
already-numbered version bumps the counter; series without a numeric
Debian release (sid, Ubuntu series, UNRELEASED) are rejected before
anything is written.
- --nmu: 1.0-1 becomes 1.0-1.1 (native 1.0 becomes 1.0+nmu1).
- --rebuild: 1.0-1 becomes 1.0-1build1.
An explicit --version overrides all three. compute_new_version went
from four positional bools to a private Bump enum; backport numbering
reuses increment_suffix with a '~bpoNN+' suffix. The CLI prints the
computed new version before opening the editor.
Upload built source packages over SFTP with host-key verification
(Launchpad fingerprints pinned in host_keys.yml, ask-to-accept
otherwise), Launchpad account discovery (git config lp.user), and
pre-flight checks the upload queue itself never does: changes file
discovery/validation, PPA existence via the Launchpad API, target
series validity, and debian/control Section validity (sections
bundled in distro_info.yml). Upload log prevents duplicate uploads
unless --force.
- pull: select the pocket to download the source package from
- deb: build against dependencies from a specific pocket:
- local mode: enable '{series}-{pocket}' suite on archive sources
- cross mode: include the pocket suite in required repositories
vhaudiquet