Two failures from resumed kernel sessions:
A resumed build re-runs debian/rules build, but the kernels keep
their wrapper-step stamps in debian/stamps and the flavour config
rule there exports .config from the annotations with no config
prerequisite: the resumed build silently kept the previous
attempt's configuration and a config edit never reached the .deb.
Dropping the stamp cache unconditionally would impose the kernel
packaging's shape on every package, so it is data-driven instead:
packages declare the caches under a resume_clear quirk, and
resumed builds remove those tree-relative paths before the build.
The inner kbuild keeps its own incremental state, so only the
cheap wrapper passes re-run and config-affected objects recompile.
A SIGKILLed build (OOM) leaves its overlay mounts and /proc bind
mount behind, and overlayfs creates root-owned work state inside
the workdir: the next plain build (recording is always on, it
replaces the session of its identity) failed to clear the
leftovers with a permission error. Session clearing now unmounts
everything under each entry at depth (re-reading /proc/mounts,
tolerating mount stacks from consecutive crashes) and escalates
through sudo; when the tree still cannot be cleared, the build
reports it and continues without a session instead of layering
over a half-cleared one.
The DEP-8 run-time contract is one page, so pkh implements it natively
instead of wrapping autopkgtest (see plans/pkh-test.md): the testbed is
a pkh context — chroot mode reuses the deb ephemeral chroots and their
cached mmdebstrap tarballs, local mode runs on the current context —
and the binaries under test are the freshness-guarded pkh deb output
next to the tree, rebuilt through the deb flow when stale or missing.
Per test: restrictions gate on the testbed (isolation-machine,
needs-reboot and container isolation skip with a pointer at --mode vm),
the stanza Depends are expanded (@, @builddeps@, @recommends@) and
installed, then the script or Test-Command runs in the staged tree with
the AUTOPKGTEST_* (and legacy ADT_*) environment, a per-test timeout
and the stderr/exit-77 verdict rules; flaky tests retry once. stdout,
stderr and $AUTOPKGTEST_ARTIFACTS are collected per test under
/var/tmp/pkh/tests/ (pruned separately). --shell/--shell-fail drop
into the testbed for debugging.
pin_pocket and install_injected_packages become crate-internal helpers
shared with the deb flow, which they mirror.
Every local build records a session under /var/tmp/pkh/sessions:
the bootstrapped chroot, the installed build dependencies, the
phase journal and the staged tree with a persistent overlay
upperdir. Rebuilding after a failure currently redoes the tarball
extraction, apt update, build-dep resolution and the whole compile;
a kernel-sized package loses half an hour per iteration.
Recording is always on, reuse is opt-in (--resume [<id>]): the
newest session of the tree, or the one matching an id as shown by
'pkh deb list' (one session per series/arch/cross identity, ids
are build-start timestamps). Resume skips the chroot bootstrap
(integrity marker + tarball check), re-mounts the previous
upperdir so make recompiles only what changed, pops the quilt
series first when debian/patches changed, and clears debian/files
before re-packaging so artifact collection stays exact. A version
bump keeps the environment and discards the build artifacts.
Explicit selectors disagreeing with the adopted session are an
error, never a silent environment switch. Failures and Ctrl+C keep
the session (the interrupt hook unmounts but preserves the tree); a
success consumes it unless --keep. Concurrent same-identity builds
serialize on a lock file kept OUTSIDE the session root: teardown
removes the root while holding the lock, and a lock inside it would
be deleted under the holder, letting the next opener lock a fresh
inode. The apt phases rerun on resume (idempotent, seconds-cheap)
rather than being stamp-gated.
An unqualified Multi-Arch: same build-dependency resolves to the
host-architecture variant only, even when a build-architecture
candidate exists: that is what dpkg's checker accepts (KnownFacts
accepts foreign, host and all instances) and what sbuild installs,
whose dummy package carries the parsed Build-Depends unqualified and
whose sbuild-cross-resolver only filters foreign M-A:foreign and
Essential packages out of the apt universe — verified by resolving a
sbuild-style dummy:arm64 against the resolute/stonking indexes, which
yields libelf-dev:arm64 & co and never the amd64 variants.
The build-architecture side of such a library is only ever installed
through an explicit :native declaration. Pin both rules so a resolver
regression towards dual-installing unqualified dependencies cannot
reappear unnoticed.
build_binary_package stages the parent of the requested cwd, then
re-derived the package directory inside the staging area from
package/version name patterns plus the calling process's working
directory. That only works by accident for interactive users sitting
in the package directory: an embedded caller whose tree lives at
<job>/tree matches no pattern, and the process cwd means nothing to
a library consumer — the bc build above failed here even though the
tree was staged correctly.
The pointed-at tree is authoritative anyway: its changelog defined
the package, version and series for this build. Resolve its staged
copy outright when it carries a debian/ tree, keep the pattern
search (with the quirks overrides) as a fallback, and hand the
resolved directory to local::build instead of searching again.
The ephemeral guard registers its chroot removal as an interrupt
cleanup hook and, once interrupted, stands down from its own teardown
so the two cannot race umount/rm; bootstrap bails out of tarball
extraction and the lockfile wait, keeping the hook registered on the
bootstrap error path so the watchdog can remove the partial tree. The
live view registers an interrupt reporter that suspends the widget and
returns the log-file hint, silences the tty rendering of the ^C
keypress (the echoed "^C" can wrap near the right edge and shift the
teardown erase by a row, leaving the first widget line on screen) and
kills the shared draw target so late log records cannot repaint the
cleared bars. Failure summaries stay quiet when interrupted: the
captured errors are just the killed children's death throes, and the
dose-builddebcheck diagnosis is skipped for a dependency failure the
user interrupted themselves.
One deb block per package cannot express two series needing different
rules. Make pull and deb lists of entries, each with its own series
scope: every matching entry applies, in file order, so a stonking
entry can carry its own dependency rules next to the resolute one.
Thread the series through find_package_directory for the
package_directory lookup, which keeps its deb-then-pull fallback.
The resolute linux and linux-riscv controls declare llvm-21-dev
unqualified while their other llvm pieces are :native, so the dpkg
cross rules resolve it against the host architecture — whose
dependency closure needs python3:riscv64, conflicting with the
python3 the control itself declares :native. No resolver can install
that set; sbuild fails on it identically.
Give the deb quirks a dependencies rule set rather than a one-off
native-qualification knob: replace rewrites a declared dependency by
name with a full dependency string (qualifier, version and
restrictions included), inject adds dependencies resolved as if
declared, drop ignores declared ones. Entries are scoped by series so
they can be dropped when the upstream packaging catches up, and rule
names that match nothing are warned about so stale quirks surface.
Ship the llvm-21-dev entry for the resolute kernels: resolve it as
llvm-21-dev:native, keeping the declared <!stage1> restriction.
Replace the apt-get build-dep passes with a native resolver: the
Build-* clauses are evaluated against the context's package state by
crate::debian::deps (the dpkg-checkbuilddeps equivalent), and the
unsatisfied ones install through explicitly architecture-qualified
names.
apt's --host-architecture build-dep resolution is coarser than the
checker it feeds: unqualified Multi-Arch: same libraries only ever
get the host-architecture variant, and there is no way to also
satisfy the build-architecture needs of a cross build without a
native re-resolution pass — which installs co-install partners that
break unpacking for -dev packages whose variants conflict on
arch-differing files (curl-config; the libcurl4-gnutls-dev
regression). The resolver now applies the same dpkg rules as the
checker, preferring the runnable build-architecture variant of
Multi-Arch: foreign tools, and passes virtual names through to apt.
Build-Conflicts are now checked before anything installs, which the
build-dep passes never did, and a failed dose3 diagnosis no longer
masks the resolver error (a latent flaw of the passes: binary-only
builds have no .dsc for dose-builddebcheck to read).
An end-to-end test pins the expected failure: a cross build whose
build-dependencies cannot satisfy dpkg's cross semantics aborts,
naming the dependency.
The chlog target-series resolution becomes changelog::series_candidates
(UNRELEASED pinning, development-series default and fallbacks modeled
by SeriesCandidates), PPA references get package_info::split_ppa
(shared by pull and deb, now also rejecting empty parts), and SSH
endpoints get context::ContextConfig::from_endpoint — so a library
consumer can resolve series, validate PPAs and build context
configurations without reimplementing the CLI's rules. All three carry
unit tests; main.rs keeps only parsing of flags and error handling.
The binary build joins the source build on the reporting ports:
build_binary_package takes a DebBuildOptions struct (replacing eleven
positional arguments), reports target, phases, progress and the
outcome through the environment-agnostic BuildView, and the Phase enum
with its default classifiers moves from the terminal widget into the
deb module (announced through the enter_phase helper). DebUi loses its
inherent event methods and only implements the port; tee logging and
the SIGINT behavior are unchanged.
No behavior change for the CLI; headless consumers pass report::Quiet.
Core flows no longer reach into the terminal UI: build_source_package
takes a BuildSourceOptions struct (source tree, domain options, view,
prompter) and reports phases, messages and outcomes through the
environment-agnostic ports in the new report module. The classifiers
move from ui/logfmt to the core logfmt module, DebUi becomes a
BuildView adapter, the re-vendor retry asks the prompter instead of
checking for a TTY, and artifact/success printing moves to the CLI.
Headless consumers pass report::Quiet; an embedding (e.g. a builder
server forwarding events to a web frontend) implements BuildView and
maps the plain-data events onto its own wire format.
The binary build exported DEB_BUILD_OPTIONS='parallel=<context nproc>
nocheck' (or the -j override) but the generated .buildinfo recomputed
the environment from host state: host core count, no nocheck, and
vendor profiles that ignored DEB_BUILD_PROFILES (a cross build recorded
no 'cross' profile). generate_binary_metadata now records the exact env
map that was exported to the build steps, and the recorded profiles
come from the exported DEB_BUILD_PROFILES when set.
Also unifies vendor parsing on one helper (the context-side copy lacked
the Origin: fallback of the source-build path).
Saving the modified source entries with save_legacy rewrote every entry
into /etc/apt/sources.list in legacy format, destroying deb-src entries
and signed-by/trusted options, duplicating every distro entry that came
from a deb822 file (which stayed in place), and hardcoding the Ubuntu
keyring on cross builds. Entries now remember the file and format they
were loaded from and are written back there; new entries (PPAs, ports)
go to a pkh-owned /etc/apt/sources.list.d/pkh-added.list, and a one-time
<path>.pkh-backup copy is made before overwriting an existing file.
Also fixes: 'Types: deb deb-src' stanzas are split instead of being
treated as binary-only, commented-out legacy entries are kept disabled
instead of deleted, debian.sources is actually read on Debian (the old
else-if never fired), and the double blank lines save_legacy emitted.
Without --host-architecture, the second build-dep pass re-resolves the
whole Build-Depends field for the native architecture: apt swaps
host-arch -dev packages for native ones (e.g. libcurl4-gnutls-dev,
whose arch-differing curl-config makes dpkg refuse the co-install) and
breaks the cross build environment.
Per dpkg-checkbuilddeps, both Build-Depends and Build-Depends-Indep
resolve for the host architecture in cross mode, so pass
--host-architecture to the second pass as well. Skip the pass entirely
when the source declares no Build-Depends-Indep.
Add an end-to-end regression test building a package that declares
libdb-dev in both fields and links a host-arch binary against it: the
test only passes if the arch-indep pass did not swap the arm64 -dev
packages for native ones.
build_binary_package_impl copied the whole parent directory into the
build root (ensure_available) and then retrieved every *.deb / *.changes
/ *.buildinfo it found there. That surfaced stale files already sitting
next to the package tree in the "Built in Ns:" summary.
Now local::build returns the exact set of artifacts produced by this
build — the binary packages registered in debian/files plus the
generated .buildinfo/.changes from generate_binary_metadata — and
build_binary_package_impl retrieves that list instead of globbing the
build root. Only files genuinely produced by the current build are
printed.
By default the number of parallel jobs is detected with nproc inside
the build context. Add a -j/--jobs option so an explicit count can be
honored instead, threading it through build_binary_package and
local::build into DEB_BUILD_OPTIONS=parallel=N.
Extend the metadata writers to binary-only uploads and wire them into
the 'pkh deb' flow:
- build/binary.rs generates <pkg>_<ver>_<arch>.buildinfo/.changes
through any Context: debian/files consumption, encounter-order
Architecture accumulation (sorted in .buildinfo like dpkg-genbuildinfo),
sorted Binary lists, dpkg-formatted Description lines with udeb
suffixes, Installed-Build-Depends closure over the context status DB,
and binNMU handling (Source: pkg (prev), Binary-Only-Changes, previous
.dsc redistribution);
- artifact digests are computed inside the context via coreutils
(md5sum/sha1sum/sha256sum/stat) so chrooted/remote trees work;
- deb/local.rs runs the generation after 'rules binary', exports
SOURCE_DATE_EPOCH from the changelog (reproducibility), and resolves
vendor/profiles inside the context; deb/mod.rs retrieves the new
artifacts alongside the debs;
- reusable helpers added: FilesList::parse/render,
parse_changelog_entry_from_str, parse_previous_version_from_str,
installed_build_depends_from_content.
Differential gate: same tree built with real 'dpkg-buildpackage -b' and
with the pkh flow; .changes/.buildinfo compared field-by-field modulo
machine-dependent fields, artifact checksums included.
'proposed' pockets are marked 'NotAutomatic' in their Release file,
which gives them an apt priority of 1: apt would ignore them during
build-dependency resolution even when the suite is enabled.
Write an apt preferences entry pinning '{series}-proposed' at
priority 600 so build-dependencies are actually resolved from the
requested pocket.
Apt preferences are global: a single 'release' pin matches the pinned
suite on every repository carrying it (archive, security and ports),
for all architectures, so this also covers cross-builds pulling
dependencies from 'ports.ubuntu.com'.
- pull: select the pocket to download the source package from
- deb: build against dependencies from a specific pocket:
- local mode: enable '{series}-{pocket}' suite on archive sources
- cross mode: include the pocket suite in required repositories
- quirks: added quirks, that does nothing for now
- deb: install arch-independant dependencies (too much is better)
- deb: added linux-riscv crossbuild test
Multiple changes:
- New contexts (schroot, unshare)
- Cross-building quirks, with ephemeral contexts and repositories management
- Contexts with parents, global context manager, better lifetime handling
- Local building of binary packages
- Pull: pulling dsc files by default
- Many small bugfixes and changes
Co-authored-by: Valentin Haudiquet <valentin.haudiquet@canonical.com>
Co-committed-by: Valentin Haudiquet <valentin.haudiquet@canonical.com>
vhaudiquet