vhaudiquet/pkh
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Compare commits

base: vhaudiquet/pkh:v0.1.0
Branches Tags
vhaudiquet/pkh:main
vhaudiquet/pkh:v0.2.0-rc2 vhaudiquet/pkh:v0.2.0-rc1 vhaudiquet/pkh:v0.1.0
...
compare: vhaudiquet/pkh:v0.2.0-rc1
Branches Tags
vhaudiquet/pkh:main
vhaudiquet/pkh:v0.2.0-rc2 vhaudiquet/pkh:v0.2.0-rc1 vhaudiquet/pkh:v0.1.0
98 Commits
v0.1.0 ... v0.2.0-rc1
Author SHA1 Message Date
vhaudiquet 537bf52656 cli: mark the tree 0.2.0-rc1
CI / snap (push) Successful in 11m56s
Details
CI / build (push) Successful in 6m22s
Details
CI / test (push) Skipped
Details
CI / publish (push) Skipped
Details 
The release-candidate version lands in 'pkh --version' and in the
build records' manifests, so an RC-tested tree is distinguishable
from the final release.
 2026-10-04 13:00:21 +02:00
vhaudiquet 946cd8e642 ci: skip crates.io publishing for prerelease tags 2026-10-04 13:00:21 +02:00
vhaudiquet 9acf650fc9 deb,config: the staged-tree budget replaces the size cap 
The size cap was the wrong shape: an --incremental user asked for a
carried tree, and the invariant that matters is which trees survive —
the newest successful one (the next run's automatic fork base) and the
newest one (what a failed run is re-forked from explicitly). The cap
is gone, with the tree measurement and manifest field it needed; the
retention planner now works on the records' outcomes alone and fills a
budget: staged_trees staged trees per target, the newer ones first,
the pair always among them. When the budget cannot hold the pair (one
staged tree configured, the newest build failed and an older
successful tree present) only one of the two is successful and that
one is kept; with no successful tree at all, the most recent one is.

The budget is a setting of the nested config file:

    deb:
      incremental:
        staged_trees: 2

(default 2; 0 clamps to 1 with a warning — the newest successful tree
always survives). The fork-base selection no longer truncates the
candidate list to the budget before ranking it: with a budget of one
and a failed newest tree, that truncation hid the successful record
and forked the failed one instead.
 2026-10-03 23:43:10 +02:00
vhaudiquet a9a6d2001c deb,config: keep the newest and the newest successful staged trees 
The retention kept the newest five staged trees per target, so the
staged trees of a kernel-scale target alone could hold three-digit
GiB. The policy is now the pair that earns its disk: the newest
staged tree (what a failed run is re-forked from explicitly) and the
newest successful one (what the next --incremental run forks
automatically) — one record when they coincide; every other staged
tree degrades to metadata-only at the next build's retention. The
retention decision moved into a pure planner over (success, size) so
the policy is table-testable.

The size cap no longer drops a tree outright: an --incremental run's
own tree is always kept by its own run — the user asked for a carried
tree, and at least one always exists — and the cap only thins the kept
pair while another member covers the resume, the newest successful
tree being the last one standing. Oversized trees left by older pkh
versions or interrupted builds come down at the next build's
retention, once their lock is free.

The cap moves to the first user settings file, config.yml in
~/.config/pkh (YAML, like the data files; missing file or keys mean
defaults, a corrupt file is backed up, PKH_STAGED_MAXSIZE overrides),
replacing the JSON sketch from the previous commit. Default 20G, 0
disables the size rule. Every retained tree's size is measured once at
retention and recorded in the manifest, so sweeps never re-walk the
trees.
 2026-10-03 21:29:07 +02:00
vhaudiquet 93943dcce0 deb: do not retain staged trees over the retention size cap 
The per-target retention kept the newest five staged trees however
large they are: five carried kernel trees are three-digit GiB, most of
it compiled objects whose resume value does not scale with their size.

A staged tree larger than the retention cap (a user setting, see
below) is now dropped at the end of its own build — the record
degrades to metadata-only and the next build re-stages from scratch —
and the retention sweep also degrades older oversized trees, covering
records kept by older pkh versions and interrupted builds. The cap of
0 disables the size rule; the count cap is unchanged.

The cap is the first setting of a user configuration file,
~/.config/pkh/config.json (the directory already holds contexts.json):
missing file and missing keys mean defaults, a corrupt file is backed
up and the defaults apply, and PKH_STAGED_MAXSIZE overrides the file.
The default is 20G, the scale where a carried kernel tree stops
earning its disk.

Every retained tree's size is measured once at retention (the same
concurrent, mount-crossing-safe walker the prune report uses) and
stored in the record manifest, so sweeps reuse it instead of re-walking
the trees on every build.
 2026-10-03 18:28:12 +02:00
vhaudiquet ed6d88008d test: rebuild a stale repo pkh binary before the e2e run 
resolve_pkh preferred target/release/pkh however old it was: after a
commit without a release rebuild, the harness measured the previous
code — the 54-package run recorded before this change exercised the
pre-nested-namespace builds and recorded no phase profiles, the
profile feature itself being newer than that binary. A repo binary is
now rebuilt when it is older than HEAD or than the newest source edit;
an explicit --pkh still pins whatever binary it names.
 2026-10-03 13:55:13 +02:00
vhaudiquet 2729f9d9e7 context,deb: run the rules as a nested-namespace build identity 
Rootless builds handed the staged tree to the build user with a
recursive chown; through the overlay staging every lower-layer file
the chown touched copy-ups — the full data movement of the tree, 19%
of a kernel build's wall clock.

The rules steps are now flagged and the unshare driver enters the
chroot by pivot_root instead of chroot(2), then creates a nested user
namespace mapping the context's root onto a fixed build id: the
chroot's files are the invoking user's host-side and the context maps
that user to uid 0, so the staged tree is owned by the build identity
by construction — no chown, no data movement. The kernel refuses user
namespace creation to chrooted processes, hence the pivot, and the
bind is recursive and taken inside the namespace: a plain bind of the
chroot's host-made submounts fails EINVAL on locked children, while
the rbind carries them across the pivot, and a fresh /proc is mounted
before the old root is shed (umount resolves through /proc/mounts).

fakeroot's chown wrappers attempt the real chown and swallow only
EPERM; as the mapped identity, root targets are unmapped and fail
EINVAL, so mapped builds export FAKEROOTDONTTRYCHOWN — fakeroot's own
knob — and the fake ownership is still recorded. Files the build
writes come out as the invoking user host-side, which also fixes the
silently missing .buildinfo/.changes of rootless builds: the host-side
rewrite of debian/files hit a subuid-owned file with EACCES.

The wrapper is probed per build; when nested namespaces are
unavailable (old util-linux, userns disabled), the previous setpriv
plus selective parallel chown handover runs unchanged. A setpriv-era
staged tree is handed to the invoking user once, host-side and
metadata-only, before the overlay mounts it — afterwards overlayfs
serves stale owners from its attribute cache.
 2026-10-03 13:00:26 +02:00
vhaudiquet 5544873107 deb: hand the staged tree to the build user selectively and in parallel 
The handover chowned the whole build root with a serial 'chown -R'.
Through the overlay staging every lower-layer file it touched is
copy-up'd — the full data movement of the staged tree — which the
profile of a kernel build showed as 15m53s in 'Preparing the build
user', 19% of the whole run.

The handover is now a 'find ! -uid | xargs -P chown' pipeline: only
entries the build user does not own yet are chowned (staging and quilt
re-application flip entries back to root; a resumed tree is already
owned and the phase collapses to a metadata-only walk), and the
copy-ups of independent files parallelize across the chroot's cores.
On a 200k-file tree the handover drops from 9.9s to 0.9s.

The ccache handover ran only when the build user was fresh, which is
every build in practice (the chroot is per-run scratch); it now rides
the same selective pipeline on every run, still best-effort — a failed
handover costs cache hits, never the build.
 2026-10-03 04:34:20 +02:00
vhaudiquet ae679f29d9 pull,ui,report: render the pull downloads as byte progress
CI / snap (push) Successful in 11m51s
Details
CI / build (push) Successful in 6m18s
Details
CI / test (push) Skipped
Details
CI / publish (push) Skipped
Details 
The pull downloads reported through the plain count port only because
byte_progress landed on main after the branch was cut. PullView now
implements it with a byte-counter template in the transfer-style
family (same shape, {bytes}/{total_bytes} counters, human-formatted by
indicatif), downloads emit through it, and the trait doc lists the
event among the ones the pull flow emits — un-duplicating the merged
'message, progress and progress' sentence.
 2026-10-03 02:28:22 +02:00
vhaudiquet a9f535bce5 Merge branch 'cleanup/errors-deps' 
* cleanup/errors-deps:
  apt/release: type the file-verification and armor errors
  new: type the scaffold validator errors
  debian: type the errors of the format primitives
  deps: add thiserror
  pull,package_info,ui,cli: report the pull flow via BuildView
  deps: trim the tokio feature set to the APIs in use
  deps: drop the lazy_static dependency
  data,apt/release,distro_info,new: replace lazy_static with LazyLock
  deps: drop the unused cmd_lib dependency

# Conflicts:
#	src/ui.rs
 2026-10-03 02:23:11 +02:00
vhaudiquet 82600b1d22 apt/release: type the file-verification and armor errors 
verify_file returns FileVerifyError (no Release file, unlisted path,
checksum mismatch, size mismatch — the structured pieces carried, so
the by-hash retry in package_info can match on the failure instead of
parsing text) and the in-process armor/base64 decoders return
ArmorError; the gpgv staging and keyring-download paths keep their
diagnostic strings, rendered at the boundary. The existing
VerifyError(String) keeps its name and downcast role for suite
signature verification. Display texts are byte-identical to the old
format! strings.
 2026-10-03 02:17:22 +02:00
vhaudiquet a59e99b107 new: type the scaffold validator errors 
The leaf validators of the wizard (source name, command, upstream
version, maintainer, homepage, Depends, orig path, template id, file
collisions) now return a shared options::ValidationError carrying the
structured pieces (the rejected string, the wrapped DebianVersion or
Deps parse failure) instead of pre-formatted Strings; the wizard
resolve flow keeps its aggregate String errors and renders at the
boundary, and the Prompter validator closures adapt with to_string()
since the port's Validator deliberately exchanges re-ask explanations
as plain strings. Display texts are unchanged, as are the tests that
assert on them.
 2026-10-03 02:17:22 +02:00
vhaudiquet 82a15c7426 debian: type the errors of the format primitives 
Give every parser of the shared format layer a proper error enum
(thiserror): VersionParseError, ArchError, deps::ParseError (wrapping
the version and arch failures as #[source] so the context chain
survives), ControlParseError, ChecksumParseError, FilesParseError and
TestsControlParseError, each carrying its structured pieces (the
offending string, the stanza number, the field name) instead of a
pre-formatted String.

Display texts are byte-identical to the old format! strings, so
user-facing output and the message-asserting tests are unchanged; the
few tests asserting on error text now render the error first. Flow
boundaries that still take String (build::env, quirks, test::expand)
render with to_string() at the edge — typing those flows is a later
step. ControlInfo's FromStr impl follows parse_content's error type.
 2026-10-03 02:17:22 +02:00
vhaudiquet 5c7627925f deps: add thiserror 2026-10-03 02:17:22 +02:00
vhaudiquet 8c8ee5beee pull,package_info,ui,cli: report the pull flow via BuildView 
Retire the ProgressCallback option-of-closure, the one pre-port
reporting idiom left in the tree: pull and package_info::lookup now
take &dyn BuildView like every other core flow, and the announcements
map onto message (unsized work), progress (item counts: series probes,
git objects, extraction) and progress again for byte downloads, which
the old callback carried in the same 4-argument shape.

The terminal side swaps the (ProgressBar, closure) pair for PullView, a
BuildView implementation owning one bar of the shared MultiProgress
with the same spinner/transfer style switching; it releases the line on
drop, which the CLI triggers explicitly on both outcomes since
exit(1) skips destructors. Headless callers (tests, internal lookups)
pass &report::Quiet. git clone sideband parsing is skipped entirely
when the view is not enabled, matching the old None-callback shortcut.
 2026-10-03 02:17:22 +02:00
vhaudiquet 9004a33bae deps: trim the tokio feature set to the APIs in use 
The runtime (multi-thread, for Runtime::new + block_on and task
spawning), time (retries, timeouts), sync (channels) and macros
(#[tokio::test]) cover every tokio:: path in the tree; 'full' also
pulled fs, io, net and process support nothing uses. Cargo.lock loses
the corresponding mio/signal deps if nothing else needs them.
 2026-10-03 02:17:22 +02:00
vhaudiquet 4d59890d5a deps: drop the lazy_static dependency 2026-10-03 02:17:22 +02:00
vhaudiquet 239bcb28df data,apt/release,distro_info,new: replace lazy_static with LazyLock 
Unify the lazily-initialized statics on std::sync::LazyLock, which the
newer code already uses: the embed_data! macro expands to a plain
static now, which also removes the visibility-forwarding workaround the
lazy_static! matcher forced on it (the @expand arm wrapping vis in
parentheses), and the 'static ref' spelling is kept at the call sites.
distro_info's HTTP client, the apt keyring cache and the new-template
registry follow the same pattern.
 2026-10-03 02:17:22 +02:00
vhaudiquet 486856da60 prune,ui: show the removal pass, prefix the sizing spinner
CI / build (push) Successful in 6m23s
Details
CI / test (push) Skipped
Details
CI / publish (push) Skipped
Details
CI / snap (push) Successful in 11m45s
Details 
Confirming a 500 GiB prune went silent under "Removing:" for minutes:
drive an indicatif bar from the plan's sizes through a new
apply_with_progress observer, in the deb live view's layout — the
running count behind the spinner on the first line, the byte bar with
its counters on the second. The removed path is not rendered: it
fights the bar on a narrow terminal, and the failed-removal errors
name the paths that matter. --force plans no sizes and gets the count
line only.

The dry-run size spinner lacked the "> " prompt prefix the other
one-line waits carry; wait_spinner renders it too now.
 2026-10-03 01:34:47 +02:00
vhaudiquet f6f2b882a8 deb,report,ui: report the staged-tree copy while a record forks 
Forking an incremental base copies its whole staged tree — bounded,
and without reflinks a matter of minutes — but the copy emitted no
build events: the live view showed the target line with nothing below
it, and an interrupt there printed raw cp errors with no hint of what
was running (the fork happens before the first bootstrap phase owns
the view).

Announce the copy through a new BuildView::byte_progress port. The
fork sizes the source up front (the copy's total), re-measures the
growing destination from a scoped thread while cp runs (cp has no
progress output to tap), and reports the completed copy when it
returns; unmeasurable or empty sources copy unreported. The terminal
view renders it like the other determinate bars, with byte-unit
counters.
 2026-10-03 01:28:30 +02:00
vhaudiquet 83a853d259 utils/size,prune: share the concurrent tree walker and byte formatting 
The incremental fork's staged-tree copy wants to report byte progress
against the same measurements the prune report is built from: move the
concurrent size walk and the human-readable byte formatting out of
prune into utils::size and rewire prune to them. Behavior is unchanged:
same-device walks, unreadable entries count as zero.
 2026-10-03 01:26:45 +02:00
vhaudiquet dc01812847 deps: drop the unused cmd_lib dependency 
No cmd_lib!/run_cmd!/run_fun! invocation remains anywhere in the tree;
the crate shells out through std::process::Command exclusively.
 2026-10-03 01:24:54 +02:00
vhaudiquet 1cc64a37da deb: make apt wait for the shared archives lock 
The shared apt download cache is bind-mounted from outside the
chroot, so when an apt of ANOTHER build holds the archives lock from
its own pid namespace, a concurrent build's apt sees the holder as
"process 0", cannot wait for it, and fails the build immediately
(observed as an e2e samba essentials install racing a kernel build's
build-deps transaction on the stonking-amd64 cache).

Set DPkg::Lock::Timeout to ten minutes in the shared-cache apt conf:
apt then waits for the holder instead of erroring. pkh itself
already serializes builds per cache with a host-side flock; the
timeout covers the window where a killed run's orphaned apt is still
winding down.
 2026-10-02 23:52:56 +02:00
vhaudiquet b85a377b24 prune,ui,docs: color the digest, add -f/--force and a sizing spinner 
- -f/--force skips the size pass and the confirmation prompt: remove
  immediately, reporting counts but no sizes. The fastest cleanup
  path; --dry-run still previews.
- The default path shows the grouped digest and asks before removing
  (the raw-mode yes/no prompt; non-interactive sessions keep their
  headless path, so scripts reach for --force).
- Sizes are computed under a spinner ("Computing sizes of
  directories to prune..."), and the digest colors the counts (bold)
  and the sizes (cyan) on terminals only.
 2026-10-02 23:43:04 +02:00
vhaudiquet 74c74f0a61 prune: size the artifacts concurrently, drop the per-path info lines 
The dry run still spammed one info line per artifact before the
grouped digest, and the size walk was serial: tens of seconds of pure
readdir/stat over the records' hundreds of thousands of files.

The per-path lines drop to debug (the grouped report is the
user-facing output), and the size pass now fans out over a shared
work queue with scoped threads — directory-level parallelism, same
du -x mount-crossing rule. 16s to 6.4s wall on a 440 GiB sessions
tree, totals unchanged.
 2026-10-02 23:29:21 +02:00
vhaudiquet 2991237c91 prune,docs: group the prune report by kind with sizes 
The prune output spammed one line per path. The report now groups
entries by artifact kind with their total size, and the size is
measured before removal whatever the mode, so --dry-run prints a
digest ("24 residual build directories for 1.2 GiB") and both modes
end with the freed total. The size walk never crosses a mount point:
residual chroots hold bind-mounted /proc trees whose virtual files
(/proc/kcore alone reports 128 TiB) would poison the total.

--all is also honest now: it removes recent build records and staged
trees (that is where the space is), and the flag description says so.
One behavior fix rides along: a record or lock a live build holds was
previously prunable with --all, which would wreck a running build;
held locks are now out of bounds whatever the policy.
 2026-10-02 23:22:14 +02:00
vhaudiquet 8485e7272b deb: sync upper entries through the privileged removal path 
A rootless build's upperdir entries are copy-up'd by whoever wrote
them during the build: the unprivileged build user, which host-side
is a subuid of the invoking user. The host-side sync of a forked
record removes the stale upper copy of every host-modified file, and
the plain fs removals failed with EPERM on those build-user-owned
entries: the kernel incremental run died with a bare
'Permission denied (os error 13)' and no context, ten minutes into
the run.

Route the sync removals through the privileged path (the same sudo
fallback the record teardowns use) and wrap the sync error at the
call site, so a failure names the operation instead of surfacing a
bare errno string.
 2026-10-02 23:03:15 +02:00
vhaudiquet f1e8244e73 docs: drop the plans directory 
The specs describe shipped features — incremental builds, the live
view, overlayfs staging, pkh new, pkh test, pkh lint phase 1 — and
the satellite-tools plan is implemented in full (no shell-outs to
dpkg-architecture, dpkg-checkbuilddeps or dpkg-parsechangelog
remain). The two unshipped proposals (the lint porting track and the
data/configuration architecture) stay in whatever notes the project
keeps outside the tree; the README remains the user-facing
documentation.
 2026-10-02 21:31:48 +02:00
vhaudiquet b246b30eac docs: tighten the README prose 
Plain-speech pass: drop the em-dash asides and mid-sentence colons,
reword the boldface digest constructions ("What --incremental buys
is ..."), and replace the fluffy intro ("aims at wrapping ... while
being smarter at integrating all workflows") with what pkh actually
does. No content changes.
 2026-10-02 21:28:21 +02:00
vhaudiquet e4612ca128 deb: give the build-user setup and resume cleanup their own phases 
The profile's build-dependency phase spanned everything between the
apt passes and the build step, and for a kernel incremental build
that hid ~13 minutes of housekeeping: the recursive chown handing
the staged tree to the build user, and the resume_clear quirk
dropping the previous attempt's stamp caches. Both now enter the
profiler (ClearingResumeCaches only when the quirk has paths and a
tree is carried, PreparingBuildUser only for rootless builds), so
the status bar and pkh deb profile show where the time actually
goes — on a large carried tree the chown dominates, which is the
honest answer to a 6m30s DEPS column next to a 19m19s phase.
 2026-10-02 20:39:24 +02:00
vhaudiquet 4535255844 logfmt: collapse the rootless chown EPERM flood into one digest line 
A rules chown failing for the unprivileged build user is expected
and harmless (see the revert of the build-step fakeroot wrap), but a
sysroot tree can print thousands of 'chown: changing ownership of
...: Operation not permitted' lines. MakeClassifier now shows one
warning digest for the flood and hides the rest; every other line,
make errors included, keeps its classification. The full tee in the
record's build.log is untouched.
 2026-10-02 18:42:36 +02:00
vhaudiquet 6050da7dbb Revert "deb: run the rootless build step under fakeroot, sharing the state" 
This reverts commit 5f36a54db42e0e5b8f5243dded89f68cd93fa689.

Fakeroot's IPC degrades under kernel-scale parallel builds: a
linux-riscv build with make -j plus LTO floods the faked daemon and
libfakeroot prints 'internal error: payload not recognized' per
mangled message, with the ownership database losing entries. The
benefit the wrap bought is vacuous anyway: the .deb is assembled in
the binary step's own fakeroot session, which records files unknown
to its database as root-owned — verified with a sysroot package
whose build chown fails: the produced .deb still carries root/root.

A rules chown failing for the unprivileged build user stays expected
and harmless; the visible noise is collapsed at the classifier level
instead (next commit). The bare build step also matches modern dpkg
semantics: with Rules-Requires-Root: binary-targets (pkh's default
export), dpkg-buildpackage itself runs the build targets unprivileged
and fakes root only for the binary targets.
 2026-10-02 18:42:36 +02:00
vhaudiquet b8adb318bd deb: stage the pointed-at tree, not its parent 
The staged tree was the parent of the tree the caller pointed at,
so building a tree sitting in a busy directory dragged every
sibling into the chroot: a /tmp checkout copied the whole host /tmp
(snap sockets, other jobs' scratch), and the build user's chown -R
then failed per foreign-owned file, or aborted the build outright.

Stage the tree itself: its content is the merge point of the build
root, the overlay lower is the tree and the record's upper stacks
on top. The package resolution treats the staged root as the
package when it carries debian/, keeping the pattern search for
nested layouts. The .debs, the upload metadata and the dose3
scratch follow dh_builddeb's .. (the package directory's parent)
instead of assuming the build root, and the host-tree snapshot
covers the tree only. Records staged the old way self-heal: their
snapshots carry parent-relative paths, so the next incremental
run's sync wipes the stale upper entries and rebuilds once.

Siblings are no longer visible to debian/rules — dpkg packaging
never crosses that boundary, and pkh test already staged the tree
alone.
 2026-10-02 17:55:22 +02:00
vhaudiquet 5f36a54e7c deb: run the rootless build step under fakeroot, sharing the state 
The rules run as the unprivileged build user, so an ownership change
they make during the build (a sysroot chown) failed per file with
EPERM, spamming the build view and log and silently not happening.

Wrap the build step in fakeroot like the binary step already is, and
share one session state between the two — the way dpkg-buildpackage
-rfakeroot wraps every rules target: the chown succeeds faked and
carries into the .deb assembly. With the context's own user the
build step stays bare. The state file lives in the staged build
root, outside the package tree, and is removed before each build so
an interrupted previous attempt cannot seed the session.
 2026-10-02 17:36:05 +02:00
vhaudiquet 6967813fa0 cli: drop pkh deb --mode and tighten the deb flag help texts 
Only one build mode exists and the flag only ever accepted its
default 'local'. The --mirror help keeps what a user acts on. The
URL feeds mmdebstrap so every apt resolves from it, and the mirror
must serve every suite and architecture the build resolves.
--incremental says the same things without the asides, and
--no-ccache is one line.
 2026-10-02 15:45:49 +02:00
vhaudiquet 2ab2ff91e9 deb: record per-phase build timings, shown by pkh deb profile 
Time every Phase the build flow announces through enter_phase — the
chroot bootstrap (tarball download, extraction, finalization), the
apt phases, patch application, the build and binary steps and the
artifact retrieval — and land the ordered profile in the record's
manifest of every run, failed builds included: their profile ends
at the phase the error propagated out of.

Timing rides the enter_phase choke point instead of new measurement
sites, so a phase added later profiles itself. The manifest field
is additive (serde default, skipped when absent), so older manifests
keep loading without a schema bump. The coarse timings stay the
source of the deb list columns: they measure the apt transactions
themselves, where the profile measures whole phases.

pkh deb profile renders a record's profile through the same
read-only record discovery as pkh deb log — the newest record of
the tree, or an id prefix as shown by pkh deb list — so
record_for_log becomes the neutral find_record. Phase keys are
storage names, distinct from the display labels; Phase::from_key
turns them back and a test keeps both in sync.
 2026-10-02 14:17:15 +02:00
vhaudiquet 7055d69152 Merge branch 'rootless-builds' 
deb: build the rules as an unprivileged user in pkh chroots, the
same way Launchpad's buildds and sbuild do - setpriv drops to an
on-demand pkh-build user owning the staged tree, the scratch HOME
and the ccache; dependency installation and quilt stay root.
Validated by the e2e canary: coreutils (which refuses a root
configure) now passes.

* rootless-builds:
  test: drop the coreutils xfail, rootless builds clear it
  deb: run the user probes through sh with stderr dropped
  deb: build the rules as an unprivileged user in pkh chroots
 2026-10-02 12:58:44 +02:00
vhaudiquet abc89c4544 test: drop the coreutils xfail, rootless builds clear it 
The e2e canary against the rootless build records coreutils as
xpass: with the rules running as the build user, its configure no
longer refuses the build.
 2026-10-02 12:43:32 +02:00
vhaudiquet 672d17136b deb: run the user probes through sh with stderr dropped 
A missing build user is the expected state in every fresh ephemeral
chroot: without the redirect the id failures leak into the build log
as errors on every run.
 2026-10-02 12:26:46 +02:00
vhaudiquet 7b47a3b32c deb: build the rules as an unprivileged user in pkh chroots 
Launchpad's buildds and sbuild compile as a non-root user, and
packages rely on it: coreutils' configure refuses to run as root
(the one remaining standard-tier e2e failure). Run the rules - and
the fakeroot-wrapped binary step - as an on-demand 'pkh-build' user
whenever the build runs in a context pkh controls (the ephemeral
chroot); ssh and schroot contexts keep their own user.

setpriv drops to the user without touching the environment, so the
dpkg architecture variables and DEB_BUILD_OPTIONS survive the
transition. The staged tree, the scratch HOME and the shared ccache
directory are handed to the user on every build; everything before
the rules (dependency installation, quilt) legitimately stays root.

Minimal mmdebstrap chroots carry no passwd package, so the user is
appended to /etc/passwd and /etc/group directly when useradd is
missing, as sbuild's build-user setup does.
 2026-10-02 12:23:02 +02:00
vhaudiquet decd092871 test: xfail gdb, grub2 and u-boot in the e2e catalog 
All three fail on the current archive, none by pkh's doing: gdb's
PDF dies in texi2dvi against texinfo 7.3; sid's grub2 is the 2.06-2
branch whose control pins gcc-10, which sid no longer ships; and
cross u-boot compiles its kbuild host tools with the arm64 flags
dpkg-buildflags serves in cross mode (-mbranch-protection is amd64
rejected), the same under sbuild cross.
 2026-10-02 12:14:55 +02:00
vhaudiquet d9c11cf0e7 test: xfail binutils in the e2e catalog 
glibc 2.42 turned strstr into a macro with C2x redirect annotations
in string.h, and gprofng's libcollector does not survive it:
iolib.c:898 fails with 'expected identifier before _Generic'. Fixed
upstream after the 2.45 snapshot this rebuild uses - a distro
toolchain change hitting every rebuild, not pkh.
 2026-10-02 11:54:53 +02:00
vhaudiquet 88fcafc274 deb: point the build HOME at a chroot-internal scratch directory 
The build context inherits pkh's environment, so the rules ran with
HOME set to the invoking user's host path - which does not exist
inside the chroot. Tools keying user config on HOME then fail
mid-build: gdb's doc build dies in texi2dvi because the TeX live
setup cannot use the nonexistent home.

Point HOME at /tmp/pkh-home, created through the build context (all
drivers: local, unshare chroot, ssh, schroot). sbuild uses a
non-writable path, Launchpad's buildd user a real one; a writable
scratch directory matches the latter.
 2026-10-02 11:54:53 +02:00
vhaudiquet 0ef0505829 pull: keep self-prefixed multi-orig components in place 
A component tarball whose entries already carry the addon prefix
('www/...' in sqlite3's 'sqlite3_3.53.4.orig-www.tar.xz') extracts
to its end state, but the wrapper detection named it a wrapper and
unwrap_into then cleared the addon target - destroying the component
just extracted into it - and unwrapped the emptied directory into
itself: no 'www' at all, and the deb failed at dh_installdocs over
the missing 'www/*' docs.

dpkg-source unpacks a component into '<root>/<addon>': when the
tarball's own layout already is that, the extraction is the end
state; leave it alone.
 2026-10-02 02:23:22 +02:00
vhaudiquet 14704db777 pull: create the addon directory when unwrapping a multi-orig component 
unwrap_into pointed the component's unwrap at '<root>/<addon>' but
never created that directory: unwrap_wrapper moves the wrapper's
entries into it by rename, and a rename whose target parent is
missing fails with ENOENT - every entry, aborting the whole pull.
The multi-orig unit test exercised unwrap_wrapper directly and could
not see it (dovecot).

Also exercise the component through unwrap_into in the test, and
assert the wrapper's own layout does not leak to the root - the
shape dpkg-source -b rejects as unexpected upstream changes.
 2026-10-02 01:43:54 +02:00
vhaudiquet 5e4b9cb231 fmt: apply rustfmt 2026-10-02 01:28:08 +02:00
vhaudiquet 61b95c0ac1 test: xfail systemd in the e2e catalog 
python3-lxml 6.1 no longer consults custom resolvers for parameter
entities, and systemd's man build leans on exactly that (the
custom-entities.ent resolver in tools/xml_helper.py):
make-directive-index fails with 'Entity entities not defined'. A
distro toolchain change hitting every rebuild, not pkh - the same
breakage forced a backport in Yocto.
 2026-10-02 01:19:45 +02:00
vhaudiquet 3038cdf5e7 deb: export DEB_RULES_REQUIRES_ROOT for the rules 
dpkg-buildpackage normally exports the rules' root requirement from
debian/control; running the rules directly does not, and tools key
their assembly strategy on it - without it debputy refuses to
assemble packages shipping root-owned paths at all (apache2). Policy
defaults to 'binary-targets' when control does not declare it.
 2026-10-02 01:19:33 +02:00
vhaudiquet bb90842ac9 deb: resolve pocket suites and serve Debian security updates 
A changelog may target a pocket suite ('trixie-security') rather than
the bare series: split the pocket off so the build resolves against
the base series with that pocket enabled - the same target an
explicit '-s trixie -p security' names. The pocket must be one of
the base series' distro-info pockets, which also guards against
series names merely containing a dash; an explicit --pocket wins.

Debian serves security updates from a dedicated archive: appending
the security suite to the main mirror would 404 on 'apt-get update'
(ubuntu serves every pocket from the same archive, where appending
works) - clone the pinned source against deb.debian.org/debian-security
instead, as mmdebstrap does.
 2026-10-02 01:19:24 +02:00
vhaudiquet 3c70ccce19 pull: include the patch output when a .diff.gz fails to apply 
patch reports rejected hunks on stdout, not stderr: the error used
to carry only the diff path and an empty stderr, naming neither the
failing file nor the reason.
 2026-10-02 01:18:58 +02:00
vhaudiquet 69bb29a291 pull: ignore the pax global header when detecting a tarball wrapper 
Git-archive tarballs open with a pax global header record
('pax_global_header'): GNU tar consumes it and never lists or
extracts it, but the tar crate yields it as an entry. Counting it
for the wrapper detection made every git-archive orig look
multi-rooted - linux-riscv's orig stayed nested under 'linux-6.8/'
and its .diff.gz failed to apply with 'can't find file to patch'.

Skip the record outright: unpack would no-op on it anyway, so this
only keeps it out of the wrapper detection (and the tree).
 2026-10-02 01:18:48 +02:00
vhaudiquet 3227c0abb9 pull: unwrap multi-orig components into their addon directory 
dpkg-source assembles a multi-orig source by unpacking the main orig
first and the extra components over it, each component's content at
'<root>/<addon>' ('pigeonhole' for 'pkg_1.0.orig-pigeonhole.tar.gz'),
so component files win where they overlap the main tree: process the
archives in that order, and unwrap each archive's wrapper into where
dpkg-source expects it - the root for the main orig, '<root>/<addon>'
for a component, replacing any pre-existing addon directory.

Recognizing the wrapper per archive (the one first component every
entry of THIS archive shares) rather than from the shape of the tree
root keeps every component of a multi-orig package unwrapping after
the first one filled the root: dovecot's pigeonhole sources used to
be left nested one level too deep, and dpkg-source -b then refused
the tree over unexpected upstream changes.
 2026-10-02 01:18:22 +02:00
vhaudiquet b143fa661d docs: document the e2e package harness 2026-10-01 22:24:08 +02:00
vhaudiquet 87f5b31602 test: add the e2e package harness 
Exercises pkh the way it is used in anger: 63 real source packages
(from hello and micro to dracut, grub2 and the kernel) run through
pull -> chlog --rebuild -> build -> deb, each stage timed, logged and
checked for its artifacts, with the catalog alternating between debian
and ubuntu within every category.

run.py keeps runs self-contained (per-package logs, progressive
results.json, a markdown summary), pulls in parallel while builds run
sequentially to keep pkh's shared chroot/apt/ccache caches warm,
resumes interrupted runs reusing the stages that already succeeded,
and marks known-broken entries as xfail with a reason.

The first catalog run caught four pkh defects within an hour - the
unstable/sid suite alias, the bz2 origs, the hard link extraction and
the wrapper unwrapping - which is exactly what it is for.
 2026-10-01 22:24:07 +02:00
vhaudiquet a4be046aaa deb: export the dpkg architecture environment for the rules 
The rules run outside dpkg-buildpackage, which exports none of the
dpkg architecture variables anyway - only sbuild does. Packages using
$(DEB_HOST_MULTIARCH) or $(DEB_BUILD_GNU_TYPE) without including
/usr/share/dpkg/architecture.mk expanded them empty: micro's rules
cd'd into 'obj-' and file configured --libdir=/usr/lib, putting its
libraries where dh_install's multiarch glob could not find them.

Run 'dpkg-architecture -s' in the build context and export the whole
DEB_* set, sbuild-style, for native builds; cross builds already carry
the target's set from cross::setup_environment, whose output parser
this reuses.
 2026-10-01 22:23:55 +02:00
vhaudiquet ca0782488f deb: resolve suite aliases in the target series 
The changelog (and -s) may carry a suite name instead of the series
codename: Debian conventionally writes 'unstable' where the series
data carries 'sid'. The name was used verbatim downstream, so every
NCR rebuild of a Debian package keyed its chroot tarball cache on
'unstable-buildd' (missing the cached sid tarball), tried to bootstrap
a tarball nobody waits for, and finally died on 'Unknown series:
unstable' in the distro-info lookup - with each queued build waiting
out the 300s stale-lock grace on the way.

Resolve the alias right where the series is picked, so the codename
identifies the target everywhere downstream; the codename is also what
the caches and distro-info data key on.
 2026-10-01 22:23:55 +02:00
vhaudiquet 913ef2027a pull: fix archive extraction and tree assembly 
Three archive-side defects broke pulled trees, each surfacing as a
later build failure rather than at pull time:

- '.tar.bz2'/'.tbz2' origs (busybox) were downloaded but never
  extracted: the dispatch only knew gz and xz. dpkg-source -b then
  packed the missing tree as a mass of ignored deletions and the
  chroot's quilt run failed on every patch. Extract bz2 through
  bzip2-rs, and match every '.tar.*' of the source package so an
  unsupported compression fails loudly instead of silently skipping
  the extraction.
- Hard link entries (jq) aborted the whole extraction: resolve their
  targets against the archive root, where tar guarantees the shared
  entry was already extracted, with the same confinement checks as
  regular entries.
- The 'package-version/' wrapper was detected from explicit tar
  directory entries (tree has none, so its sources stayed nested one
  level too deep) and relocated through a copy that flattened
  top-level symlinks (htop's 'README -> README.md' became a plain
  file, which dpkg-source refuses to represent). Recognize the
  wrapper by the shape of the extraction result - exactly one real
  directory - and unwrap it by rename, preserving links and
  timestamps.
 2026-10-01 22:23:43 +02:00
vhaudiquet 2d57b1f0bc deps: add bzip2-rs for bz2 orig tarballs 
Decode-only pure-Rust bzip2, matching flate2's pure-Rust default: no
new system packages for building pkh, and the archive-side origin of
the need is the busybox class of '.tar.bz2' upstream tarballs.
 2026-10-01 22:23:34 +02:00
vhaudiquet 8ab69991ce fmt: apply rustfmt
CI / build (push) Successful in 6m25s
Details
CI / test (push) Skipped
Details
CI / publish (push) Skipped
Details
CI / snap (push) Successful in 12m9s
Details
2026-10-01 16:41:28 +02:00
vhaudiquet ed76b3ef7f deb,build: export ddebs from binary builds
CI / build (push) Failing after 54s
Details
CI / test (push) Skipped
Details
CI / snap (push) Skipped
Details
CI / publish (push) Skipped
Details 
dh_strip registers debug-symbol packages in debian/files under their
renamed .ddeb name; they are deliverables of the build just like the
regular .debs. Both artifact-type filters, the one collecting the
build's artifacts and the one feeding the .buildinfo/.changes
checksum sections, accepted only deb and udeb, so every ddeb a build
produced was silently discarded: nothing exported next to the .debs,
nothing in the upload metadata.

The filter is now a shared FilesEntry::is_binary_artifact() predicate
(plus unit tests) used by both call sites, so the two cannot drift
apart again.

One dpkg distinction has to be copied rather than shared:
dpkg-genchanges lists only deb/udeb packages in the changes' Binary:
and Description (a ddeb travels in Files and the checksum sections
but is not a listed binary of the upload), while dpkg-genbuildinfo
lists every binary artifact's package, ddebs included. The package
list is therefore accumulated twice, once per consumer.

The binary-metadata differential fixture gains a Package-Type: ddeb
package, so the native .changes/.buildinfo generation is compared
against real dpkg with a ddeb present. Without the fix the comparison
fails on the checksum sections (golden distributes the ddeb, ours
dropped it); with it every field matches.
 2026-09-30 14:18:45 +02:00
vhaudiquet 72992f6ec8 deb: create the chroot's device nodes as char devices 
The buildd tarballs carry their /dev entries as char-device members,
but the tar crate cannot create special files: unrecognized entry
types are written out as regular ones, so every device entry in the
chroot extracts to an empty regular file. The old readiness guard
only checked that /dev/null existed, which a regular file satisfies,
so the rm + mknod repair never ran.

Redirects into a regular /dev/null still succeed, so the build
proceeds until something reads the node back. The first reader in
practice is dh_strip's dbgsym path, where dpkg-gencontrol gets
-T/dev/null (dbgsym packages have no substvars file) and aborts on
"bad line in substvars file /dev/null at line 1". Packages without
debug symbols never reach that call, which is why earlier builds
looked fine.

The guard now requires every node to be a char device, and the node
set covers all entries the tarballs ship (console, full, null, ptmx,
random, tty, urandom, zero), not just null and zero: a regular-file
/dev/urandom or /dev/console is a silent hazard for anything reading
it later in the build.

Regression tests pin both shapes: the regular files extraction
leaves behind must not count as ready, and real char devices (via
symlinks to the host's nodes, so the test runs unprivileged) must.
 2026-09-30 13:47:39 +02:00
vhaudiquet d946af8bb6 deb,cli,test,docs: record and show dep-install and build durations 
Every binary build measures its dependency installation (apt-get
update, the essential and injected packages, each build-dependency
pass) and its package build (debian/rules build plus binary) into a
timings accumulator; the record manifest stores both as an optional
`timings` field when the run ends, whatever the outcome — a failed
build keeps its partial durations, and a run that died before any
timed command records nothing.

`pkh deb list` shows the two durations as DEPS and BUILD columns,
formatted like the age ("45s", "2m41s", "1h02m"); records predating
the field show dashes. The field is serde-optional, so the manifest
schema stays at 1 and existing records remain discoverable and
forkable.
 2026-09-30 12:45:13 +02:00
vhaudiquet 6385cf6dc1 deb,cli,test: add --mirror to build and resolve from a pinned mirror 
Pinned runners (a caching proxy, a chosen mirror) must resolve every
package from one base URL. 'pkh deb --mirror <url>' passes the mirror
to mmdebstrap as its mirror positional, so the chroot tarball's apt
sources carry it from the bootstrap on and every apt inside the build
follows - no per-build source rewriting. The mirror must serve every
suite and architecture the build resolves (the security pocket and,
for cross builds, the target archive included); that contract is
documented, not enforced.

The tarball cache is keyed per mirror (a short sha prefix before the
'-buildd.tar.xz' suffix prune matches): a --mirror run over a stale
distro-default tarball would silently resolve from the distro, and
'--incremental' records must never fork across a mirror change.

The runtime source passes recognize the pinned URL as an archive
source (it carries no official host for the host-based checks): the
pocket pass and the universe pass treat it as the archive, and the
cross setup gives it every component and suite, no arch scoping (the
mirror serves every architecture by contract) and no ports entry.
'pkh test' passes no mirror: its tarballs stay distro-default.
 2026-09-30 00:04:50 +02:00
vhaudiquet 8cf9489072 logfmt: teach the make classifier kbuild status tags and warnings 
Kernel builds abbreviate every invocation to an uppercase status tag
(CC, LD, HOSTCC, MODPOST, ...) instead of printing the full command
line, so the substring rules that catch classic make output never
fired and the rolling pane went static for the whole kbuild stream.
Recognize the common tags after leading whitespace and show those
lines.

Also surface warnings instead of hiding them: gcc's lowercase
'file.c:12:3: warning:' diagnostics and kbuild's line-leading
'WARNING:' banners (modpost, etc.) now map to Action::Warning.
 2026-09-29 16:35:30 +02:00
vhaudiquet 2de191cf6d deb: print raw build output again with --verbose 
The record-log wrapper provides a sink for every command, which removed
the runner's subprocess-inherit fallback -- and with it the raw output
of --verbose runs (and of piped stdout, where the live view disables
itself): Quiet's None sink got wrapped inside, every line went to
build.log, and the terminal showed nothing but pkh's own logs, breaking
what --help still promises. When the caller's view has no sink of its
own, a raw terminal sink (stdout/stderr writers, one per stream) now
takes its place inside the record-log wrapper: the terminal shows the
raw lines again while build.log keeps the full tee. A view that
consumes lines itself (the live view on a tty) is wrapped exactly as
before.
 2026-09-29 15:19:17 +02:00
vhaudiquet 562626e878 deb: report a kept record honestly when no staged tree exists 
Every --incremental run claimed "kept its staged tree" and set
kept_upper=true at teardown — even when the run died before staging
created tree-upper, leaving a manifest that lied about the record being
a fork candidate and a message contradicting the disk. The retention
now follows the disk through upper_retention(requested, has_upper()):
a keep with a staged tree keeps it as before, a plain run drops it, and
a keep with nothing staged keeps the record as metadata + log with
kept_upper=false and says so ("no staged tree was created — the run
ended before staging"). The interrupt path records the same honest
state — failed outcome, kept_upper only if a tree exists — and the
SIGINT hook's message no longer claims a tree that is not there.
 2026-09-29 15:05:08 +02:00
vhaudiquet ab0c65dc37 deb: skip unreadable paths while snapshotting the staged parent 
The snapshot walk runs over the whole staged PARENT, which contains
trees unrelated to the build; one unreadable path anywhere under it —
a root-owned sysroot left there by the user's own setup script — made
fs::read_dir or symlink_metadata fail and aborted the whole build
seven seconds in, before any staging could happen. The walk is now
best-effort below its root: unreadable directories and entries lost to
a mid-walk race are warned about and skipped, and only the root itself
being unreadable propagates (the package tree being invisible is a
real error). Absent entries are already the handled case for the
whiteout sync: they are simply untracked.
 2026-09-29 14:57:39 +02:00
vhaudiquet 6a8c301705 deb: integrate ccache for content-keyed compile reuse 
Fresh chroots also mean fresh compiles: nothing survived to make the
recompile-heavy middle of a run cheap. Every deb build now wires a
shared ccache — /var/tmp/pkh/ccache/<slug>/, bind-mounted at the
chroot's /var/cache/pkh-ccache — through a PATH shim rather than CC
environment variables: gcc/g++/cc/c++ symlinks in /usr/lib/pkh-ccache-
shim resolve to the ccache the essentials install, so rules that call
the plain compiler names are cached while explicit exotic compilers
stay untouched. CCACHE_COMPILERCHECK=content is critical — the default
is mtime-based and would never hit across fresh chroots, where every
toolchain's mtime is the extraction's — with CCACHE_BASEDIR=/tmp/pkh-
build and CCACHE_NOHASHDIR=true keying the staged tree's relative
paths, and CCACHE_MAXSIZE=10G (PKH_CCACHE_MAXSIZE overrides).

No mtime clamping anywhere: the tarball extraction preserves mtimes,
so the toolchain and libc headers stay older than carried objects and
make keeps them, while build-dependency reinstalls are newer and make
rebuilds their dependents — ccache turns those into hits, and a real
archive library change is a miss that rebuilds them, exactly the
signal clamping would silence. `pkh deb --no-ccache` skips the shim,
the env and the mount entirely: an escape hatch for rules that
mishandle compiler wrappers (kbuild-style flavour configs).
 2026-09-29 14:39:54 +02:00
vhaudiquet 0e1e37b2e6 deb: share the apt download cache across builds 
With the chroot now per-run scratch, every build re-downloaded the
packages its dependency install needed — the very cost the chroot reuse
used to hide. Every chroot's /var/cache/apt/archives is now a bind
mount over a shared cache at /var/tmp/pkh/apt-cache/<series>-<arch>/
(the chroot's architecture, like the tarball cache), which one run's
installs fill and the next runs reuse: apt keeps downloaded packages
through a dropped conf (APT::Keep-Downloaded-Packages), and an
apt-get autoclean at the end of each run drops the debs the archive no
longer serves. pkh prune removes caches untouched for a week.

The run holds a host-side flock on the cache for its whole lifetime:
apt's own archives lock is acquired inside the chroot's pid namespace,
where a concurrent build's apt shows the holder as "process 0" — an
invisible process it refuses to wait for — so pkh serializes the
traffic itself; concurrent builds of the same target wait for each
other. Any bind-mount failure falls back to the chroot's own cache
with a warning (correct, just slower), and the temp-chroot teardown now
unmounts everything under the chroot before rm -rf, so the removal can
never reach through the cache bind mount and delete the shared cache
itself.
 2026-09-29 14:22:49 +02:00
vhaudiquet dd2634abe3 deb: shrink records to upper, journal and log; keep metadata for every build 
Records carried a full chroot each, and space turned out to be the hard
constraint: five records per target multiplied by a multi-GB chroot does
not fit. The chroot is now per-run scratch — every build extracts a
fresh one from the sha-verified mmdebstrap tarball cache into its
record's chroot/ dir and removes it at the end (success, failure and
interrupt included); resume_chroot, the .pkh-ready probe and
chroot_is_usable are gone, and crashed-run leftovers are swept when the
next run of that identity starts.

What a record keeps instead: manifest.json (renamed, now with the
tarball sha it used, a kept_upper flag and a tree-only journal — the
environment phases rerun in every fresh chroot), a log/build.log full
tee of the build output (kept for EVERY build; plain runs become
metadata-only records, so keep-iff-incremental now applies to the
staged tree alone), and — for --incremental runs — tree-upper/ with its
host-files.list. The fork gate carries a base's staged tree only while
its recorded tarball sha still matches the cache and the changelog
version is unchanged; a fork candidate is any record with a staged tree.

The keep-last-5 cap now applies to staged trees: a sixth retention
degrades the oldest record to metadata-only instead of deleting it.
New `pkh deb log [-f] [<id>]` prints (or follows) a record's build log,
and `pkh deb list` shows the staged-tree state and plain-run records
too. README rewritten around the shrunk model.
 2026-09-29 14:00:34 +02:00
vhaudiquet 81f0d52f85 deb: rework build sessions into immutable per-build records 
The single mutable session per identity loses the last-good state: a
failed run after a good one adopts the session in place and leaves only
the failed state behind, so "continue from the last success" was
impossible. Each build now records an immutable record under
/var/tmp/pkh/sessions/<slug>/<id>/ (chroot, tree-upper, journal,
manifest), and --incremental forks a NEW record — a full copy of the
base's chroot and, at the same changelog version, its staged artifacts
— under the base's flock, never mutating the base itself.

Selection picks the newest successful record of the identity among the
last five usable, falling back to the newest when none succeeded, and
--incremental <id> pins an exact record (failed ones allowed: their
environment may be fine). The lifecycle is strict keep-iff-incremental:
plain runs build in a throwaway record discarded on success, failure
and interrupt alike (the SIGINT cleanup hook follows the same policy),
while --incremental records are always kept. Each identity keeps its
newest five records; pkh prune's 7-day sweep and orphaned-lock cleanup
follow the nested layout.

--resume and --keep are removed (no aliases): their adoption-in-place
and keep-on-success semantics are exactly what the record model
replaces. pkh deb list shows id, outcome, tree version, fork lineage
and age. README rewritten around the record workflow.
 2026-09-29 12:56:29 +02:00
vhaudiquet d65d3264f4 deb: add opt-in --incremental flag for single-flag iterate loops 
Iterating on a package currently needs two flags (--resume --keep):
without --keep a successful build consumes the session, and without
--resume nothing is reused, so incremental iteration never "just
works" opt-in.

--incremental is the single-flag form: with no --resume it adopts the
newest usable session of this tree (like `pkh deb --resume`), and it
always keeps the session on success, so the loop (edit, `pkh deb
--incremental`, edit, ...) reuses the chroot, build deps and objects
and only recompiles what changed. With an explicit `--resume <id>` it
only adds keep.

A plain `pkh deb` is unchanged: fresh session, consumed on success.
 2026-09-29 12:17:05 +02:00
vhaudiquet a3f6b97335 clippy: elide the needless lifetimes of series_patches_to_apply 
CI lints with -Dwarnings; the explicit 'a lifetime on
series_patches_to_apply triggers needless_lifetimes and would fail
the gate.
 2026-09-29 12:12:38 +02:00
vhaudiquet 6bb31ea533 chlog: add -f/--file to target a specific changelog 
The changelog path was hardcoded to debian/changelog in the series
detection, the entry generation and the editor invocation. Packages
keeping the changelog elsewhere (e.g. a staging copy) could not use
chlog on it.

Add -f/--file defaulting to debian/changelog, threaded through all
three uses. The library side was already parameterized on the file,
so no changelog.rs behavior changes beyond a test pinning the
non-default path.
 2026-09-29 12:03:49 +02:00
vhaudiquet babddad505 chlog: cap the no-tag history search with --limit 
Without a version tag, the boundary of a new entry was found by walking
the whole history and diffing each commit's tree against the changelog
path. On large repositories that per-commit tree diff takes 10+ minutes.

Add --limit (default 500, 0 = unlimited) capping that walk: when no
changelog commit is found within the last N commits, the oldest visited
commit becomes the boundary — the version tag probe is skipped there,
since it could jump the boundary arbitrarily far back and defeat the
cap — and a warning explains the fallback. A found version tag still
takes precedence and is unaffected by the limit.
 2026-09-29 12:02:43 +02:00
vhaudiquet e7cad0b1ca deb: apply real patches in single-debian-patch trees 
The f055b70 skip assumed a single-debian-patch tree carries all patch
content ambient in the working tree. A tree that declares
single-debian-patch while listing real quilt patches in
debian/patches/series (e.g. a pkh-scaffolded package that grew patches,
like lite-xl) was built from unpatched sources: the binary silently
lost the patch content, while plain dpkg-buildpackage - and Launchpad,
which never sees debian/source/local-options - applies the series.

Mirror dpkg-source --before-build semantics: apply the series minus the
ambient 'debian-changes[-<version>]' entries, dropping those entries
from the staged series so 'quilt push -a' tolerates the ambient content.
A series holding only the ambient entry keeps skipping, as before
(covered by apply_phase_skips_single_debian_patch_trees); the
classification core is unit-tested in
series_patches_to_apply_classification.
 2026-09-28 14:04:22 +02:00
vhaudiquet d927dc93ab deb: survive crashed sessions and config-blind resumes
CI / build (push) Successful in 3m9s
Details
CI / test (push) Skipped
Details
CI / publish (push) Skipped
Details
CI / snap (push) Successful in 6m46s
Details 
Two failures from resumed kernel sessions:

A resumed build re-runs debian/rules build, but the kernels keep
their wrapper-step stamps in debian/stamps and the flavour config
rule there exports .config from the annotations with no config
prerequisite: the resumed build silently kept the previous
attempt's configuration and a config edit never reached the .deb.
Dropping the stamp cache unconditionally would impose the kernel
packaging's shape on every package, so it is data-driven instead:
packages declare the caches under a resume_clear quirk, and
resumed builds remove those tree-relative paths before the build.
The inner kbuild keeps its own incremental state, so only the
cheap wrapper passes re-run and config-affected objects recompile.

A SIGKILLed build (OOM) leaves its overlay mounts and /proc bind
mount behind, and overlayfs creates root-owned work state inside
the workdir: the next plain build (recording is always on, it
replaces the session of its identity) failed to clear the
leftovers with a permission error. Session clearing now unmounts
everything under each entry at depth (re-reading /proc/mounts,
tolerating mount stacks from consecutive crashes) and escalates
through sudo; when the tree still cannot be cleared, the build
reports it and continues without a session instead of layering
over a half-cleared one.
 2026-09-28 00:05:14 +02:00
vhaudiquet 6bb8ab5016 deb,prune: clean up orphaned session lock files 
Session locks live next to the session root, and failed or
interrupted runs left them behind after the root was gone (seven
stray locks on a used machine). Session removal now unlinks its own
lock file, and prune collects the orphans: by default only the ones
no live build holds (probed with a non-blocking flock), with --all
every one of them. Locks guarding an existing session root are
never touched.
 2026-09-27 23:32:21 +02:00
vhaudiquet 961f6bc234 deb: wipe the staged tree when overlay staging falls back to a copy 
On a resumed session whose overlay mount fails (no overlayfs, mount
error, non-overlay driver), the fallback used to copy the host tree
into the staged tree of the previous attempt: a merge that keeps the
old build stamps and resurrects files deleted on the host — a
silently half-stale build, exactly what resume must never produce.
The driver now reports the failure instead of copying, and the
caller wipes the staged build root and re-stages a fresh copy
(environment resumable, build artifacts discarded).
 2026-09-27 23:32:16 +02:00
vhaudiquet 318bcea341 docs: document pkh test
CI / build (push) Successful in 3m6s
Details
CI / test (push) Skipped
Details
CI / publish (push) Skipped
Details
CI / snap (push) Successful in 6m22s
Details 
Add the DEP-8 runner to the command list and a usage section: default
chroot testbed sharing the deb tarball cache, binary reuse from pkh deb
with in-process rebuilds, the local and vm modes, and the 0/1/2 exit
contract. autopkgtest joins the shell-out list (vm mode) and the
roadmap bullet comes off — the feature exists.
 2026-09-27 21:33:11 +02:00
vhaudiquet 8513805f98 test: point at root in the local-mode apt failure 
A local run fails apt-get update exactly like a local build does when
the user is not root; say so in the error instead of leaving a bare
failure (the chroot modes bootstrap their own root environment).
 2026-09-27 21:33:11 +02:00
vhaudiquet d35ffefacb test: add the differential harness against autopkgtest 
The native runner's trust anchor: on a tree of the caller's choosing
(PKH_TEST_DIFF_TREE, --ignored, deliberate: both runs land on the host
testbed), the harness runs pkh's local mode and the installed
autopkgtest with its null runner over the same tree and binaries, then
fails on any per-test verdict disagreement — pinning the semantics the
DEP-8 documents under-specify (verdict classes, argid names, edge-case
expansions). The comparison itself is pure and always tested.
 2026-09-27 21:33:11 +02:00
vhaudiquet 7790b9452d test: add --mode vm, the autopkgtest escape hatch 
VM virtualization is the one backend pkh does not reimplement: the qemu
runner carries the serial-console protocol, image overlay management
and reboot support, and it is the only one satisfying
isolation-machine/needs-reboot tests. pkh execs the installed
autopkgtest with its qemu runner (--image required, --cpus/--ram-size
passthrough) in its own process group so Ctrl+C reaches the testbed
cleanup, parses the --summary file back into the pkh report model and
maps autopkgtest's exit classes onto the 0/1/2 contract (12/14/16/20
are runtime errors; the raw code rides the JSON report). Pockets, PPAs
and injection are refused in vm mode for now: they would need the
keyring setup inside the image.
 2026-09-27 21:33:11 +02:00
vhaudiquet 15cc19007f prune: garbage-collect pkh test run directories 
Test artifacts land under /var/tmp/pkh/tests (<source>-<series>-<arch>-
<timestamp>/), one directory per run: they are disposable results, not
state worth keeping. Prune removes them on the same 7-day retention as
build sessions, and with --all; the root is overridable through
PKH_TESTS_DIR like the sessions root.
 2026-09-27 21:08:22 +02:00
vhaudiquet 2a9287d942 test,cli: wire pkh test into the CLI 
Argument wiring only, per the report ports: the live view carries
phases and the outcome, the per-test report renders after the run
(text or --json) and decides the exit code; runtime errors exit 2.
--list-tests prints the parsed debian/tests/control and exits without
booting a testbed.
 2026-09-27 21:04:03 +02:00
vhaudiquet 299b5c28c3 test: add pkh test, the native DEP-8 runner for local and chroot 
The DEP-8 run-time contract is one page, so pkh implements it natively
instead of wrapping autopkgtest (see plans/pkh-test.md): the testbed is
a pkh context — chroot mode reuses the deb ephemeral chroots and their
cached mmdebstrap tarballs, local mode runs on the current context —
and the binaries under test are the freshness-guarded pkh deb output
next to the tree, rebuilt through the deb flow when stale or missing.

Per test: restrictions gate on the testbed (isolation-machine,
needs-reboot and container isolation skip with a pointer at --mode vm),
the stanza Depends are expanded (@, @builddeps@, @recommends@) and
installed, then the script or Test-Command runs in the staged tree with
the AUTOPKGTEST_* (and legacy ADT_*) environment, a per-test timeout
and the stderr/exit-77 verdict rules; flaky tests retry once. stdout,
stderr and $AUTOPKGTEST_ARTIFACTS are collected per test under
/var/tmp/pkh/tests/ (pruned separately). --shell/--shell-fail drop
into the testbed for debugging.

pin_pocket and install_injected_packages become crate-internal helpers
shared with the deb flow, which they mirror.
 2026-09-27 21:00:34 +02:00
vhaudiquet d30b179430 report,ui: name the tee log by flow kind 
BuildTarget::source_only carried two states for what is a three-way
distinction now that pkh test exists: replace it with a TargetKind
(Source/Binary/Test) and derive the tee log prefix (build-/deb-/test-)
from it, so test runs get their own log family instead of borrowing
the deb one.
 2026-09-27 20:41:40 +02:00
vhaudiquet 067e0614e6 debian: add the DEP-8 tests-control parser 
Parse debian/tests/control into runnable test descriptions: one test
per Tests: script and per Test-Command value, carrying the shared
Depends/Restrictions/Tests-Directory stanza fields and the autopkgtest
command1 enumeration (test-name overrides advance the counter).
Depends substitution is deliberately left raw: expanding @ and friends
needs the built binary list, so it belongs to the runner.

First piece of pkh test (see plans/pkh-test.md); the runner expands
these stanzas against the deb822 primitives.
 2026-09-27 20:35:01 +02:00
vhaudiquet 1c8f4cf35b docs: document incremental builds and the session design
CI / build (push) Successful in 3m6s
Details
CI / test (push) Skipped
Details
CI / publish (push) Skipped
Details
CI / snap (push) Successful in 6m25s
Details 
README: the build-session workflow (auto-recording, --resume,
'pkh deb list', --keep, prune retention) under a new section.
plans/: the design spec with the decisions taken along the way
(opt-in resume, --keep for the iteration loop, /var/tmp/pkh/
sessions, Ctrl+C keeping the session) and the implementation notes
recording the as-built deviations.
 2026-09-26 11:52:07 +02:00
vhaudiquet 1b71a6814c cli: add deb --resume/--keep and the deb list subcommand 
Wiring for the session feature: --resume optionally takes a session
id (bare --resume adopts the newest session of the tree), --keep
keeps the session after a successful build, and 'pkh deb list'
renders the read-only session table for the current tree. Listing
is a subcommand, not a mode-switching flag: pkh deb with flags
always starts a build.
 2026-09-26 11:51:55 +02:00
vhaudiquet aaa6723d2f prune: garbage-collect resumable build sessions 
Sessions accumulate under /var/tmp/pkh/sessions for every failed,
interrupted or --keep build and can reach tens of gigabytes for
kernel-sized chroots: prune is their primary GC. By default remove
the sessions untouched for longer than the retention window (7
days) plus the corrupt leftovers; --all removes everything. Session
roots hold the same mounts as residual chroots (/proc bind mount,
overlays), so they are unmounted before removal.

prune_in() (the testable core) deliberately stays session-free:
the test suite runs prune tests and e2e builds concurrently, and
scanning the real sessions root from the tests deleted live
sessions mid-build. Session pruning is an explicit opt-in of
prune_in_roots(), used by the production prune().
 2026-09-26 11:51:46 +02:00
vhaudiquet 8ff423ffb8 deb: add resumable build sessions 
Every local build records a session under /var/tmp/pkh/sessions:
the bootstrapped chroot, the installed build dependencies, the
phase journal and the staged tree with a persistent overlay
upperdir. Rebuilding after a failure currently redoes the tarball
extraction, apt update, build-dep resolution and the whole compile;
a kernel-sized package loses half an hour per iteration.

Recording is always on, reuse is opt-in (--resume [<id>]): the
newest session of the tree, or the one matching an id as shown by
'pkh deb list' (one session per series/arch/cross identity, ids
are build-start timestamps). Resume skips the chroot bootstrap
(integrity marker + tarball check), re-mounts the previous
upperdir so make recompiles only what changed, pops the quilt
series first when debian/patches changed, and clears debian/files
before re-packaging so artifact collection stays exact. A version
bump keeps the environment and discards the build artifacts.

Explicit selectors disagreeing with the adopted session are an
error, never a silent environment switch. Failures and Ctrl+C keep
the session (the interrupt hook unmounts but preserves the tree); a
success consumes it unless --keep. Concurrent same-identity builds
serialize on a lock file kept OUTSIDE the session root: teardown
removes the root while holding the lock, and a lock inside it would
be deleted under the holder, letting the next opener lock a fresh
inode. The apt phases rerun on resume (idempotent, seconds-cheap)
rather than being stamp-gated.
 2026-09-26 11:51:06 +02:00
vhaudiquet 0f1446ad4d context: stage trees over a caller-supplied overlay upperdir 
Resumable build sessions need to re-mount the staged tree over the
same overlay upperdir on every attempt: the build artifacts written
inside the chroot (object files) live there, while the host tree
stays visible as the live lowerdir. Add an optional
ensure_available_with_overlay staging path (OverlayStaging reports
whether the persistent upperdir was actually mounted or a fresh copy
was staged instead) with a default implementation rejecting the
request, so drivers without overlay support degrade cleanly.

The overlay mount itself is factored out of try_overlay_mount into
mount_overlay, taking optional pinned upper/work directories; a
failed mount only cleans up the directories it created, never the
caller-pinned ones.
 2026-09-26 11:50:39 +02:00
vhaudiquet 02415fa450 pkh: move version to 0.2.0 2026-09-26 11:49:08 +02:00
vhaudiquet 46407375d0 deb/cross: stop exporting PKG_CONFIG_LIBDIR for cross builds
CI / build (push) Successful in 3m13s
Details
CI / test (push) Skipped
Details
CI / publish (push) Skipped
Details
CI / snap (push) Successful in 6m15s
Details 
dpkg-buildpackage and sbuild export no pkg-config redirection: the
environment is the package's to set.  Pointing the whole build at the
target multiarch pkgconfig dirs makes every pkg-config-consuming tool
resolve against the target libraries, the host-side tools of the same
build included: the host linker is then handed target-arch -L paths
(the kernel's resolve_btfids logging 'skipping incompatible
/usr/lib/aarch64-linux-gnu/libelf.so') and only survives through
fallbacks when the build-architecture libraries happen to be installed
too.

Packages whose cross builds need target pkg-config data must arrange
for it themselves, the way the kernel packaging builds rtla statically
against no target system libraries.
 2026-09-25 20:54:06 +02:00
vhaudiquet ce19c5648b quirks: inject the kernel's host-tool libraries as :native for cross 
The resolute and stonking kernels declare libelf-dev, libdw-dev and
libssl-dev unqualified, unlike the Debian control which carries the
same names qualified :native for the host-side tools (resolve_btfids,
gendwarfksyms, sign-file).  The dpkg cross rules resolve an unqualified
Multi-Arch: same name against the host architecture only, so a cross
build installs no build-architecture variants and the kernel's
host-side tools cannot link (-ldw, -lelf, -lssl all fail).

Inject the :native variants for the affected series — the declared
unqualified dependencies stay, so the target-side tools keep their
host-architecture libraries.  Series-scoped so the entry can be
dropped once the control is fixed upstream.
 2026-09-25 20:53:10 +02:00
vhaudiquet 99c6f63cd4 deb: pin the cross variant choice of M-A:same build-deps in tests 
An unqualified Multi-Arch: same build-dependency resolves to the
host-architecture variant only, even when a build-architecture
candidate exists: that is what dpkg's checker accepts (KnownFacts
accepts foreign, host and all instances) and what sbuild installs,
whose dummy package carries the parsed Build-Depends unqualified and
whose sbuild-cross-resolver only filters foreign M-A:foreign and
Essential packages out of the apt universe — verified by resolving a
sbuild-style dummy:arm64 against the resolute/stonking indexes, which
yields libelf-dev:arm64 & co and never the amd64 variants.

The build-architecture side of such a library is only ever installed
through an explicit :native declaration.  Pin both rules so a resolver
regression towards dual-installing unqualified dependencies cannot
reappear unnoticed.
 2026-09-25 20:53:02 +02:00
vhaudiquet 4b52f8e9e9 docs: add crates.io installation instructions
CI / build (push) Successful in 3m1s
Details
CI / test (push) Skipped
Details
CI / publish (push) Skipped
Details
CI / snap (push) Successful in 6m26s
Details 
pkh v0.1.0 is published to crates.io, so cargo install pkh is now the
primary installation path; building from source stays as the
alternative. Drop the 'no distribution channel' notice.
 2026-09-24 22:19:24 +02:00
60 changed files with 15655 additions and 1164 deletions
Expand all files Collapse all files
.github/workflows/ci.yml
+4 -2
View File
@@ -124,11 +124,13 @@ jobs:
if-no-files-found: error
publish:
# Publishes the crate to crates.io on a v* tag. Trusted publishing
# Publishes the crate to crates.io on a release tag (a v* tag without
# a prerelease suffix: v0.2.0 publishes, v0.2.0-rc1 does not — release
# candidates are never published). Trusted publishing
# (OIDC) is GitHub-Actions-only, so authentication goes through a
# crates.io API token stored as the CARGO_REGISTRY_TOKEN secret,
# scoped to the pkh crate.
if: startsWith(github.ref, 'refs/tags/v')
if: startsWith(github.ref, 'refs/tags/v') && !contains(github.ref_name, '-')
needs: build
runs-on: ubuntu-latest
container:
Cargo.lock
Generated
+64 -109
View File
@@ -94,6 +94,12 @@ version = "0.22.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6"
[[package]]
name = "base64"
version = "0.23.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ac07cdecf99051d9a5238b80f35af32cdeba5b336e55d957b318b50137e18da5"
[[package]]
name = "bitflags"
version = "1.3.2"
@@ -150,10 +156,20 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04"
[[package]]
name = "cc"
version = "1.4.6"
name = "bzip2-rs"
version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a3eb0f42d6c360dc3f8a821f6bf2fdea7f72bfd36b3076eb0e6d1e9e0752fff4"
checksum = "beeb59e7e4c811ab37cc73680c798c7a5da77fc9989c62b09138e31ee740f735"
dependencies = [
"crc32fast",
"tinyvec",
]
[[package]]
name = "cc"
version = "1.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f360145194ee8e21db5ee7f3fcd4fe52210864c75c985dae33218202c8bbe040"
dependencies = [
"find-msvc-tools",
"jobserver",
@@ -217,30 +233,6 @@ version = "1.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1c133bc6a41be0d194c306b5506d15e6feeea7b1d6604bd3f8310dfb2ca96486"
[[package]]
name = "cmd_lib"
version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dc93f60133215cad2ef47e47db5a386c78874a2630c2c2be68f0f543b3ce0c2c"
dependencies = [
"cmd_lib_macros",
"env_logger",
"faccess",
"log",
"os_pipe",
]
[[package]]
name = "cmd_lib_macros"
version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bd39b5c2657c1a0064aa5d73636dac4a217edeeb6c5f8a5f52c123a758462ece"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
]
[[package]]
name = "colorchoice"
version = "1.0.5"
@@ -471,13 +463,12 @@ checksum = "34aa73646ffb006b8f5147f3dc182bd4bcb190227ce861fc4a4844bf8e3cb2c0"
[[package]]
name = "encoding_rs"
version = "0.8.41"
version = "0.8.42"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7b5ef0006ac9ab233c38522f5ae99cae3625151de8f706cacee1cba4b8e2832a"
checksum = "8e985e0451871ad22fb8d2b6b076e2028a502a0d3950998c2c5c0a4f9b5d9679"
dependencies = [
"cfg-if",
"core_detect",
"multiversion",
"multiversion_no_op",
"rustversion",
"scopeguard",
@@ -523,17 +514,6 @@ dependencies = [
"windows-sys 0.61.2",
]
[[package]]
name = "faccess"
version = "0.2.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "59ae66425802d6a903e268ae1a08b8c38ba143520f227a205edf4e9c7e3e26d5"
dependencies = [
"bitflags 1.3.2",
"libc",
"winapi",
]
[[package]]
name = "fastrand"
version = "2.5.0"
@@ -552,9 +532,9 @@ dependencies = [
[[package]]
name = "find-msvc-tools"
version = "0.1.12"
version = "0.1.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3e0f1c7c3a72c66fd80abe965175f7523475c0489a87d3ff9d6e8c87d87a9d2d"
checksum = "aedcfb3409746eddb02b9e19ebda1c3394f759a152e48ee875a0844d1b955484"
[[package]]
name = "flate2"
@@ -863,9 +843,9 @@ dependencies = [
[[package]]
name = "hyper-rustls"
version = "0.27.9"
version = "0.27.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "33ca68d021ef39cf6463ab54c1d0f5daf03377b70561305bb89a8f83aab66e0f"
checksum = "dfa8e654703247911e29c23fbeaa261834bd9bb74efba2f9acddc37bfb127f53"
dependencies = [
"http",
"hyper",
@@ -894,16 +874,17 @@ dependencies = [
[[package]]
name = "hyper-util"
version = "0.1.20"
version = "0.1.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0"
checksum = "ddc03d96684f9226b8a787cdb71488417b53ab5ea8fdb1dac946cb9431cc8bff"
dependencies = [
"base64",
"base64 0.23.1",
"bytes 1.12.1",
"futures-channel",
"futures-util",
"http",
"http-body",
"httparse",
"hyper",
"ipnet",
"libc",
@@ -1154,9 +1135,9 @@ dependencies = [
[[package]]
name = "js-sys"
version = "0.3.105"
version = "0.3.106"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ce57d20d1ea864ce2ac172ab472d409214f4fd359f0b2a2775abdf522e2af99e"
checksum = "7883d941dae510fb2d978fc3fe018c71c9e2892fd38854de3e8b92c2e5ad9cc5"
dependencies = [
"cfg-if",
"futures-util",
@@ -1225,9 +1206,9 @@ dependencies = [
[[package]]
name = "libredox"
version = "0.1.24"
version = "0.1.25"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6480ccc157a1389bb2e4891b24751b0f798ba640d22386f23143fbcc89da195a"
checksum = "61ff90caf6077a803a240f62fdbe88645a890bbca49ef8174c3cb0404362171d"
dependencies = [
"libc",
]
@@ -1364,27 +1345,6 @@ dependencies = [
"windows-sys 0.61.2",
]
[[package]]
name = "multiversion"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b4ca4bea16ffc3f443cf7d866912118196bfef4c6a1556ca00f9f9b00bb43f7c"
dependencies = [
"multiversion-macros",
]
[[package]]
name = "multiversion-macros"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0d416831a7317ef4b08bee00b69cbbb9c8763da7959a7026244d6266869f9c83"
dependencies = [
"proc-macro2",
"quote",
"rustversion",
"syn 3.0.6",
]
[[package]]
name = "multiversion_no_op"
version = "1.0.0"
@@ -1493,16 +1453,6 @@ version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "04744f49eae99ab78e0d5c0b603ab218f515ea8cfe5a456d7629ad883a3b6e7d"
[[package]]
name = "os_pipe"
version = "1.2.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7d8fae84b431384b68627d0f9b3b1245fcf9f46f6c0e3dc902e9dce64edd1967"
dependencies = [
"libc",
"windows-sys 0.61.2",
]
[[package]]
name = "parking_lot"
version = "0.12.5"
@@ -1546,11 +1496,11 @@ checksum = "f6b464fbc74e149a392436b17d523f769e057cb6877f6a5c4618bc6f11800548"
[[package]]
name = "pkh"
version = "0.1.0"
version = "0.2.0-rc1"
dependencies = [
"bzip2-rs",
"chrono",
"clap",
"cmd_lib",
"crossterm",
"csv",
"directories",
@@ -1562,7 +1512,6 @@ dependencies = [
"hex",
"indicatif",
"indicatif-log-bridge",
"lazy_static",
"libc",
"log",
"md-5",
@@ -1580,6 +1529,7 @@ dependencies = [
"tar",
"tempfile",
"test-log",
"thiserror",
"tokio",
"unicode-width",
"xz2",
@@ -1687,7 +1637,7 @@ version = "0.12.28"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "eddd3ca559203180a307f12d114c268abf583f59b03cb906fd0b3ff8646c1147"
dependencies = [
"base64",
"base64 0.22.1",
"bytes 1.12.1",
"encoding_rs",
"futures-channel",
@@ -2038,9 +1988,9 @@ checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5"
[[package]]
name = "smallvec"
version = "1.16.1"
version = "1.16.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ba467056f1b547ed52077911161fc86985becbc60e8e1857c8a144dab0def891"
checksum = "f9395f0f0eee849a9b707b2f06bb92a6a422090e2123bb2ef8e87a0e61892a8e"
[[package]]
name = "socket2"
@@ -2090,9 +2040,9 @@ checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292"
[[package]]
name = "suppaftp"
version = "12.0.1"
version = "12.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ba93b140f4d07f1b3f3fb4d7264b2d422d5c8b4e925eb6a2646e1342d5d2070a"
checksum = "d79f066c76927a775d61b5642e0b712a17582ebe7f62b7e5abba246e619878f7"
dependencies = [
"chrono",
"lazy-regex",
@@ -2240,18 +2190,18 @@ dependencies = [
[[package]]
name = "thiserror"
version = "2.0.20"
version = "2.0.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ec86235f5fcc2a73650310756d2ac5b138a5780bbbdfae3eeccec992c435ba4f"
checksum = "09e52cb86a36cede5cb101bf8908837b3e4c6e5e59fe7fd85c23fb56200d189e"
dependencies = [
"thiserror-impl",
]
[[package]]
name = "thiserror-impl"
version = "2.0.20"
version = "2.0.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bc04cd3e1236dd4a98afca4569f2deb3f120e5422a4023be2cb683f8486292af"
checksum = "fe5197923287db20a58125f0bc85c062f7f2c892de97b18c356f9efb14b28524"
dependencies = [
"proc-macro2",
"quote",
@@ -2277,6 +2227,12 @@ dependencies = [
"zerovec",
]
[[package]]
name = "tinyvec"
version = "1.13.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fd3ca314f692efd6c868f8408f53fe444634a845f96c028b97d35f6a1f79f0ee"
[[package]]
name = "tokio"
version = "1.53.1"
@@ -2286,9 +2242,7 @@ dependencies = [
"bytes 1.12.1",
"libc",
"mio",
"parking_lot",
"pin-project-lite",
"signal-hook-registry",
"socket2",
"tokio-macros",
"windows-sys 0.61.2",
@@ -2584,9 +2538,9 @@ checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b"
[[package]]
name = "wasm-bindgen"
version = "0.2.128"
version = "0.2.129"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "aecb87a33d3b0c5e3b7aa46336eaf486cffafbd281b195e4c8b80d50df2351bf"
checksum = "9bb54f33acc68fd454578d9820b0bde1a1a3d17aa17bb7b6595806d02886d409"
dependencies = [
"cfg-if",
"once_cell",
@@ -2597,19 +2551,20 @@ dependencies = [
[[package]]
name = "wasm-bindgen-futures"
version = "0.4.78"
version = "0.4.79"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6ef4c5d3d2cdf5c54f4231181768f5510842e350db025faf1f7163b1030ed928"
checksum = "3cbab34de2d982e9b48e18d216d04c4a6f641066ff19ffb699980f591ee3610e"
dependencies = [
"js-sys",
"tokio",
"wasm-bindgen",
]
[[package]]
name = "wasm-bindgen-macro"
version = "0.2.128"
version = "0.2.129"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a690d511e3c1a8b3a55e33511e3c2c00c78415cd23650f32b808627f5696b9ed"
checksum = "2e29d0c35b16e224a7eeb5cd2d25e3e1968fbd65604117b44d3b789d00ee8535"
dependencies = [
"quote",
"wasm-bindgen-macro-support",
@@ -2617,9 +2572,9 @@ dependencies = [
[[package]]
name = "wasm-bindgen-macro-support"
version = "0.2.128"
version = "0.2.129"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "411e4887f0071ef2d2164a9d5fdf2d20efbef78fccd3a78b0c10a1dc5295e48a"
checksum = "6f501a8bc3719dba86ef8ae4728879c08001bea749eb1333ac5b91e040e2a6b7"
dependencies = [
"bumpalo",
"proc-macro2",
@@ -2630,9 +2585,9 @@ dependencies = [
[[package]]
name = "wasm-bindgen-shared"
version = "0.2.128"
version = "0.2.129"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "81941cd78d0c92026c33e5e01312845a4cb1e9af3407f9134b100dd03144103e"
checksum = "23f0c9c52aa7cd7d77769a4cfe2a9adb1b331f489a41d912ce14513d5ab995c6"
dependencies = [
"unicode-ident",
]
@@ -2652,9 +2607,9 @@ dependencies = [
[[package]]
name = "web-sys"
version = "0.3.105"
version = "0.3.106"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9fbddc4a036f00ec4f18c83445bd3115cb306a91da554919a099d9222fe4a7f8"
checksum = "88261b9deccee56594c11a3460c462c41f58d148598fe70ad77070126a68aba4"
dependencies = [
"js-sys",
"wasm-bindgen",
Cargo.toml
+4 -4
View File
@@ -1,6 +1,6 @@
[package]
name = "pkh"
version = "0.1.0"
version = "0.2.0-rc1"
edition = "2024"
authors = ["vhaudiquet"]
description = "pkh is a packaging helper for Debian/Ubuntu packages"
@@ -10,7 +10,6 @@ readme = "README.md"
[dependencies]
clap = { version = "4.5.51", features = ["cargo"] }
cmd_lib = "2.0.0"
crossterm = "0.28"
flate2 = "1.1.5"
serde = { version = "1.0.228", features = ["derive"] }
@@ -20,7 +19,7 @@ reqwest = { version = "0.12.9", features = ["blocking", "json", "stream"] }
git2 = "0.20.2"
regex = "1"
chrono = "0.4"
tokio = { version = "1.41.1", features = ["full"] }
tokio = { version = "1.41.1", features = ["rt-multi-thread", "macros", "time", "sync"] }
sha2 = "0.10.8"
sha1 = "0.10"
md-5 = "0.10"
@@ -37,10 +36,11 @@ directories = "6.0.0"
ssh2 = "0.9.5"
gpgme = "0.11"
serde_yaml = "0.9"
lazy_static = "1.4.0"
unicode-width = "0.2"
parking_lot = "0.12"
suppaftp = "12"
bzip2-rs = "0.1.2"
thiserror = "2"
[dev-dependencies]
test-log = "0.2.19"
README.md
+173 -8
View File
@@ -6,7 +6,13 @@
## Installation
No distribution channel is published yet; build from source:
From crates.io:
```
cargo install pkh
```
Or build from source (the same system packages are needed either way):
```
sudo apt install pkg-config libssl-dev libgpg-error-dev libgpgme-dev
@@ -16,20 +22,21 @@ cargo install --path .
```
At runtime pkh shells out to the Debian packaging toolchain (git,
dpkg-dev, quilt, mmdebstrap, lintian, pristine-tar, ...): install the
ones your workflows use, or build the classic snap from
dpkg-dev, quilt, mmdebstrap, lintian, pristine-tar, autopkgtest, ...).
Install the ones your workflows use, or build the classic snap from
`snap/snapcraft.yaml` (`snapcraft pack`), which carries them.
## Usage and features
### Basic concepts
`pkh` aims at wrapping the different debian tools and workflows
`pkh` wraps the different Debian tools and workflows
(`git`, `git-ubuntu`, `pull-debian-sources`, `pull-lp-sources`, `pull-ppa-sources`,
`dch`, `dpkg-buildpackage`, `sbuild`, `dpkg-source`, `quilt`, ...) into one tool, that
would have the same interface for everything, while being smarter at integrating all workflows.
`dch`, `dpkg-buildpackage`, `sbuild`, `dpkg-source`, `quilt`, ...) into
one tool with the same interface for everything, which also handles each
workflow's specifics.
Thus, `pkh` uses similar options for all subcommands (with very few command-specific options):
The same options work across subcommands (with very few command-specific options):
```
Options:
-s, --series <series> Target package distribution series
@@ -49,6 +56,7 @@ Commands:
build Build the source package (into a .dsc)
put Upload the built source package to a PPA
deb Build the source package into binary package (.deb)
test Run the package's DEP-8 as-installed tests (autopkgtests)
lint Lint the package (lintian wrapper + pkh-native checks)
prune Prune residual pkh build artifacts and caches
help Print this message or the help of the given subcommand(s)
@@ -105,11 +113,168 @@ pkh put --ppa user/hello_xxx
git push xxx user-fork
```
`pkh chlog` edits `debian/changelog` by default; point it at another file
with `--file <path>` (`-f`).
`pkh chlog` walks the commit history back to the previous changelog entry.
Without a version tag, that search is capped at the last 500 commits
(`--limit <n>`, `0` = unlimited): beyond the cap, the entry simply starts
at the oldest commit searched, which keeps the run fast on large histories.
### Incremental builds (build records)
Every `pkh deb` build records its metadata under
`/var/tmp/pkh/sessions/<target>/<id>/`: the manifest (identity, fork
lineage, outcome, the sha of the chroot tarball it used) and a full
`log/build.log` tee of the build output. The chroot itself is per-run
scratch, extracted fresh from the sha-verified mmdebstrap tarball cache
on every run and removed at the end, plain and `--incremental` alike,
so records stay small. `--incremental` keeps the record's staged tree
(`tree-upper/`), the overlay upperdir where the build's writes land,
holding the compiled objects.
Fresh chroots share their downloads. Every chroot's
`/var/cache/apt/archives` is a bind mount over a shared apt download
cache at `/var/tmp/pkh/apt-cache/<series>-<arch>/`, which the installs
of one run fill and the next runs reuse (apt is configured to keep the
downloaded packages, and an `apt-get autoclean` at the end of each run
drops the debs the archive no longer serves). `pkh prune` removes the
caches untouched for a week.
Fresh chroots share their compiles too. Every run's chroot
gets a shared ccache (at `/var/tmp/pkh/ccache/<target>/`, mounted at
`/var/cache/pkh-ccache`) wired in through a compiler PATH shim that
resolves `gcc`, `g++`, `cc` and `c++` to ccache, with content-keyed
cache checks (`CCACHE_COMPILERCHECK=content`), so cache hits survive
the fresh toolchain of a new chroot. The cache is capped at 10G per
target (override with `PKH_CCACHE_MAXSIZE`); `pkh deb --no-ccache`
bypasses the shim and cache entirely, an escape hatch for rules that
mishandle compiler wrappers (kbuild-style flavour configs).
`--incremental` forks a copy of a previous record and never modifies
the base, so a failed run cannot destroy the last-good state it was
built from. The staged tree is carried only when the base's recorded
tarball sha still matches the cache (the toolchain did not move) and the
changelog version is unchanged; otherwise the fork builds from a fresh
tree:
```
pkh deb # plain run: fresh chroot, fresh tree; the
# record keeps only metadata and the log
# (nothing is kept even on failure or
# interrupt)
pkh deb --incremental # fork the newest successful record of the
# target (pkh keeps the newest and the newest
# successful staged trees) and keep this run's
# staged tree, for the next fork to carry
pkh deb --incremental <id>
# fork that exact record (see `pkh deb list`);
# a failed record is fine if its staged tree is
pkh deb list # the records of this tree: id, outcome,
# version, fork lineage, staged-tree state,
# dep-install and build times, age
pkh deb log [-f] [<id>] # print a record's build log; -f follows a
# live build
pkh deb profile [<id>] # a record's per-phase timing profile: how
# long every step took (chroot bootstrap,
# apt phases, patch application, build-user
# setup, build and binary steps, artifact
# retrieval), with their shares of the total
pkh deb --no-ccache # build without the shared compile cache
```
Each target keeps two staged trees by default: the newest one and the
newest successful one — the tree the next `--incremental` run forks
automatically, and the tree a failed run can be re-forked from
explicitly — with any newer trees filling the budget. An
`--incremental` run always keeps its own tree, so at least one staged
tree exists even when the newest build failed; every other staged tree
degrades to a metadata-only record (manifest and log stay). The
budget is a setting:
```yaml
# ~/.config/pkh/config.yml (missing file or keys = defaults)
deb:
incremental:
staged_trees: 2
```
`pkh prune` removes records older than a week. `pkh prune --dry-run` previews everything it would remove,
grouped by kind with sizes; `--all` ignores the age and removes
everything a live build does not hold, recent records and staged trees
included; `-f/--force` skips the size pass and the confirmation
prompt, removing immediately. `pkh deb --incremental` never forks a
record built for a different series/architecture.
Packages whose `debian/rules` keeps an input-untracked stamp cache (the
kernels' `debian/stamps`) declare it in the quirks data
(`resume_clear`). Forked builds drop the cache, so the wrapper steps,
including the kernel's flavour config export, re-run over the
incremental inner builds and a config edit is never silently ignored.
A build killed hard (OOM) can leave its chroot behind; the next run of
that target sweeps it (unmounting and escalating through sudo when
needed), and `pkh prune` collects the rest.
On a pinned runner, a machine that must resolve every package from one
mirror (typically a caching proxy), `pkh deb --mirror <url>` builds the
chroot and resolves every package from that base URL. The mirror is
passed to mmdebstrap at bootstrap, so the chroot tarball's apt sources
carry it and every apt inside the build follows. The mirror must serve
every suite and architecture the build resolves (the security pocket
and, for `--cross` builds, the target architecture's archive included);
PPAs and other third-party sources are left alone. The tarball cache is
keyed per mirror, so pinned and unpinned builds keep separate tarballs
(and `--incremental` records never fork across a mirror change).
### Running the DEP-8 tests
`pkh test` runs the package's as-installed tests (`debian/tests/control`,
declared by `Testsuite: autopkgtest` in `debian/control`) through a
native runner on a pkh testbed, and reports one verdict per test:
```
pkh test # test this tree: builds it first if needed
pkh test --mode local # on the current context, no isolation
pkh test --mode vm --image img.qcow2 # in a VM, via autopkgtest
pkh test --json --fail-on skip # CI-shaped
pkh test --list-tests # what does this package test?
```
By default the tests run in an ephemeral unshare chroot bootstrapped
from the same cached tarballs as `pkh deb`, against the binaries pkh deb
just built next to the tree (stale or missing output is rebuilt first;
`--no-build` refuses, `--debs` overrides). Pockets and PPAs resolve test
dependencies like in `pkh deb` (`-p`, `--ppa`), and `--test-name`,
`--skip-test`, `--shell-fail` and `--setup-commands` cover the day-to-day
debugging loop. Tests needing a real machine (`isolation-machine`,
`needs-reboot`) run only in `--mode vm`, which execs the installed
`autopkgtest` with its qemu runner.
Exit codes: 0 all tests passed (skips and flaky allowed), 1 at least one
test failed (or a `--fail-on` trigger), 2 runtime errors (no declared
tests, no usable testbed, ...).
## End-to-end package harness
`tests/e2e/` exercises pkh the way it is used in anger: a catalog of ~60
real source packages (from `hello` and `micro` to `dracut`, `grub2` and
the kernel) run through the full `pull` → `chlog --rebuild` → `build` →
`deb` workflow, each stage timed, logged and checked for artifacts:
```
tests/e2e/run.py --list # what would run
tests/e2e/run.py # smoke + standard tiers
tests/e2e/run.py --tier long # the multi-hour builds
```
See [tests/e2e/README.md](tests/e2e/README.md) for prerequisites,
filters (`--only`, `--category`, `--tag`), resume support and the run
output layout.
## Future improvement ideas
- pull: try to fetch the correct git branch for series on Debian
- deb: asynchronous build, detachable and monitorable
- put: allow uploads to Debian or Ubuntu archives
- test: add 'pkh test' to run autopkgtests
- pull: cache Sources.gz files to improve speed
- pull: 'pkh pull' in a package tree should git pull and re-fetch orig tgz
data/quirks.yml
+35
View File
@@ -15,18 +15,53 @@ quirks:
# against the host architecture, whose dependency closure conflicts with
# the `:native` python3. Resolve it against the build architecture
# until the control is fixed upstream.
# The resolute and stonking kernels declare the host-tool libraries
# unqualified (libelf-dev for resolve_btfids, libdw-dev for
# gendwarfksyms, libssl-dev for sign-file), unlike the Debian control,
# which carries the same names qualified `:native`. The dpkg cross
# rules resolve an unqualified Multi-Arch: same name against the host
# architecture only, so a cross build installs no build-architecture
# variants and the kernel's host-side tools cannot link. Inject the
# build-architecture variants until the control is fixed upstream.
# The kernel's debian/rules keeps its wrapper-step stamps in
# debian/stamps, and the flavour config rule there exports .config from
# the annotations with no config prerequisite — a resumed build would
# keep the previous attempt's configuration and silently ignore config
# edits. List the stamp cache under `resume_clear` so resumed builds
# drop it and re-run the wrapper steps; the inner kbuild keeps its own
# incremental state, so only the cheap wrapper passes re-run. This is
# the packaging's long-standing shape rather than a fixable regression,
# so the entry applies to every series.
linux:
deb:
- series: []
resume_clear:
- debian/stamps
- series: [resolute]
dependencies:
replace:
llvm-21-dev: llvm-21-dev:native <!stage1>
- series: [resolute, stonking]
dependencies:
inject:
- libelf-dev:native
- libdw-dev:native
- libssl-dev:native
linux-riscv:
deb:
- series: []
resume_clear:
- debian/stamps
- series: [resolute]
dependencies:
replace:
llvm-21-dev: llvm-21-dev:native <!stage1>
- series: [resolute, stonking]
dependencies:
inject:
- libelf-dev:native
- libdw-dev:native
- libssl-dev:native
# Add more packages and their quirks as needed
# example-package:
data/skill/SKILL.md
+7 -3
View File
@@ -86,9 +86,13 @@ act on the package in the current directory.
- `pkh deb` builds the binary packages in an isolated context with the
build dependencies installed. `--ppa` (repeatable) adds dependency
sources, `--inject <package|.deb>` preinstalls a package, and `-j`
caps parallel jobs. `--cross` cross-compiles instead of using
qemu-binfmt, but most packages cannot cross-compile, so prefer qemu.
Leave `--mode` unset unless you need a specific build context.
caps parallel jobs. `--mirror <url>` builds the chroot and resolves
every package from that base URL (a caching proxy on a pinned runner;
it must serve every suite and architecture the build resolves).
`--cross`
cross-compiles instead of using qemu-binfmt, but most packages cannot
cross-compile, so prefer qemu. Leave `--mode` unset unless you need a
specific build context.
- `pkh lint [path]` runs lintian plus the pkh-native checks. `--json`
emits a machine-readable report and `--list-tags` prints the native
src/apt/release.rs
+126 -46
View File
@@ -39,13 +39,14 @@
//! the missing authenticity only produces a `warn!`.
//!
//! Only actual tampering evidence — an invalid signature, or an index whose
//! hash does not match the signed Release — is reported as [`VerifyError`].
//! hash does not match the signed Release — is reported as an error:
//! [`VerifyError`] for the suite signature, [`FileVerifyError`] for an
//! index that fails its Release checksums.
//!
//! ASCII-armored keyrings (Debian distributes `.asc` keyrings, and
//! keyserver responses are armored) are dearmored in-process, so no `gpg`
//! binary is needed.
use lazy_static::lazy_static;
use log::{debug, warn};
use serde::Deserialize;
use std::collections::HashMap;
@@ -53,6 +54,7 @@ use std::error::Error;
use std::fmt;
use std::path::Path;
use std::path::PathBuf;
use std::sync::LazyLock;
use std::sync::Mutex;
use std::sync::PoisonError;
@@ -227,48 +229,101 @@ impl VerifiedRelease {
/// or, for flat repositories, 'Sources.xz'). Both the digest and the
/// size are checked; a file not listed in the Release file is refused,
/// as it is not covered by the repository signature.
pub fn verify_file(&self, rel_path: &str, data: &[u8]) -> Result<(), String> {
pub fn verify_file(&self, rel_path: &str, data: &[u8]) -> Result<(), FileVerifyError> {
if !self.available {
return Err(format!(
"no Release file was found for '{}': cannot checksum-verify '{rel_path}'",
self.suite_url
));
return Err(FileVerifyError::NoReleaseFile {
suite_url: self.suite_url.clone(),
path: rel_path.to_string(),
});
}
let entry = self.checksums.get(rel_path).ok_or_else(|| {
format!(
"'{rel_path}' is not listed in the Release file of '{}'; refusing \
to use an index that is not covered by the repository metadata",
self.suite_url
)
})?;
let entry = self
.checksums
.get(rel_path)
.ok_or_else(|| FileVerifyError::NotListed {
path: rel_path.to_string(),
suite_url: self.suite_url.clone(),
})?;
let digest = entry.kind.digest(data);
if !digest.eq_ignore_ascii_case(&entry.hash) {
return Err(format!(
"checksum mismatch for '{rel_path}' of '{}': the Release file \
expects {} {} but the downloaded file hashes to {}; the index \
does not match the signed repository metadata",
self.suite_url,
entry.kind.name(),
entry.hash,
digest
));
return Err(FileVerifyError::ChecksumMismatch {
path: rel_path.to_string(),
suite_url: self.suite_url.clone(),
algorithm: entry.kind.name().to_string(),
expected: entry.hash.clone(),
actual: digest,
});
}
if entry.size > 0 && entry.size != data.len() as u64 {
return Err(format!(
"size mismatch for '{rel_path}' of '{}': expected {} bytes, got {}",
self.suite_url,
entry.size,
data.len()
));
return Err(FileVerifyError::SizeMismatch {
path: rel_path.to_string(),
suite_url: self.suite_url.clone(),
expected: entry.size,
actual: data.len(),
});
}
Ok(())
}
}
/// Failure of [`VerifiedRelease::verify_file`]: the downloaded index is not
/// covered by the suite's signed Release metadata.
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
pub enum FileVerifyError {
/// No Release file was seen on the suite, so there is nothing to
/// verify against.
#[error("no Release file was found for '{suite_url}': cannot checksum-verify '{path}'")]
NoReleaseFile {
/// URL of the suite directory.
suite_url: String,
/// The path that was to be verified.
path: String,
},
/// The Release file does not list the path: it is not covered by the
/// repository signature.
#[error(
"'{path}' is not listed in the Release file of '{suite_url}'; refusing to use an index that is not covered by the repository metadata"
)]
NotListed {
/// The unlisted path.
path: String,
/// URL of the suite directory.
suite_url: String,
},
/// The downloaded file does not hash to the Release entry.
#[error(
"checksum mismatch for '{path}' of '{suite_url}': the Release file expects {algorithm} {expected} but the downloaded file hashes to {actual}; the index does not match the signed repository metadata"
)]
ChecksumMismatch {
/// The verified path.
path: String,
/// URL of the suite directory.
suite_url: String,
/// Name of the checksum algorithm (`SHA256`, ...).
algorithm: String,
/// The digest the Release file lists.
expected: String,
/// The digest of the downloaded bytes.
actual: String,
},
/// The downloaded file's length differs from the Release entry (a
/// zeroed entry skips the check).
#[error("size mismatch for '{path}' of '{suite_url}': expected {expected} bytes, got {actual}")]
SizeMismatch {
/// The verified path.
path: String,
/// URL of the suite directory.
suite_url: String,
/// The size the Release file lists.
expected: u64,
/// The size of the downloaded bytes.
actual: usize,
},
}
/// Parse a Release file body (the signed payload of an InRelease, or a
/// plain Release file) into a [`VerifiedRelease`]
fn parse_release_body(suite_url: &str, body: &str, authenticated: bool) -> VerifiedRelease {
@@ -793,11 +848,10 @@ async fn fetch_optional(url: &str) -> Option<Vec<u8>> {
None
}
lazy_static! {
/// Downloaded keyrings, keyed by URL: keyrings are small but would
/// otherwise be re-downloaded once per series/pocket lookup
static ref KEYRING_CACHE: Mutex<HashMap<String, Vec<u8>>> = Mutex::new(HashMap::new());
}
/// Downloaded keyrings, keyed by URL: keyrings are small but would
/// otherwise be re-downloaded once per series/pocket lookup
static KEYRING_CACHE: LazyLock<Mutex<HashMap<String, Vec<u8>>>> =
LazyLock::new(|| Mutex::new(HashMap::new()));
/// Read the keyring cache, ignoring poisoning (the map is only corrupted on
/// panics, in which case a fresh map is just as good)
@@ -914,9 +968,33 @@ pub async fn ppa_keyring_bytes(
Ok(keyring)
}
/// Failure of the OpenPGP armor and base64 decoders ([`dearmor`],
/// [`base64_decode`]).
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
pub enum ArmorError {
/// The armored input is not valid UTF-8.
#[error("the armored keyring is not valid UTF-8")]
NotUtf8,
/// The armor CRC-24 does not match the decoded payload.
#[error("the armor CRC-24 checksum does not match")]
CrcMismatch,
/// An armor marker was present but no payload followed it.
#[error("no armored OpenPGP data found")]
NoArmoredData,
/// A byte outside the base64 alphabet.
#[error("invalid base64 character '{c}'")]
InvalidBase64 {
/// The offending character.
c: char,
},
/// The base64 payload ends mid-group.
#[error("truncated base64 data")]
Truncated,
}
/// Decode an OpenPGP ASCII-armored keyring into its binary form (the
/// equivalent of 'gpg --dearmor'). Binary input passes through unchanged.
fn dearmor(data: &[u8]) -> Result<Vec<u8>, String> {
fn dearmor(data: &[u8]) -> Result<Vec<u8>, ArmorError> {
// Only input starting with an armor marker is armored; anything else is
// assumed to already be a binary keyring
let first = data
@@ -927,8 +1005,7 @@ fn dearmor(data: &[u8]) -> Result<Vec<u8>, String> {
return Ok(data.to_vec());
}
let text = std::str::from_utf8(data)
.map_err(|_| "the armored keyring is not valid UTF-8".to_string())?;
let text = std::str::from_utf8(data).map_err(|_| ArmorError::NotUtf8)?;
#[derive(PartialEq)]
enum State {
@@ -964,7 +1041,7 @@ fn dearmor(data: &[u8]) -> Result<Vec<u8>, String> {
| (u32::from(expected[1]) << 8)
| u32::from(expected[2]);
if value != crc24(&decoded) {
return Err("the armor CRC-24 checksum does not match".to_string());
return Err(ArmorError::CrcMismatch);
}
}
}
@@ -982,21 +1059,21 @@ fn dearmor(data: &[u8]) -> Result<Vec<u8>, String> {
}
if out.is_empty() {
return Err("no armored OpenPGP data found".to_string());
return Err(ArmorError::NoArmoredData);
}
Ok(out)
}
/// Decode standard base64, ignoring whitespace; '=' padding is optional
fn base64_decode(input: &str) -> Result<Vec<u8>, String> {
fn value(byte: u8) -> Result<u32, String> {
fn base64_decode(input: &str) -> Result<Vec<u8>, ArmorError> {
fn value(byte: u8) -> Result<u32, ArmorError> {
match byte {
b'A'..=b'Z' => Ok(u32::from(byte - b'A')),
b'a'..=b'z' => Ok(u32::from(byte - b'a' + 26)),
b'0'..=b'9' => Ok(u32::from(byte - b'0' + 52)),
b'+' => Ok(62),
b'/' => Ok(63),
_ => Err(format!("invalid base64 character '{}'", byte as char)),
_ => Err(ArmorError::InvalidBase64 { c: byte as char }),
}
}
@@ -1016,7 +1093,7 @@ fn base64_decode(input: &str) -> Result<Vec<u8>, String> {
}
3 => (value(chunk[0])? << 18) | (value(chunk[1])? << 12) | (value(chunk[2])? << 6),
2 => (value(chunk[0])? << 18) | (value(chunk[1])? << 12),
_ => return Err("truncated base64 data".to_string()),
_ => return Err(ArmorError::Truncated),
};
out.push((n >> 16) as u8);
if chunk.len() >= 3 {
@@ -1155,20 +1232,23 @@ garbage
// Tampered content fails with a checksum mismatch
let err = verified
.verify_file("main/source/Sources.gz", b"tampered!")
.unwrap_err();
.unwrap_err()
.to_string();
assert!(err.contains("checksum mismatch"), "got: {err}");
// A correct hash with a wrong size also fails
let err = verified
.verify_file("main/source/Sizes.gz", b"test")
.unwrap_err();
.unwrap_err()
.to_string();
assert!(err.contains("size mismatch"), "got: {err}");
// A file not listed in the Release file is refused: it is not
// covered by the repository signature
let err = verified
.verify_file("evil/source/Sources.gz", b"test")
.unwrap_err();
.unwrap_err()
.to_string();
assert!(err.contains("not listed"), "got: {err}");
// Without a Release file nothing can be verified
src/build/binary.rs
+18 -6
View File
@@ -92,7 +92,7 @@ pub fn generate_binary_metadata(
// ------------------------------------------------------------------
let artifact_names: Vec<String> = files_list
.iter()
.filter(|e| matches!(e.package_type.as_deref(), Some("deb") | Some("udeb")))
.filter(|e| e.is_binary_artifact())
.map(|e| e.filename.clone())
.collect();
@@ -159,16 +159,28 @@ pub fn generate_binary_metadata(
// ------------------------------------------------------------------
// Binary package names and descriptions
// ------------------------------------------------------------------
// dpkg-genchanges lists only deb/udeb packages in `Binary:` (a ddeb
// is distributed through Files but is not a listed binary of the
// upload), while dpkg-genbuildinfo lists every binary artifact's
// package, ddebs included.
let mut binaries: Vec<String> = Vec::new();
let mut buildinfo_binaries: Vec<String> = Vec::new();
for name in &artifact_names {
if let Some(e) = files_list.get(name)
&& let Some(pkg) = &e.package
&& !binaries.contains(pkg)
{
let Some(e) = files_list.get(name) else {
continue;
};
let Some(pkg) = &e.package else {
continue;
};
if !buildinfo_binaries.contains(pkg) {
buildinfo_binaries.push(pkg.clone());
}
if e.package_type.as_deref() != Some("ddeb") && !binaries.contains(pkg) {
binaries.push(pkg.clone());
}
}
binaries.sort();
buildinfo_binaries.sort();
// Description lines: first line of each binary stanza's Description,
// formatted exactly like dpkg-genchanges, sorted.
@@ -238,7 +250,7 @@ pub fn generate_binary_metadata(
let buildinfo_doc =
crate::build::buildinfo::render_buildinfo(&crate::build::buildinfo::BuildInfoInput {
source: source_display.clone(),
binaries: binaries.clone(),
binaries: buildinfo_binaries.clone(),
architecture: buildinfo_arch_values.join(" "),
version: entry.version.full(),
binary_only_changes: binary_only_changes.clone(),
src/build/env.rs
+1 -1
View File
@@ -81,7 +81,7 @@ pub fn build_env(
/// (`*_ARCH`, `*_OS`, `*_CPU`, `*_MULTIARCH`, `*_GNU_TYPE`, ...), exactly as
/// `dpkg-buildpackage` does.
pub fn arch_env(host_arch: Option<&str>) -> Result<BTreeMap<String, String>, String> {
crate::debian::arch::arch_env(host_arch)
crate::debian::arch::arch_env(host_arch).map_err(|e| e.to_string())
}
/// Read the current vendor name from the active dpkg origins `default` file
src/build/mod.rs
+9 -6
View File
@@ -314,7 +314,7 @@ pub fn run_source_build(
entry.version.full(),
entry.distribution
),
source_only: true,
kind: crate::report::TargetKind::Source,
tee_log: true,
});
@@ -1889,19 +1889,22 @@ Provides: virtual-thing (= 2.0), plain-virtual
}
/// debian/control shared by the binary-metadata differential cases: one
/// arch:any deb and one arch:all udeb.
/// arch:any deb, one arch:all udeb and one arch:all ddeb (the
/// debug-symbol shape dh_strip leaves behind: the package type
/// drives the file name in debian/files).
fn binary_test_control(name: &str) -> String {
format!(
"Source: {name}\nSection: utils\nPriority: optional\nMaintainer: {MAINTAINER}\nBuild-Depends: libc6\n\n\
Package: {name}\nArchitecture: any\nDescription: test package main\n long description\n\n\
Package: {name}-u\nPackage-Type: udeb\nArchitecture: all\nDescription: test udeb\n short\n"
Package: {name}-u\nPackage-Type: udeb\nArchitecture: all\nDescription: test udeb\n short\n\n\
Package: {name}-d\nPackage-Type: ddeb\nArchitecture: all\nDescription: test ddeb\n short\n"
)
}
/// debian/rules driving dpkg-gencontrol/dpkg-deb directly (no debhelper).
fn binary_test_rules(name: &str) -> String {
format!(
"#!/usr/bin/make -f\nV = $(shell dpkg-parsechangelog -S Version)\nA = $(shell dpkg-architecture -qDEB_HOST_ARCH)\n\nbuild:\n\tmkdir -p debian/tmp/usr/bin\n\tprintf '#!/bin/sh\\necho hi\\n' > debian/tmp/usr/bin/hello\n\tchmod 755 debian/tmp/usr/bin/hello\n\ttouch $@\n\nbinary: build\n\trm -rf debian/{name} debian/{name}-u\n\tmkdir -p debian/{name}/usr/bin debian/{name}/DEBIAN\n\tcp -r debian/tmp/. debian/{name}/\n\tdpkg-gencontrol -p{name} -Pdebian/{name}\n\tdpkg-deb --build debian/{name} ..\n\tmkdir -p debian/{name}-u/usr/share debian/{name}-u/DEBIAN\n\techo data > debian/{name}-u/usr/share/data.txt\n\tdpkg-gencontrol -p{name}-u -Pdebian/{name}-u\n\tdpkg-deb --build debian/{name}-u ..\n\tmv ../{name}-u_$(V)_all.deb ../{name}-u_$(V)_all.udeb\n\nclean:\n\trm -rf debian/tmp debian/{name} debian/{name}-u build-stamp debian/files debian/*.substvars\n\n.PHONY: build binary clean\n"
"#!/usr/bin/make -f\nV = $(shell dpkg-parsechangelog -S Version)\nA = $(shell dpkg-architecture -qDEB_HOST_ARCH)\n\nbuild:\n\tmkdir -p debian/tmp/usr/bin\n\tprintf '#!/bin/sh\\necho hi\\n' > debian/tmp/usr/bin/hello\n\tchmod 755 debian/tmp/usr/bin/hello\n\ttouch $@\n\nbinary: build\n\trm -rf debian/{name} debian/{name}-u debian/{name}-d\n\tmkdir -p debian/{name}/usr/bin debian/{name}/DEBIAN\n\tcp -r debian/tmp/. debian/{name}/\n\tdpkg-gencontrol -p{name} -Pdebian/{name}\n\tdpkg-deb --build debian/{name} ..\n\tmkdir -p debian/{name}-u/usr/share debian/{name}-u/DEBIAN\n\techo data > debian/{name}-u/usr/share/data.txt\n\tdpkg-gencontrol -p{name}-u -Pdebian/{name}-u\n\tdpkg-deb --build debian/{name}-u ..\n\tmv ../{name}-u_$(V)_all.deb ../{name}-u_$(V)_all.udeb\n\tmkdir -p debian/{name}-d/usr/share debian/{name}-d/DEBIAN\n\techo data > debian/{name}-d/usr/share/data.txt\n\tdpkg-gencontrol -p{name}-d -Pdebian/{name}-d\n\tdpkg-deb --build debian/{name}-d ..\n\tmv ../{name}-d_$(V)_all.deb ../{name}-d_$(V)_all.ddeb\n\nclean:\n\trm -rf debian/tmp debian/{name} debian/{name}-u debian/{name}-d build-stamp debian/files debian/*.substvars\n\n.PHONY: build binary clean\n"
)
}
@@ -2281,13 +2284,13 @@ Provides: virtual-thing (= 2.0), plain-virtual
Some(dist),
None,
None,
None,
&crate::report::Quiet,
))
.unwrap_or_else(|e| panic!("package lookup failed for {}: {}", package, e));
rt.block_on(crate::pull::pull(
&package_info,
Some(fetch_dir.path()),
None,
&crate::report::Quiet,
true,
))
.unwrap_or_else(|e| panic!("pull failed for {}: {}", package, e));
src/changelog.rs
+297 -15
View File
@@ -60,6 +60,7 @@ pub async fn generate_entry(
user_version: Option<&str>,
target_series: Option<&str>,
kind: EntryKind,
history_limit: usize,
) -> Result<GeneratedEntry, Box<dyn std::error::Error>> {
let changelog_path = if let Some(path) = cwd {
path.join(changelog_file)
@@ -79,7 +80,12 @@ pub async fn generate_entry(
let commits = match Repository::open(&repo_path) {
Ok(repo) => {
let repo_changelog = changelog_in_repo(&repo, &changelog_path);
get_commits_since_version(&repo, &old_version, repo_changelog.as_deref())?
get_commits_since_version(
&repo,
&old_version,
repo_changelog.as_deref(),
history_limit,
)?
}
// If there is no git repo (e.g. package downloaded from archive),
// we just generate an empty list of changes
@@ -459,6 +465,7 @@ fn get_commits_since_version(
repo: &Repository,
version: &str,
changelog_path: Option<&Path>,
history_limit: usize,
) -> Result<Vec<String>, Box<dyn std::error::Error>> {
// Preferred boundary: the tag of the current (previous) version
if let Some(tag_id) = find_version_tag(repo, version) {
@@ -472,17 +479,44 @@ fn get_commits_since_version(
let Some(rel_path) = changelog_path else {
return Ok(Vec::new());
};
let Some(boundary) = last_commit_touching(repo, rel_path)? else {
match last_commit_touching(repo, rel_path, history_limit)? {
// The changelog was never committed; nothing to walk from
return Ok(Vec::new());
};
Boundary::None => Ok(Vec::new()),
// The walk hit the limit without finding a changelog commit:
// cap the entry at the oldest visited commit rather than keep
// diffing trees over the whole history (which on large repos can
// take minutes). The version_at_commit tag probe is skipped: it
// could jump the boundary arbitrarily far back.
Boundary::Capped(oldest_visited) => {
log::warn!(
"No changelog commit found within the last {} commits; \
the entry only covers the commits since then.",
history_limit
);
commit_summaries_since(repo, oldest_visited)
}
Boundary::Found(boundary) => {
// Prefer a tag of the version the committed changelog carries:
// it may point past the changelog commit itself (e.g. a later
// release commit)
let boundary_id = version_at_commit(repo, &boundary, rel_path)
.and_then(|committed_version| find_version_tag(repo, &committed_version))
.unwrap_or_else(|| boundary.id());
commit_summaries_since(repo, boundary_id)
}
}
}
// Prefer a tag of the version the committed changelog carries: it may
// point past the changelog commit itself (e.g. a later release commit)
let boundary_id = version_at_commit(repo, &boundary, rel_path)
.and_then(|committed_version| find_version_tag(repo, &committed_version))
.unwrap_or_else(|| boundary.id());
commit_summaries_since(repo, boundary_id)
/// The result of searching the history for the last commit that modified
/// the changelog
enum Boundary<'a> {
/// That commit was found
Found(git2::Commit<'a>),
/// The walk visited `history_limit` commits without a match; carries
/// the id of the oldest visited commit
Capped(git2::Oid),
/// The changelog was never committed (or the history is empty)
None,
}
/// The commit names a Debian version may be tagged under in a repository
@@ -552,26 +586,36 @@ fn changelog_in_repo(repo: &Repository, changelog_path: &Path) -> Option<std::pa
.map(|p| p.to_path_buf())
}
/// The last commit that modified the file at `rel_path`, newest first
/// The last commit that modified the file at `rel_path`, newest first,
/// walking at most `history_limit` commits (0 = the whole history)
fn last_commit_touching<'a>(
repo: &'a Repository,
rel_path: &Path,
) -> Result<Option<git2::Commit<'a>>, Box<dyn std::error::Error>> {
history_limit: usize,
) -> Result<Boundary<'a>, Box<dyn std::error::Error>> {
let mut revwalk = repo.revwalk()?;
revwalk.set_sorting(Sort::TIME)?;
if repo.head().is_err() {
// Unborn HEAD: the repository has no commits yet
return Ok(None);
return Ok(Boundary::None);
}
revwalk.push_head()?;
let mut visited = 0;
for id in revwalk {
let commit = repo.find_commit(id?)?;
if commit_touches_path(repo, &commit, rel_path)? {
return Ok(Some(commit));
return Ok(Boundary::Found(commit));
}
if history_limit > 0 {
visited += 1;
if visited >= history_limit {
// The oldest commit visited so far is the walk boundary
return Ok(Boundary::Capped(commit.id()));
}
}
}
Ok(None)
Ok(Boundary::None)
}
/// Whether `commit` modified the file at `rel_path` relative to its first
@@ -919,6 +963,23 @@ mod tests {
.unwrap();
}
/// A commit with an explicit, strictly increasing timestamp: the
/// revwalks sort by commit time, which is otherwise ambiguous when
/// several commits land within the same second
fn commit_dated(dir: &Path, message: &str, seconds: u64) {
let date = format!("{} +0000", 1_700_000_000 + seconds);
Command::new("git")
.arg("commit")
.arg("--allow-empty")
.arg("-m")
.arg(message)
.env("GIT_AUTHOR_DATE", &date)
.env("GIT_COMMITTER_DATE", &date)
.current_dir(dir)
.output()
.unwrap();
}
fn tag(dir: &Path, name: &str) {
Command::new("git")
.arg("tag")
@@ -966,6 +1027,7 @@ mod tests {
None,
None,
EntryKind::Normal,
0,
)
.await
.unwrap();
@@ -1023,6 +1085,7 @@ mod tests {
None,
None,
EntryKind::Normal,
0,
)
.await
.unwrap();
@@ -1062,6 +1125,7 @@ mod tests {
None,
None,
EntryKind::Normal,
0,
)
.await
.unwrap();
@@ -1119,6 +1183,7 @@ mod tests {
None,
None,
EntryKind::Normal,
0,
)
.await
.unwrap();
@@ -1133,6 +1198,217 @@ mod tests {
assert!(content.contains("* Add feature B"));
}
/// A deep history of commits that never touch the changelog caps the
/// boundary search at `--limit`: the entry covers only the commits
/// after the oldest visited one, instead of walking the whole history.
#[tokio::test]
async fn test_generate_entry_history_limit_capped() {
let temp_dir = TempDir::new().unwrap();
let repo_dir = temp_dir.path();
setup_repo(repo_dir);
let changelog_path = repo_dir.join("debian/changelog");
std::fs::create_dir_all(repo_dir.join("debian")).unwrap();
std::fs::write(
&changelog_path,
"mypackage (0.1.0-1) unstable; urgency=medium\n\n * Initial release\n\n -- Maintainer <m@e.com> Wed, 01 Jan 2020 00:00:00 +0000\n",
)
.unwrap();
Command::new("git")
.arg("add")
.arg(".")
.current_dir(repo_dir)
.output()
.unwrap();
commit_dated(repo_dir, "Initial commit", 0);
for i in 1..=10 {
commit_dated(repo_dir, &format!("Fix bug {i}"), i);
}
let _identity = IDENTITY_LOCK.lock().await;
unsafe {
std::env::set_var("DEBFULLNAME", "Maintainer Maintainer");
std::env::set_var("DEBEMAIL", "maintainer@maintainer.com");
}
generate_entry(
"debian/changelog",
Some(repo_dir),
None,
None,
EntryKind::Normal,
5,
)
.await
.unwrap();
unsafe {
std::env::remove_var("DEBFULLNAME");
std::env::remove_var("DEBEMAIL");
}
// Boundary = oldest of the 5 visited commits ("Fix bug 6"): the
// entry covers the 4 commits after it, and nothing older
let content = std::fs::read_to_string(&changelog_path).unwrap();
assert!(content.contains("mypackage (0.1.0-2) unstable; urgency=medium"));
assert!(content.contains("* Fix bug 7"));
assert!(content.contains("* Fix bug 10"));
assert!(!content.contains("* Fix bug 6"));
assert!(!content.contains("* Fix bug 5"));
assert!(!content.contains("* Initial commit"));
}
/// an unbounded walk: the boundary is still the changelog commit.
#[tokio::test]
async fn test_generate_entry_history_limit_above_depth() {
let temp_dir = TempDir::new().unwrap();
let repo_dir = temp_dir.path();
setup_repo(repo_dir);
let changelog_path = repo_dir.join("debian/changelog");
std::fs::create_dir_all(repo_dir.join("debian")).unwrap();
std::fs::write(
&changelog_path,
"mypackage (0.1.0-1) unstable; urgency=medium\n\n * Initial release\n\n -- Maintainer <m@e.com> Wed, 01 Jan 2020 00:00:00 +0000\n",
)
.unwrap();
Command::new("git")
.arg("add")
.arg(".")
.current_dir(repo_dir)
.output()
.unwrap();
commit_dated(repo_dir, "Initial commit", 0);
for i in 1..=10 {
commit_dated(repo_dir, &format!("Fix bug {i}"), i);
}
let _identity = IDENTITY_LOCK.lock().await;
unsafe {
std::env::set_var("DEBFULLNAME", "Maintainer Maintainer");
std::env::set_var("DEBEMAIL", "maintainer@maintainer.com");
}
generate_entry(
"debian/changelog",
Some(repo_dir),
None,
None,
EntryKind::Normal,
100,
)
.await
.unwrap();
unsafe {
std::env::remove_var("DEBFULLNAME");
std::env::remove_var("DEBEMAIL");
}
// The changelog commit ("Initial commit") is the boundary: all
// later commits are in the entry, the boundary itself is not
let content = std::fs::read_to_string(&changelog_path).unwrap();
assert!(content.contains("mypackage (0.1.0-2) unstable; urgency=medium"));
assert!(content.contains("* Fix bug 1"));
assert!(content.contains("* Fix bug 10"));
assert!(!content.contains("* Initial commit"));
}
/// A limit of 0 means unlimited: the walk reaches the deep changelog
/// commit like today's behavior.
#[tokio::test]
async fn test_generate_entry_history_limit_zero_unlimited() {
let temp_dir = TempDir::new().unwrap();
let repo_dir = temp_dir.path();
setup_repo(repo_dir);
let changelog_path = repo_dir.join("debian/changelog");
std::fs::create_dir_all(repo_dir.join("debian")).unwrap();
std::fs::write(
&changelog_path,
"mypackage (0.1.0-1) unstable; urgency=medium\n\n * Initial release\n\n -- Maintainer <m@e.com> Wed, 01 Jan 2020 00:00:00 +0000\n",
)
.unwrap();
Command::new("git")
.arg("add")
.arg(".")
.current_dir(repo_dir)
.output()
.unwrap();
commit_dated(repo_dir, "Initial commit", 0);
for i in 1..=10 {
commit_dated(repo_dir, &format!("Fix bug {i}"), i);
}
let _identity = IDENTITY_LOCK.lock().await;
unsafe {
std::env::set_var("DEBFULLNAME", "Maintainer Maintainer");
std::env::set_var("DEBEMAIL", "maintainer@maintainer.com");
}
generate_entry(
"debian/changelog",
Some(repo_dir),
None,
None,
EntryKind::Normal,
0,
)
.await
.unwrap();
unsafe {
std::env::remove_var("DEBFULLNAME");
std::env::remove_var("DEBEMAIL");
}
let content = std::fs::read_to_string(&changelog_path).unwrap();
assert!(content.contains("mypackage (0.1.0-2) unstable; urgency=medium"));
assert!(content.contains("* Fix bug 1"));
assert!(content.contains("* Fix bug 10"));
assert!(!content.contains("* Initial commit"));
}
/// A non-default changelog file (pkh chlog -f): the entry is
/// prepended to that file, and its version is parsed from it
#[tokio::test]
async fn test_generate_entry_custom_changelog_file() {
let temp_dir = TempDir::new().unwrap();
let repo_dir = temp_dir.path();
setup_repo(repo_dir);
let changelog_path = repo_dir.join("debian/changelog.local");
std::fs::create_dir_all(repo_dir.join("debian")).unwrap();
std::fs::write(
&changelog_path,
"mypackage (0.1.0-1) unstable; urgency=medium\n\n * Initial release\n\n -- Maintainer <m@e.com> Wed, 01 Jan 2020 00:00:00 +0000\n",
)
.unwrap();
let _identity = IDENTITY_LOCK.lock().await;
unsafe {
std::env::set_var("DEBFULLNAME", "Maintainer Maintainer");
std::env::set_var("DEBEMAIL", "maintainer@maintainer.com");
}
let entry = generate_entry(
"debian/changelog.local",
Some(repo_dir),
None,
None,
EntryKind::Normal,
0,
)
.await
.unwrap();
unsafe {
std::env::remove_var("DEBFULLNAME");
std::env::remove_var("DEBEMAIL");
}
assert_eq!(entry.previous_version, "0.1.0-1");
assert_eq!(entry.new_version, "0.1.0-2");
let content = std::fs::read_to_string(&changelog_path).unwrap();
assert!(content.starts_with("mypackage (0.1.0-2) unstable; urgency=medium"));
assert!(content.contains("mypackage (0.1.0-1) unstable; urgency=medium"));
// The default file is not created next to it
assert!(!repo_dir.join("debian/changelog").exists());
}
#[test]
fn test_compute_new_version() {
// Debian upload
@@ -1303,6 +1579,7 @@ mod tests {
None,
Some("bookworm"),
EntryKind::Backport,
0,
)
.await
.unwrap();
@@ -1338,6 +1615,7 @@ mod tests {
None,
Some("sid"),
EntryKind::Backport,
0,
)
.await;
assert!(result.is_err());
@@ -1371,6 +1649,7 @@ mod tests {
None,
Some("noble"),
EntryKind::Backport,
0,
)
.await
.unwrap();
@@ -1407,6 +1686,7 @@ mod tests {
None,
Some("noble"),
EntryKind::Normal,
0,
)
.await
.unwrap();
@@ -1420,6 +1700,7 @@ mod tests {
None,
Some("noble"),
EntryKind::Normal,
0,
)
.await
.unwrap();
@@ -1452,6 +1733,7 @@ mod tests {
None,
Some("noble"),
EntryKind::Normal,
0,
)
.await
.unwrap();
src/config.rs
+144
View File
@@ -0,0 +1,144 @@
//! User settings: the optional `config.yml` in the pkh configuration
//! directory (the same `~/.config/pkh/` that holds `contexts.json`),
//! and their defaults.
//!
//! A missing file carries the defaults — the file is written only when
//! a user creates it, never generated. A file that cannot be read or
//! parsed must not take the whole program down: like the context
//! configuration, it is backed up to `<path>.bak` (best effort) so a
//! later save cannot silently destroy it, and the defaults apply. Every
//! struct level is individually defaulted (`#[serde(default)]`), so any
//! hand-written file may carry a subset of the settings:
//!
//! ```yaml
//! deb:
//! incremental:
//! staged_trees: 2
//! ```
use serde::Deserialize;
use std::path::PathBuf;
/// The pkh configuration directory (the one holding `contexts.json`
/// and `config.yml`).
pub fn config_dir() -> Option<PathBuf> {
directories::ProjectDirs::from("com", "pkh", "pkh").map(|dirs| dirs.config_dir().to_path_buf())
}
/// The settings file in [`config_dir`].
fn config_path() -> Option<PathBuf> {
config_dir().map(|dir| dir.join("config.yml"))
}
/// The user-adjustable settings, as stored in `config.yml`. New fields
/// must default — the file always may carry a subset.
#[derive(Debug, Clone, Deserialize, Default)]
#[serde(default)]
pub struct Settings {
/// `pkh deb` settings.
pub deb: DebSettings,
}
/// The `pkh deb` settings.
#[derive(Debug, Clone, Deserialize, Default)]
#[serde(default)]
pub struct DebSettings {
/// `pkh deb --incremental` settings.
pub incremental: IncrementalSettings,
}
/// The `pkh deb --incremental` settings.
#[derive(Debug, Clone, Deserialize, Default)]
#[serde(default)]
pub struct IncrementalSettings {
/// How many staged trees `pkh deb` retains per target: the newer
/// ones first, with the newest successful one and the newest one
/// always among them.
pub staged_trees: Option<usize>,
}
/// How many staged trees `pkh deb` retains per target: the setting's
/// value, or the default (2 — the newest tree and the newest successful
/// one). A configured 0 cannot honor the retention invariant (the
/// newest successful tree always survives), so it clamps to 1 with a
/// warning.
pub fn staged_trees() -> usize {
const DEFAULT: usize = 2;
let Some(path) = config_path() else {
return DEFAULT;
};
let Ok(text) = std::fs::read_to_string(&path) else {
return DEFAULT;
};
let settings: Settings = match serde_yaml::from_str(&text) {
Ok(settings) => settings,
Err(e) => {
log::error!(
"Settings file {} is corrupt ({e}); using the defaults",
path.display()
);
backup(&path);
return DEFAULT;
}
};
match settings.deb.incremental.staged_trees {
Some(0) => {
log::warn!(
"Settings file {} carries staged_trees: 0; the newest \
successful staged tree is always retained, so 1 applies",
path.display()
);
1
}
Some(count) => count,
None => DEFAULT,
}
}
/// Back up a corrupt settings file next to itself (best effort), so a
/// later save cannot silently destroy its content.
fn backup(path: &std::path::Path) {
let mut os = path.as_os_str().to_os_string();
os.push(".bak");
let backup_path = PathBuf::from(os);
match std::fs::copy(path, &backup_path) {
Ok(_) => log::warn!(
"Corrupt settings file backed up to {}",
backup_path.display()
),
Err(e) => log::warn!(
"Could not back up corrupt settings file to {}: {e}",
backup_path.display()
),
}
}
#[cfg(test)]
mod tests {
use super::*;
/// The nested setting parses; every level may be missing and every
/// missing level defaults.
#[test]
fn settings_parse_the_nested_shape() {
let settings: Settings =
serde_yaml::from_str("deb:\n incremental:\n staged_trees: 3").unwrap();
assert_eq!(settings.deb.incremental.staged_trees, Some(3));
let settings: Settings = serde_yaml::from_str("deb: {}").unwrap();
assert_eq!(settings.deb.incremental.staged_trees, None);
let settings: Settings = serde_yaml::from_str("{}").unwrap();
assert_eq!(settings.deb.incremental.staged_trees, None);
}
/// A value of the wrong type is a file error, not a default for the
/// field: the whole file is corrupt (the caller backs it up and
/// applies the defaults).
#[test]
fn a_mistyped_value_is_a_corrupt_file() {
let result =
serde_yaml::from_str::<Settings>("deb:\n incremental:\n staged_trees: two\n");
assert!(result.is_err());
}
}
src/context/api.rs
+68
View File
@@ -29,9 +29,58 @@ use super::schroot::SchrootDriver;
use super::ssh::SshDriver;
use super::unshare::UnshareDriver;
/// Outcome of an overlay-aware staging
/// ([`ContextDriver::ensure_available_with_overlay`]).
#[derive(Debug, Clone)]
pub struct OverlayStaging {
/// The staged path as it appears inside the context.
pub path: PathBuf,
/// Whether the staging reused the persistent overlay upperdir (build
/// artifacts of a previous attempt are visible); `false` means a fresh
/// copy was staged instead.
pub overlay: bool,
}
/// Marker environment value (set to `1`) flagging a command as a rules
/// step of a rootless build: unshare-based contexts then enter the chroot
/// by pivoting and drop into a nested user namespace mapping the chroot's
/// root ownership onto the build identity ([`MAPPED_BUILD_ID`]), instead
/// of chrooting into the namespace's root — so the staged tree is already
/// owned and no chown is needed. See `crate::deb::local`, which sets it.
pub(crate) const MAPPED_IDENTITY_ENV: &str = "PKH_MAPPED_IDENTITY";
/// The build identity inside the nested namespace of a rootless build:
/// the namespace maps the chroot's root onto this id, and the chroot's
/// passwd/group entries carry it, so `getpwuid` resolves the identity for
/// the build tools. It must stay outside the context namespace's own
/// mapping (which covers 0..=65535), so the two identities never alias.
pub(crate) const MAPPED_BUILD_ID: u32 = 65536;
const _: () = assert!(
MAPPED_BUILD_ID > 65535,
"the mapped identity must not alias a context-namespace uid"
);
/// A ContextDriver is the interface for the logic happening inside a context
pub trait ContextDriver {
fn ensure_available(&self, src: &Path, dest_root: &str) -> io::Result<PathBuf>;
/// Stage `src` at `dest_root` over the given persistent overlay
/// upperdir/workdir (host-side paths), for resumable build sessions.
///
/// The default implementation rejects the request; drivers that cannot
/// overlay-mount fall back to a plain copy in their caller.
fn ensure_available_with_overlay(
&self,
_src: &Path,
_dest_root: &str,
_upper: &Path,
_work: &Path,
) -> io::Result<OverlayStaging> {
Err(io::Error::new(
io::ErrorKind::Unsupported,
"overlay staging is not supported by this context driver",
))
}
fn retrieve_path(&self, src: &Path, dest: &Path) -> io::Result<()>;
fn list_files(&self, path: &Path) -> io::Result<Vec<PathBuf>>;
fn run(
@@ -281,6 +330,25 @@ impl Context {
.ensure_available(src, dest_root)
}
/// Stage `src` at `dest_root` reusing the given overlay upperdir and
/// workdir (host-side paths), so build artifacts written inside the
/// context during a previous attempt survive into this one. Falls back
/// to a fresh copy when the driver does not support overlay staging or
/// the mount fails (the returned [`OverlayStaging::overlay`] says
/// which happened).
pub fn ensure_available_with_overlay(
&self,
src: &Path,
dest_root: &str,
upper: &Path,
work: &Path,
) -> io::Result<OverlayStaging> {
self.driver()
.as_ref()
.unwrap()
.ensure_available_with_overlay(src, dest_root, upper, work)
}
/// Create a temp directory inside context
pub fn create_temp_dir(&self) -> io::Result<String> {
self.driver().as_ref().unwrap().create_temp_dir()
src/context/mod.rs
+2 -1
View File
@@ -7,7 +7,8 @@ pub(crate) mod shell;
mod ssh;
mod unshare;
pub use api::{Context, ContextCommand, ContextConfig, LineSink, Stream};
pub use api::{Context, ContextCommand, ContextConfig, LineSink, OverlayStaging, Stream};
pub(crate) use api::{MAPPED_BUILD_ID, MAPPED_IDENTITY_ENV};
// The driver trait is implementation detail of the context API; it is only
// needed crate-internally (test-run capture wrapper), so keep it out of the
// public surface (and its documentation requirement).
src/context/unshare.rs
+263 -25
View File
@@ -1,4 +1,6 @@
use super::api::{Context, ContextCommand, ContextDriver, LineSink};
use super::api::{
Context, ContextCommand, ContextDriver, LineSink, MAPPED_BUILD_ID, MAPPED_IDENTITY_ENV,
};
use super::shell::shell_quote;
use log::debug;
use std::fs;
@@ -176,7 +178,7 @@ impl ContextDriver for UnshareDriver {
// Try overlayfs for directories first — avoids a potentially expensive full copy
if src.is_dir() && is_overlayfs_available() {
match self.try_overlay_mount(src, &dest_path) {
match self.mount_overlay(src, &dest_path, None, None) {
Ok(()) => {
debug!(
"Mounted overlay for {} at {}",
@@ -218,6 +220,52 @@ impl ContextDriver for UnshareDriver {
Ok(Path::new(dest_root).join(filename))
}
fn ensure_available_with_overlay(
&self,
src: &Path,
dest_root: &str,
upper: &Path,
work: &Path,
) -> io::Result<super::api::OverlayStaging> {
let dest_dir = Path::new(&self.path).join(dest_root.trim_start_matches('/'));
std::fs::create_dir_all(&dest_dir)?;
let filename = src
.file_name()
.ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "Invalid source path"))?;
let dest_path = dest_dir.join(filename);
// The upper/work dirs are the persistent ones of a build session:
// mounting them again exposes the previous attempt's build
// artifacts over the (live) host tree. Mount failure is an ERROR,
// never a silent copy fallback: a copy would merge the host tree
// into the stale staged tree of the previous attempt (old build
// stamps, dropped host deletions) — the caller decides to wipe and
// re-stage instead.
if !src.is_dir() {
return Err(io::Error::new(
io::ErrorKind::Unsupported,
"overlay staging only applies to directories",
));
}
if !is_overlayfs_available() {
return Err(io::Error::new(
io::ErrorKind::Unsupported,
"overlayfs is not available on this system",
));
}
self.mount_overlay(src, &dest_path, Some(upper), Some(work))?;
debug!(
"Mounted overlay for {} at {} (reused upper {})",
src.display(),
dest_path.display(),
upper.display()
);
Ok(super::api::OverlayStaging {
path: Path::new(dest_root).join(filename),
overlay: true,
})
}
fn cleanup(&self) -> io::Result<()> {
let mounts = self.overlay_mounts.lock().unwrap();
for mount_path in mounts.iter() {
@@ -370,19 +418,33 @@ impl UnshareDriver {
.expect("UnshareDriver requires a parent context")
}
/// Try to mount `src` as an overlay at `dest_path` inside the chroot.
/// Mount `src` as an overlay at `dest_path` inside the chroot.
///
/// On success, the overlay is tracked in `overlay_mounts` for later cleanup.
/// On failure, the caller should fall back to the copy-based approach.
fn try_overlay_mount(&self, src: &Path, dest_path: &Path) -> io::Result<()> {
// Create unique upper/work dirs inside the chroot
/// `upper`/`work` override the per-mount random directories: the
/// resumable-session staging passes its persistent upperdir so build
/// artifacts survive across attempts. On success, the overlay is
/// tracked in `overlay_mounts` for later cleanup. On failure, the
/// caller should fall back to the copy-based approach.
fn mount_overlay(
&self,
src: &Path,
dest_path: &Path,
upper: Option<&Path>,
work: Option<&Path>,
) -> io::Result<()> {
// Create unique upper/work dirs inside the chroot unless the caller
// pinned them (session reuse).
let overlay_base = Path::new(&self.path).join("pkh-overlay");
let id = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos();
let upper_dir = overlay_base.join(format!("upper-{}", id));
let work_dir = overlay_base.join(format!("work-{}", id));
let upper_dir = upper
.map(Path::to_path_buf)
.unwrap_or_else(|| overlay_base.join(format!("upper-{}", id)));
let work_dir = work
.map(Path::to_path_buf)
.unwrap_or_else(|| overlay_base.join(format!("work-{}", id)));
fs::create_dir_all(&upper_dir)?;
fs::create_dir_all(&work_dir)?;
@@ -421,9 +483,13 @@ impl UnshareDriver {
.output()?;
if !output.status.success() {
// Clean up dirs we created
let _ = fs::remove_dir_all(&upper_dir);
let _ = fs::remove_dir_all(&work_dir);
// Clean up dirs we created (never the caller-pinned persistent
// upper/work of a session: they hold the previous attempt's
// build artifacts)
if upper.is_none() {
let _ = fs::remove_dir_all(&upper_dir);
let _ = fs::remove_dir_all(&work_dir);
}
let stderr = String::from_utf8_lossy(&output.stderr);
return Err(io::Error::other(format!(
"Overlay mount of '{}' at '{}' failed: {}",
@@ -470,6 +536,9 @@ impl UnshareDriver {
env: &[(String, String)],
cwd: Option<&str>,
) -> ContextCommand<'_> {
let mapped = env
.iter()
.any(|(key, value)| key == MAPPED_IDENTITY_ENV && value == "1");
let mut cmd = self.parent().command("unshare");
cmd.envs(env.iter().cloned());
@@ -484,25 +553,112 @@ impl UnshareDriver {
.arg("--map-auto")
.arg("-r")
.arg("--mount")
.arg("--fork")
.arg("-R")
.arg(&self.path);
.arg("--fork");
if let Some(dir) = cwd {
cmd.arg("-w").arg(dir);
if mapped {
// The rules steps of a rootless build (see the marker in
// `crate::deb::local`): no chroot(2) — the wrapper pivots into
// the chroot and only then creates the nested build-identity
// namespace, because a user namespace cannot be created from a
// chrooted process (the kernel refuses with EPERM) and
// chroot(2) cannot run inside one. The cwd is taken inside the
// script, after the pivot.
cmd.arg("--").arg("bash").arg("-c").arg(build_pivot_script(
&self.path,
cwd,
program,
args,
MAPPED_BUILD_ID,
));
} else {
cmd.arg("-R").arg(&self.path);
if let Some(dir) = cwd {
cmd.arg("-w").arg(dir);
}
// Build the bash command: set up /dev/pts and run the program
// /proc should already be bind-mounted from the host before entering the namespace
cmd.arg("--")
.arg("bash")
.arg("-c")
.arg(build_namespace_script(program, args));
}
// Build the bash command: set up /dev/pts and run the program
// /proc should already be bind-mounted from the host before entering the namespace
cmd.arg("--")
.arg("bash")
.arg("-c")
.arg(build_namespace_script(program, args));
cmd
}
}
/// Build the rules-mode wrapper script: pivot into the chroot, bring up
/// `/proc` and `/dev/pts` inside it, then drop into the nested
/// build-identity namespace (mapping the namespace's root onto
/// `build_id`) and exec `program`.
///
/// The pivot replaces chroot(2): it keeps the process root at the root of
/// the mount namespace, which is what makes the nested user namespace
/// creatable at all. The bind is recursive (`--rbind`) and taken fresh
/// inside the namespace: a plain bind of a tree carrying mounts made
/// outside the namespace (the chroot's `/proc`, apt-cache and ccache
/// binds) is denied with EINVAL — its locked children would have to move
/// with it — while a recursive bind carries them, which is exactly what
/// the rules need to keep working after the pivot. A bind made outside
/// the namespace would also carry MNT_LOCKED and fail with EINVAL. The
/// fresh `/proc` must be mounted before the old root is unmounted: umount
/// resolves its target through /proc/mounts, which lives on the old root.
///
/// Everything before the pivot runs as the namespace's root with the
/// host's binaries and paths; everything after it resolves inside the
/// chroot with the chroot's own binaries (plain `unshare`, an Essential
/// tool). The program and every argument are POSIX-shell-quoted (see
/// [`shell_quote`]).
fn build_pivot_script(
chroot: &str,
cwd: Option<&str>,
program: &str,
args: &[String],
build_id: u32,
) -> String {
let chroot = shell_quote(chroot);
let mut script = format!(
"set -e; \
mount --make-rprivate /; \
mount --rbind {chroot} {chroot}; \
cd {chroot}; \
putold=$(mktemp -d .pkh-pivot.XXXXXX); \
pivot_root . \"$putold\"; \
mount -t proc proc /proc; \
umount -l \"$putold\"; \
rmdir \"$putold\" 2>/dev/null || true; \
mkdir -p /dev/pts; \
mount -t devpts devpts /dev/pts 2>/dev/null || true; \
touch /dev/ptmx; \
mount --bind /dev/pts/ptmx /dev/ptmx 2>/dev/null || true; "
);
// The nested namespace's command line, itself shell-quoted once as the
// `sh -c` argument: the cwd is taken inside the new root, then the
// program runs exec'd, as the build identity. Note that the identity
// is the only mapped id: uid/gid 0 stay unmapped, and the build env
// carries FAKEROOTDONTTRYCHOWN for the binary step — fakeroot's chown
// wrappers attempt the real chown and swallow only EPERM, so an
// unmapped root target would fail them with EINVAL (dh_fixperms
// chowns to root:root).
let mut inner = String::new();
if let Some(dir) = cwd {
inner.push_str(&format!("cd {}; ", shell_quote(dir)));
}
inner.push_str(&format!("exec {}", shell_quote(program)));
for arg in args {
inner.push(' ');
inner.push_str(&shell_quote(arg));
}
script.push_str(&format!(
"exec unshare --user --fork --map-user={build_id} --map-group={build_id} -- sh -c {}",
shell_quote(&inner)
));
script
}
/// Build the shell script executed by `bash -c` inside the user namespace:
/// bring up `/dev/pts`, then run `program` with `args`.
///
@@ -526,7 +682,7 @@ fn build_namespace_script(program: &str, args: &[String]) -> String {
#[cfg(test)]
mod tests {
use super::build_namespace_script;
use super::{MAPPED_BUILD_ID, build_namespace_script, build_pivot_script, shell_quote};
fn tail_after_devpts_setup(script: &str) -> &str {
script
@@ -575,4 +731,86 @@ mod tests {
let script = build_namespace_script("prog", &[String::new(), "x".to_string()]);
assert_eq!(tail_after_devpts_setup(&script), "'prog' '' 'x'");
}
/// The pivot script pivots before creating the nested namespace (a
/// user namespace cannot be created from a chrooted process, so no
/// chroot(2) may precede it), mounts a fresh /proc before shedding the
/// old root (umount resolves through /proc/mounts), and execs the
/// program as the mapped build identity.
#[test]
fn pivot_script_pivots_then_maps_the_identity() {
let script = build_pivot_script(
"/var/tmp/pkh/chroot",
Some("/tmp/pkh-build/tree"),
"debian/rules",
&["build".to_string()],
MAPPED_BUILD_ID,
);
// pivot strictly before the nested unshare
let pivot = script.find("pivot_root").expect("pivot step");
let nested = script.find("unshare --user").expect("nested step");
assert!(pivot < nested, "{script}");
assert!(script.contains("mount -t proc proc /proc;"), "{script}");
assert!(script.contains("umount -l"), "{script}");
assert!(
script.contains(&format!("--map-user={MAPPED_BUILD_ID}")),
"{script}"
);
// the inner command line is the single shell-quoted `sh -c`
// argument: cwd inside the new root, then the exec'd program
let inner = format!(
"cd {}; exec {} {}",
shell_quote("/tmp/pkh-build/tree"),
shell_quote("debian/rules"),
shell_quote("build")
);
assert!(
script.ends_with(&format!("sh -c {}", shell_quote(&inner))),
"{script}"
);
// and the whole script must parse as shell
std::process::Command::new("bash")
.arg("-n")
.arg("-c")
.arg(&script)
.status()
.expect("run bash")
.success()
.then_some(())
.expect("the pivot script must parse: {script}");
}
/// Paths and arguments with shell metacharacters survive the two
/// quoting levels (outer script, inner `sh -c`).
#[test]
fn pivot_script_quotes_the_chroot_and_the_command() {
let script = build_pivot_script(
"/tmp/some'chroot",
None,
"fakeroot",
&["debian/rules".to_string(), "b'n".to_string()],
MAPPED_BUILD_ID,
);
assert!(
script.contains(&format!("cd {}", shell_quote("/tmp/some'chroot"))),
"{script}"
);
let inner = format!(
"exec {} {} {}",
shell_quote("fakeroot"),
shell_quote("debian/rules"),
shell_quote("b'n")
);
assert!(script.contains(&shell_quote(&inner)), "{script}");
std::process::Command::new("bash")
.arg("-n")
.arg("-c")
.arg(&script)
.status()
.expect("run bash")
.success()
.then_some(())
.expect("the pivot script must parse: {script}");
}
}
src/data.rs
+13 -24
View File
@@ -13,7 +13,7 @@
//! data/quirks.yml) through the [`embed_data!`] macro below, so data
//! and its accessors stay together and a diff touching one domain cannot
//! half-touch another. The macro embeds the file at compile time and
//! parses it once into a `lazy_static` on first use; since the data ships
//! parses it once into a [`LazyLock`] on first use; since the data ships
//! inside the binary, a parse failure is a build-time bug that cannot be
//! recovered from at runtime, and the macro panics on it.
//!
@@ -30,8 +30,9 @@
/// the invoking source file (`"../data/distro_info.yml"` from
/// `src/distro_info.rs`, `"../../data/host_keys.yml"` from
/// `src/put/ssh.rs`, ...), and expands to the house `include_str!` →
/// `lazy_static` → parse pattern — only the embed+parse boilerplate is
/// generated.
/// parse-on-first-use pattern — only the embed+parse boilerplate is
/// generated. The `static ref` spelling is kept from the macro's
/// `lazy_static!` ancestry; it expands to a plain `LazyLock` static.
///
/// ```ignore
/// embed_data! {
@@ -39,31 +40,19 @@
/// }
/// ```
macro_rules! embed_data {
// Internal arm: the visibility arrives wrapped in parentheses (empty for
// private statics) because `lazy_static!` only re-matches literal
// `pub`/`pub(...)` token sequences, not an opaque forwarded `vis`.
(@expand ($($vis:tt)*) static ref $name:ident : $ty:ty = $path:literal) => {
lazy_static::lazy_static! {
// The YAML is include_str!'d at compile time and statically
// valid; if it ever failed to parse it would be a build-time bug
// that cannot be recovered from at runtime, so panicking here is
// acceptable.
$($vis)* static ref $name: $ty = serde_yaml::from_str(include_str!($path))
($vis:vis static ref $name:ident : $ty:ty = $path:literal) => {
// The YAML is include_str!'d at compile time and statically
// valid; if it ever failed to parse it would be a build-time bug
// that cannot be recovered from at runtime, so panicking here is
// acceptable.
$vis static $name: ::std::sync::LazyLock<$ty> = ::std::sync::LazyLock::new(|| {
serde_yaml::from_str(include_str!($path))
.expect(concat!(
"built-in ",
$path,
" data is statically valid and must parse"
));
}
};
(static ref $name:ident : $ty:ty = $path:literal) => {
$crate::data::embed_data!(@expand () static ref $name : $ty = $path);
};
(pub static ref $name:ident : $ty:ty = $path:literal) => {
$crate::data::embed_data!(@expand (pub) static ref $name : $ty = $path);
};
(pub ($($vis:tt)+) static ref $name:ident : $ty:ty = $path:literal) => {
$crate::data::embed_data!(@expand (pub ($($vis)+)) static ref $name : $ty = $path);
))
});
};
}
src/deb/cross.rs
+31 -31
View File
@@ -7,7 +7,9 @@ use std::sync::Arc;
/// Parse 'dpkg-architecture' output (KEY=value lines) into a set of
/// environment variables. Unexpected lines (e.g. warnings on stderr leaking
/// into stdout) are skipped instead of causing a failure.
fn parse_dpkg_architecture_output(output: &str, env: &mut HashMap<String, String>) {
/// Parse the `VAR=value; ...; export ...` output of `dpkg-architecture -s`
/// into `env`.
pub fn parse_dpkg_architecture_output(output: &str, env: &mut HashMap<String, String>) {
let env_var_regex = regex::Regex::new(r"(?<key>.*)=(?<value>.*)").unwrap();
for l in output.lines() {
let Some(capture) = env_var_regex.captures(l) else {
@@ -62,20 +64,6 @@ pub fn setup_environment(
.map_err(|e| format!("Invalid UTF-8 in dpkg-architecture output: {e}"))?;
parse_dpkg_architecture_output(&dpkg_architecture, env);
// In-tree tools locate their libraries with the *host* pkg-config during
// cross builds (the kernel's tools/build feature checks derive their
// cflags/ldflags from `pkg-config --cflags/--libs`), whose search path
// only covers the build architecture's pkgconfig dirs. Point it at the
// target's so `libtraceevent` & co resolve to target-arch libraries:
// linux-riscv cross builds die in rtla's Makefile.config otherwise, even
// with the target -dev packages installed.
if let Some(multiarch) = env.get("DEB_HOST_MULTIARCH").cloned() {
env.insert(
"PKG_CONFIG_LIBDIR".to_string(),
format!("/usr/lib/{multiarch}/pkgconfig:/usr/share/pkgconfig"),
);
}
env.insert("DEB_BUILD_PROFILES".to_string(), "cross".to_string());
Ok(())
@@ -113,6 +101,10 @@ fn cross_suites(
/// are scoped to it and carry every component and cross-build suite, and
/// the mirror serving the target architecture (ports, for the non-local
/// ones) is added when no existing source serves the arch from it yet.
/// Under a mirror pin (`mirror`), the tarball's own source already points
/// at the pinned URL: it gets the components and suites, no arch scoping
/// (the mirror serves every architecture by contract), and no ports
/// entry — a source at the pinned URL counts as serving the target.
/// Debian hosts are left alone: one mirror serves every architecture, so
/// the host's own sources already cover the target — the os-release gate
/// below is what makes that a data conclusion instead of hardcoding.
@@ -120,6 +112,7 @@ pub fn ensure_repositories(
arch: &str,
series: &str,
pocket: Option<&str>,
mirror: Option<&str>,
ctx: Arc<Context>,
) -> Result<(), Box<dyn Error>> {
let local_arch = crate::get_current_arch();
@@ -160,14 +153,19 @@ pub fn ensure_repositories(
// Ensure all components are enabled for the primary architecture
for source in &mut sources {
// Official sources served by the local mirror (the primary archive
// and its security sibling); ports serves the other architectures
// and its security sibling), or the mirror-pinned source itself
// (the chroot tarball was bootstrapped from it — see
// `DebBuildOptions::mirror`); ports serves the other architectures
// and is configured below instead
if !crate::distro_info::is_mirror_source(local_mirror, &source.uri) {
let pinned = mirror.is_some_and(|m| source.uri == m);
if !pinned && !crate::distro_info::is_mirror_source(local_mirror, &source.uri) {
continue;
}
// Scope to local_arch if not already scoped
if source.architectures.is_empty() {
// Scope to local_arch if not already scoped — never under a pin:
// the pinned mirror serves the target architecture too (its
// contract), scoping it would cut the target apt off
if !pinned && source.architectures.is_empty() {
source.architectures.push(local_arch.clone());
}
@@ -188,9 +186,14 @@ pub fn ensure_repositories(
// Check whether an existing source already serves the target
// architecture from its mirror (e.g. the ports mirror for a
// non-local arch); when cross-building for the local architecture,
// the primary sources above already do
// non-local arch), or from the pinned mirror — which serves every
// architecture by contract, so its (unscoped) source counts; when
// cross-building for the local architecture, the primary sources
// above already do
let has_target = sources.iter().any(|s| {
if mirror.is_some_and(|m| s.uri == m) {
return s.architectures.is_empty() || s.architectures.contains(&arch.to_string());
}
crate::distro_info::is_mirror_source(target_mirror, &s.uri)
&& s.architectures.contains(&arch.to_string())
});
@@ -304,21 +307,18 @@ mod tests {
assert!(cross_suites("noble", None, "not-a-distro").is_err());
}
/// setup_environment exports the target multiarch pkg-config libdir:
/// tools' feature checks run the *host* pkg-config, which must find the
/// target's .pc files (rtla hard-errors on libtraceevent otherwise,
/// failing linux-riscv cross builds despite the target -dev packages
/// being installed).
/// setup_environment exports the dpkg cross variables and the 'cross'
/// build profile, and nothing beyond what dpkg-buildpackage exports:
/// no pkg-config redirection (the environment is the package's to
/// set, and target-arch pkgconfig paths would also poison the
/// host-side tools of the same build).
#[test]
fn test_setup_environment_exports_cross_pkg_config_libdir() {
fn test_setup_environment_exports_dpkg_cross_variables_only() {
let mut env = HashMap::new();
let ctx = Arc::new(Context::new(crate::context::ContextConfig::Local).unwrap());
setup_environment(&mut env, "riscv64", ctx).unwrap();
assert_eq!(
env.get("PKG_CONFIG_LIBDIR").map(String::as_str),
Some("/usr/lib/riscv64-linux-gnu/pkgconfig:/usr/share/pkgconfig")
);
assert!(!env.contains_key("PKG_CONFIG_LIBDIR"));
assert_eq!(
env.get("DEB_BUILD_PROFILES").map(String::as_str),
Some("cross")
src/deb/ephemeral.rs
+987 -133
View File
File diff suppressed because it is too large Load Diff
src/deb/local.rs
+1067 -67
View File
File diff suppressed because it is too large Load Diff
src/deb/mod.rs
+1175 -77
View File
File diff suppressed because it is too large Load Diff
src/deb/session.rs
+3617
View File
File diff suppressed because it is too large Load Diff
src/debian/arch.rs
+56 -18
View File
@@ -682,14 +682,45 @@ pub fn is_invalid(arch: &str, positive: bool) -> bool {
}
}
/// Failure of the architecture helpers: an invalid list entry, an
/// undetectable native architecture, or an architecture unknown to dpkg's
/// tables.
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
pub enum ArchError {
/// An entry of the list is not a valid architecture name.
#[error("'{arch}' is not a valid architecture in list '{list}'")]
InvalidInList {
/// The offending architecture name.
arch: String,
/// The whole list.
list: String,
},
/// Neither dpkg nor uname could name the native architecture.
#[error("cannot determine native Debian architecture")]
NativeUndetermined,
/// The architecture is not in dpkg's tables.
#[error("unknown Debian architecture '{arch}'")]
Unknown {
/// The unknown architecture name.
arch: String,
},
/// A GNU triplet without the expected `cpu-system` shape.
#[error("invalid GNU triplet '{triplet}'")]
InvalidTriplet {
/// The malformed triplet.
triplet: String,
},
}
/// Parse a whitespace-separated architecture list, validating every entry.
pub fn list_parse(list: &str) -> Result<Vec<String>, String> {
pub fn list_parse(list: &str) -> Result<Vec<String>, ArchError> {
let arches: Vec<String> = list.split_whitespace().map(str::to_string).collect();
for arch in &arches {
if is_invalid(arch, false) {
return Err(format!(
"'{arch}' is not a valid architecture in list '{list}'"
));
return Err(ArchError::InvalidInList {
arch: arch.clone(),
list: list.to_string(),
});
}
}
Ok(arches)
@@ -802,7 +833,7 @@ fn from_uname() -> Option<String> {
/// the `DEB_BUILD_*` variables): the authoritative answer comes from the
/// dpkg database itself; if the `dpkg` frontend cannot be executed, the
/// architecture is derived from `uname` through the same tables.
pub fn native() -> Result<String, String> {
pub fn native() -> Result<String, ArchError> {
if let Ok(output) = Command::new("dpkg").arg("--print-architecture").output()
&& output.status.success()
{
@@ -811,7 +842,7 @@ pub fn native() -> Result<String, String> {
return Ok(arch);
}
}
from_uname().ok_or_else(|| "cannot determine native Debian architecture".to_string())
from_uname().ok_or(ArchError::NativeUndetermined)
}
/// Compute the complete architecture environment, the equivalent of
@@ -821,7 +852,7 @@ pub fn native() -> Result<String, String> {
/// The target architecture defaults to the host architecture, and the host
/// architecture defaults to the native build architecture, exactly like
/// dpkg-architecture.
pub fn arch_env(host_arch: Option<&str>) -> Result<BTreeMap<String, String>, String> {
pub fn arch_env(host_arch: Option<&str>) -> Result<BTreeMap<String, String>, ArchError> {
let build_arch = native()?;
let host_arch = host_arch.unwrap_or(&build_arch).to_string();
let target_arch = host_arch.clone();
@@ -832,8 +863,9 @@ pub fn arch_env(host_arch: Option<&str>) -> Result<BTreeMap<String, String>, Str
("HOST", host_arch),
("TARGET", target_arch),
] {
let tuple = debarch_to_debtuple(&arch)
.ok_or_else(|| format!("unknown Debian architecture '{arch}'"))?;
let tuple = debarch_to_debtuple(&arch).ok_or_else(|| ArchError::Unknown {
arch: arch.to_string(),
})?;
env.insert(format!("DEB_{role}_ARCH"), arch.clone());
env.insert(format!("DEB_{role}_ARCH_ABI"), tuple.abi.clone());
@@ -841,20 +873,26 @@ pub fn arch_env(host_arch: Option<&str>) -> Result<BTreeMap<String, String>, Str
env.insert(format!("DEB_{role}_ARCH_OS"), tuple.os.clone());
env.insert(format!("DEB_{role}_ARCH_CPU"), tuple.cpu.clone());
let (bits, endian) =
abi_attrs(&arch).ok_or_else(|| format!("unknown Debian architecture '{arch}'"))?;
let (bits, endian) = abi_attrs(&arch).ok_or_else(|| ArchError::Unknown {
arch: arch.to_string(),
})?;
env.insert(format!("DEB_{role}_ARCH_BITS"), bits.to_string());
env.insert(format!("DEB_{role}_ARCH_ENDIAN"), endian.to_string());
let multi =
multiarch(&arch).ok_or_else(|| format!("unknown Debian architecture '{arch}'"))?;
let multi = multiarch(&arch).ok_or_else(|| ArchError::Unknown {
arch: arch.to_string(),
})?;
env.insert(format!("DEB_{role}_MULTIARCH"), multi);
let gnu_type = debarch_to_gnutriplet(&arch)
.ok_or_else(|| format!("unknown Debian architecture '{arch}'"))?;
let (gnu_cpu, gnu_system) = gnu_type
.split_once('-')
.ok_or_else(|| format!("invalid GNU triplet '{gnu_type}'"))?;
let gnu_type = debarch_to_gnutriplet(&arch).ok_or_else(|| ArchError::Unknown {
arch: arch.to_string(),
})?;
let (gnu_cpu, gnu_system) =
gnu_type
.split_once('-')
.ok_or_else(|| ArchError::InvalidTriplet {
triplet: gnu_type.clone(),
})?;
env.insert(format!("DEB_{role}_GNU_CPU"), gnu_cpu.to_string());
env.insert(format!("DEB_{role}_GNU_SYSTEM"), gnu_system.to_string());
env.insert(format!("DEB_{role}_GNU_TYPE"), gnu_type);
src/debian/checksums.rs
+72 -23
View File
@@ -100,6 +100,48 @@ pub struct FileChecksums {
index: HashMap<String, usize>,
}
/// Failure of [`FileChecksums::parse_field`]: a malformed checksum field
/// line, naming the field, the offending line and what is wrong with it.
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
pub enum ChecksumParseError {
/// The line has other than `checksum size name` columns.
#[error(
"malformed '{field}' line (expected 'checksum size name', got {columns} columns): '{line}'"
)]
MalformedLine {
/// The checksum field the line came from (`Checksums-Sha256`, ...).
field: String,
/// The actual number of whitespace-separated columns.
columns: usize,
/// The offending line.
line: String,
},
/// The size column is not a number.
#[error("malformed '{field}' line (size '{size}' is not a number): '{line}'")]
BadSize {
/// The checksum field the line came from.
field: String,
/// The non-numeric size text.
size: String,
/// The offending line.
line: String,
},
/// The digest is not the expected number of lowercase hex characters.
#[error(
"malformed '{field}' line (digest '{digest}' is not {len} lowercase hex characters): '{line}'"
)]
BadDigest {
/// The checksum field the line came from.
field: String,
/// The malformed digest.
digest: String,
/// The expected digest length.
len: usize,
/// The offending line.
line: String,
},
}
impl FileChecksums {
/// Create an empty registry.
pub fn new() -> Self {
@@ -234,7 +276,11 @@ impl FileChecksums {
/// `build::parse_checksum_field` helper (which additionally accepts the
/// legacy 5-column `Files` layout); the two are intentionally not unified
/// across modules.
pub fn parse_field(kind: ChecksumKind, value: &str) -> Result<Vec<(String, Entry)>, String> {
pub fn parse_field(
kind: ChecksumKind,
value: &str,
) -> Result<Vec<(String, Entry)>, ChecksumParseError> {
let field = kind.field_name().to_string();
let mut entries = Vec::new();
for line in value.lines() {
if line.trim().is_empty() {
@@ -242,30 +288,28 @@ impl FileChecksums {
}
let tokens: Vec<&str> = line.split_whitespace().collect();
let [digest, size, name] = tokens.as_slice() else {
return Err(format!(
"malformed '{}' line (expected 'checksum size name', got {} \
columns): '{line}'",
kind.field_name(),
tokens.len()
));
return Err(ChecksumParseError::MalformedLine {
field,
columns: tokens.len(),
line: line.to_string(),
});
};
let size: u64 = size.parse().map_err(|_| {
format!(
"malformed '{}' line (size '{size}' is not a number): '{line}'",
kind.field_name()
)
let size: u64 = size.parse().map_err(|_| ChecksumParseError::BadSize {
field: field.clone(),
size: size.to_string(),
line: line.to_string(),
})?;
let digest_ok = digest.len() == kind.digest_len()
&& digest
.bytes()
.all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b));
if !digest_ok {
return Err(format!(
"malformed '{}' line (digest '{digest}' is not {} lowercase \
hex characters): '{line}'",
kind.field_name(),
kind.digest_len()
));
return Err(ChecksumParseError::BadDigest {
field,
digest: digest.to_string(),
len: kind.digest_len(),
line: line.to_string(),
});
}
let mut entry = Entry {
size,
@@ -401,27 +445,32 @@ mod tests {
assert_eq!(entries[0].1.size, 12);
// 2 columns: missing the name.
let err =
FileChecksums::parse_field(ChecksumKind::Sha512, &format!("{digest} 12")).unwrap_err();
let err = FileChecksums::parse_field(ChecksumKind::Sha512, &format!("{digest} 12"))
.unwrap_err()
.to_string();
assert!(err.contains("Checksums-Sha512"), "{err}");
assert!(err.contains(&format!("{digest} 12")), "{err}");
// 4 columns.
let err =
FileChecksums::parse_field(ChecksumKind::Sha512, &format!(" {digest} 12 bogus a.txt"))
.unwrap_err();
.unwrap_err()
.to_string();
assert!(err.contains("Checksums-Sha512"), "{err}");
assert!(err.contains("a.txt"), "{err}");
// Non-numeric size.
let err =
FileChecksums::parse_field(ChecksumKind::Sha512, &format!(" {digest} twelve a.txt"))
.unwrap_err();
.unwrap_err()
.to_string();
assert!(err.contains("not a number"), "{err}");
assert!(err.contains("twelve"), "{err}");
// Digest that is not 128 lowercase hex characters.
let err = FileChecksums::parse_field(ChecksumKind::Sha512, " abc123 12 a.txt").unwrap_err();
let err = FileChecksums::parse_field(ChecksumKind::Sha512, " abc123 12 a.txt")
.unwrap_err()
.to_string();
assert!(err.contains("lowercase hex"), "{err}");
}
src/debian/control.rs
+20 -7
View File
@@ -403,6 +403,21 @@ pub struct ControlInfo {
pub binaries: Vec<Paragraph>,
}
/// Failure of [`ControlInfo::parse_content`]: the content is not shaped
/// like a `debian/control` file.
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
pub enum ControlParseError {
/// The content has no paragraphs at all.
#[error("control file has no paragraphs")]
NoParagraphs,
/// The first (source) stanza carries no `Source` field.
#[error("first control paragraph has no 'Source' field")]
NoSourceField,
/// A binary stanza carries no `Package` field.
#[error("binary control paragraph has no 'Package' field")]
NoBinaryPackageField,
}
impl ControlInfo {
/// Parse a `debian/control` file.
pub fn parse(path: &Path) -> Result<ControlInfo, Box<dyn std::error::Error>> {
@@ -416,19 +431,17 @@ impl ControlInfo {
/// Parse control content from a string.
///
/// Prefer [`std::str::FromStr`] (`"...".parse::<ControlInfo>()`).
pub fn parse_content(content: &str) -> Result<ControlInfo, String> {
pub fn parse_content(content: &str) -> Result<ControlInfo, ControlParseError> {
let paragraphs = parse_paragraphs(content);
let mut iter = paragraphs.into_iter();
let source = iter
.next()
.ok_or_else(|| "control file has no paragraphs".to_string())?;
let source = iter.next().ok_or(ControlParseError::NoParagraphs)?;
if source.get("Source").is_none() {
return Err("first control paragraph has no 'Source' field".to_string());
return Err(ControlParseError::NoSourceField);
}
let binaries: Vec<Paragraph> = iter.collect();
for bin in &binaries {
if bin.get("Package").is_none() {
return Err("binary control paragraph has no 'Package' field".to_string());
return Err(ControlParseError::NoBinaryPackageField);
}
}
Ok(ControlInfo { source, binaries })
@@ -451,7 +464,7 @@ impl ControlInfo {
}
impl std::str::FromStr for ControlInfo {
type Err = String;
type Err = ControlParseError;
fn from_str(content: &str) -> Result<Self, Self::Err> {
ControlInfo::parse_content(content)
src/debian/deps.rs
+135 -33
View File
@@ -167,22 +167,96 @@ fn profile_name_regex() -> &'static Regex {
})
}
/// Parse one simple dependency alternative.
///
/// Fails when the text does not match the grammar, when `:native` appears in
/// a non-build-dependency context, or when the architecture/version/profile
/// syntax is invalid.
pub fn parse_simple(dep: &str, build_dep: bool) -> Result<PkgRelation, String> {
/// Failure of the dependency parsers: a syntactically invalid relation or
/// restriction formula, an invalid parse option, or an unreadable dpkg
/// status file.
#[derive(Debug, thiserror::Error)]
pub enum ParseError {
/// The dependency text does not match the package relation grammar.
#[error("cannot parse dependency '{dep}'")]
Syntax {
/// The unparsable dependency text.
dep: String,
},
/// `:native` appeared outside a build-dependency context.
#[error("':native' qualifier only allowed in build dependencies: '{dep}'")]
NativeOutsideBuildDep {
/// The offending dependency text.
dep: String,
},
/// The version relation operator is not one of `<< <= = >= >>`.
#[error("invalid relation '{relation}' in '{dep}'")]
InvalidRelation {
/// The unrecognized operator.
relation: String,
/// The dependency text carrying it.
dep: String,
},
/// The constrained version does not parse.
#[error("invalid version in dependency '{dep}': {source}")]
InvalidVersion {
/// The dependency text carrying it.
dep: String,
/// The version parse failure.
source: crate::debian::version::VersionParseError,
},
/// The `[...]` architecture list does not parse.
#[error("invalid architecture list in '{dep}': {source}")]
InvalidArchList {
/// The dependency text carrying it.
dep: String,
/// The architecture list failure.
source: crate::debian::arch::ArchError,
},
/// A `[...]` restriction formula reduced to nothing.
#[error("empty restriction formula in '{dep}'")]
EmptyRestrictions {
/// The dependency text carrying it.
dep: String,
},
/// A `<...>` formula carries a non-profile term.
#[error("'{formula}' is not a valid build profile restriction formula")]
InvalidProfileFormula {
/// The offending formula.
formula: String,
},
/// [`ParseOpts::host_arch`] is empty or not a valid architecture.
#[error("invalid host_arch {host_arch}")]
InvalidHostArch {
/// The offending host architecture.
host_arch: String,
},
/// `union` mode saw a clause with several alternatives.
#[error("an union dependency can only contain simple dependencies")]
UnionNotSimple,
/// The dpkg status file could not be read.
#[error("cannot read status file '{path}': {source}")]
ReadStatus {
/// The status file path.
path: String,
/// The read failure.
source: std::io::Error,
},
}
/// Parse one simple (no alternatives) package relation: `pkg`
/// (`pkg:qualifier`, `pkg (op version)`, `pkg [arch list]`,
/// `pkg <restrictions>` combinations), validating the pieces: `:native`
/// is rejected in a non-build-dependency context, or when the
/// architecture/version/profile syntax is invalid.
pub fn parse_simple(dep: &str, build_dep: bool) -> Result<PkgRelation, ParseError> {
let caps = dep_regex()
.captures(dep)
.ok_or_else(|| format!("cannot parse dependency '{dep}'"))?;
.ok_or_else(|| ParseError::Syntax {
dep: dep.to_string(),
})?;
let package = caps[2].to_string();
let arch_qualifier = caps.get(3).map(|m| m.as_str().to_string());
if arch_qualifier.as_deref() == Some("native") && !build_dep {
return Err(format!(
"':native' qualifier only allowed in build dependencies: '{dep}'"
));
return Err(ParseError::NativeOutsideBuildDep {
dep: dep.to_string(),
});
}
let constraint = match (caps.get(5), caps.get(6)) {
@@ -197,36 +271,49 @@ pub fn parse_simple(dep: &str, build_dep: bool) -> Result<PkgRelation, String> {
"=" => Relation::Eq,
">" | ">=" => Relation::Ge,
">>" => Relation::Gt,
other => return Err(format!("invalid relation '{other}' in '{dep}'")),
other => {
return Err(ParseError::InvalidRelation {
relation: other.to_string(),
dep: dep.to_string(),
});
}
};
let version = DebianVersion::parse(version.as_str())
.map_err(|e| format!("invalid version in dependency '{dep}': {e}"))?;
let version = DebianVersion::parse(version.as_str()).map_err(|source| {
ParseError::InvalidVersion {
dep: dep.to_string(),
source,
}
})?;
Some(VersionConstraint { relation, version })
}
_ => None,
};
let arches = match caps.get(8) {
Some(list) => Some(
arch::list_parse(list.as_str())
.map_err(|e| format!("invalid architecture list in '{dep}': {e}"))?,
),
None => None,
};
let arches =
match caps.get(8) {
Some(list) => Some(arch::list_parse(list.as_str()).map_err(|source| {
ParseError::InvalidArchList {
dep: dep.to_string(),
source,
}
})?),
None => None,
};
let mut restrictions = Vec::new();
if let Some(formula) = caps.get(9) {
for group in restriction_group_regex().captures_iter(formula.as_str()) {
let terms: Vec<String> = group[1].split_whitespace().map(str::to_string).collect();
if terms.is_empty() {
return Err(format!("empty restriction formula in '{dep}'"));
return Err(ParseError::EmptyRestrictions {
dep: dep.to_string(),
});
}
for term in &terms {
if !profile_name_regex().is_match(term) {
return Err(format!(
"'{}' is not a valid build profile restriction formula",
formula.as_str()
));
return Err(ParseError::InvalidProfileFormula {
formula: formula.as_str().to_string(),
});
}
}
restrictions.push(terms);
@@ -291,13 +378,19 @@ impl Deps {
/// alternatives whose architecture or profile restrictions do not apply
/// are dropped first; clauses losing all their alternatives disappear,
/// and a fully reduced field yields an empty [`Deps`].
pub fn parse(input: &str, opts: &ParseOpts) -> Result<Deps, String> {
pub fn parse(input: &str, opts: &ParseOpts) -> Result<Deps, ParseError> {
Self::parse_inner(input, opts, false)
}
fn parse_inner(input: &str, opts: &ParseOpts, reduce_arch_only: bool) -> Result<Deps, String> {
fn parse_inner(
input: &str,
opts: &ParseOpts,
reduce_arch_only: bool,
) -> Result<Deps, ParseError> {
if opts.host_arch.is_empty() || arch::is_invalid(&opts.host_arch, true) {
return Err(format!("invalid host_arch {}", opts.host_arch));
return Err(ParseError::InvalidHostArch {
host_arch: opts.host_arch.clone(),
});
}
// Merge continuation lines and normalize whitespace.
@@ -336,7 +429,7 @@ impl Deps {
continue;
}
if opts.union && alternatives.len() > 1 {
return Err("an union dependency can only contain simple dependencies".to_string());
return Err(ParseError::UnionNotSimple);
}
clauses.push(alternatives);
}
@@ -652,9 +745,15 @@ impl Facts {
/// Load a dpkg status file (e.g. `/var/lib/dpkg/status`), binding the
/// facts to the given host/build architectures.
pub fn load_status(path: &Path, host_arch: &str, build_arch: &str) -> Result<Facts, String> {
let content = std::fs::read_to_string(path)
.map_err(|e| format!("cannot read status file '{}': {}", path.display(), e))?;
pub fn load_status(
path: &Path,
host_arch: &str,
build_arch: &str,
) -> Result<Facts, ParseError> {
let content = std::fs::read_to_string(path).map_err(|source| ParseError::ReadStatus {
path: path.display().to_string(),
source,
})?;
Ok(Facts::from_status(&content, host_arch, build_arch))
}
@@ -936,7 +1035,10 @@ impl UnmetReport {
///
/// On success the report is empty; otherwise its [`UnmetReport::message`]
/// carries the dpkg-compatible diagnostics.
pub fn check_build_depends(control: &ControlInfo, opts: &CheckOpts) -> Result<UnmetReport, String> {
pub fn check_build_depends(
control: &ControlInfo,
opts: &CheckOpts,
) -> Result<UnmetReport, ParseError> {
let source = &control.source;
let mut bd_parts: Vec<&str> = Vec::new();
src/debian/files.rs
+55 -4
View File
@@ -46,6 +46,21 @@ impl FilesEntry {
entry.priority = priority.to_string();
entry
}
/// Whether this entry is a binary-package artifact that must be
/// exported from the build: a `.deb`, a `.udeb`, or a `.ddeb`.
///
/// Debug-symbol packages are registered in `debian/files` by
/// `dh_strip` under their renamed `.ddeb` name; they are as much
/// a deliverable of the build as the regular `.deb`s (they belong
/// in the upload set), so any filter keyed on artifact type has
/// to accept them.
pub fn is_binary_artifact(&self) -> bool {
matches!(
self.package_type.as_deref(),
Some("deb") | Some("udeb") | Some("ddeb")
)
}
}
/// Derive metadata from an artifact file name following the common
@@ -106,6 +121,23 @@ pub struct FilesList {
files: BTreeMap<String, FilesEntry>,
}
/// Failure of [`FilesList::parse`]: a malformed `debian/files` line.
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
pub enum FilesParseError {
/// The line has fewer than `filename section priority` columns.
#[error("badly formed line: {line}")]
BadLine {
/// The offending line.
line: String,
},
/// The file name does not match any known artifact shape.
#[error("badly formed file name: {name}")]
BadFileName {
/// The offending file name.
name: String,
},
}
impl FilesList {
/// An empty registry.
pub fn new() -> Self {
@@ -126,7 +158,7 @@ impl FilesList {
/// Parse a `debian/files` registry from its textual content
/// (`filename section priority [key=value...]` lines).
pub fn parse(content: &str) -> Result<FilesList, String> {
pub fn parse(content: &str) -> Result<FilesList, FilesParseError> {
let mut list = FilesList::new();
for line in content.lines() {
if line.trim().is_empty() {
@@ -134,10 +166,14 @@ impl FilesList {
}
let tokens: Vec<&str> = line.split_whitespace().collect();
if tokens.len() < 3 {
return Err(format!("badly formed line: {line}"));
return Err(FilesParseError::BadLine {
line: line.to_string(),
});
}
let mut entry = parse_filename(tokens[0])
.ok_or_else(|| format!("badly formed file name: {}", tokens[0]))?;
let mut entry =
parse_filename(tokens[0]).ok_or_else(|| FilesParseError::BadFileName {
name: tokens[0].to_string(),
})?;
entry.section = tokens[1].to_string();
entry.priority = tokens[2].to_string();
for attr in &tokens[3..] {
@@ -243,6 +279,21 @@ mod tests {
assert_eq!(e.package_type, None);
}
#[test]
fn binary_artifact_filter_accepts_ddebs() {
let deb = parse_filename("hello_2.10-3_amd64.deb").unwrap();
let udeb = parse_filename("hello_2.10-3_amd64.udeb").unwrap();
let ddeb = parse_filename("hello-dbgsym_2.10-3_amd64.ddeb").unwrap();
let buildinfo = parse_filename("hello_2.10-3_amd64.buildinfo").unwrap();
let changes = parse_filename("hello_2.10-3_amd64.changes").unwrap();
assert!(deb.is_binary_artifact());
assert!(udeb.is_binary_artifact());
assert!(ddeb.is_binary_artifact(), "ddebs must be exported");
assert!(!buildinfo.is_binary_artifact());
assert!(!changes.is_binary_artifact());
}
#[test]
fn load_save_roundtrip() {
let dir = tempfile::tempdir().unwrap();
src/debian/mod.rs
+3
View File
@@ -10,6 +10,7 @@
//! - [`files`]: `debian/files` artifact registry (`Dpkg::Dist::Files`)
//! - [`version`]: Debian version splitting/validation/comparison
//! - [`changelog`]: `debian/changelog` entry parsing
//! - [`tests_control`]: DEP-8 test metadata (`debian/tests/control`)
pub mod arch;
pub mod changelog;
@@ -17,6 +18,7 @@ pub mod checksums;
pub mod control;
pub mod deps;
pub mod files;
pub mod tests_control;
pub mod version;
pub use changelog::{
@@ -28,4 +30,5 @@ pub use control::{
ControlInfo, Paragraph, parse_paragraphs, strip_clearsigned_armour, write_paragraph,
};
pub use files::{FilesEntry, FilesList};
pub use tests_control::{Test, TestsControl};
pub use version::DebianVersion;
src/debian/tests_control.rs
+286
View File
@@ -0,0 +1,286 @@
//! DEP-8 as-installed test metadata (`debian/tests/control`).
//!
//! Parses the test control file into runnable test descriptions: one
//! [`Test`] per `Tests:` script and per `Test-Command:` value, carrying
//! the stanza fields they share (`Depends:`, `Restrictions:`,
//! `Tests-Directory:`). The `@`-family substitution of `Depends:` needs
//! the list of built binaries and is runner-side logic — this module
//! only hands over the raw value.
use std::path::Path;
use crate::debian::control::parse_paragraphs;
/// One runnable DEP-8 test.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Test {
/// Test name: the `Tests:` script path as written, the `test-name:`
/// value, or the `command1`, `command2`, ... enumeration of
/// `Test-Command:` stanzas. The enumeration advances once per
/// `Test-Command:` stanza whether or not `test-name:` overrides the
/// name.
pub name: String,
/// Script path relative to the tests directory (`Tests:` stanzas):
/// `debian/tests/` or the stanza's [`Test::tests_directory`], with
/// subdirectories as written (`upstream/foo`).
pub script: Option<String>,
/// Inline shell command run under `bash -e` (`Test-Command:` stanzas).
pub command: Option<String>,
/// Raw `Depends:` value as written; `None` when absent, meaning the
/// DEP-8 default `@` (all built binaries).
pub depends: Option<String>,
/// Declared restrictions, in declaration order, as written
/// (e.g. `allow-stderr`).
pub restrictions: Vec<String>,
/// `Tests-Directory:` replacing the `debian/tests` path segment for
/// this stanza's scripts (relative path).
pub tests_directory: Option<String>,
}
impl Test {
/// Whether the test declares `restriction` (case-insensitive).
pub fn has_restriction(&self, restriction: &str) -> bool {
self.restrictions
.iter()
.any(|r| r.eq_ignore_ascii_case(restriction))
}
}
/// Parsed `debian/tests/control`: the runnable tests, in declaration
/// order.
#[derive(Debug, Clone, PartialEq, Eq, Default)]
pub struct TestsControl {
/// One entry per `Tests:` script and per `Test-Command:` stanza.
pub tests: Vec<Test>,
}
/// Failure of [`TestsControl::parse_content`]: a malformed stanza, naming
/// it by its 1-based position.
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
pub enum TestsControlParseError {
/// The stanza carries both `Tests:` and `Test-Command:`.
#[error("stanza {stanza}: Tests and Test-Command are mutually exclusive")]
MutuallyExclusive {
/// The 1-based stanza position.
stanza: usize,
},
/// The stanza carries neither `Tests:` nor `Test-Command:`.
#[error("stanza {stanza}: missing a Tests or Test-Command field")]
MissingField {
/// The 1-based stanza position.
stanza: usize,
},
/// `test-name:` appeared on a `Tests:` (script list) stanza.
#[error("stanza {stanza}: test-name only applies to a Test-Command stanza")]
TestNameOnScripts {
/// The 1-based stanza position.
stanza: usize,
},
}
impl TestsControl {
/// Parse the test control file at `path`.
pub fn parse(path: &Path) -> Result<TestsControl, Box<dyn std::error::Error>> {
let content = std::fs::read_to_string(path)?;
Ok(TestsControl::parse_content(&content)?)
}
/// Parse the test control file from its content.
///
/// Every non-empty stanza must carry exactly one of `Tests:` or
/// `Test-Command:` (a script list expands to one test per script,
/// sharing the stanza fields); `test-name:` is only valid on a
/// `Test-Command:` stanza. A file without stanzas parses to an empty
/// [`TestsControl`] — whether the package declares tests at all is
/// the caller's check (`Testsuite: autopkgtest`).
pub fn parse_content(content: &str) -> Result<TestsControl, TestsControlParseError> {
let mut tests_control = TestsControl::default();
let mut command_index = 0usize;
for (index, paragraph) in parse_paragraphs(content).iter().enumerate() {
let stanza = index + 1;
let tests_field = non_empty(paragraph.get("Tests"));
let command_field = non_empty(paragraph.get("Test-Command"));
let test_name = non_empty(paragraph.get("test-name"));
let depends = non_empty(paragraph.get("Depends")).map(String::from);
let restrictions: Vec<String> = non_empty(paragraph.get("Restrictions"))
.map(|value| value.split_whitespace().map(String::from).collect())
.unwrap_or_default();
let tests_directory = non_empty(paragraph.get("Tests-Directory")).map(String::from);
match (tests_field, command_field) {
(Some(_), Some(_)) => {
return Err(TestsControlParseError::MutuallyExclusive { stanza });
}
(None, None) => {
return Err(TestsControlParseError::MissingField { stanza });
}
(Some(scripts), None) => {
if test_name.is_some() {
return Err(TestsControlParseError::TestNameOnScripts { stanza });
}
for script in scripts.split_whitespace() {
tests_control.tests.push(Test {
name: script.to_string(),
script: Some(script.to_string()),
command: None,
depends: depends.clone(),
restrictions: restrictions.clone(),
tests_directory: tests_directory.clone(),
});
}
}
(None, Some(command)) => {
command_index += 1;
let name = test_name
.map(String::from)
.unwrap_or_else(|| format!("command{command_index}"));
tests_control.tests.push(Test {
name,
script: None,
command: Some(command.to_string()),
depends: depends.clone(),
restrictions: restrictions.clone(),
tests_directory: tests_directory.clone(),
});
}
}
}
Ok(tests_control)
}
}
/// A field value with its surrounding whitespace trimmed; `None` when the
/// field is absent or whitespace-only.
fn non_empty(value: Option<&str>) -> Option<&str> {
value.map(str::trim).filter(|value| !value.is_empty())
}
#[cfg(test)]
mod tests {
use super::*;
/// A realistic control file: a restrictions stanza, a multi-script
/// stanza sharing its fields, a Tests-Directory stanza and two
/// Test-Command stanzas (one enumerated, one named).
#[test]
fn parse_golden_multi_stanza() {
let control = TestsControl::parse_content(
"# a leading comment\n\
\n\
Tests: smoke\n\
Restrictions: allow-stderr needs-root\n\
Depends: @, build-essential\n\
\n\
Tests: one two\n\
Depends: @\n\
\n\
Tests: checks/cli\n\
Tests-Directory: tests\n\
\n\
Test-Command: hello --version\n\
Depends: @builddeps@\n\
\n\
Test-Command: python3 -c \"print('ok')\"\n\
test-name: py-smoke\n\
Restrictions: superficial skippable\n",
)
.unwrap();
let names: Vec<&str> = control.tests.iter().map(|t| t.name.as_str()).collect();
assert_eq!(
names,
["smoke", "one", "two", "checks/cli", "command1", "py-smoke"]
);
let smoke = &control.tests[0];
assert_eq!(smoke.script.as_deref(), Some("smoke"));
assert_eq!(smoke.depends.as_deref(), Some("@, build-essential"));
assert!(smoke.has_restriction("NEEDS-ROOT"));
assert!(!smoke.has_restriction("skippable"));
// The multi-script stanza's fields are shared by every test.
assert_eq!(control.tests[1].depends, control.tests[2].depends);
assert_eq!(control.tests[2].script.as_deref(), Some("two"));
let cli = &control.tests[3];
assert_eq!(cli.tests_directory.as_deref(), Some("tests"));
assert_eq!(cli.depends, None); // absent: the runner defaults to @
let enumerated = &control.tests[4];
assert_eq!(enumerated.command.as_deref(), Some("hello --version"));
assert_eq!(enumerated.script, None);
let named = &control.tests[5];
assert_eq!(named.name, "py-smoke");
assert!(named.has_restriction("skippable") && named.has_restriction("superficial"));
}
/// The command enumeration advances per Test-Command stanza, even
/// when test-name overrides the name.
#[test]
fn command_numbering_skips_named_stanzas() {
let control = TestsControl::parse_content(
"Test-Command: a\n\
test-name: named\n\
\n\
Test-Command: b\n",
)
.unwrap();
assert_eq!(
control
.tests
.iter()
.map(|t| t.name.as_str())
.collect::<Vec<_>>(),
["named", "command2"]
);
}
/// Continuation lines join field values; comments inside a value are
/// not possible, but comment lines between fields are ignored.
#[test]
fn continuation_lines_join_values() {
let control = TestsControl::parse_content(concat!(
"Tests: smoke\n",
"Depends: @,\n",
" glib-network-service,\n",
"# comment line\n",
" locales-all\n",
))
.unwrap();
assert_eq!(
control.tests[0].depends.as_deref(),
Some("@,\nglib-network-service,\nlocales-all")
);
}
/// An empty file (or only comments) parses to no tests.
#[test]
fn empty_file_parses_to_no_tests() {
assert!(TestsControl::parse_content("").unwrap().tests.is_empty());
assert!(
TestsControl::parse_content("# nothing\n\n")
.unwrap()
.tests
.is_empty()
);
}
/// Malformed stanzas are errors, naming the stanza.
#[test]
fn malformed_stanzas_are_errors() {
for (content, expected) in [
("Depends: @\n", "missing a Tests or Test-Command"),
("Tests: a\nTest-Command: b\n", "mutually exclusive"),
(
"Test-Command: b\ntest-name: x\n\nTests: a\ntest-name: y\n",
"only applies",
),
] {
let error = TestsControl::parse_content(content)
.unwrap_err()
.to_string();
assert!(error.contains(expected), "{error}");
}
}
}
src/debian/version.rs
+66 -13
View File
@@ -12,19 +12,65 @@ pub struct DebianVersion {
pub debian_revision: Option<String>,
}
/// Failure of [`DebianVersion::parse`]: the string is outside the
/// `[epoch:]upstream[-revision]` syntax.
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
pub enum VersionParseError {
/// The string, once trimmed, is empty.
#[error("empty version string")]
Empty,
/// The epoch (before the first `:`) is not a number.
#[error("invalid epoch '{epoch}' in version '{raw}'")]
InvalidEpoch {
/// The non-numeric epoch text.
epoch: String,
/// The whole version string.
raw: String,
},
/// Nothing precedes the revision separator: no upstream version.
#[error("missing upstream version in '{raw}'")]
MissingUpstream {
/// The whole version string.
raw: String,
},
/// A character outside dpkg's upstream alphabet.
#[error("invalid character '{c}' in version '{raw}'")]
InvalidUpstreamChar {
/// The offending character.
c: char,
/// The whole version string.
raw: String,
},
/// A trailing hyphen: the revision number is empty (dpkg rejects it).
#[error("empty debian revision in '{raw}'")]
EmptyRevision {
/// The whole version string.
raw: String,
},
/// A character outside dpkg's revision alphabet.
#[error("invalid character '{c}' in revision of version '{raw}'")]
InvalidRevisionChar {
/// The offending character.
c: char,
/// The whole version string.
raw: String,
},
}
impl DebianVersion {
/// Parse and validate a Debian version string.
pub fn parse(raw: &str) -> Result<DebianVersion, String> {
pub fn parse(raw: &str) -> Result<DebianVersion, VersionParseError> {
let raw = raw.trim();
if raw.is_empty() {
return Err("empty version string".to_string());
return Err(VersionParseError::Empty);
}
let (epoch, rest) = match raw.split_once(':') {
Some((e, r)) => {
let epoch: u32 = e
.parse()
.map_err(|_| format!("invalid epoch '{}' in version '{}'", e, raw))?;
let epoch: u32 = e.parse().map_err(|_| VersionParseError::InvalidEpoch {
epoch: e.to_string(),
raw: raw.to_string(),
})?;
(Some(epoch), r)
}
None => (None, raw),
@@ -37,14 +83,19 @@ impl DebianVersion {
};
if upstream.is_empty() {
return Err(format!("missing upstream version in '{}'", raw));
return Err(VersionParseError::MissingUpstream {
raw: raw.to_string(),
});
}
for c in upstream.chars() {
if !(c.is_ascii_alphanumeric()
|| matches!(c, '.' | '+' | '-' | '~' | ':')
|| !c.is_ascii())
{
return Err(format!("invalid character '{}' in version '{}'", c, raw));
return Err(VersionParseError::InvalidUpstreamChar {
c,
raw: raw.to_string(),
});
}
}
if let Some(rev) = &debian_revision {
@@ -52,14 +103,16 @@ impl DebianVersion {
// dpkg rejects a trailing hyphen: "bad syntax: revision
// number is empty". Native versions (no `-` at all) are
// handled above and stay valid.
return Err(format!("empty debian revision in '{}'", raw));
return Err(VersionParseError::EmptyRevision {
raw: raw.to_string(),
});
}
for c in rev.chars() {
if !(c.is_ascii_alphanumeric() || matches!(c, '.' | '+' | '~') || !c.is_ascii()) {
return Err(format!(
"invalid character '{}' in revision of version '{}'",
c, raw
));
return Err(VersionParseError::InvalidRevisionChar {
c,
raw: raw.to_string(),
});
}
}
}
@@ -340,7 +393,7 @@ mod tests {
/// the *last* hyphen (upstream `1.0-` + revision `1`).
#[test]
fn version_empty_revision() {
let err = DebianVersion::parse("1.0-").unwrap_err();
let err = DebianVersion::parse("1.0-").unwrap_err().to_string();
assert!(err.contains("empty"), "unexpected message: {err}");
assert!(DebianVersion::parse("1.0-").is_err());
src/distro_info.rs
+10 -10
View File
@@ -1,10 +1,10 @@
use crate::data::embed_data;
use chrono::NaiveDate;
use lazy_static::lazy_static;
use serde::Deserialize;
use std::collections::HashMap;
use std::error::Error;
use std::path::Path;
use std::sync::LazyLock;
use std::time::Duration;
#[derive(Debug, Clone)]
@@ -104,20 +104,20 @@ embed_data! {
static ref DATA: Data = "../data/distro_info.yml"
}
lazy_static! {
// Shared HTTP client used for all outgoing plain requests: timeouts keep
// a hanging remote (connect or transfer) from stalling pkh indefinitely.
// The short pool idle timeout and TCP keepalive avoid reusing keep-alive
// connections that the remote closed in the meantime, which surfaces as
// spurious 'error sending request' failures on busy mirrors/CDNs.
static ref HTTP_CLIENT: reqwest::Client = reqwest::Client::builder()
// Shared HTTP client used for all outgoing plain requests: timeouts keep
// a hanging remote (connect or transfer) from stalling pkh indefinitely.
// The short pool idle timeout and TCP keepalive avoid reusing keep-alive
// connections that the remote closed in the meantime, which surfaces as
// spurious 'error sending request' failures on busy mirrors/CDNs.
static HTTP_CLIENT: LazyLock<reqwest::Client> = LazyLock::new(|| {
reqwest::Client::builder()
.connect_timeout(Duration::from_secs(10))
.timeout(Duration::from_secs(30))
.pool_idle_timeout(Duration::from_secs(10))
.tcp_keepalive(Duration::from_secs(30))
.build()
.expect("building the shared HTTP client with static options cannot fail");
}
.expect("building the shared HTTP client with static options cannot fail")
});
/// Shared HTTP client with a connect timeout (10s) and a total request
/// timeout (30s), to be used for all outgoing plain HTTP(S) requests
src/lib.rs
+6 -8
View File
@@ -9,13 +9,16 @@ pub mod apt;
pub mod build;
/// Parse or edit a Debian changelog of a source package
pub mod changelog;
/// User settings: the optional `config.yml` in the pkh configuration
/// directory, its defaults and the environment overrides
pub mod config;
/// Embedding convention for static reference data (`data/*.yml`), applied
/// by each owning module via the `embed_data!` macro
pub(crate) mod data;
/// Build a Debian package into a binary (.deb)
pub mod deb;
/// Reusable Debian format primitives (control/deb822, checksums, versions,
/// changelog entries, artifact registries)
/// changelog entries, test metadata, artifact registries)
pub mod debian;
/// Obtain general information about distribution, series, etc
pub mod distro_info;
@@ -38,6 +41,8 @@ pub mod pull;
pub mod put;
/// Handle package-specific quirks and workarounds
pub mod quirks;
/// Run a source package's DEP-8 as-installed tests (`pkh test`)
pub mod test;
/// Line classifiers rewriting raw subprocess output into display actions
/// and countable progress (pure logic, shared by build views)
@@ -61,13 +66,6 @@ pub(crate) mod test_support;
/// Utility functions
pub(crate) mod utils;
/// Optional callback function (taking 4 arguments)
/// - Name of the current main operation (e.g. pulling package)
/// - Name of the current nested operation (e.g. cloning git repo)
/// - Progress, position, index of current operation (e.g. amount of data downloaded)
/// - Total amount for current operation (e.g. size of the file to download)
pub type ProgressCallback<'a> = Option<&'a dyn Fn(&str, &str, usize, usize)>;
/// Returns the architecture of current CPU, debian-compatible
pub fn get_current_arch() -> String {
match std::env::consts::ARCH {
src/lint/mod.rs
+1 -1
View File
@@ -1,6 +1,6 @@
//! `pkh lint`: lint a Debian source tree, with lintian feature parity.
//!
//! The strategy is *wrap first, port second* (see `plans/pkh-lint.md`): the
//! The strategy is *wrap first, port second*: the
//! [wrapper](wrapper) runs the installed lintian over an ephemeral source
//! package for day-one parity with every lintian check, while the native
//! engine (check registry, collectors, emitter) hosts pkh-specific workflow
src/logfmt.rs
+166 -7
View File
@@ -5,7 +5,7 @@
//! errors) and whether the line carries countable progress. Classifiers are
//! pure with respect to the UI: they only return [`Action`]s.
use std::sync::OnceLock;
use std::sync::{LazyLock, OnceLock};
use crate::context::Stream;
use regex::Regex;
@@ -226,9 +226,18 @@ impl Classifier for QuiltClassifier {
/// Classifier for make/cmake-based builds (`debian/rules build`, dh helpers)
///
/// Detects `[ 42%]`-style progress markers, hides directory enter/leave
/// noise, and shows compile/link/dh lines.
/// noise, and shows compile/link/dh lines as well as kbuild status-tag
/// lines and warnings. Collapses the per-file `chown: ... Operation not
/// permitted` flood of a rootless build into one digest line: a rules
/// chown failing for the unprivileged build user is expected and harmless
/// (the .deb is assembled in the binary step's fakeroot session, which
/// records root ownership anyway), but a sysroot can emit thousands of
/// lines.
#[derive(Default)]
pub struct MakeClassifier {}
pub struct MakeClassifier {
/// How many chown-EPERM lines were collapsed so far.
suppressed_chown: usize,
}
impl MakeClassifier {
/// Create a new classifier
@@ -242,6 +251,23 @@ impl Classifier for MakeClassifier {
static PERCENT_RE: OnceLock<Regex> = OnceLock::new();
let percent_re = PERCENT_RE.get_or_init(|| Regex::new(r"\[\s*(\d+)%\]").unwrap());
// kbuild abbreviates each invocation to an uppercase status tag
// (` CC [M] drivers/x.o`, ` HOSTCC scripts/foo`) instead of
// printing the full command line, so the classic-make substring
// rules below never fire for kernel builds; this tag list is the
// kbuild counterpart of those substrings.
static KBUILD_TAG_RE: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(
r"^\s*(CC|HOSTCC|HOSTLD|HOSTAR|LD|AR|AS|OBJCOPY|STRIP|GEN|UPD|DESCEND|MODPOST|MODORDER|BTF|SHIPPED|INSTALL|INSTALLMOD|DEPMOD|SIGN|WRAP|SYSHDR|SYSTBL|KSYMS)\b",
)
.unwrap()
});
static CHOWN_EPERM_RE: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"^chown: (changing ownership|cannot read directory|cannot operate on)")
.unwrap()
});
if let Some(caps) = percent_re.captures(line) {
let pct: u64 = caps[1].parse().unwrap_or(0);
return vec![
@@ -253,10 +279,16 @@ impl Classifier for MakeClassifier {
];
}
if line.contains("make[")
&& (line.contains("Entering directory") || line.contains("Leaving directory"))
{
return vec![Action::Hidden];
if CHOWN_EPERM_RE.is_match(line) {
self.suppressed_chown += 1;
return if self.suppressed_chown == 1 {
vec![Action::Warning(truncate(
"chown: ownership changes skipped by the unprivileged build user \
(harmless for the .deb; further messages collapsed)",
))]
} else {
vec![Action::Hidden]
};
}
if line.contains("error:")
@@ -267,6 +299,19 @@ impl Classifier for MakeClassifier {
return vec![Action::Error(truncate(line))];
}
// gcc-style diagnostics (`file.c:12:3: warning: ...`) and kbuild's
// uppercase banners (`WARNING: modpost: ...`) must not be drowned
// out by the activity checks below.
if line.contains("warning:") || line.trim_start().starts_with("WARNING:") {
return vec![Action::Warning(truncate(line))];
}
if line.contains("make[")
&& (line.contains("Entering directory") || line.contains("Leaving directory"))
{
return vec![Action::Hidden];
}
if line.starts_with("dh_")
|| line.contains("gcc ")
|| line.contains("g++ ")
@@ -274,6 +319,7 @@ impl Classifier for MakeClassifier {
|| line.contains("clang")
|| line.contains("ld ")
|| line.contains("ar ")
|| KBUILD_TAG_RE.is_match(line)
{
return vec![Action::Shown(truncate(line))];
}
@@ -517,6 +563,119 @@ mod tests {
);
}
/// A rootless build's per-file chown failures collapse into one
/// digest warning; every other line keeps its normal classification.
#[test]
fn test_make_collapses_chown_eperm_flood() {
let mut c = MakeClassifier::new();
let first = "chown: changing ownership of \
'/tmp/pkh-build/k3/demos/sysroot/usr/share/zoneinfo/Asia/Sao_Paulo': \
Operation not permitted";
assert_eq!(
feed_one(&mut c, first),
vec![Action::Warning(
"chown: ownership changes skipped by the unprivileged build user \
(harmless for the .deb; further messages collapsed)"
.to_string()
)]
);
assert_eq!(
feed_one(
&mut c,
"chown: changing ownership of \
'/tmp/pkh-build/k3/demos/sysroot/usr/share/zoneinfo/Asia/Tokyo': \
Operation not permitted"
),
vec![Action::Hidden]
);
assert_eq!(
feed_one(
&mut c,
"chown: cannot read directory '/tmp/pkh-build/k3/demos': Permission denied"
),
vec![Action::Hidden]
);
// A non-chown line keeps its own classification after the flood.
assert_eq!(
feed_one(&mut c, " CC [M] drivers/net/foo.o"),
vec![Action::Shown(" CC [M] drivers/net/foo.o".to_string())]
);
// The digest must not swallow real make errors.
assert_eq!(
feed_one(&mut c, "make[1]: *** [Makefile:531: all] Error 2"),
vec![Action::Error(
"make[1]: *** [Makefile:531: all] Error 2".to_string()
)]
);
}
#[test]
fn test_make_shows_kbuild_status_tags() {
let mut c = MakeClassifier::new();
assert_eq!(
feed_one(&mut c, " CC [M] drivers/net/foo.o"),
vec![Action::Shown(" CC [M] drivers/net/foo.o".to_string())]
);
assert_eq!(
feed_one(&mut c, " LD [M] drivers/bar.ko"),
vec![Action::Shown(" LD [M] drivers/bar.ko".to_string())]
);
assert_eq!(
feed_one(&mut c, " HOSTCC scripts/tool"),
vec![Action::Shown(" HOSTCC scripts/tool".to_string())]
);
assert_eq!(
feed_one(&mut c, " MODPOST 1 modules"),
vec![Action::Shown(" MODPOST 1 modules".to_string())]
);
// kbuild tag lines still roll the pane while make progress markers
// update the progress bar
assert_eq!(
feed_one(&mut c, "[ 45%] Building C object x.o"),
vec![
Action::Shown("[ 45%] Building C object x.o".to_string()),
Action::Progress {
pos: 45,
total: 100
}
]
);
// plain noise stays hidden
assert_eq!(
feed_one(&mut c, "test -x debian/scripts/foo"),
vec![Action::Hidden]
);
}
#[test]
fn test_make_shows_warnings_and_keeps_regressions() {
let mut c = MakeClassifier::new();
assert_eq!(
feed_one(&mut c, "drivers/x.c:12:3: warning: unused variable 'y'"),
vec![Action::Warning(
"drivers/x.c:12:3: warning: unused variable 'y'".to_string()
)]
);
assert_eq!(
feed_one(&mut c, "WARNING: modpost: missing MODULE_LICENSE"),
vec![Action::Warning(
"WARNING: modpost: missing MODULE_LICENSE".to_string()
)]
);
// configure-stage probe lines embed the toolchain name and must
// stay visible
assert_eq!(
feed_one(
&mut c,
"checking whether riscv64-linux-gnu-gcc understands -c and -o together... (cached) yes"
),
vec![Action::Shown(
"checking whether riscv64-linux-gnu-gcc understands -c and -o together... (cached) yes"
.to_string()
)]
);
}
#[test]
fn test_mmdebstrap_prefixes() {
let mut c = MmdebstrapClassifier::new();
src/main.rs
+464 -65
View File
@@ -1,5 +1,5 @@
use std::env;
use std::io::Write;
use std::io::{IsTerminal, Write};
extern crate clap;
use clap::{Command, arg, command};
@@ -12,14 +12,35 @@ use log::{error, info};
/// Obtain the current working directory, exiting with a helpful message on failure.
fn current_dir_or_exit() -> std::path::PathBuf {
match std::env::current_dir() {
Ok(p) => p,
Err(e) => {
error!("Could not determine the current working directory: {}", e);
Ok(dir) => dir,
Err(err) => {
error!("cannot determine the current directory: {err}");
std::process::exit(1);
}
}
}
/// Wrap `text` in the ANSI `code` (e.g. "1" for bold) when stderr is a
/// terminal; return it unchanged otherwise (piped output keeps no escape
/// codes).
fn ansi_wrap(code: &str, text: &str) -> String {
if std::io::stderr().is_terminal() {
format!("\x1b[{code}m{text}\x1b[0m")
} else {
text.to_string()
}
}
/// Bold text for the prune digest.
fn bold(text: &str) -> String {
ansi_wrap("1", text)
}
/// Cyan text for the prune digest.
fn cyan(text: &str) -> String {
ansi_wrap("36", text)
}
/// CLI-side Ctrl+C wiring. The passive state (interrupted flag, cleanup
/// hook registry, reporter slot) lives in `pkh::interrupt`; everything that
/// installs, prints or exits lives here: the SIGINT handler only wakes a
@@ -328,7 +349,10 @@ fn main() {
.arg(arg!(--nmu "Number the entry as a non-maintainer upload (1.0-1 becomes 1.0-1.1, native 1.0 becomes 1.0+nmu1)").required(false)
.conflicts_with("rebuild"))
.arg(arg!(--rebuild "Number the entry as a no-change rebuild (1.0-1 becomes 1.0-1build1)").required(false))
.arg(arg!(-v --version <version> "Target version (overrides the --backport/--nmu/--rebuild numbering)").required(false)),
.arg(arg!(-v --version <version> "Target version (overrides the --backport/--nmu/--rebuild numbering)").required(false))
.arg(arg!(-f --file <path> "Path to the changelog file to generate into and edit (default: debian/changelog)").required(false).default_value("debian/changelog"))
.arg(arg!(--limit <n> "Cap how far back the history is searched for the previous changelog entry when no version tag exists (0 = unlimited)").required(false)
.value_parser(clap::value_parser!(usize)).default_value("500")),
)
.subcommand(
Command::new("build")
@@ -352,17 +376,125 @@ fn main() {
.arg(arg!(-a --arch <arch> "Target architecture").required(false))
.arg(arg!(-p --pocket <pocket> "Build against dependencies from a specific distribution pocket (updates, security, proposed)").required(false)
.long_help("Build against dependencies from a specific distribution pocket (e.g. updates, security, proposed).\nThe '<series>-<pocket>' suite will be enabled on archive sources when resolving build-dependencies."))
.arg(arg!(--mirror <url> "Resolve every package of the build from the given base URL").required(false)
.long_help("Build the chroot and resolve every package from the given base URL instead of the distro default. The URL is passed to mmdebstrap at bootstrap, so the chroot's apt sources carry it and every apt in the build resolves from it.\nThe mirror must serve every suite and architecture the build resolves, the security pocket and cross-build archives included. The tarball cache is keyed per mirror, so pinned and unpinned builds keep separate tarballs."))
.arg(arg!(--ppa <ppa> "Build the package adding a specific PPA for dependencies (can be specified multiple times)")
.long_help("Build the package adding a specific PPA for dependencies. Can be specified multiple times.").required(false).action(clap::ArgAction::Append))
.arg(arg!(--inject <package> "Inject a package into the build environment (can be specified multiple times)")
.long_help("Inject a package into the build environment before build-dep. Can be a .deb file path, a package name from the archive, or a package from a previously added PPA. Can be specified multiple times.").required(false).action(clap::ArgAction::Append))
.arg(arg!(--cross "Cross-compile for target architecture (instead of qemu-binfmt)")
.long_help("Cross-compile for target architecture (instead of using qemu-binfmt)\nNote that most packages cannot be cross-compiled").required(false))
.arg(arg!(--mode <mode> "Change build mode [local]").required(false)
.long_help("Change build mode [local]\nDefault will chose depending on other parameters, don't provide if unsure"))
.arg(arg!(-j --jobs <jobs> "Number of parallel build jobs (default: number of CPUs available in the build context)").required(false))
.arg(
clap::Arg::new("incremental")
.long("incremental")
.value_name("ID")
.num_args(0..=1)
.require_equals(false)
.default_missing_value("")
.help("Incremental build: fork a previous build record of this target and keep its staged tree")
.long_help("Incremental build: fork a previous build record of this target and keep its staged tree, so make recompiles only what changed. The base's staged tree carries over only while the changelog version and the chroot tarball are unchanged.\nWithout a value, the base is the newest successful record of the target — pkh keeps the newest and the newest successful staged trees per target (two by default, 'deb.incremental.staged_trees' in ~/.config/pkh/config.yml). With an ID as shown by 'pkh deb list', that exact record. A failed record is fine as a base if its staged tree is still there.\nEvery --incremental run keeps its staged tree, success or failure, so the next run can fork it. A plain 'pkh deb' keeps only metadata and the build log."),
)
.arg(arg!(--"no-ccache" "Build without the shared ccache").required(false))
.arg(arg!(--verbose "Show raw tool output instead of the live build view").required(false)
.long_help("Show raw tool output instead of the live build view.\nAlso implied by RUST_LOG=debug for pkh's own logs.")),
.long_help("Show raw tool output instead of the live build view.\nAlso implied by RUST_LOG=debug for pkh's own logs."))
.subcommand(
Command::new("list")
.about("List the build records recorded from this tree")
.long_about("List the build records recorded from this tree (plain runs too, as metadata records), with their outcome, version, fork lineage, whether their staged tree is kept, how long the dependency installation and the package build took, and the id to pass to 'pkh deb --incremental <id>'.\nRead-only: never builds or removes anything (the newest and the newest successful staged trees are kept per target by default, 'pkh prune' removes records older than a week)."),
)
.subcommand(
Command::new("log")
.about("Print a recorded build's log")
.long_about("Print the build log of a recorded build: the full tee of its output.\nWithout an ID, the newest record of this tree; with an ID (as shown by 'pkh deb list'), that exact record.\n--follow keeps reading a live build's log as it is written and exits once the build's record reaches a terminal state.")
.arg(arg!([id] "Record id (default: the newest record of this tree)").required(false))
.arg(arg!(-f --follow "Keep reading the log as the build writes it")),
)
.subcommand(
Command::new("profile")
.about("Print a recorded build's per-phase timing profile")
.long_about("Print the timing profile of a recorded build: how long every phase took, from the chroot bootstrap (tarball download, extraction, finalization) through the apt phases (package lists, essential packages, injected packages, build dependencies) and patch application to the build and binary steps and the artifact retrieval, each with its share of the total.\nWithout an ID, the newest record of this tree; with an ID (as shown by 'pkh deb list'), that exact record. A failed build's profile ends at the phase the error surfaced from.")
.arg(arg!([id] "Record id (default: the newest record of this tree)").required(false)),
)
)
.subcommand(
Command::new("test")
.about("Run the package's DEP-8 as-installed tests (autopkgtests)")
.arg(arg!([path] "Source tree to test (default: the current directory)").required(false))
.arg(arg!(-s --series <series> "Target distribution series").required(false))
.arg(arg!(-d --dist <dist> "Target distribution (debian, ubuntu)").required(false))
.arg(arg!(-a --arch <arch> "Target architecture").required(false))
.arg(arg!(-p --pocket <pocket> "Resolve test dependencies from a specific distribution pocket (updates, security, proposed)").required(false))
.arg(arg!(--ppa <ppa> "Add a PPA as test dependency source (can be specified multiple times)").required(false).action(clap::ArgAction::Append))
.arg(arg!(--inject <package> "Inject a package into the testbed before the tests (.deb path or package name, can be specified multiple times)").required(false).action(clap::ArgAction::Append))
.arg(arg!(--mode <mode> "Testbed mode: chroot (default, an ephemeral unshare chroot), local (the current context, no isolation) or vm (a qemu VM through autopkgtest)").required(false))
.arg(arg!(--image <file> "VM image for --mode vm (required there)").required(false))
.arg(arg!(--cpus <n> "VM virtual CPUs (vm mode)").required(false).value_parser(clap::value_parser!(u32)))
.arg(arg!(--"ram-size" <mib> "VM memory size in MiB (vm mode)").required(false).value_parser(clap::value_parser!(u32)))
.arg(arg!(--debs <deb> "Explicit binary package to test, overriding the pkh deb output reuse (can be specified multiple times)").required(false).action(clap::ArgAction::Append))
.arg(
clap::Arg::new("no_build")
.long("no-build")
.action(clap::ArgAction::SetTrue)
.help("Never build: fail when no fresh 'pkh deb' output exists next to the tree"),
)
.arg(arg!(--build "Ignore fresh pkh deb output and rebuild the binaries first").required(false))
.arg(
clap::Arg::new("test_name")
.long("test-name")
.value_name("NAME")
.action(clap::ArgAction::Append)
.help("Run only this test (can be specified multiple times)"),
)
.arg(
clap::Arg::new("skip_test")
.long("skip-test")
.value_name("NAME")
.action(clap::ArgAction::Append)
.help("Skip this test, reporting it as skipped (can be specified multiple times)"),
)
.arg(
clap::Arg::new("setup_commands")
.long("setup-commands")
.value_name("CMD")
.action(clap::ArgAction::Append)
.help("Shell command run in the testbed before the tests (can be specified multiple times)"),
)
.arg(
clap::Arg::new("apt_upgrade")
.long("apt-upgrade")
.action(clap::ArgAction::SetTrue)
.help("Update, upgrade and autopurge the testbed before the tests"),
)
.arg(arg!(--shell "Drop into an interactive testbed shell after the run").required(false))
.arg(
clap::Arg::new("shell_fail")
.long("shell-fail")
.action(clap::ArgAction::SetTrue)
.help("Drop into an interactive testbed shell when a test failed"),
)
.arg(arg!(--timeout <secs> "Per-test timeout in seconds (default: 10000)").required(false).value_parser(clap::value_parser!(u64)))
.arg(
clap::Arg::new("timeout_factor")
.long("timeout-factor")
.value_name("FACTOR")
.value_parser(clap::value_parser!(f64))
.help("Scale the default timeouts (an explicit --timeout stands as given)"),
)
.arg(
clap::Arg::new("fail_on")
.long("fail-on")
.value_name("TRIGGERS")
.help("Comma-separated extra failure triggers: skip, no-tests (default: none)"),
)
.arg(
clap::Arg::new("list_tests")
.long("list-tests")
.action(clap::ArgAction::SetTrue)
.help("Print the tests declared in debian/tests/control and exit"),
)
.arg(arg!(--json "Emit the report as JSON").required(false))
.arg(arg!(--verbose "Show raw test output instead of the summarized live view").required(false)),
)
.subcommand(
Command::new("lint")
@@ -438,7 +570,16 @@ fn main() {
clap::Arg::new("all")
.long("all")
.action(clap::ArgAction::SetTrue)
.help("Also remove cached chroot tarballs (expensive to re-download)"),
.help("Remove everything, including recent build records and staged trees")
.long_help("Remove everything, whatever its age: recent build records and their staged trees (the incremental bases `pkh deb --incremental` forks) included, plus the cached chroot tarballs, which are expensive to re-download. Records and locks a live build holds are never touched."),
)
.arg(
clap::Arg::new("force")
.short('f')
.long("force")
.action(clap::ArgAction::SetTrue)
.help("Skip the size computation and the confirmation prompt")
.long_help("Skip the size computation and the confirmation prompt: remove immediately, reporting counts but no sizes. The fastest cleanup path; a --dry-run still previews, and -a decides the policy as usual."),
)
)
.get_matches();
@@ -529,7 +670,7 @@ fn main() {
.unwrap_or("");
let archive = sub_matches.get_one::<bool>("archive").unwrap_or(&false);
let (pb, progress_callback) = pkh::ui::create_progress_bar(&multi);
let view = pkh::ui::PullView::new(&multi);
// Convert PPA to base URL if provided
let base_url = match ppa.map(pkh::package_info::split_ppa) {
@@ -551,20 +692,28 @@ fn main() {
dist,
base_url.as_deref(),
repository,
Some(&progress_callback),
&view,
)
.await?;
pkh::pull::pull(&package_info, None, Some(&progress_callback), *archive).await
pkh::pull::pull(&package_info, None, &view, *archive).await
}) {
pb.finish_and_clear();
// exit(1) below would skip the view's Drop
drop(view);
error!("{}", e);
std::process::exit(1);
}
pb.finish_and_clear();
multi.remove(&pb);
drop(view);
info!("Done.");
}
Some(("chlog", sub_matches)) => {
let history_limit = sub_matches
.get_one::<usize>("limit")
.copied()
.unwrap_or(500);
let changelog_file = sub_matches
.get_one::<String>("file")
.map(|s| s.to_string())
.unwrap_or_else(|| "debian/changelog".to_string());
let cwd = current_dir_or_exit();
let version = sub_matches.get_one::<String>("version").map(|s| s.as_str());
let cli_series = sub_matches.get_one::<String>("series").map(|s| s.as_str());
@@ -590,7 +739,7 @@ fn main() {
let target_series = if let Some(s) = cli_series {
Some(s.to_string())
} else {
let changelog_path = cwd.join("debian/changelog");
let changelog_path = cwd.join(&changelog_file);
match rt.block_on(pkh::changelog::series_candidates(&changelog_path)) {
Some(pkh::changelog::SeriesCandidates::Choose {
options,
@@ -618,11 +767,12 @@ fn main() {
};
let entry = match rt.block_on(pkh::changelog::generate_entry(
"debian/changelog",
&changelog_file,
Some(&cwd),
version,
target_series.as_deref(),
kind,
history_limit,
)) {
Ok(entry) => entry,
Err(e) => {
@@ -649,7 +799,7 @@ fn main() {
};
let _status = std::process::Command::new(&editor)
.current_dir(&cwd)
.args(["debian/changelog"])
.arg(&changelog_file)
.status()
.map_err(|e| {
error!(
@@ -780,6 +930,64 @@ fn main() {
}
Some(("deb", sub_matches)) => {
let cwd = current_dir_or_exit();
// `pkh deb list`: read-only listing of the build records
// recorded from this tree; never touches the build machinery.
if let Some(("list", _)) = sub_matches.subcommand() {
let tree = cwd.canonicalize().unwrap_or(cwd.clone());
let rows = pkh::deb::session::list_for_tree(&tree);
print!("{}", pkh::deb::session::render_session_list(&tree, &rows));
return;
}
// `pkh deb log`: print a recorded build's log (optionally
// following a live build); read-only too.
if let Some(("log", log_matches)) = sub_matches.subcommand() {
let tree = cwd.canonicalize().unwrap_or_else(|_| cwd.clone());
let id = log_matches.get_one::<String>("id").map(String::as_str);
let follow = log_matches.get_flag("follow");
let record = match pkh::deb::session::find_record(
&pkh::deb::session::sessions_root(),
&tree,
id,
) {
Ok(record) => record,
Err(e) => {
error!("{e}");
std::process::exit(1);
}
};
let mut out = std::io::stdout();
if let Err(e) = pkh::deb::session::print_record_log(&record, follow, &mut out) {
error!(
"cannot print the build log of {}: {e}",
record.root().display()
);
std::process::exit(1);
}
return;
}
// `pkh deb profile`: print a recorded build's per-phase
// timing profile; read-only too.
if let Some(("profile", profile_matches)) = sub_matches.subcommand() {
let tree = cwd.canonicalize().unwrap_or_else(|_| cwd.clone());
let id = profile_matches.get_one::<String>("id").map(String::as_str);
let record = match pkh::deb::session::find_record(
&pkh::deb::session::sessions_root(),
&tree,
id,
) {
Ok(record) => record,
Err(e) => {
error!("{e}");
std::process::exit(1);
}
};
print!("{}", pkh::deb::session::render_record_profile(&record));
return;
}
// Ctrl+C during the build must say what happened and release the
// ephemeral chroot instead of dying on the default disposition.
// The live view (when enabled) registers its own reporter on top
@@ -800,11 +1008,6 @@ fn main() {
.get_many::<String>("inject")
.map(|v| v.cloned().collect())
.unwrap_or_default();
let mode: Option<&str> = sub_matches.get_one::<String>("mode").map(|s| s.as_str());
let mode: Option<pkh::deb::BuildMode> = match mode {
Some("local") => Some(pkh::deb::BuildMode::Local),
_ => None,
};
let verbose = sub_matches
.get_one::<bool>("verbose")
.copied()
@@ -818,6 +1021,19 @@ fn main() {
})
});
// Incremental request: `--incremental` (no value) forks the
// auto-selected base, `--incremental <id>` that exact record;
// without the flag the run is fully ephemeral (see the deb
// subcommand help).
let incremental = match sub_matches
.get_one::<String>("incremental")
.map(String::as_str)
{
Some("") => pkh::deb::Incremental::Auto,
Some(id) => pkh::deb::Incremental::Id(id.to_string()),
None => pkh::deb::Incremental::Off,
};
// Live build view, unless --verbose (DebUi additionally disables
// itself when stdout is not a terminal)
let quiet = pkh::report::Quiet;
@@ -836,12 +1052,14 @@ fn main() {
arch,
series,
pocket,
mirror: sub_matches.get_one::<String>("mirror").cloned(),
cwd: Some(cwd.clone()),
cross,
mode,
ppa,
inject,
jobs,
incremental,
ccache: !sub_matches.get_flag("no-ccache"),
view,
..Default::default()
})
@@ -863,57 +1081,238 @@ fn main() {
}
}
}
Some(("test", sub_matches)) => {
let path = sub_matches
.get_one::<String>("path")
.map(std::path::PathBuf::from);
// Read-only listing of the declared tests; never boots a
// testbed.
if sub_matches.get_flag("list_tests") {
let tree = path.unwrap_or_else(current_dir_or_exit);
match pkh::test::list_tests(&tree) {
Ok(listing) => print!("{listing}"),
Err(e) => {
error!("{e}");
std::process::exit(2);
}
}
return;
}
// Ctrl+C must release the ephemeral chroot instead of dying
// on the default disposition (the deb pattern).
interrupt::install();
let mode = match sub_matches.get_one::<String>("mode").map(String::as_str) {
None => None,
Some(value) => match pkh::test::TestMode::parse(value) {
Some(mode) => Some(mode),
None => {
error!("Invalid --mode value '{value}': expected chroot or local");
std::process::exit(1);
}
},
};
let (fail_on_skip, fail_on_no_tests) =
match sub_matches.get_one::<String>("fail_on").map(String::as_str) {
None => (false, false),
Some(list) => {
let mut skip = false;
let mut no_tests = false;
for trigger in list.split(',').map(str::trim).filter(|t| !t.is_empty()) {
match trigger {
"skip" => skip = true,
"no-tests" => no_tests = true,
other => {
error!(
"Unknown --fail-on trigger '{other}': \
expected skip or no-tests"
);
std::process::exit(1);
}
}
}
(skip, no_tests)
}
};
// Live build view, unless --verbose (DebUi additionally
// disables itself when stdout is not a terminal).
let quiet = pkh::report::Quiet;
let live = if sub_matches.get_flag("verbose") {
None
} else {
Some(pkh::ui::deb::DebUi::new(&multi))
};
let view: &dyn pkh::report::BuildView = live
.as_ref()
.map(|v| v as &dyn pkh::report::BuildView)
.unwrap_or(&quiet);
let options = pkh::test::TestOptions {
tree: path,
series: sub_matches.get_one::<String>("series").cloned(),
dist: sub_matches.get_one::<String>("dist").cloned(),
arch: sub_matches.get_one::<String>("arch").cloned(),
pocket: sub_matches.get_one::<String>("pocket").cloned(),
ppa: sub_matches
.get_many::<String>("ppa")
.map(|v| v.cloned().collect())
.unwrap_or_default(),
inject: sub_matches
.get_many::<String>("inject")
.map(|v| v.cloned().collect())
.unwrap_or_default(),
mode,
image: sub_matches
.get_one::<String>("image")
.map(std::path::PathBuf::from),
cpus: sub_matches.get_one::<u32>("cpus").copied(),
ram_size: sub_matches
.get_one::<u32>("ram-size")
.map(|v| v.to_string()),
debs: sub_matches
.get_many::<String>("debs")
.map(|v| v.map(std::path::PathBuf::from).collect())
.unwrap_or_default(),
no_build: sub_matches.get_flag("no_build"),
build: sub_matches.get_flag("build"),
test_names: sub_matches
.get_many::<String>("test_name")
.map(|v| v.cloned().collect())
.unwrap_or_default(),
skip_tests: sub_matches
.get_many::<String>("skip_test")
.map(|v| v.cloned().collect())
.unwrap_or_default(),
setup_commands: sub_matches
.get_many::<String>("setup_commands")
.map(|v| v.cloned().collect())
.unwrap_or_default(),
apt_upgrade: sub_matches.get_flag("apt_upgrade"),
shell: sub_matches.get_flag("shell"),
shell_fail: sub_matches.get_flag("shell_fail"),
timeout: sub_matches.get_one::<u64>("timeout").copied(),
timeout_factor: sub_matches.get_one::<f64>("timeout_factor").copied(),
fail_on_skip,
fail_on_no_tests,
json: sub_matches.get_flag("json"),
verbose: sub_matches.get_flag("verbose"),
view,
};
let json = options.json;
match rt.block_on(pkh::test::run(options)) {
Ok(report) => {
// The per-test report is the deliverable; the live
// view carries phases and the outcome banner only.
if json {
println!("{}", report.render_json());
} else {
print!("{}", report.render_text());
}
std::process::exit(report.exit_code(fail_on_skip, fail_on_no_tests));
}
Err(e) => {
// On Ctrl+C the interrupt watchdog owns the shutdown
// (see `pkh deb`): park here instead of racing it.
if pkh::interrupt::interrupted() {
interrupt::wait_for_shutdown();
}
error!("{e}");
std::process::exit(2);
}
}
}
Some(("prune", sub_matches)) => {
let dry_run = sub_matches
.get_one::<bool>("dry_run")
.copied()
.unwrap_or(false);
let all = sub_matches.get_one::<bool>("all").copied().unwrap_or(false);
let force = sub_matches
.get_one::<bool>("force")
.copied()
.unwrap_or(false);
let options = pkh::prune::PruneOptions { dry_run, all };
match pkh::prune::prune(options) {
Ok(report) => {
if report.is_empty() {
info!("Nothing to prune.");
} else {
let action = if report.dry_run {
"Would remove"
} else {
"Removed"
};
for path in &report.removed {
info!("{} {}", action, path.display());
}
for (path, err) in &report.failed {
error!("Failed to remove {}: {}", path.display(), err);
}
if report.dry_run {
info!(
"(dry run) {} item(s) would be removed.",
report.removed.len()
);
} else {
info!(
"Pruned {} item(s){} ({} failure(s)).",
report.removed.len(),
if all {
" including cached chroot tarballs"
} else {
""
},
report.failed.len()
);
}
}
if !report.failed.is_empty() {
std::process::exit(1);
}
let options = pkh::prune::PruneOptions {
dry_run,
all,
force,
};
// The size pass spins on a terminal; --force skips it entirely.
let spinner = (!force)
.then(|| pkh::ui::wait_spinner("Computing sizes of directories to prune..."));
let plan = pkh::prune::plan(options);
if let Some(spinner) = spinner {
spinner.finish_and_clear();
}
let digest = plan.report();
if digest.is_empty() {
info!("Nothing to prune.");
return;
}
// The digest: one grouped line per kind. Counts are bold and
// sizes cyan — when there are sizes at all (--force skips the
// walk and reports counts only).
let header = if dry_run {
"Dry run - would remove:"
} else {
"Would remove:"
};
info!("{}", bold(header));
for group in &digest.groups {
let size = if group.bytes > 0 {
format!(" for {}", cyan(&pkh::prune::format_bytes(group.bytes)))
} else {
String::new()
};
info!("{} {}{size}", bold(&group.count.to_string()), group.label());
}
let total = cyan(&pkh::prune::format_bytes(digest.total_bytes));
if dry_run {
// --force skips the size pass: counts without a total.
if !force {
info!("Would have freed {total} total.");
}
Err(e) => {
error!("{}", e);
std::process::exit(1);
return;
}
// Confirmation for the interactive case (--force and
// non-interactive sessions skip it).
if !force {
let confirmed =
pkh::ui::prompt::confirm("Proceed with the removal?", false).unwrap_or(false);
if !confirmed {
info!("Aborted.");
return;
}
}
info!("Removing:");
// The removal is slow (the big session trees take minutes):
// the plan's sizes drive a byte bar in the deb view's layout;
// --force planned no sizes and gets the count line only.
let bar = pkh::ui::removal_bar(&multi, digest.total_bytes);
let report = pkh::prune::apply_with_progress(
plan,
Some(&|idx, count, _path, done| {
pkh::ui::removal_progress(&bar, idx, count, done);
}),
);
bar.finish_and_clear();
for (path, err) in &report.failed {
error!("Failed to remove {}: {}", path.display(), err);
}
if !force {
info!("Freed {} total.", total);
}
if !report.failed.is_empty() {
std::process::exit(1);
}
}
Some(("lint", sub_matches)) => {
if sub_matches.get_flag("list_tags") {
src/new/mod.rs
+1 -2
View File
@@ -1,5 +1,4 @@
//! `pkh new`: interactive-first package scaffolding (see
//! `plans/pkh-new.md`).
//! `pkh new`: interactive-first package scaffolding.
//!
//! This module orchestrates a scaffold run: target directory checks, project
//! detection, in-memory rendering of every file (all-or-nothing write), the
src/new/options.rs
+167 -69
View File
@@ -63,22 +63,12 @@ impl TemplateId {
}
/// Parse a CLI identifier, accepting exactly the registered spellings.
pub fn parse(s: &str) -> Result<TemplateId, String> {
pub fn parse(s: &str) -> Result<TemplateId, ValidationError> {
Self::all()
.iter()
.copied()
.find(|id| id.as_str() == s)
.ok_or_else(|| {
format!(
"Unknown language/template '{}'. Supported values are: {}.",
s,
Self::all()
.iter()
.map(|id| id.as_str())
.collect::<Vec<_>>()
.join(", ")
)
})
.ok_or_else(|| ValidationError::UnknownTemplate { id: s.to_string() })
}
/// Human-readable menu label of this template, as offered by the wizard
@@ -407,21 +397,132 @@ pub struct NewCli {
pub defaults: bool,
}
/// Failure of one of the scaffold option validators ([`validate_source_name`],
/// [`validate_command`], [`validate_upstream_version`], [`parse_maintainer`],
/// [`validate_homepage`], [`validate_depends`], [`validate_orig_path`],
/// [`check_file_collisions`], [`TemplateId::parse`]): what the wizard re-asks
/// about, carrying the pieces callers may want to match on.
#[derive(Debug, thiserror::Error)]
pub enum ValidationError {
/// The source package name is shorter than dpkg's 2-character minimum.
#[error("'{name}' is too short: package names need at least 2 characters")]
SourceNameTooShort {
/// The rejected name.
name: String,
},
/// The source package name is outside dpkg's charset or shape.
#[error(
"'{}' is not a valid package name: names must be lowercase, start with a letter or digit and may only contain [a-z0-9+.-]",
name
)]
InvalidSourceName {
/// The rejected name.
name: String,
},
/// No installed command was given.
#[error("the command name must not be empty")]
EmptyCommand,
/// The command name is not a lowercase identifier.
#[error(
"'{}' is not a valid command name: commands must be a lowercase identifier (letters, digits, + - . _), e.g. 'mytool'",
command
)]
InvalidCommand {
/// The rejected command name.
command: String,
},
/// Upstream versions should start with a digit (dpkg recommendation,
/// enforced here).
#[error("upstream versions should start with a digit")]
UpstreamNotDigit,
/// `-` is the Debian revision separator.
#[error(
"upstream versions must not contain '-': the Debian revision is appended automatically"
)]
UpstreamContainsDash,
/// The upstream version composed with the Debian revision does not
/// parse as a Debian version.
#[error("'{version}' is not a valid Debian version: {source}")]
InvalidVersion {
/// The composed `upstream-revision` string.
version: String,
/// The version parse failure.
source: crate::debian::version::VersionParseError,
},
/// The maintainer is not of the form `Name <email>`.
#[error("'{s}' is not a valid maintainer: expected the form 'Name <email>'")]
MaintainerShape {
/// The rejected maintainer string.
s: String,
},
/// The maintainer name (before the angle brackets) is empty.
#[error("'{s}': the maintainer name must not be empty")]
MaintainerNameEmpty {
/// The rejected maintainer string.
s: String,
},
/// The maintainer email does not carry exactly one `@`.
#[error("'{s}': the maintainer email must contain exactly one '@'")]
MaintainerEmailAt {
/// The rejected maintainer string.
s: String,
},
/// The maintainer email carries whitespace.
#[error("'{s}': the maintainer email must not contain whitespace")]
MaintainerEmailWhitespace {
/// The rejected maintainer string.
s: String,
},
/// The homepage is not an http(s) URL.
#[error("'{url}' is not a valid homepage: expected an http:// or https:// URL")]
InvalidHomepage {
/// The rejected URL.
url: String,
},
/// The Depends list does not parse as package relations.
#[error("Invalid Depends list '{list}': {source}")]
InvalidDepends {
/// The rejected list.
list: String,
/// The relation parse failure.
source: crate::debian::deps::ParseError,
},
/// The orig tarball is neither an http(s) URL nor an existing file.
#[error(
"'{path}' is not an http(s) URL and not an existing file: --orig-path must point at a tarball to repack"
)]
InvalidOrigPath {
/// The rejected path or URL.
path: String,
},
/// The language/template id is not one of the registered spellings.
#[error("Unknown language/template '{}'. Supported values are: {}.", id, TemplateId::all().iter().map(|id| id.as_str()).collect::<Vec<_>>().join(", "))]
UnknownTemplate {
/// The unknown id.
id: String,
},
/// A template generated the same file twice.
#[error("internal error: file '{path}' was generated more than once")]
GeneratedTwice {
/// The duplicated path.
path: String,
},
}
/// Validate a Debian source package name: `^[a-z0-9][a-z0-9+.\-]+$` with a
/// minimum length of 2 (dpkg rules).
pub fn validate_source_name(name: &str) -> Result<(), String> {
pub fn validate_source_name(name: &str) -> Result<(), ValidationError> {
static NAME_REGEX: std::sync::OnceLock<Regex> = std::sync::OnceLock::new();
let regex = NAME_REGEX.get_or_init(|| Regex::new(r"^[a-z0-9][a-z0-9+.\-]+$").unwrap());
if name.len() < 2 {
return Err(format!(
"'{name}' is too short: package names need at least 2 characters"
));
return Err(ValidationError::SourceNameTooShort {
name: name.to_string(),
});
}
if !regex.is_match(name) {
return Err(format!(
"'{name}' is not a valid package name: names must be lowercase, \
start with a letter or digit and may only contain [a-z0-9+.-]"
));
return Err(ValidationError::InvalidSourceName {
name: name.to_string(),
});
}
Ok(())
}
@@ -473,17 +574,16 @@ pub fn sanitize_name(input: &str) -> Option<String> {
/// `meson.build` and the `[project.scripts]` table, so uppercase letters,
/// whitespace, quotes and shell metacharacters are rejected here instead of
/// generating broken install lines and build files.
pub fn validate_command(command: &str) -> Result<(), String> {
pub fn validate_command(command: &str) -> Result<(), ValidationError> {
static COMMAND_REGEX: std::sync::OnceLock<Regex> = std::sync::OnceLock::new();
let regex = COMMAND_REGEX.get_or_init(|| Regex::new(r"^[a-z0-9][a-z0-9+.\-_]*$").unwrap());
if command.is_empty() {
return Err("the command name must not be empty".to_string());
return Err(ValidationError::EmptyCommand);
}
if !regex.is_match(command) {
return Err(format!(
"'{command}' is not a valid command name: commands must be a \
lowercase identifier (letters, digits, + - . _), e.g. 'mytool'"
));
return Err(ValidationError::InvalidCommand {
command: command.to_string(),
});
}
Ok(())
}
@@ -492,65 +592,60 @@ pub fn validate_command(command: &str) -> Result<(), String> {
/// recommendation, enforced here) and survive [`DebianVersion::parse`] once
/// composed with the Debian revision. It must not contain `-` (the revision
/// separator).
pub fn validate_upstream_version(upstream: &str, revision: u32) -> Result<(), String> {
pub fn validate_upstream_version(upstream: &str, revision: u32) -> Result<(), ValidationError> {
if !upstream.starts_with(|c: char| c.is_ascii_digit()) {
return Err("upstream versions should start with a digit".to_string());
return Err(ValidationError::UpstreamNotDigit);
}
if upstream.contains('-') {
return Err(
"upstream versions must not contain '-': the Debian revision is \
appended automatically"
.to_string(),
);
return Err(ValidationError::UpstreamContainsDash);
}
let composed = format!("{}-{}", upstream, revision);
DebianVersion::parse(&composed)
.map(|_| ())
.map_err(|e| format!("'{composed}' is not a valid Debian version: {e}"))
.map_err(|source| ValidationError::InvalidVersion {
version: composed.clone(),
source,
})
}
/// Parse a `Name <email>` maintainer string: the name must be non-empty and
/// the email must contain exactly one `@` with no whitespace inside the
/// angle brackets.
pub fn parse_maintainer(s: &str) -> Result<(String, String), String> {
pub fn parse_maintainer(s: &str) -> Result<(String, String), ValidationError> {
static MAINTAINER_REGEX: std::sync::OnceLock<Regex> = std::sync::OnceLock::new();
let regex = MAINTAINER_REGEX.get_or_init(|| Regex::new(r"^(.+?)\s*<([^<>]*)>\s*$").unwrap());
let caps = regex.captures(s.trim()).ok_or_else(|| {
format!("'{s}' is not a valid maintainer: expected the form 'Name <email>'")
})?;
let caps = regex
.captures(s.trim())
.ok_or_else(|| ValidationError::MaintainerShape { s: s.to_string() })?;
let name = caps[1].trim().to_string();
let email = caps[2].trim().to_string();
if name.is_empty() {
return Err(format!("'{s}': the maintainer name must not be empty"));
return Err(ValidationError::MaintainerNameEmpty { s: s.to_string() });
}
if !email.contains('@') || email.matches('@').count() > 1 {
return Err(format!(
"'{s}': the maintainer email must contain exactly one '@'"
));
return Err(ValidationError::MaintainerEmailAt { s: s.to_string() });
}
if email.chars().any(char::is_whitespace) {
return Err(format!(
"'{s}': the maintainer email must not contain whitespace"
));
return Err(ValidationError::MaintainerEmailWhitespace { s: s.to_string() });
}
Ok((name, email))
}
/// Validate a homepage URL: it must carry an `http(s)://` prefix.
pub fn validate_homepage(url: &str) -> Result<(), String> {
pub fn validate_homepage(url: &str) -> Result<(), ValidationError> {
if url.starts_with("http://") || url.starts_with("https://") {
Ok(())
} else {
Err(format!(
"'{url}' is not a valid homepage: expected an http:// or https:// URL"
))
Err(ValidationError::InvalidHomepage {
url: url.to_string(),
})
}
}
/// Validate a comma-separated runtime Depends list (the metapackage flavor)
/// by parsing it with [`Deps::parse`] and re-rendering every clause
/// canonically.
pub fn validate_depends(list: &str) -> Result<Vec<String>, String> {
pub fn validate_depends(list: &str) -> Result<Vec<String>, ValidationError> {
let opts = ParseOpts {
host_arch: crate::get_current_arch(),
build_arch: crate::get_current_arch(),
@@ -559,8 +654,10 @@ pub fn validate_depends(list: &str) -> Result<Vec<String>, String> {
union: false,
build_dep: false,
};
let deps =
Deps::parse(list, &opts).map_err(|e| format!("Invalid Depends list '{list}': {e}"))?;
let deps = Deps::parse(list, &opts).map_err(|source| ValidationError::InvalidDepends {
list: list.to_string(),
source,
})?;
Ok(deps
.clauses()
.map(|clause| {
@@ -620,7 +717,7 @@ pub async fn resolve(cli: NewCli) -> Result<NewOptions, String> {
// (nothing to detect for a fresh skeleton), else --defaults picks the
// empty template.
let template = match &cli.lang {
Some(lang) => Some(TemplateId::parse(lang)?),
Some(lang) => Some(TemplateId::parse(lang).map_err(|e| e.to_string())?),
None => match &source_dir {
SourceDir::Skeleton => {
if cli.defaults {
@@ -706,7 +803,7 @@ pub async fn resolve(cli: NewCli) -> Result<NewOptions, String> {
}
// Everything below has a default and is validated as it is resolved.
validate_source_name(&name)?;
validate_source_name(&name).map_err(|e| e.to_string())?;
// Source format: explicit flags win (--native / --quilt, mutually
// exclusive), otherwise the mode decides — a fresh, self-authored
@@ -769,7 +866,7 @@ pub async fn resolve(cli: NewCli) -> Result<NewOptions, String> {
let homepage = match &cli.homepage {
Some(h) => {
validate_homepage(h)?;
validate_homepage(h).map_err(|e| e.to_string())?;
Some(h.clone())
}
None => None,
@@ -780,10 +877,10 @@ pub async fn resolve(cli: NewCli) -> Result<NewOptions, String> {
let command = cli.command.unwrap_or_else(|| name.clone());
// The default is the already-validated package name (a subset of the
// command charset), so only an explicit --command/wizard answer can fail.
validate_command(&command)?;
validate_command(&command).map_err(|e| e.to_string())?;
let maintainer = match &cli.maintainer {
Some(m) => parse_maintainer(m)?,
Some(m) => parse_maintainer(m).map_err(|e| e.to_string())?,
None => crate::changelog::get_maintainer_info().map_err(|e| {
format!(
"Could not determine the maintainer ({e}). \
@@ -815,7 +912,7 @@ pub async fn resolve(cli: NewCli) -> Result<NewOptions, String> {
let depends = if joined.trim().is_empty() {
Vec::new()
} else {
validate_depends(&joined)?
validate_depends(&joined).map_err(|e| e.to_string())?
};
Ok(NewOptions {
@@ -874,7 +971,7 @@ fn resolve_orig_origin(
let source = cli.orig_path.clone().ok_or_else(|| {
"--orig-from path requires --orig-path <tarball file or URL>".to_string()
})?;
validate_orig_path(&source)?;
validate_orig_path(&source).map_err(|e| e.to_string())?;
OrigOrigin::Provided { source }
}
Some("git") => {
@@ -935,27 +1032,26 @@ fn resolve_orig_origin(
/// Validate a `--orig-path` value (and the wizard's tarball answer): either
/// an http(s) URL (downloaded at scaffold time) or an existing local
/// tarball.
pub fn validate_orig_path(source: &str) -> Result<(), String> {
pub fn validate_orig_path(source: &str) -> Result<(), ValidationError> {
if source.starts_with("http://") || source.starts_with("https://") {
return Ok(());
}
if Path::new(source).is_file() {
return Ok(());
}
Err(format!(
"'{source}' is not an http(s) URL and not an existing file: \
--orig-path must point at a tarball to repack"
))
Err(ValidationError::InvalidOrigPath {
path: source.to_string(),
})
}
/// Set of names no template may produce twice (collision check).
pub(crate) fn check_file_collisions(paths: &[String]) -> Result<(), String> {
pub(crate) fn check_file_collisions(paths: &[String]) -> Result<(), ValidationError> {
let mut seen: HashSet<&String> = HashSet::new();
for path in paths {
if !seen.insert(path) {
return Err(format!(
"internal error: file '{path}' was generated more than once"
));
return Err(ValidationError::GeneratedTwice {
path: path.to_string(),
});
}
}
Ok(())
@@ -1222,7 +1318,9 @@ mod tests {
#[test]
fn collision_detection() {
assert!(check_file_collisions(&["a".into(), "b".into()]).is_ok());
let err = check_file_collisions(&["a".into(), "b".into(), "a".into()]).unwrap_err();
let err = check_file_collisions(&["a".into(), "b".into(), "a".into()])
.unwrap_err()
.to_string();
assert!(err.contains('a'));
}
src/new/questions.rs
+26 -19
View File
@@ -96,12 +96,9 @@ async fn run_wizard(
// basename of the current directory.
if cli.name.is_none() {
let default = default_package_name(&cwd, probe.as_ref());
let answer = ask_text(
prompter,
"Package name",
&default,
options::validate_source_name,
)?;
let answer = ask_text(prompter, "Package name", &default, |name: &str| {
options::validate_source_name(name).map_err(|e| e.to_string())
})?;
cli.name = Some(answer);
}
@@ -265,7 +262,9 @@ async fn run_wizard(
.or_else(|| origin.as_ref().and_then(GitOrigin::git_version))
.unwrap_or_else(|| "0.1.0".to_string());
let revision = cli.revision.unwrap_or(1);
let validate = move |version: &str| options::validate_upstream_version(version, revision);
let validate = move |version: &str| {
options::validate_upstream_version(version, revision).map_err(|e| e.to_string())
};
let answer = ask_text(prompter, "Upstream version", &default, validate)?;
cli.upstream_version = Some(answer.clone());
@@ -321,7 +320,8 @@ async fn run_wizard(
.expect("answer comes from the choice list");
cli.orig_from = Some(chosen.to_string());
if chosen == "path" {
let validator = |path: &str| options::validate_orig_path(path);
let validator =
|path: &str| options::validate_orig_path(path).map_err(|e| e.to_string());
let path = prompter.text("Tarball path or URL", "", Some(&validator))?;
cli.orig_path = Some(path);
}
@@ -354,7 +354,7 @@ async fn run_wizard(
if url.is_empty() {
Ok(())
} else {
options::validate_homepage(url)
options::validate_homepage(url).map_err(|e| e.to_string())
}
};
let answer = ask_text(prompter, "Homepage (blank to skip)", &default, validate)?;
@@ -400,12 +400,9 @@ async fn run_wizard(
.as_ref()
.and_then(|p| p.command.clone())
.unwrap_or_else(|| cli.name.clone().unwrap_or_default());
let command = ask_text(
prompter,
"Command name",
&default,
options::validate_command,
)?;
let command = ask_text(prompter, "Command name", &default, |command: &str| {
options::validate_command(command).map_err(|e| e.to_string())
})?;
cli.command = Some(command);
}
@@ -430,7 +427,11 @@ async fn run_wizard(
DEBEMAIL and no git user config): answer as 'Name <email>'"
);
}
let validate = |answer: &str| options::parse_maintainer(answer).map(|_| ());
let validate = |answer: &str| {
options::parse_maintainer(answer)
.map(|_| ())
.map_err(|e| e.to_string())
};
cli.maintainer = Some(ask_text(prompter, "Maintainer", &default, validate)?);
}
@@ -480,7 +481,11 @@ async fn run_wizard(
// 13. Metapackage Depends (empty template only).
if template == TemplateId::EMPTY && cli.depends.is_empty() {
let validate = |answer: &str| options::validate_depends(answer).map(|_| ());
let validate = |answer: &str| {
options::validate_depends(answer)
.map(|_| ())
.map_err(|e| e.to_string())
};
let answer = ask_text(
prompter,
"Depends (metapackage, comma-separated, blank for an empty base)",
@@ -1496,7 +1501,9 @@ mod tests {
assert!(options::validate_upstream_version("v1.0", 1).is_err());
assert!(options::validate_upstream_version("1.0.0", 1).is_ok());
let validate = |version: &str| options::validate_upstream_version(version, 1);
let validate = |version: &str| {
options::validate_upstream_version(version, 1).map_err(|e| e.to_string())
};
// Invalid probes are not offered as the default...
assert_eq!(offered_default("1.0-2", &validate), "");
assert_eq!(offered_default("v1.0", &validate), "");
@@ -1532,7 +1539,7 @@ mod tests {
// The upstream-version scenario end to end: Enter on an invalid
// probed default (offered as none) fails instead of taking it
// verbatim, while a valid default is taken.
let version = |v: &str| options::validate_upstream_version(v, 1);
let version = |v: &str| options::validate_upstream_version(v, 1).map_err(|e| e.to_string());
assert!(accept_answer("", "1.0-2", &version).is_err());
assert_eq!(
accept_answer("", "1.0.0", &version),
src/new/templates/mod.rs
+8 -9
View File
@@ -52,6 +52,7 @@ pub mod rust;
pub mod shell;
use std::path::{Path, PathBuf};
use std::sync::LazyLock;
use serde::Deserialize;
@@ -478,15 +479,13 @@ static TEMPLATE_SOURCES: &[TemplateSources] = &[
},
];
lazy_static::lazy_static! {
/// The parsed registry: manifest data plus resolved hooks, in the
/// order of [`TEMPLATE_SOURCES`] (wizard menu = detection priority).
static ref REGISTRY: Vec<Template> = build_registry();
/// The registry ids ([`Template::id`]), cached for the `TemplateId`
/// accessors.
static ref REGISTRY_IDS: Vec<TemplateId> =
REGISTRY.iter().map(Template::id).collect();
}
/// The parsed registry: manifest data plus resolved hooks, in the
/// order of [`TEMPLATE_SOURCES`] (wizard menu = detection priority).
static REGISTRY: LazyLock<Vec<Template>> = LazyLock::new(build_registry);
/// The registry ids ([`Template::id`]), cached for the `TemplateId`
/// accessors.
static REGISTRY_IDS: LazyLock<Vec<TemplateId>> =
LazyLock::new(|| REGISTRY.iter().map(Template::id).collect());
/// Parse and cross-check every embedded manifest into the registry. The
/// files ship inside the binary, so anything malformed here is a build-time
src/package_info.rs
+43 -59
View File
@@ -4,8 +4,8 @@ use std::error::Error;
use std::io::Read;
use xz2::read::XzDecoder;
use crate::ProgressCallback;
use crate::apt::release::{self, VerifiedRelease};
use crate::report::BuildView;
use crossterm::style::Stylize;
use log::{debug, warn};
@@ -490,7 +490,7 @@ async fn find_package(
version: Option<&str>,
base_url: Option<&str>,
launchpad_vcs: Option<String>,
progress: ProgressCallback<'_>,
view: &dyn BuildView,
) -> Result<PackageInfo, Box<dyn Error>> {
let series_list = crate::distro_info::get_ordered_series_name(dist).await?;
@@ -500,9 +500,7 @@ async fn find_package(
let mut attempt_errors: Vec<String> = Vec::new();
for (i, series) in series_list.iter().enumerate() {
if let Some(cb) = progress {
cb("", &format!("Checking {}...", series), i, series_list.len());
}
view.progress(&format!("Checking {series}..."), i, series_list.len());
let pockets = if pocket.is_empty() {
crate::distro_info::get_dist_pockets(dist)?
@@ -770,16 +768,9 @@ pub async fn lookup_repository(
version: Option<&str>,
repo_url: &str,
series: Option<&str>,
progress: ProgressCallback<'_>,
view: &dyn BuildView,
) -> Result<PackageInfo, Box<dyn Error>> {
if let Some(cb) = progress {
cb(
&format!("Fetching repository info from {}...", repo_url),
"",
0,
0,
);
}
view.message(&format!("Fetching repository info from {repo_url}..."));
// Attempt to authenticate the repository. In non-strict mode only the
// absence of a Release file (or of a way to verify it) is tolerated:
@@ -814,14 +805,7 @@ pub async fn lookup_repository(
get_flat_repo_series(repo_url).await?
};
if let Some(cb) = progress {
cb(
&format!("Resolving package info for {}...", package),
"",
0,
0,
);
}
view.message(&format!("Resolving package info for {package}..."));
let sources = get_flat_repo_sources(repo_url, verified.as_ref()).await?;
let stanza = parse_sources(&sources, package, version)?
@@ -855,7 +839,7 @@ pub async fn lookup_repository(
/// * `dist` - Optional distribution name (e.g., "ubuntu", "debian")
/// * `base_url` - Optional base URL for the package archive (e.g., "https://ppa.launchpadcontent.net/user/ppa/ubuntu/")
/// * `repository` - Optional URL of an external flat repository suite (e.g., "https://pkg.noctalia.dev/deb/resolute/")
/// * `progress` - Optional progress callback
/// * `view` - Build view receiving the lookup progress
#[allow(clippy::too_many_arguments)]
pub async fn lookup(
package: &str,
@@ -865,12 +849,12 @@ pub async fn lookup(
dist: Option<&str>,
base_url: Option<&str>,
repository: Option<&str>,
progress: ProgressCallback<'_>,
view: &dyn BuildView,
) -> Result<PackageInfo, Box<dyn Error>> {
// External flat repository: the URL is the suite root, distribution
// resolution and pocket logic do not apply
if let Some(repo_url) = repository {
return lookup_repository(package, version, repo_url, series, progress).await;
return lookup_repository(package, version, repo_url, series, view).await;
}
// Resolve the distribution early so we can check Launchpad once
@@ -887,9 +871,7 @@ pub async fn lookup(
// Check Launchpad repository once for Ubuntu packages, rather than
// once per series/pocket combination inside get().
let launchpad_vcs = if resolved_dist == "ubuntu" && base_url.is_none() {
if let Some(cb) = progress {
cb("", &format!("Checking Launchpad for {}...", package), 0, 0);
}
view.message(&format!("Checking Launchpad for {package}..."));
match check_launchpad_repo(package).await {
Ok(Some(lp_url)) => {
debug!("Found Launchpad URL: {}", lp_url);
@@ -910,34 +892,20 @@ pub async fn lookup(
// Obtain the package information, either directly in a series or with a search in all series
let package_info = if let Some(s) = series {
if let Some(cb) = progress {
cb(
&format!("Resolving package info for {}...", package),
"",
0,
0,
);
}
view.message(&format!("Resolving package info for {package}..."));
// Get the package information from that series and pocket
get(package, s, pocket, version, base_url, launchpad_vcs).await?
} else {
if let Some(cb) = progress {
cb(
&format!(
"Searching for package {} in {}...",
package,
if base_url.is_none() {
resolved_dist
} else {
"ppa"
}
),
"",
0,
0,
);
}
view.message(&format!(
"Searching for package {} in {}...",
package,
if base_url.is_none() {
resolved_dist
} else {
"ppa"
}
));
// Try to find the package in all series from that dist
find_package(
@@ -947,7 +915,7 @@ pub async fn lookup(
version,
base_url,
launchpad_vcs,
progress,
view,
)
.await?
};
@@ -1142,9 +1110,17 @@ Directory: pool/main/h/hello
#[tokio::test]
async fn test_find_package_fallback() {
// python2.7 is in bullseye but not above
let info = find_package("python2.7", "debian", "", None, None, None, None)
.await
.unwrap();
let info = find_package(
"python2.7",
"debian",
"",
None,
None,
None,
&crate::report::Quiet,
)
.await
.unwrap();
assert_eq!(info.stanza.package, "python2.7");
assert_eq!(info.series, "bullseye")
}
@@ -1152,9 +1128,17 @@ Directory: pool/main/h/hello
#[tokio::test]
async fn test_find_package_devel() {
// hello is in sid
let info = find_package("hello", "debian", "", None, None, None, None)
.await
.unwrap();
let info = find_package(
"hello",
"debian",
"",
None,
None,
None,
&crate::report::Quiet,
)
.await
.unwrap();
assert_eq!(info.stanza.package, "hello");
assert_eq!(info.series, "sid")
}
src/prune.rs
+847 -26
View File
File diff suppressed because it is too large Load Diff
src/pull.rs
+637 -190
View File
File diff suppressed because it is too large Load Diff
src/put/mod.rs
+1 -1
View File
@@ -94,7 +94,7 @@ async fn put_impl(opts: &PutOptions<'_>) -> Result<(), Box<dyn std::error::Error
"Uploading {} {} to {}",
changes.source, changes.version, target.label
),
source_only: false,
kind: crate::report::TargetKind::Binary,
// An upload runs no subprocess: nothing to tee
tee_log: false,
});
src/quirks.rs
+109 -2
View File
@@ -54,6 +54,20 @@ pub struct OperationQuirks {
/// like linux packages that use directories like "linux-main" or other custom names
#[serde(default)]
pub package_directory: Vec<String>,
/// Paths (relative to the package tree) removed before
/// `debian/rules build` when the build resumes a previous attempt.
///
/// For packages whose debian/rules keeps a stamp cache that does not
/// track its inputs (the Ubuntu/Debian kernels' `debian/stamps`: the
/// flavour config rule exports `.config` from the annotations with no
/// config prerequisite), a resumed build would keep building from the
/// previous attempt's configuration. Listing the stamp directories
/// here drops the cache so the wrapper steps re-run, while the inner
/// build system (kbuild) keeps its own incremental state. Only list
/// caches whose re-run is cheap or whose inner build is incremental.
#[serde(default)]
pub resume_clear: Vec<String>,
}
/// Quirks for a specific package
@@ -166,6 +180,33 @@ pub fn get_package_directories(package: &str, series: &str) -> Vec<String> {
directories
}
/// Get the paths to clear before a resumed `debian/rules build`
///
/// Every matching deb entry contributes its paths, in file order. See
/// [`OperationQuirks::resume_clear`] for what belongs here.
///
/// # Arguments
/// * `package` - The package name
/// * `series` - The distribution series (e.g. "resolute")
///
/// # Returns
/// * `Vec<String>` - Tree-relative paths to remove, empty when the package
/// opts out
pub fn get_resume_clear_paths(package: &str, series: &str) -> Vec<String> {
let Some(quirks) = get_package_quirks(&QUIRKS_DATA, package) else {
return Vec::new();
};
let mut paths = Vec::new();
for deb in quirks
.deb
.iter()
.filter(|q| entry_applies_to_series(q, series))
{
paths.extend(deb.resume_clear.iter().cloned());
}
paths
}
/// Apply the dependency quirks of `package` in `series` to parsed
/// build-dependency clauses
///
@@ -225,7 +266,7 @@ fn apply_rules(
}
for injected in &deps.inject {
let parsed = Deps::parse(injected, opts)?;
let parsed = Deps::parse(injected, opts).map_err(|e| e.to_string())?;
clauses.extend(parsed.clauses().map(<[PkgRelation]>::to_vec));
}
@@ -265,15 +306,62 @@ mod tests {
fn linux_dependency_quirks_are_series_scoped() {
for package in ["linux", "linux-riscv"] {
let rules = get_deb_dependency_quirks(package, "resolute");
assert_eq!(rules.len(), 1, "the resolute entry applies");
assert_eq!(rules.len(), 2, "the resolute entries apply");
assert_eq!(
rules[0].replace.get("llvm-21-dev").map(String::as_str),
Some("llvm-21-dev:native <!stage1>")
);
// Only the stonking entry exists there: the llvm-21-dev
// replacement above is a resolute-only packaging state.
let rules = get_deb_dependency_quirks(package, "stonking");
assert_eq!(rules.len(), 1);
assert!(rules[0].replace.is_empty());
assert!(get_deb_dependency_quirks(package, "noble").is_empty());
}
}
/// The kernels of the series whose control dropped the Debian-style
/// `:native` qualifiers on the host-tool libraries inject the
/// build-architecture variants: the dpkg cross rules resolve the
/// unqualified Multi-Arch: same names against the host architecture
/// only, leaving nothing for the kernel's host-side tools to link.
#[test]
fn linux_injects_native_host_tool_libraries_for_cross() {
for series in ["resolute", "stonking"] {
let mut clauses = vec![vec![parse("libelf-dev <!stage1>")]];
crate::quirks::apply_dependency_quirks("linux", series, &mut clauses, &opts()).unwrap();
let injected: Vec<&PkgRelation> = clauses[1..]
.iter()
.flatten()
.filter(|rel| {
["libelf-dev", "libdw-dev", "libssl-dev"].contains(&rel.package.as_str())
})
.collect();
assert_eq!(
injected.len(),
3,
"one :native clause per host-tool library"
);
assert!(
injected
.iter()
.all(|rel| rel.arch_qualifier.as_deref() == Some("native"))
);
// The declared dependency itself is untouched: the host
// (target) variant still installs for the checker.
assert_eq!(clauses[0][0].arch_qualifier, None);
}
}
/// Other kernel series keep their declared dependencies untouched.
#[test]
fn noble_linux_dependencies_are_not_rewritten() {
let mut clauses = vec![vec![parse("libelf-dev")]];
crate::quirks::apply_dependency_quirks("linux", "noble", &mut clauses, &opts()).unwrap();
assert_eq!(clauses.len(), 1);
assert_eq!(clauses[0][0].arch_qualifier, None);
}
/// `replace` rewrites exactly the dependencies whose declared name
/// matches, wholesale: the replacement carries its own qualifier and
/// restrictions.
@@ -327,4 +415,23 @@ mod tests {
};
assert!(apply_rules(&mut clauses, &deps, &opts()).is_err());
}
/// The kernels opt in to clearing their stamp cache on resume, for
/// every series; packages without the entry opt out by default.
#[test]
fn resume_clear_paths_are_data_driven() {
for package in ["linux", "linux-riscv"] {
assert_eq!(
get_resume_clear_paths(package, "resolute"),
vec!["debian/stamps".to_string()]
);
assert_eq!(
get_resume_clear_paths(package, "some-future-series"),
vec!["debian/stamps".to_string()],
"the stamp-cache entry applies to every series"
);
}
assert!(get_resume_clear_paths("hello", "resolute").is_empty());
assert!(get_resume_clear_paths("no-such-package", "resolute").is_empty());
}
}
src/report.rs
+26 -7
View File
@@ -28,6 +28,18 @@ use crate::logfmt::Classifier;
/// why it is rejected (the implementation re-asks with the explanation).
pub type Validator = dyn Fn(&str) -> Result<(), String>;
/// Which flow a target belongs to; names the terminal adapter's tee log
/// (`build-*.log`, `deb-*.log`, `test-*.log`).
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum TargetKind {
/// Source-only build (produces a `.dsc`).
Source,
/// Binary build (produces `.debs`).
Binary,
/// DEP-8 test run (`pkh test`).
Test,
}
/// Identity of the build whose events follow, as announced through
/// [`BuildView::target`].
#[derive(Debug, Clone)]
@@ -45,9 +57,8 @@ pub struct BuildTarget<'a> {
/// "Uploading hello (2.10-3) to ppa:user/ppa"): the wording is the
/// flow's, adapters render it verbatim.
pub display: String,
/// Whether this is a source-only build (producing a `.dsc`); names the
/// terminal adapter's tee log (`build-*.log` vs `deb-*.log`).
pub source_only: bool,
/// The flow driving the events; names the terminal adapter's tee log.
pub kind: TargetKind,
/// Whether the view should tee raw subprocess output to its log file.
/// Flows without subprocess output (uploads) pass `false` and create
/// no log file.
@@ -57,10 +68,12 @@ pub struct BuildTarget<'a> {
/// Observer of a running build: target identification, phases, status
/// messages, progress and the final outcome.
///
/// Implement this to observe the [`crate::build`] and [`crate::deb`] flows
/// from any frontend. All events arrive in order from the build thread;
/// long-lived views are expected to be `Send + Sync` because builds may run
/// inside async tasks.
/// Implement this to observe the [`crate::build`], [`crate::deb`] and
/// [`crate::pull`] flows from any frontend. All events arrive in order from
/// the build thread; long-lived views are expected to be `Send + Sync`
/// because builds may run inside async tasks. The pull flow only emits the
/// [`BuildView::message`], [`BuildView::progress`] and
/// [`BuildView::byte_progress`] events — no target, phases or outcome.
pub trait BuildView: Send + Sync {
/// The build target was identified; the events that follow belong to it
/// (including the earliest subprocess output, e.g. a chroot download).
@@ -81,6 +94,12 @@ pub trait BuildView: Send + Sync {
/// retrieval); `pos` runs from 0 to `total`.
fn progress(&self, _label: &str, _pos: usize, _total: usize) {}
/// Determinate progress of a byte transfer within the current phase
/// (e.g. the staged-tree copy forked from an incremental base):
/// `pos` of `total` bytes. Like [`BuildView::progress`], but adapters
/// render byte units for it.
fn byte_progress(&self, _label: &str, _pos: u64, _total: u64) {}
/// Sink receiving every raw subprocess line while a build command runs,
/// when this view consumes the lines itself (live rewriting, tee to a
/// log file, forwarding over the network). `None` lets the caller fall
src/test/bed.rs
+93
View File
@@ -0,0 +1,93 @@
//! Testbed lifecycle: where and how the tests run.
//!
//! A testbed is a [`Context`] plus its teardown. chroot mode reuses the
//! `pkh deb` machinery wholesale (an ephemeral unshare context guarded by
//! [`crate::deb::ephemeral::EphemeralContextGuard`], bootstrapped from the
//! same cached mmdebstrap tarballs); local mode is the current context
//! used directly. The testbed itself is disposable per run: autopkgtests
//! may break it, and the next run starts from a clean bootstrap.
use std::error::Error;
use std::sync::Arc;
use crate::context::Context;
use crate::deb::BuildProfiler;
use crate::deb::ephemeral::EphemeralContextGuard;
use crate::report::BuildView;
use crate::test::options::TestMode;
/// A prepared testbed, torn down on drop.
pub(crate) enum Bed {
/// An ephemeral unshare chroot.
Chroot(EphemeralContextGuard),
/// The current context, used directly.
Local(Arc<Context>),
}
impl Bed {
/// Prepare a testbed of `mode` for `series`/`arch` on `base_ctx`.
pub(crate) async fn prepare(
mode: TestMode,
series: &str,
arch: &str,
base_ctx: Arc<Context>,
view: &dyn BuildView,
) -> Result<Bed, Box<dyn Error>> {
match mode {
TestMode::Chroot => {
// Mirror the deb flow: only a foreign target requests a
// foreign-arch chroot, so the native case shares deb's
// unsuffixed `<series>-buildd.tar.xz` cache entry.
let chroot_arch = (arch != crate::get_current_arch()).then(|| arch.to_string());
let guard = EphemeralContextGuard::new_with_context(
series,
chroot_arch.as_deref(),
None,
base_ctx,
view,
&mut BuildProfiler::new(),
)
.await?;
Ok(Bed::Chroot(guard))
}
TestMode::Local => Ok(Bed::Local(base_ctx)),
// The vm mode never builds a Bed: run_vm execs autopkgtest.
TestMode::Vm => unreachable!("vm mode does not prepare a Bed"),
}
}
/// The context to run testbed commands in.
pub(crate) fn ctx(&self) -> Arc<Context> {
match self {
Bed::Chroot(guard) => guard.context(),
Bed::Local(ctx) => Arc::clone(ctx),
}
}
/// A scratch root inside the testbed for the staged tree, test
/// temporary directories and artifacts.
pub(crate) fn scratch_root(&self) -> Result<String, Box<dyn Error>> {
match self {
// A stable root inside the chroot; the guard creates a fresh
// temp directory for anonymous chroots.
Bed::Chroot(guard) => guard.build_root(),
Bed::Local(ctx) => Ok(ctx.create_temp_dir()?),
}
}
/// Whether root-requiring tests can run: always inside a chroot
/// (commands run as mapped root), on the local context only when the
/// context itself runs as root.
pub(crate) fn supports_root(&self) -> bool {
match self {
Bed::Chroot(_) => true,
Bed::Local(ctx) => matches!(
ctx.command("id")
.arg("-u")
.output()
.map(|output| String::from_utf8_lossy(&output.stdout).trim().to_string()),
Ok(id) if id == "0"
),
}
}
}
src/test/expand.rs
+248
View File
@@ -0,0 +1,248 @@
//! DEP-8 `Depends:` substitution: `@`, `@builddeps@` and `@recommends@`
//! expanded into concrete package specs for `apt-get install`.
//!
//! The built binaries themselves are installed separately (from the
//! staged `.deb` files, so tests exercise exactly what pkh built);
//! expansion substitutes bare binary *names*, which apt then resolves
//! against the installed binaries first and the archive second.
use crate::debian::control::ControlInfo;
use crate::debian::deps::{Deps, ParseOpts};
/// Expand a test stanza's `Depends:` value into apt package specs.
///
/// `raw` is the value as written (`None` = absent, the DEP-8 default
/// `@`). The substitutions, per DEP-8:
///
/// - `@` in an or-clause replicates the clause once per binary package
/// built from the source, with the name substituted (other
/// alternatives and version constraints are kept);
/// - `@builddeps@` becomes the source's Build-Depends,
/// Build-Depends-Indep and Build-Depends-Arch plus `build-essential`;
/// - `@recommends@` becomes the Recommends of every binary package,
/// with `${...}` substvar references dropped (autopkgtest's
/// "variables stripped") but versions kept.
///
/// Returns one apt spec per comma-separated clause, in order; an empty
/// result means nothing to install.
pub fn expand_depends(
raw: Option<&str>,
control: &ControlInfo,
binaries: &[String],
host_arch: &str,
) -> Result<Vec<String>, String> {
let raw = raw.unwrap_or("@");
let mut specs = Vec::new();
for token in split_top_level_commas(raw) {
match token.as_str() {
"@builddeps@" => {
for field in ["Build-Depends", "Build-Depends-Indep", "Build-Depends-Arch"] {
if let Some(value) = non_empty(control.source.get(field)) {
for clause in split_top_level_commas(value) {
if !clause.is_empty() {
specs.push(clause);
}
}
}
}
specs.push("build-essential".to_string());
}
"@recommends@" => {
for paragraph in &control.binaries {
if let Some(value) = non_empty(paragraph.get("Recommends")) {
specs.extend(name_only_clauses(value, host_arch)?);
}
}
}
token if token.split_whitespace().any(|word| word == "@") => {
// Textual substitution: DEP-8 @ is a word-level
// placeholder the deps grammar does not know. The clause
// replicates once per binary with the name substituted,
// constraints and alternatives kept verbatim.
for binary in binaries {
specs.push(
token
.split_whitespace()
.map(|word| {
if word == "@" {
binary.clone()
} else {
word.to_string()
}
})
.collect::<Vec<_>>()
.join(" "),
);
}
}
token => specs.push(token.to_string()),
}
}
Ok(specs)
}
/// Split on commas that sit outside any parentheses: or-clauses carry
/// their own commas inside `(...)` constraints, top-level commas separate
/// dependencies.
fn split_top_level_commas(input: &str) -> Vec<String> {
let mut parts = Vec::new();
let mut depth = 0usize;
let mut current = String::new();
for character in input.chars() {
match character {
'(' => {
depth += 1;
current.push(character);
}
')' => {
depth = depth.saturating_sub(1);
current.push(character);
}
',' if depth == 0 => {
parts.push(current.trim().to_string());
current.clear();
}
_ => current.push(character),
}
}
parts.push(current.trim().to_string());
parts
}
/// Render each or-clause of a dependency list, dropping alternatives
/// that are `${...}` substvar references (autopkgtest's "variables
/// stripped") and keeping everything else verbatim.
fn name_only_clauses(input: &str, host_arch: &str) -> Result<Vec<String>, String> {
let opts = ParseOpts::build_deps(host_arch.to_string(), Vec::new());
let deps = Deps::parse(input, &opts).map_err(|e| e.to_string())?;
let mut clauses = Vec::new();
for clause in deps.clauses() {
let kept: Vec<String> = clause
.iter()
.filter(|relation| !relation.package.contains('$'))
.map(|relation| relation.output())
.collect();
if !kept.is_empty() {
clauses.push(kept.join(" | "));
}
}
Ok(clauses)
}
/// A field value with its surrounding whitespace trimmed; `None` when the
/// field is absent or whitespace-only.
fn non_empty(value: Option<&str>) -> Option<&str> {
value.map(str::trim).filter(|value| !value.is_empty())
}
/// The binary package names declared by a control file, in stanza order.
pub fn binary_names(control: &ControlInfo) -> Vec<String> {
control
.binaries
.iter()
.filter_map(|paragraph| paragraph.get("Package"))
.map(|name| name.trim().to_string())
.collect()
}
#[cfg(test)]
mod tests {
use super::*;
use std::str::FromStr;
fn control() -> ControlInfo {
ControlInfo::from_str(
"Source: hello\n\
Build-Depends: debhelper-compat (= 13),\n meson\n\
Build-Depends-Indep: dh-python\n\
\n\
Package: hello\n\
Architecture: any\n\
Recommends: hello-doc, pulseaudio (>= 15)\n\
\n\
Package: hello-doc\n\
Architecture: all\n\
Recommends: hello\n",
)
.unwrap()
}
fn binaries() -> Vec<String> {
vec!["hello".to_string(), "hello-doc".to_string()]
}
/// The default (absent) Depends is `@`: one spec per binary.
#[test]
fn default_is_at() {
let specs = expand_depends(None, &control(), &binaries(), "amd64").unwrap();
assert_eq!(specs, ["hello", "hello-doc"]);
}
/// An or-clause containing @ replicates once per binary, keeping the
/// other alternatives and version constraints.
#[test]
fn or_clauses_replicate_per_binary() {
let specs = expand_depends(
Some("@ (>= 1.0) | fallback, jq"),
&control(),
&binaries(),
"amd64",
)
.unwrap();
assert_eq!(
specs,
[
"hello (>= 1.0) | fallback",
"hello-doc (>= 1.0) | fallback",
"jq"
]
);
}
/// @builddeps@ pulls the three Build-* fields plus build-essential.
#[test]
fn builddeps_expansion() {
let specs =
expand_depends(Some("@builddeps@, @"), &control(), &binaries(), "amd64").unwrap();
assert_eq!(
specs,
[
"debhelper-compat (= 13)",
"meson",
"dh-python",
"build-essential",
"hello",
"hello-doc"
]
);
}
/// @recommends@ collects every binary's Recommends, stripped to names.
#[test]
fn recommends_expansion() {
let specs = expand_depends(Some("@recommends@"), &control(), &binaries(), "amd64").unwrap();
assert_eq!(specs, ["hello-doc", "pulseaudio (>= 15)", "hello"]);
}
/// Plain dependencies pass through untouched; a token mentioning @ in
/// a version constraint is still a substitution.
#[test]
fn plain_tokens_pass_through() {
let specs = expand_depends(
Some("ca-certificates, python3:any"),
&control(),
&binaries(),
"amd64",
)
.unwrap();
assert_eq!(specs, ["ca-certificates", "python3:any"]);
}
/// Commas inside version constraints do not split clauses.
#[test]
fn parenthesised_commas_stay_together() {
let specs =
expand_depends(Some("foo (>= 1) , bar"), &control(), &binaries(), "amd64").unwrap();
assert_eq!(specs, ["foo (>= 1)", "bar"]);
}
}
src/test/mod.rs
+875
View File
@@ -0,0 +1,875 @@
//! Run a source package's DEP-8 as-installed tests (`pkh test`).
//!
//! The runner is native for the chroot and local testbeds: the testbed
//! is a pkh context (an ephemeral unshare chroot bootstrapped exactly
//! like a `pkh deb` build chroot, or the current context directly), and
//! the DEP-8 per-test contract lives in [`runner`]. `--mode vm` execs
//! the installed autopkgtest with its qemu runner instead — the one
//! backend pkh does not reimplement (serial-console protocol, image
//! management, reboot support).
//!
//! Binaries under test are the fresh `pkh deb` output next to the tree
//! (freshness-guarded like the lint wrapper); when they are missing or
//! stale, the deb flow builds them in-process first.
pub(crate) mod bed;
pub mod expand;
pub mod options;
pub mod outcome;
pub(crate) mod runner;
pub(crate) mod vm;
pub use options::{TestMode, TestOptions};
pub use outcome::TestRunReport;
use std::collections::HashMap;
use std::fs;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use crate::context::{LineSink, Stream};
use crate::debian::control::{ControlInfo, parse_paragraphs};
use crate::debian::tests_control::TestsControl;
use crate::logfmt::GenericClassifier;
use crate::report::{BuildTarget, BuildView, TargetKind};
use crate::test::bed::Bed;
use crate::test::outcome::Verdict;
/// Where the built binaries under test came from.
struct Binaries {
/// Host-side paths of the .deb files.
paths: Vec<PathBuf>,
/// Provenance note for the report.
note: String,
}
/// Run the DEP-8 tests of a source package tree.
///
/// Errors are runtime errors (no declared tests, unusable testbed,
/// failed in-process build); test failures are a report, not an error.
pub async fn run(opts: TestOptions<'_>) -> Result<TestRunReport, Box<dyn std::error::Error>> {
let view = opts.view;
let tree = opts
.tree
.clone()
.unwrap_or_else(|| std::env::current_dir().unwrap_or_else(|_| PathBuf::from(".")));
let tree = tree
.canonicalize()
.map_err(|e| format!("cannot access the source tree '{}': {e}", tree.display()))?;
// Package identity and target, resolved like a deb build.
let entry = crate::debian::parse_changelog_entry(&tree.join("debian/changelog"))
.map_err(|e| format!("cannot read the package identity: {e}"))?;
let package = entry.source.clone();
let version = entry.version.full();
let dist = opts
.dist
.clone()
.unwrap_or_else(crate::build::env::current_vendor);
let series = match opts.series.clone() {
Some(series) => series,
None if crate::distro_info::is_unreleased(&entry.distribution) => {
let resolved = crate::distro_info::effective_series(&entry.distribution, &dist).await?;
log::info!(
"Changelog is UNRELEASED, testing against series {}",
resolved
);
resolved
}
None => entry.distribution.clone(),
};
let arch = opts.arch.clone().unwrap_or_else(crate::get_current_arch);
// Pre-check: the package must declare tests, before any testbed is
// booted.
let control_path = tree.join("debian/tests/control");
if !control_path.exists() {
return Err(
"the package declares no autopkgtests (no debian/tests/control). \
Add debian/tests/control and 'Testsuite: autopkgtest' to debian/control \
(pkh lint flags this too), or point pkh test at another tree."
.into(),
);
}
let tests_control = TestsControl::parse(&control_path)
.map_err(|e| format!("cannot parse debian/tests/control: {e}"))?;
if tests_control.tests.is_empty() {
return Err("debian/tests/control declares no tests".into());
}
// Binaries under test: fresh pkh deb output, an in-process build, or
// an explicit --debs list.
let binaries = resolve_binaries(&opts, &tree, &package, &version, &arch, view).await?;
view.target(BuildTarget {
package: &package,
version: &version,
target: &format!("{series}/{arch}"),
display: format!(
"Testing {} ({}) on {}/{} ({})",
package,
version,
series,
arch,
opts.mode.unwrap_or_default().label()
),
kind: TargetKind::Test,
tee_log: true,
});
let mode = opts.mode.unwrap_or_default();
// The vm escape hatch goes through the installed autopkgtest; it
// owns the artifacts directory (passed as -o, which must not exist
// yet), so the path is only computed, not created.
if mode == TestMode::Vm {
let run_dir = run_dir_path(&package, &series, &arch);
let (outcomes, raw_code) = vm::run_vm(
&tree,
&binaries.paths,
&tests_control.tests,
&run_dir,
opts.image.as_deref(),
&opts,
view,
)?;
return finish(
TestRunReport {
package,
version,
series,
arch,
mode: mode.label().to_string(),
binaries: binaries.paths,
binaries_note: binaries.note,
outcomes,
artifacts_dir: run_dir,
raw_exit_code: Some(raw_code),
},
view,
);
}
if mode == TestMode::Local {
log::warn!(
"Tests run directly on the current context: they install packages \
and can rewrite it. Use the default chroot mode for isolation."
);
}
let run_dir = create_run_dir(&package, &series, &arch)?;
let sink = view.sink().or_else(|| {
// Verbose runs without a live view still stream test output live.
opts.verbose
.then(|| Arc::new(TerminalLines) as Arc<dyn LineSink>)
});
let base_ctx = crate::context::current();
let bed = Bed::prepare(mode, &series, &arch, base_ctx, view).await?;
let ctx = bed.ctx();
let mut env = HashMap::new();
env.insert("LANG".to_string(), "C".to_string());
env.insert("DEBIAN_FRONTEND".to_string(), "noninteractive".to_string());
if matches!(bed, Bed::Chroot(_)) {
env.insert("HOME".to_string(), "/root".to_string());
env.insert("USER".to_string(), "root".to_string());
env.insert("LOGNAME".to_string(), "root".to_string());
}
configure_testbed(
&ctx,
&env,
&series,
&arch,
&opts,
matches!(bed, Bed::Local(_)),
view,
)
.await?;
// Stage the tree and the binaries into the testbed. The copy also
// protects the host tree: tests may write into it (rw-build-tree).
view.message("Staging the tree and binaries");
let staging_root = bed.scratch_root()?;
let staged_tree = ctx.ensure_available(&tree, &staging_root)?;
let staged_tree = staged_tree.to_string_lossy().into_owned();
let mut staged_debs = Vec::new();
for deb in &binaries.paths {
staged_debs.push(ctx.ensure_available(deb, &staging_root)?);
}
// Install the exact binaries under test before the tests run.
if !staged_debs.is_empty() {
view.phase(
"Installing built binaries",
Box::new(GenericClassifier::new()),
);
let mut cmd = ctx.command("apt-get");
cmd.envs(env.clone())
.arg("-y")
.arg("--allow-downgrades")
.arg("install");
for deb in &staged_debs {
cmd.arg(format!(
"./{}",
deb.to_string_lossy().trim_start_matches('/')
));
}
let status = cmd.status().map_err(|e| e.to_string())?;
if !status.success() {
return Err("installing the built binaries into the testbed failed".into());
}
}
// What @ expands to: the names of the binaries actually under test,
// from their .deb file names (<binary>_<version>_<arch>.deb).
let control_info = ControlInfo::parse(&tree.join("debian/control"))
.map_err(|e| format!("cannot parse debian/control: {e}"))?;
let binary_names: Vec<String> = staged_debs
.iter()
.filter_map(|deb| deb_binary_name(deb))
.collect();
let run_ctx = runner::RunContext {
ctx,
tree: staged_tree.clone(),
staging_root,
arch: arch.clone(),
host_arch: crate::get_current_arch(),
supports_root: bed.supports_root(),
timeout_secs: opts.timeout_secs(),
sink,
view,
control: control_info,
binaries: binary_names,
env: env.clone(),
artifacts_dir: run_dir.clone(),
};
let outcomes = runner::run_tests(
&run_ctx,
&tests_control.tests,
&opts.test_names,
&opts.skip_tests,
);
// Debugging escape hatches.
let has_failures = outcomes
.iter()
.any(|outcome| outcome.verdict == Verdict::Fail);
if opts.shell || (opts.shell_fail && has_failures) {
drop_shell(&bed, &staged_tree, &env, view)?;
}
finish(
TestRunReport {
package,
version,
series,
arch,
mode: mode.label().to_string(),
binaries: binaries.paths,
binaries_note: binaries.note,
outcomes,
artifacts_dir: run_dir,
raw_exit_code: None,
},
view,
)
}
/// Announce the outcome on the view and hand the report back.
fn finish(
report: TestRunReport,
view: &dyn BuildView,
) -> Result<TestRunReport, Box<dyn std::error::Error>> {
if report.has_failures() {
view.finish_failure();
} else {
view.finish_success(std::slice::from_ref(&report.artifacts_dir));
}
Ok(report)
}
/// Resolve the binaries under test: `--debs` overrides everything;
/// otherwise the fresh `pkh deb` output next to the tree (freshness-
/// guarded, the lint wrapper's artifact strategy); otherwise the deb
/// flow builds them in-process. `--no-build` refuses instead.
async fn resolve_binaries(
opts: &TestOptions<'_>,
tree: &Path,
package: &str,
version: &str,
arch: &str,
view: &dyn BuildView,
) -> Result<Binaries, Box<dyn std::error::Error>> {
if !opts.debs.is_empty() {
for deb in &opts.debs {
if !deb.exists() {
return Err(format!("--deb file '{}' does not exist", deb.display()).into());
}
}
return Ok(Binaries {
paths: opts.debs.clone(),
note: "explicitly passed with --debs".to_string(),
});
}
if !opts.build
&& let Some(paths) = fresh_deb_output(tree, package, version, arch)
{
log::info!("Testing the fresh pkh deb output next to the tree");
return Ok(Binaries {
paths,
note: "fresh pkh deb output".to_string(),
});
}
if opts.no_build {
return Err(
"no fresh pkh deb output next to the tree and --no-build given: \
run 'pkh deb' first, drop --no-build, or pass --debs"
.into(),
);
}
view.message("No fresh pkh deb output: building the binaries first");
let artifacts = crate::deb::build_binary_package(crate::deb::DebBuildOptions {
series: opts.series.clone(),
arch: opts.arch.clone(),
pocket: opts.pocket.clone(),
ppa: opts.ppa.clone(),
cwd: Some(tree.to_path_buf()),
view,
..Default::default()
})
.await?;
let debs: Vec<PathBuf> = artifacts
.into_iter()
.filter(|path| path.extension().is_some_and(|ext| ext == "deb"))
.collect();
if debs.is_empty() {
return Err("the build produced no binary packages to test".into());
}
Ok(Binaries {
paths: debs,
note: "built by pkh deb just now".to_string(),
})
}
/// The fresh `pkh deb` output next to the tree, when it matches the
/// current changelog entry and no tree content is newer than it: the
/// `<source>_<version>_<arch>.changes` file gates, its checksum lists
/// name the debs. Stale or missing output tests nothing.
fn fresh_deb_output(tree: &Path, package: &str, version: &str, arch: &str) -> Option<Vec<PathBuf>> {
let entry = crate::debian::parse_changelog_entry(&tree.join("debian/changelog")).ok()?;
let parent = tree.parent()?;
let changes_path = parent.join(format!(
"{}_{}_{}.changes",
package,
entry.version.no_epoch(),
arch
));
let changes = fs::read_to_string(&changes_path).ok()?;
// Only the current entry's binaries are the ones under test.
let paragraph = parse_paragraphs(&changes).into_iter().next()?;
let matches = paragraph.get("Source").map(str::trim) == Some(package)
&& paragraph.get("Version").map(str::trim) == Some(version);
if !matches {
return None;
}
let mut debs = Vec::new();
for (field, value) in paragraph.iter() {
if !field.starts_with("Checksums-") && field != "Files" {
continue;
}
for line in value.lines() {
// `<checksum> <size> <section> [<priority>] <filename>`
if let Some(name) = line.split_whitespace().last()
&& name.ends_with(".deb")
{
debs.push(parent.join(name));
}
}
}
if debs.is_empty() || debs.iter().any(|deb| !deb.exists()) {
return None;
}
// A tree file written after the changes file makes the output stale.
let built = fs::metadata(&changes_path).ok()?.modified().ok()?;
if tree_newer_than(tree, built) {
return None;
}
Some(debs)
}
/// Whether any tree content is newer than `built`. Skips `.git` and
/// `.pc`: commits and quilt bookkeeping churn their mtimes without
/// touching what the package contains.
fn tree_newer_than(root: &Path, built: std::time::SystemTime) -> bool {
let mut stack = vec![root.to_path_buf()];
while let Some(dir) = stack.pop() {
let Ok(entries) = fs::read_dir(&dir) else {
continue;
};
for entry in entries.flatten() {
let path = entry.path();
let name = entry.file_name();
if name == ".git" || name == ".pc" {
continue;
}
let Ok(meta) = fs::symlink_metadata(&path) else {
continue;
};
if meta.is_dir() {
stack.push(path);
} else if meta.modified().is_ok_and(|mtime| mtime > built) {
return true;
}
}
}
false
}
/// Configure the testbed's APT setup and update it: pockets and PPAs on
/// the archive sources (the same rules as the deb build), an optional
/// upgrade, the caller's setup commands and injected packages.
async fn configure_testbed(
ctx: &Arc<crate::context::Context>,
env: &HashMap<String, String>,
series: &str,
arch: &str,
opts: &TestOptions<'_>,
local_context: bool,
view: &dyn BuildView,
) -> Result<(), Box<dyn std::error::Error>> {
let mut sources = crate::apt::sources::load(Some(ctx.clone()))?;
let mut modified = false;
let mut added_ppas: Vec<(&str, &str)> = Vec::new();
for ppa in &opts.ppa {
let (ppa_user, ppa_name) = crate::package_info::split_ppa(ppa)?;
let base_url = crate::package_info::ppa_to_base_url(ppa_user, ppa_name);
if sources.iter().any(|s| s.uri.contains(&base_url)) {
continue;
}
let mut architectures = vec![crate::get_current_arch()];
if arch != crate::get_current_arch() {
architectures.push(arch.to_string());
}
sources.push(crate::apt::sources::SourceEntry {
enabled: true,
kind: crate::apt::sources::SourceKind::Deb,
components: vec!["main".to_string()],
architectures,
signed_by: None,
trusted: None,
suite: vec![series.to_string()],
uri: base_url,
origin: None,
});
added_ppas.push((ppa_user, ppa_name));
modified = true;
}
// Ubuntu: test dependencies live in universe as often as in main.
let ubuntu_components = crate::distro_info::get_dist_components("ubuntu")?;
if ubuntu_components.iter().any(|c| c == "universe") {
for source in &mut sources {
if crate::distro_info::is_official_source("ubuntu", &source.uri)
&& !source.components.contains(&"universe".to_string())
{
source.components.push("universe".to_string());
modified = true;
}
}
}
if let Some(pocket) = &opts.pocket {
let pocket_suite = format!("{series}-{pocket}");
log::info!("Enabling pocket '{}' for test dependencies", pocket_suite);
for source in &mut sources {
if crate::deb::is_archive_source(&source.uri) && !source.suite.contains(&pocket_suite) {
source.suite.push(pocket_suite.clone());
modified = true;
}
}
// proposed is NotAutomatic: without a pin apt would ignore it.
if pocket.starts_with("proposed") {
crate::deb::local::pin_pocket(&pocket_suite, ctx)?;
}
}
if modified {
crate::apt::sources::save(Some(ctx.clone()), sources)?;
for (user, ppa_name) in added_ppas {
if let Err(e) =
crate::apt::keyring::download_trust_ppa_key(Some(ctx.clone()), user, ppa_name).await
{
log::warn!("Failed to download PPA key for {user}/{ppa_name}: {e}");
}
}
}
view.phase("Configuring testbed", Box::new(GenericClassifier::new()));
let status = ctx
.command("apt-get")
.envs(env.clone())
.arg("update")
.status()
.map_err(|e| format!("apt-get update failed inside the testbed: {e}"))?;
if !status.success() {
return Err(format!(
"apt-get update failed inside the testbed{}",
if local_context {
" (a local run needs root to install packages: \
use sudo, or the default chroot mode)"
} else {
""
}
)
.into());
}
if opts.apt_upgrade {
for action in ["upgrade", "autopurge"] {
let status = ctx
.command("apt-get")
.envs(env.clone())
.arg("-y")
.arg(action)
.status()
.map_err(|e| e.to_string())?;
if !status.success() {
return Err(format!("apt-get {action} failed inside the testbed").into());
}
}
}
for setup in &opts.setup_commands {
let status = ctx
.command("bash")
.envs(env.clone())
.arg("-e")
.arg("-c")
.arg(setup)
.status()
.map_err(|e| e.to_string())?;
if !status.success() {
return Err(format!("setup command failed: {setup}").into());
}
}
if !opts.inject.is_empty() {
// Testbed setup is not a profiled build: the durations and the
// phase profile go nowhere.
crate::deb::local::install_injected_packages(
&opts.inject,
env,
ctx.clone(),
view,
&None,
&mut crate::deb::session::BuildTimings::default(),
&mut crate::deb::BuildProfiler::new(),
)?;
}
Ok(())
}
/// Drop into an interactive testbed shell (the `--shell`/
/// `--shell-fail` debugging escape hatch).
fn drop_shell(
bed: &Bed,
staged_tree: &str,
env: &HashMap<String, String>,
view: &dyn BuildView,
) -> Result<(), Box<dyn std::error::Error>> {
view.suspend();
log::info!("Dropping into the testbed shell (exit to continue)");
let status = bed
.ctx()
.command("bash")
.envs(env.iter())
.current_dir(staged_tree)
.status()?;
if !status.success() {
log::warn!("Testbed shell exited with {}", status);
}
Ok(())
}
/// Create the run's host artifacts directory:
/// `/var/tmp/pkh/tests/<source>-<series>-<arch>-<timestamp>/`.
fn create_run_dir(package: &str, series: &str, arch: &str) -> std::io::Result<PathBuf> {
let timestamp = chrono::Utc::now().format("%Y%m%dT%H%M%S");
let dir = runs_root().join(format!("{package}-{series}-{arch}-{timestamp}"));
fs::create_dir_all(&dir)?;
Ok(dir)
}
/// The run directory path for an identity, without creating it.
fn run_dir_path(package: &str, series: &str, arch: &str) -> PathBuf {
let timestamp = chrono::Utc::now().format("%Y%m%dT%H%M%S");
runs_root().join(format!("{package}-{series}-{arch}-{timestamp}"))
}
/// The test-run artifacts root: `$PKH_TESTS_DIR`, defaulting to
/// `/var/tmp/pkh/tests`.
pub fn runs_root() -> PathBuf {
if let Ok(dir) = std::env::var("PKH_TESTS_DIR")
&& !dir.is_empty()
{
return PathBuf::from(dir);
}
PathBuf::from("/var/tmp/pkh/tests")
}
/// Render the tests declared in the tree's `debian/tests/control`, one
/// block per test (`pkh test --list-tests`).
pub fn list_tests(tree: &Path) -> Result<String, Box<dyn std::error::Error>> {
let tests_control = TestsControl::parse(&tree.join("debian/tests/control"))
.map_err(|e| format!("cannot parse debian/tests/control: {e}"))?;
let mut out = String::new();
for test in &tests_control.tests {
out.push_str(&test.name);
out.push('\n');
if let Some(command) = &test.command {
out.push_str(&format!(" Test-Command: {command}\n"));
}
if !test.restrictions.is_empty() {
out.push_str(&format!(
" Restrictions: {}\n",
test.restrictions.join(" ")
));
}
if let Some(depends) = &test.depends {
out.push_str(&format!(" Depends: {depends}\n"));
}
if let Some(directory) = &test.tests_directory {
out.push_str(&format!(" Tests-Directory: {directory}\n"));
}
}
Ok(out)
}
/// The binary package name of a staged .deb file name.
fn deb_binary_name(path: &Path) -> Option<String> {
let name = path.file_name()?.to_string_lossy();
let stem = name.strip_suffix(".deb")?;
Some(stem.split('_').next()?.to_string())
}
/// Live terminal passthrough of captured lines, for `--verbose` runs
/// without a live view.
struct TerminalLines;
impl LineSink for TerminalLines {
fn line(&self, stream: Stream, line: &str) {
match stream {
Stream::Stdout => println!("{line}"),
Stream::Stderr => eprintln!("{line}"),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
/// The freshness guard accepts the changes file matching the current
/// changelog entry with its debs, and rejects a stale one.
#[test]
fn fresh_deb_output_freshness() {
let base = tempfile::tempdir().unwrap();
let tree = base.path().join("hello");
fs::create_dir_all(tree.join("debian")).unwrap();
fs::write(
tree.join("debian/changelog"),
"hello (2.10-4) noble; urgency=medium\n\n * bump\n\n -- A <a@b.c> Thu, 01 Jan 2026 00:00:00 +0000\n",
)
.unwrap();
let changes_name = "hello_2.10-4_amd64.changes";
let write_changes = |fresh: bool| {
let content = format!(
"Format: 1.8\nSource: hello\nVersion: 2.10-4\nChecksums-Sha256:\n \
abc 123 utils {} \n",
changes_name.replace(".changes", ".deb")
);
let path = base.path().join(changes_name);
fs::write(&path, content).unwrap();
if !fresh {
// Push the tree strictly newer than the changes file.
let newer = tree.join("debian/control");
fs::write(&newer, "Source: hello\n").unwrap();
let time = std::time::SystemTime::now() + std::time::Duration::from_secs(3600);
let file = fs::File::options().append(true).open(&newer).unwrap();
file.set_times(std::fs::FileTimes::new().set_modified(time))
.unwrap();
}
};
// No changes file at all: nothing to reuse.
assert!(fresh_deb_output(&tree, "hello", "2.10-4", "amd64").is_none());
// The deb the changes file lists must exist for the reuse.
fs::write(base.path().join("hello_2.10-4_amd64.deb"), "deb").unwrap();
write_changes(true);
let debs = fresh_deb_output(&tree, "hello", "2.10-4", "amd64").unwrap();
assert_eq!(debs, vec![base.path().join("hello_2.10-4_amd64.deb")]);
// A different version or arch tests nothing.
assert!(fresh_deb_output(&tree, "hello", "2.10-3", "amd64").is_none());
assert!(fresh_deb_output(&tree, "hello", "2.10-4", "arm64").is_none());
// Tree content newer than the output makes it stale.
write_changes(false);
assert!(fresh_deb_output(&tree, "hello", "2.10-4", "amd64").is_none());
}
/// The run directory is created fresh under the pkh tests root.
#[test]
fn run_dir_layout() {
let dir = create_run_dir("hello", "noble", "amd64").unwrap();
assert_eq!(dir.parent(), Some(Path::new("/var/tmp/pkh/tests")));
assert!(
dir.file_name()
.and_then(|name| name.to_str())
.is_some_and(|name| name.starts_with("hello-noble-amd64-"))
);
assert!(dir.is_dir());
fs::remove_dir(&dir).ok();
}
/// The differential harness: pkh's native verdicts must match the
/// installed autopkgtest's on the same tree and binaries. This is
/// the trust anchor for the native runner — it pins the semantics
/// the DEP-8 documents under-specify. Deliberate, heavy and
/// host-mutating (both runners fall back to the host testbed), so
/// it only runs with --ignored and a tree to work on:
///
/// ```sh
/// PKH_TEST_DIFF_TREE=../hello cargo test -- --ignored \
/// test::tests::differential_harness_against_autopkgtest
/// ```
#[test]
fn differential_harness_against_autopkgtest() {
let Some(tree) = std::env::var("PKH_TEST_DIFF_TREE")
.ok()
.map(PathBuf::from)
.map(|tree| tree.canonicalize().unwrap())
else {
return; // not requested: nothing to compare
};
// pkh's native runner, on the host context (autopkgtest's null
// runner equivalent).
let rt = tokio::runtime::Runtime::new().unwrap();
let native = rt.block_on(crate::test::run(TestOptions {
tree: Some(tree.clone()),
mode: Some(TestMode::Local),
..Default::default()
}));
let native = match native {
Ok(report) => report,
Err(e) => panic!("native run failed: {e}"),
};
let native_map: HashMap<String, Verdict> = native
.outcomes
.iter()
.map(|outcome| (outcome.name.clone(), outcome.verdict))
.collect();
// The installed autopkgtest, same tree, same binaries, null
// runner (no isolation, like pkh's local mode).
let dir = tempfile::tempdir().unwrap();
let summary = dir.path().join("summary");
let mut command = std::process::Command::new("autopkgtest");
command
.arg("-o")
.arg(dir.path())
.arg("--summary")
.arg(&summary);
for deb in &native.binaries {
command.arg(deb);
}
command.arg(&tree).arg("--").arg("null");
let status = command.status().expect("autopkgtest must be installed");
let raw_code = status.code().unwrap_or(16);
assert!(
!matches!(raw_code, 12 | 14 | 16 | 20),
"autopkgtest failed outright (exit {raw_code})"
);
let tests_control = TestsControl::parse(&tree.join("debian/tests/control")).unwrap();
let wrapped = vm::parse_summary(&summary, &tests_control.tests).unwrap();
let wrapped_map: HashMap<String, Verdict> = wrapped
.iter()
.map(|outcome| (outcome.name.clone(), outcome.verdict))
.collect();
let delta = verdict_delta(&native_map, &wrapped_map);
assert!(
delta.is_empty(),
"native and autopkgtest disagree:\n{}",
delta.join("\n")
);
}
/// Compare per-test verdict maps: failures first (a verdict class
/// change matters), then tests only one runner reported. Returns one
/// human-readable line per difference.
fn verdict_delta(
native: &HashMap<String, Verdict>,
wrapped: &HashMap<String, Verdict>,
) -> Vec<String> {
let mut delta = Vec::new();
for (name, verdict) in native {
match wrapped.get(name) {
Some(wrapped) if wrapped == verdict => {}
Some(wrapped) => delta.push(format!(
"{name}: pkh={}, autopkgtest={}",
verdict.as_str(),
wrapped.as_str()
)),
None => delta.push(format!(
"{name}: pkh={}, autopkgtest did not report it",
verdict.as_str()
)),
}
}
for (name, verdict) in wrapped {
if !native.contains_key(name) {
delta.push(format!(
"{name}: autopkgtest={}, pkh did not report it",
verdict.as_str()
));
}
}
delta.sort();
delta
}
#[test]
fn verdict_delta_reports_only_disagreements() {
let native = HashMap::from([
("passing".to_string(), Verdict::Pass),
("failing".to_string(), Verdict::Fail),
("pkh-only".to_string(), Verdict::Skip),
]);
let wrapped = HashMap::from([
("passing".to_string(), Verdict::Pass),
("failing".to_string(), Verdict::Skip),
("wrapped-only".to_string(), Verdict::Fail),
]);
let delta = verdict_delta(&native, &wrapped);
assert_eq!(delta.len(), 3, "{delta:?}");
assert!(
delta[0].contains("failing: pkh=fail, autopkgtest=skip"),
"{delta:?}"
);
assert!(delta[1].contains("pkh-only"), "{delta:?}");
assert!(delta[2].contains("wrapped-only"), "{delta:?}");
}
}
src/test/options.rs
+219
View File
@@ -0,0 +1,219 @@
//! Options of a test run (`pkh test`), and their pure resolution rules.
use crate::report::BuildView;
use std::path::PathBuf;
/// Testbed mode: where the tests run.
///
/// The default is [`TestMode::Chroot`]: `pkh deb` defaults to the local
/// context because builds are non-destructive, but autopkgtests install
/// packages and are destructive by design — isolation is the default and
/// running on the host is the explicit choice.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub enum TestMode {
/// An ephemeral unshare chroot, bootstrapped like a `pkh deb` build
/// chroot (shared tarball cache).
#[default]
Chroot,
/// The current context directly: no isolation, tests can rewrite it.
Local,
/// A qemu virtual machine, through the installed autopkgtest's qemu
/// runner — the one backend pkh does not reimplement.
Vm,
}
impl TestMode {
/// Label used in the report and the view status.
pub fn label(&self) -> &'static str {
match self {
TestMode::Chroot => "chroot",
TestMode::Local => "local",
TestMode::Vm => "vm",
}
}
/// Parse a `--mode` value.
pub fn parse(value: &str) -> Option<TestMode> {
match value {
"chroot" => Some(TestMode::Chroot),
"local" => Some(TestMode::Local),
"vm" => Some(TestMode::Vm),
_ => None,
}
}
}
/// Parameters of one [`crate::test::run`] call.
pub struct TestOptions<'a> {
/// Source tree to test; defaults to the process working directory.
pub tree: Option<PathBuf>,
/// Target distribution series; defaults to the changelog series
/// (UNRELEASED resolves to the vendor's development series).
pub series: Option<String>,
/// Target distribution (debian/ubuntu); defaults to the host vendor.
pub dist: Option<String>,
/// Target architecture; defaults to the host architecture.
pub arch: Option<String>,
/// Distribution pocket to resolve test dependencies from.
pub pocket: Option<String>,
/// PPAs to add as dependency sources (`user/ppa_name`).
pub ppa: Vec<String>,
/// Packages to inject into the testbed before the tests (.deb paths
/// or archive names).
pub inject: Vec<String>,
/// Testbed mode; defaults to [`TestMode::Chroot`].
pub mode: Option<TestMode>,
/// VM image for [`TestMode::Vm`] (required there).
pub image: Option<PathBuf>,
/// VM virtual CPUs ([`TestMode::Vm`], passthrough).
pub cpus: Option<u32>,
/// VM memory size in MiB ([`TestMode::Vm`], passthrough).
pub ram_size: Option<String>,
/// Explicit binary packages to test; overrides the `pkh deb` output
/// reuse.
pub debs: Vec<PathBuf>,
/// Never build: error out when no fresh `pkh deb` output exists.
pub no_build: bool,
/// Ignore fresh `pkh deb` output and rebuild through the deb flow.
pub build: bool,
/// Run only these tests (exact names); empty = all.
pub test_names: Vec<String>,
/// Never run these tests (they are reported as skipped).
pub skip_tests: Vec<String>,
/// Extra shell commands run in the testbed after the APT setup.
pub setup_commands: Vec<String>,
/// Upgrade the testbed (update/upgrade/autopurge) before the tests.
pub apt_upgrade: bool,
/// Drop into an interactive testbed shell after the run.
pub shell: bool,
/// Drop into an interactive testbed shell when a test failed.
pub shell_fail: bool,
/// Per-test timeout in seconds; defaults to autopkgtest's 10 000 s,
/// scaled by `timeout_factor`.
pub timeout: Option<u64>,
/// Scale the default timeouts (`--timeout-factor`).
pub timeout_factor: Option<f64>,
/// Exit 1 when tests were skipped or flaky (`--fail-on skip`).
pub fail_on_skip: bool,
/// Exit 1 when no tests ran (`--fail-on no-tests`).
pub fail_on_no_tests: bool,
/// Render the report as JSON instead of text.
pub json: bool,
/// Raw test output instead of the summarized live view.
pub verbose: bool,
/// Where run events (phases, progress, outcome) are reported.
pub view: &'a dyn BuildView,
}
impl Default for TestOptions<'_> {
fn default() -> Self {
static QUIET: crate::report::Quiet = crate::report::Quiet;
TestOptions {
tree: None,
series: None,
dist: None,
arch: None,
pocket: None,
ppa: Vec::new(),
inject: Vec::new(),
mode: None,
image: None,
cpus: None,
ram_size: None,
debs: Vec::new(),
no_build: false,
build: false,
test_names: Vec::new(),
skip_tests: Vec::new(),
setup_commands: Vec::new(),
apt_upgrade: false,
shell: false,
shell_fail: false,
timeout: None,
timeout_factor: None,
fail_on_skip: false,
fail_on_no_tests: false,
json: false,
verbose: false,
view: &QUIET,
}
}
}
impl TestOptions<'_> {
/// Whether a declared test is selected for the run: an explicit
/// `--test-name` list must contain it, and no `--skip-test` may.
pub fn is_selected(&self, name: &str) -> bool {
(self.test_names.is_empty() || self.test_names.iter().any(|n| n == name))
&& !self.skip_tests.iter().any(|n| n == name)
}
/// The per-test timeout in seconds: an explicit `--timeout` stands as
/// given; otherwise the autopkgtest default is scaled by
/// `--timeout-factor`.
pub fn timeout_secs(&self) -> u64 {
match (self.timeout, self.timeout_factor) {
(Some(secs), _) => secs,
(None, factor) => (10_000_f64 * factor.unwrap_or(1.0)) as u64,
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn options(
test_names: &[&str],
skip_tests: &[&str],
timeout: Option<u64>,
factor: Option<f64>,
) -> TestOptions<'static> {
TestOptions {
test_names: test_names.iter().map(|s| s.to_string()).collect(),
skip_tests: skip_tests.iter().map(|s| s.to_string()).collect(),
timeout,
timeout_factor: factor,
..Default::default()
}
}
#[test]
fn selection_filters() {
let all = options(&[], &[], None, None);
assert!(all.is_selected("smoke"));
let only = options(&["smoke", "lint"], &[], None, None);
assert!(only.is_selected("smoke"));
assert!(!only.is_selected("other"));
let skipped = options(&[], &["heavy"], None, None);
assert!(skipped.is_selected("smoke"));
assert!(!skipped.is_selected("heavy"));
// An explicit list and a skip compose: the skip wins.
let both = options(&["smoke", "heavy"], &["heavy"], None, None);
assert!(both.is_selected("smoke"));
assert!(!both.is_selected("heavy"));
}
#[test]
fn timeout_resolution() {
// autopkgtest's default test timeout, unscaled.
assert_eq!(options(&[], &[], None, None).timeout_secs(), 10_000);
// The factor scales only the default.
assert_eq!(options(&[], &[], None, Some(2.0)).timeout_secs(), 20_000);
// An explicit timeout stands as given.
assert_eq!(options(&[], &[], Some(60), Some(2.0)).timeout_secs(), 60);
}
#[test]
fn mode_parsing() {
assert_eq!(TestMode::parse("chroot"), Some(TestMode::Chroot));
assert_eq!(TestMode::parse("local"), Some(TestMode::Local));
assert_eq!(TestMode::parse("vm"), Some(TestMode::Vm));
assert_eq!(TestMode::parse("other"), None);
assert_eq!(TestMode::default(), TestMode::Chroot);
assert_eq!(TestMode::Vm.label(), "vm");
}
}
src/test/outcome.rs
+332
View File
@@ -0,0 +1,332 @@
//! The test run report: per-test outcomes, text/JSON rendering and the
//! pkh exit-code contract.
use std::path::PathBuf;
use std::time::Duration;
/// Outcome of one DEP-8 test.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Verdict {
/// The test passed (exit 0, no unexpected stderr).
Pass,
/// The test failed (nonzero exit, unexpected stderr, timeout, or its
/// dependencies could not be installed).
Fail,
/// The test did not run: unmet restriction on this testbed, filtered
/// by `--skip-test`, or self-skipped via exit 77 under `skippable`.
Skip,
/// A `flaky`-restricted test failed; the failure is tolerated (like a
/// skip for the exit code) but reported.
Flaky,
}
impl Verdict {
/// Lowercase name used in the JSON report and the summary.
pub fn as_str(&self) -> &'static str {
match self {
Verdict::Pass => "pass",
Verdict::Fail => "fail",
Verdict::Skip => "skip",
Verdict::Flaky => "flaky",
}
}
}
/// One test's recorded outcome.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct TestOutcome {
/// Test name (the `Tests:` path or resolved `Test-Command` name).
pub name: String,
/// How the test ended.
pub verdict: Verdict,
/// Skip or failure reason, when there is one.
pub reason: Option<String>,
/// Wall-clock time the test ran for (zero for skipped tests).
pub duration: Duration,
}
/// The whole run's result, as returned by [`crate::test::run`].
#[derive(Debug, Clone)]
pub struct TestRunReport {
/// Source package name.
pub package: String,
/// Changelog version the run tested.
pub version: String,
/// Distribution series the testbed ran.
pub series: String,
/// Architecture the testbed ran.
pub arch: String,
/// Testbed mode used (`chroot`, `local`, `vm`).
pub mode: String,
/// The binary packages installed for the tests (host-side paths).
pub binaries: Vec<PathBuf>,
/// Where the binaries came from, for the report (freshness reuse,
/// in-process build, explicit request).
pub binaries_note: String,
/// One outcome per selected test, in declaration order.
pub outcomes: Vec<TestOutcome>,
/// Host directory holding the run's artifacts (log, per-test
/// artifacts).
pub artifacts_dir: PathBuf,
/// Raw autopkgtest exit code, when the run went through the vm
/// escape hatch.
pub raw_exit_code: Option<i32>,
}
impl TestRunReport {
/// Whether at least one test failed.
pub fn has_failures(&self) -> bool {
self.outcomes
.iter()
.any(|outcome| outcome.verdict == Verdict::Fail)
}
/// The pkh exit code: 0 pass/skip/flaky (unless `--fail-on`), 1 test
/// failure (or `--fail-on` trigger), 2 for runtime errors (those
/// abort the run with an `Err` before a report exists).
pub fn exit_code(&self, fail_on_skip: bool, fail_on_no_tests: bool) -> i32 {
if self.has_failures() {
return 1;
}
if fail_on_skip
&& self
.outcomes
.iter()
.any(|outcome| matches!(outcome.verdict, Verdict::Skip | Verdict::Flaky))
{
return 1;
}
if fail_on_no_tests && self.outcomes.is_empty() {
return 1;
}
0
}
/// Human-readable text report: one line per test, then the summary.
pub fn render_text(&self) -> String {
let mut out = String::new();
for outcome in &self.outcomes {
let secs = outcome.duration.as_secs_f32();
match outcome.verdict {
Verdict::Pass => out.push_str(&format!("✓ {} ({secs:.1}s)\n", outcome.name)),
Verdict::Fail => {
out.push_str(&format!("✗ {}", outcome.name));
if let Some(reason) = &outcome.reason {
out.push_str(&format!(" — {reason}"));
}
out.push_str(&format!(" ({secs:.1}s)\n"));
}
Verdict::Skip => {
out.push_str(&format!("- {} (skipped", outcome.name));
if let Some(reason) = &outcome.reason {
out.push_str(&format!(": {reason}"));
}
out.push_str(")\n");
}
Verdict::Flaky => {
out.push_str(&format!("~ {} (flaky", outcome.name));
if let Some(reason) = &outcome.reason {
out.push_str(&format!(": {reason}"));
}
out.push_str(&format!(") ({secs:.1}s)\n"));
}
}
}
let pass = self
.outcomes
.iter()
.filter(|outcome| outcome.verdict == Verdict::Pass)
.count();
let fail = self
.outcomes
.iter()
.filter(|outcome| outcome.verdict == Verdict::Fail)
.count();
let skipped = self.outcomes.len() - pass - fail;
if self.outcomes.is_empty() {
out.push_str("No tests ran.\n");
} else if fail > 0 {
out.push_str(&format!(
"{fail} of {} tests failed{}.\n",
self.outcomes.len(),
if skipped > 0 {
format!(" ({skipped} skipped or flaky)")
} else {
String::new()
}
));
} else {
out.push_str(&format!(
"All {} tests passed{}.\n",
self.outcomes.len(),
if skipped > 0 {
format!(" ({skipped} skipped or flaky)")
} else {
String::new()
}
));
}
out.push_str(&format!("Artifacts: {}\n", self.artifacts_dir.display()));
out
}
/// Schema-versioned JSON report, same content as the text rendering.
pub fn render_json(&self) -> String {
#[derive(serde::Serialize)]
#[serde(rename_all = "snake_case")]
struct Outcome<'a> {
name: &'a str,
verdict: &'a str,
#[serde(skip_serializing_if = "Option::is_none")]
reason: Option<&'a String>,
duration_secs: f64,
}
#[derive(serde::Serialize)]
#[serde(rename_all = "snake_case")]
struct Report<'a> {
schema: &'static str,
package: &'a str,
version: &'a str,
series: &'a str,
arch: &'a str,
mode: &'a str,
binaries: Vec<String>,
binaries_note: &'a str,
outcomes: Vec<Outcome<'a>>,
artifacts_dir: String,
#[serde(skip_serializing_if = "Option::is_none")]
raw_exit_code: Option<i32>,
}
let report = Report {
schema: "pkh-test-report-v1",
package: &self.package,
version: &self.version,
series: &self.series,
arch: &self.arch,
mode: &self.mode,
binaries: self
.binaries
.iter()
.map(|path| path.display().to_string())
.collect(),
binaries_note: &self.binaries_note,
outcomes: self
.outcomes
.iter()
.map(|outcome| Outcome {
name: &outcome.name,
verdict: outcome.verdict.as_str(),
reason: outcome.reason.as_ref(),
duration_secs: outcome.duration.as_secs_f64(),
})
.collect(),
artifacts_dir: self.artifacts_dir.display().to_string(),
raw_exit_code: self.raw_exit_code,
};
serde_json::to_string_pretty(&report).unwrap_or_else(|_| "{}".to_string())
}
}
#[cfg(test)]
mod tests {
use super::*;
fn outcome(name: &str, verdict: Verdict, reason: Option<&str>) -> TestOutcome {
TestOutcome {
name: name.to_string(),
verdict,
reason: reason.map(String::from),
duration: Duration::from_millis(1500),
}
}
fn report(outcomes: Vec<TestOutcome>) -> TestRunReport {
TestRunReport {
package: "hello".to_string(),
version: "2.10-4".to_string(),
series: "noble".to_string(),
arch: "amd64".to_string(),
mode: "chroot".to_string(),
binaries: vec![PathBuf::from("../hello_2.10-4_amd64.deb")],
binaries_note: "fresh pkh deb output".to_string(),
outcomes,
artifacts_dir: PathBuf::from("/var/tmp/pkh/tests/hello-noble-amd64-x/"),
raw_exit_code: None,
}
}
/// The exit-code contract: failures dominate, `--fail-on` promotes
/// skips and empty runs.
#[test]
fn exit_code_matrix() {
let pass = report(vec![outcome("a", Verdict::Pass, None)]);
assert_eq!(pass.exit_code(false, false), 0);
assert_eq!(pass.exit_code(true, true), 0);
let skip = report(vec![
outcome("a", Verdict::Pass, None),
outcome("b", Verdict::Skip, Some("needs root")),
]);
assert_eq!(skip.exit_code(false, false), 0);
assert_eq!(skip.exit_code(true, false), 1);
let fail = report(vec![
outcome("a", Verdict::Pass, None),
outcome("b", Verdict::Fail, Some("exit code 1")),
]);
assert_eq!(fail.exit_code(false, false), 1);
// A failure dominates every other trigger.
assert_eq!(fail.exit_code(true, true), 1);
let empty = report(vec![]);
assert_eq!(empty.exit_code(false, false), 0);
assert_eq!(empty.exit_code(false, true), 1);
let flaky = report(vec![outcome("a", Verdict::Flaky, Some("passed on retry"))]);
assert_eq!(flaky.exit_code(false, false), 0);
assert_eq!(flaky.exit_code(true, false), 1);
}
/// The text rendering shows every verdict with reasons and a summary.
#[test]
fn text_rendering() {
let run = report(vec![
outcome("smoke", Verdict::Pass, None),
outcome("l10n", Verdict::Fail, Some("exit code 1")),
outcome("heavy", Verdict::Skip, Some("requires a VM")),
]);
let text = run.render_text();
assert!(text.contains("✓ smoke"), "{text}");
assert!(text.contains("✗ l10n — exit code 1"), "{text}");
assert!(text.contains("- heavy (skipped: requires a VM)"), "{text}");
assert!(
text.contains("1 of 3 tests failed (1 skipped or flaky)"),
"{text}"
);
assert!(
text.contains("Artifacts: /var/tmp/pkh/tests/hello-noble-amd64-x/"),
"{text}"
);
let empty = report(vec![]).render_text();
assert!(empty.contains("No tests ran."), "{empty}");
let clean = report(vec![outcome("smoke", Verdict::Pass, None)]).render_text();
assert!(clean.contains("All 1 tests passed."), "{clean}");
}
/// The JSON report round-trips the report shape with a schema marker.
#[test]
fn json_rendering() {
let run = report(vec![outcome("smoke", Verdict::Pass, None)]);
let json: serde_json::Value = serde_json::from_str(&run.render_json()).unwrap();
assert_eq!(json["schema"], "pkh-test-report-v1");
assert_eq!(json["package"], "hello");
assert_eq!(json["mode"], "chroot");
assert_eq!(json["outcomes"][0]["verdict"], "pass");
assert_eq!(json["outcomes"][0]["duration_secs"], 1.5);
assert!(json.get("reason").is_none() && json["outcomes"][0].get("reason").is_none());
assert!(json.get("raw_exit_code").is_none());
}
}
src/test/runner.rs
+544
View File
@@ -0,0 +1,544 @@
//! The DEP-8 per-test contract: restriction gates, dependency
//! installation, the run environment, verdicts and artifact retrieval.
//!
//! The contract (autopkgtest 6.x `README.package-tests`): tests run in
//! the unpacked source tree with a fresh `$AUTOPKGTEST_TMP`; exit 0 with
//! empty stderr passes, any stderr output fails unless the
//! `allow-stderr` restriction is declared, exit 77 skips under
//! `skippable`, and a per-test timeout kills the test.
use std::collections::HashMap;
use std::fs;
use std::io::Write;
use std::path::PathBuf;
use std::sync::Arc;
use std::sync::Mutex;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::{Duration, Instant};
use crate::context::{Context, LineSink, Stream};
use crate::debian::control::ControlInfo;
use crate::debian::tests_control::Test;
use crate::logfmt::GenericClassifier;
use crate::report::BuildView;
use crate::test::expand::expand_depends;
use crate::test::outcome::{TestOutcome, Verdict};
/// Grace period `timeout --kill-after` gives a test after SIGTERM.
const KILL_AFTER_SECS: u64 = 15;
/// Everything one test run needs from the orchestration.
pub(crate) struct RunContext<'a> {
/// The testbed context.
pub ctx: Arc<Context>,
/// Context-side path of the staged source tree.
pub tree: String,
/// Context-side scratch root (temporary directories, artifacts).
pub staging_root: String,
/// The testbed architecture.
pub arch: String,
/// The host architecture (for `skip-foreign-architecture`).
pub host_arch: String,
/// Whether root-requiring tests can run.
pub supports_root: bool,
/// Per-test timeout in seconds.
pub timeout_secs: u64,
/// The view's tee sink, when the view consumes subprocess lines.
pub sink: Option<Arc<dyn LineSink>>,
/// Where run events are reported.
pub view: &'a dyn BuildView,
/// The source control file, for `Depends:` expansion.
pub control: ControlInfo,
/// The binary package names built for the tests.
pub binaries: Vec<String>,
/// Base environment for testbed commands.
pub env: HashMap<String, String>,
/// Host directory holding the run's artifacts.
pub artifacts_dir: PathBuf,
}
/// Run the tests in declaration order, filtered by the caller's
/// selection, and return one outcome per selected test.
pub(crate) fn run_tests(
run: &RunContext<'_>,
tests: &[Test],
only: &[String],
skipped: &[String],
) -> Vec<TestOutcome> {
let selected: Vec<&Test> = tests
.iter()
.filter(|test| only.is_empty() || only.iter().any(|name| name == &test.name))
.collect();
let total = selected.len();
run.view
.phase("Running tests", Box::new(GenericClassifier::new()));
let mut outcomes = Vec::new();
for (index, test) in selected.into_iter().enumerate() {
let outcome = if skipped.iter().any(|name| name == &test.name) {
TestOutcome {
name: test.name.clone(),
verdict: Verdict::Skip,
reason: Some("excluded by --skip-test".to_string()),
duration: Duration::ZERO,
}
} else {
run_one(run, test)
};
run.view.progress("Tests", index + 1, total);
outcomes.push(outcome);
}
outcomes
}
/// The result of one raw test execution, as seen by the verdict logic.
pub(crate) struct RawRun {
/// Process exit code (`None` when killed by a signal).
pub code: Option<i32>,
/// Whether the test produced any stderr output.
pub stderr_nonempty: bool,
/// Whether the `timeout` wrapper killed the test.
pub timed_out: bool,
}
/// The verdict for one raw run, per the DEP-8 contract.
pub(crate) fn decide(test: &Test, raw: &RawRun, timeout_secs: u64) -> (Verdict, Option<String>) {
if raw.timed_out {
return (
Verdict::Fail,
Some(format!("timed out after {timeout_secs}s")),
);
}
match raw.code {
// `timeout` reports its kill with 124; a test exiting 124 by
// itself is indistinguishable and counted as a timeout.
Some(124) => (
Verdict::Fail,
Some(format!("timed out after {timeout_secs}s")),
),
None => (Verdict::Fail, Some("killed by a signal".to_string())),
// A self-skipping test may be chatty: the skip verdict comes
// before the stderr rule.
Some(77) if test.has_restriction("skippable") => (
Verdict::Skip,
Some("the test skipped itself (exit 77)".to_string()),
),
Some(0) if !raw.stderr_nonempty || test.has_restriction("allow-stderr") => {
(Verdict::Pass, None)
}
Some(0) => (Verdict::Fail, Some("the test wrote to stderr".to_string())),
Some(code) => (Verdict::Fail, Some(format!("exit code {code}"))),
}
}
/// The static reason a test cannot run on this testbed, if any.
pub(crate) fn restriction_skip(
test: &Test,
supports_root: bool,
foreign_arch: bool,
) -> Option<String> {
if test.has_restriction("isolation-machine") {
return Some("requires a virtual machine (isolation-machine); use --mode vm".to_string());
}
if test.has_restriction("needs-reboot") {
return Some("requires reboot support (needs-reboot); use --mode vm".to_string());
}
if test.has_restriction("isolation-container") {
return Some("requires container isolation (isolation-container)".to_string());
}
if test.has_restriction("skip-foreign-architecture") && foreign_arch {
return Some("skipped on a foreign architecture (skip-foreign-architecture)".to_string());
}
if (test.has_restriction("needs-root") || test.has_restriction("needs-sudo")) && !supports_root
{
return Some("requires root on this testbed".to_string());
}
None
}
/// Run one test through the DEP-8 contract.
fn run_one(run: &RunContext<'_>, test: &Test) -> TestOutcome {
let started = Instant::now();
if let Some(reason) = restriction_skip(test, run.supports_root, run.arch != run.host_arch) {
return TestOutcome {
name: test.name.clone(),
verdict: Verdict::Skip,
reason: Some(reason),
duration: Duration::ZERO,
};
}
// Test dependencies: expanded per stanza, installed before the test.
if let Err(reason) = install_test_deps(run, test) {
return outcome_fail(test, started, reason);
}
// build-needed: the tests need the tree in its built state.
if test.has_restriction("build-needed")
&& let Err(reason) = build_tree(run)
{
return outcome_fail(test, started, reason);
}
let (raw, duration) = execute(run, test);
let (mut verdict, mut reason) = decide(test, &raw, run.timeout_secs);
// flaky: one tolerated failure is retried once and reported as FLAKY
// rather than FAIL (autopkgtest's semantics).
if verdict == Verdict::Fail && test.has_restriction("flaky") {
let (retry_raw, _) = execute(run, test);
let (retry_verdict, retry_reason) = decide(test, &retry_raw, run.timeout_secs);
reason = if retry_verdict == Verdict::Pass {
Some("passed on retry".to_string())
} else {
reason.or(retry_reason)
};
verdict = Verdict::Flaky;
}
TestOutcome {
name: test.name.clone(),
verdict,
reason,
duration,
}
}
fn outcome_fail(test: &Test, started: Instant, reason: String) -> TestOutcome {
TestOutcome {
name: test.name.clone(),
verdict: Verdict::Fail,
reason: Some(reason),
duration: started.elapsed(),
}
}
/// Execute a test once, returning the raw result and its duration.
fn execute(run: &RunContext<'_>, test: &Test) -> (RawRun, Duration) {
let started = Instant::now();
// Fresh per-test scratch and artifact directories.
let tmp_dir = run
.ctx
.create_temp_dir()
.unwrap_or_else(|_| format!("{}/tmp-{}", run.staging_root, safe_name(&test.name)));
let artifacts_dir = format!("{}/artifacts/{}", run.staging_root, safe_name(&test.name));
let _ = run
.ctx
.command("mkdir")
.arg("-p")
.arg(&artifacts_dir)
.status();
// The test's argv, wrapped in the per-test timeout.
let mut argv: Vec<String> = vec![
"timeout".to_string(),
"--foreground".to_string(),
format!("--kill-after={KILL_AFTER_SECS}"),
run.timeout_secs.to_string(),
];
match (&test.script, &test.command) {
(Some(script), _) => {
let path = test_script_path(run, test, script);
// Scripts may be shipped without the executable bit.
let _ = run.ctx.command("chmod").arg("+x").arg(&path).status();
argv.push(path);
}
(None, Some(command)) => {
argv.push("bash".to_string());
argv.push("-e".to_string());
argv.push("-c".to_string());
argv.push(command.clone());
}
(None, None) => unreachable!("parser rejects stanzas with neither Tests nor Test-Command"),
}
// The DEP-8 environment. AUTOPKGTEST_* are the current names; the
// ADT_* aliases are still read by older tests.
let mut env = run.env.clone();
for (key, value) in [
("AUTOPKGTEST_TMP", tmp_dir.as_str()),
("ADTTMP", tmp_dir.as_str()),
("AUTOPKGTEST_ARTIFACTS", artifacts_dir.as_str()),
("ADT_ARTIFACTS", artifacts_dir.as_str()),
("AUTOPKGTEST_TESTBED_ARCH", run.arch.as_str()),
] {
env.insert(key.to_string(), value.to_string());
}
let capture = Arc::new(TestSink::new(
host_artifacts_dir(run, &test.name),
run.sink.clone(),
));
let mut cmd = run.ctx.command(&argv[0]);
cmd.args(&argv[1..])
.envs(env.iter())
.current_dir(&run.tree)
.capture(capture.clone());
let status = cmd.status().ok();
// The scratch directory is disposable; the artifacts directory was
// already retrieved by the sink's host-side files plus the copy
// below (tests may have written into $AUTOPKGTEST_ARTIFACTS).
let _ = run.ctx.command("rm").arg("-rf").arg(&tmp_dir).status();
retrieve_artifacts(run, &test.name);
let raw = RawRun {
code: status.as_ref().and_then(|status| status.code()),
stderr_nonempty: capture.stderr_lines() > 0,
timed_out: status.as_ref().and_then(|status| status.code()) == Some(124),
};
(raw, started.elapsed())
}
/// Install a test stanza's expanded dependencies.
fn install_test_deps(run: &RunContext<'_>, test: &Test) -> Result<(), String> {
let specs = expand_depends(
test.depends.as_deref(),
&run.control,
&run.binaries,
&run.arch,
)
.map_err(|e| format!("cannot expand the test dependencies: {e}"))?;
if specs.is_empty() {
return Ok(());
}
run.view
.message(&format!("Installing dependencies of {}", test.name));
let mut cmd = run.ctx.command("apt-get");
cmd.envs(run.env.clone())
.arg("-y")
.arg("--no-install-recommends")
.arg("install");
for spec in &specs {
cmd.arg(spec);
}
let status = cmd.status().map_err(|e| e.to_string())?;
if status.success() {
Ok(())
} else {
Err(format!(
"dependency installation failed ({})",
specs.join(", ")
))
}
}
/// Bring the staged tree to its built state (`build-needed` tests): the
/// autopkgtest build, with the nocheck profile — the tests need the
/// build products, not another set of binaries.
fn build_tree(run: &RunContext<'_>) -> Result<(), String> {
let mut env = run.env.clone();
env.insert("DEB_BUILD_OPTIONS".to_string(), "nocheck".to_string());
let mut cmd = run.ctx.command("dpkg-buildpackage");
cmd.envs(env.iter())
.arg("-us")
.arg("-uc")
.arg("-b")
.current_dir(&run.tree);
let status = cmd.status().map_err(|e| e.to_string())?;
if status.success() {
Ok(())
} else {
Err("building the source tree failed (build-needed)".to_string())
}
}
/// The context-side path of a test script: `Tests-Directory:` replaces
/// the `debian/tests` path segment.
fn test_script_path(run: &RunContext<'_>, test: &Test, script: &str) -> String {
let base = test.tests_directory.as_deref().unwrap_or("debian/tests");
format!("{}/{base}/{script}", run.tree)
}
/// Retrieve a test's artifacts directory from the testbed into the run's
/// host directory (best-effort: tests may not have written anything).
fn retrieve_artifacts(run: &RunContext<'_>, test_name: &str) {
let remote = PathBuf::from(format!(
"{}/artifacts/{}",
run.staging_root,
safe_name(test_name)
));
if !run.ctx.exists(&remote).unwrap_or(false) {
return;
}
let host = host_artifacts_dir(run, test_name);
let _ = fs::create_dir_all(&host);
let _ = run.ctx.retrieve_path(&remote, &host);
}
/// The host-side artifacts directory of one test.
fn host_artifacts_dir(run: &RunContext<'_>, test_name: &str) -> PathBuf {
run.artifacts_dir
.join("artifacts")
.join(safe_name(test_name))
}
/// Filesystem-safe directory name for a test name (script paths may
/// contain slashes).
fn safe_name(name: &str) -> String {
name.replace('/', "_")
}
/// Per-test capture: appends stdout/stderr lines to files in the run's
/// artifacts directory, counts stderr output (the DEP-8 failure rule)
/// and forwards every line to the view's tee sink.
struct TestSink {
stdout: Mutex<fs::File>,
stderr: Mutex<fs::File>,
stderr_lines: AtomicUsize,
forward: Option<Arc<dyn LineSink>>,
}
impl TestSink {
/// Build the sink; the log files live under the test's artifacts
/// directory on the host.
fn new(host_dir: PathBuf, forward: Option<Arc<dyn LineSink>>) -> TestSink {
let _ = fs::create_dir_all(&host_dir);
let open = |name: &str| {
fs::OpenOptions::new()
.create(true)
.append(true)
.open(host_dir.join(name))
.unwrap_or_else(|_| {
// No artifacts directory: capture into a null file.
fs::File::create("/dev/null").expect("/dev/null is always openable")
})
};
TestSink {
stdout: Mutex::new(open("stdout.log")),
stderr: Mutex::new(open("stderr.log")),
stderr_lines: AtomicUsize::new(0),
forward,
}
}
/// Number of stderr lines seen.
fn stderr_lines(&self) -> usize {
self.stderr_lines.load(Ordering::Relaxed)
}
}
impl LineSink for TestSink {
fn line(&self, stream: Stream, line: &str) {
let mut file = match stream {
Stream::Stdout => self
.stdout
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner),
Stream::Stderr => {
self.stderr_lines.fetch_add(1, Ordering::Relaxed);
self.stderr
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
}
};
let _ = writeln!(file, "{line}");
if let Some(forward) = &self.forward {
forward.line(stream, line);
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn test(restrictions: &[&str]) -> Test {
Test {
name: "smoke".to_string(),
script: Some("smoke".to_string()),
command: None,
depends: None,
restrictions: restrictions.iter().map(|s| s.to_string()).collect(),
tests_directory: None,
}
}
fn raw(code: Option<i32>, stderr: bool) -> RawRun {
RawRun {
code,
stderr_nonempty: stderr,
timed_out: false,
}
}
/// The verdict matrix: exit/stderr/skip/timeout rules.
#[test]
fn verdict_matrix() {
let plain = test(&[]);
assert_eq!(
decide(&plain, &raw(Some(0), false), 10),
(Verdict::Pass, None)
);
assert_eq!(
decide(&plain, &raw(Some(0), true), 10),
(Verdict::Fail, Some("the test wrote to stderr".to_string()))
);
assert_eq!(
decide(&plain, &raw(Some(1), false), 10),
(Verdict::Fail, Some("exit code 1".to_string()))
);
assert_eq!(
decide(&plain, &raw(None, false), 10),
(Verdict::Fail, Some("killed by a signal".to_string()))
);
assert_eq!(
decide(&plain, &raw(Some(124), false), 42),
(Verdict::Fail, Some("timed out after 42s".to_string()))
);
// exit 77 skips only under `skippable` — and a chatty
// self-skipping test still skips (the verdict comes first).
assert_eq!(decide(&plain, &raw(Some(77), false), 10).0, Verdict::Fail);
let skippable = test(&["skippable"]);
assert_eq!(
decide(&skippable, &raw(Some(77), true), 10).0,
Verdict::Skip
);
// allow-stderr lifts the stderr rule.
let chatty = test(&["allow-stderr"]);
assert_eq!(decide(&chatty, &raw(Some(0), true), 10).0, Verdict::Pass);
}
/// Restriction gates: machine isolation and root requirements skip
/// with an actionable reason.
#[test]
fn restriction_gates() {
let plain = test(&[]);
assert_eq!(restriction_skip(&plain, true, false), None);
assert_eq!(restriction_skip(&plain, false, true), None);
let vm_only = test(&["isolation-machine"]);
assert!(
restriction_skip(&vm_only, true, false)
.unwrap()
.contains("--mode vm")
);
let reboot = test(&["needs-reboot"]);
assert!(restriction_skip(&reboot, true, false).is_some());
let container = test(&["isolation-container"]);
assert!(restriction_skip(&container, true, false).is_some());
let root = test(&["needs-root"]);
assert!(restriction_skip(&root, false, false).is_some());
assert_eq!(restriction_skip(&root, true, false), None);
let sudo = test(&["needs-sudo"]);
assert!(restriction_skip(&sudo, false, false).is_some());
let native_only = test(&["skip-foreign-architecture"]);
assert!(restriction_skip(&native_only, true, true).is_some());
assert_eq!(restriction_skip(&native_only, true, false), None);
}
/// Test names map to filesystem-safe artifact directory names.
#[test]
fn names_are_sanitized() {
assert_eq!(safe_name("checks/cli"), "checks_cli");
assert_eq!(safe_name("smoke"), "smoke");
}
}
src/test/vm.rs
+242
View File
@@ -0,0 +1,242 @@
//! The `--mode vm` escape hatch: exec the installed `autopkgtest` with
//! its qemu runner.
//!
//! VM virtualization is the one backend pkh deliberately does not
//! reimplement: autopkgtest-virt-qemu carries a serial-console control
//! protocol, image overlay management and the reboot protocol, and it is
//! the only runner satisfying `isolation-machine`/`needs-reboot` tests.
//! Everything around it stays pkh's: binary resolution, the report
//! model, the exit-code contract and the artifacts directory.
use std::path::{Path, PathBuf};
use std::process::{Command, Stdio};
use crate::debian::tests_control::Test;
use crate::interrupt;
use crate::report::BuildView;
use crate::test::outcome::{TestOutcome, Verdict};
/// Run the tests through autopkgtest's qemu runner.
///
/// `tree` is the source tree, `binaries` the fresh binary packages under
/// test (may be empty: autopkgtest then builds inside the VM),
/// `run_dir` the artifacts directory (passed as `-o`; autopkgtest
/// requires it to not exist yet) and `image` the VM image path. Returns
/// the per-test outcomes and autopkgtest's raw exit code; the caller
/// builds the full report around them.
pub(crate) fn run_vm(
tree: &Path,
binaries: &[PathBuf],
tests: &[Test],
run_dir: &Path,
image: Option<&Path>,
opts: &crate::test::options::TestOptions<'_>,
view: &dyn BuildView,
) -> Result<(Vec<TestOutcome>, i32), Box<dyn std::error::Error>> {
let Some(image) = image else {
return Err(
"--mode vm needs a VM image: pass --image, or build one with \
'autopkgtest-build-qemu <series> <image>' (Debian) or \
'autopkgtest-buildvm-ubuntu-cloud -r <series> -a <arch>' (Ubuntu)"
.into(),
);
};
if !image.exists() {
return Err(format!(
"VM image '{}' does not exist: pass --image, or build one with \
'autopkgtest-build-qemu <series> <image>' (Debian) or \
'autopkgtest-buildvm-ubuntu-cloud -r <series> -a <arch>' (Ubuntu)",
image.display()
)
.into());
}
if !opts.ppa.is_empty() || opts.pocket.is_some() || !opts.inject.is_empty() {
return Err(
"--pocket, --ppa and --inject are not supported with --mode vm yet: \
configure them inside the image or use a chroot/local run"
.into(),
);
}
view.message("Running autopkgtest with the qemu runner");
let mut command = Command::new("autopkgtest");
command.arg("-o").arg(run_dir);
let summary_path = run_dir.join("autopkgtest-summary");
command.arg("--summary").arg(&summary_path);
if opts.apt_upgrade {
command.arg("--apt-upgrade");
}
for name in &opts.test_names {
command.arg("--test-name").arg(name);
}
for name in &opts.skip_tests {
command.arg("--skip-test").arg(name);
}
for setup in &opts.setup_commands {
command.arg("--setup-commands").arg(setup);
}
if opts.shell {
command.arg("--shell");
}
if opts.shell_fail {
command.arg("--shell-fail");
}
if let Some(secs) = opts.timeout {
command.arg("--timeout").arg(secs.to_string());
}
if let Some(factor) = opts.timeout_factor {
command.arg("--timeout-factor").arg(factor.to_string());
}
for deb in binaries {
// Absolute paths: autopkgtest reads a bare name as an archive
// source package.
command.arg(deb.canonicalize()?);
}
command.arg(tree.canonicalize()?);
command.arg("--");
command.arg("qemu");
if let Some(cpus) = opts.cpus {
command.arg("--cpus").arg(cpus.to_string());
}
if let Some(ram) = &opts.ram_size {
command.arg("--ram-size").arg(ram);
}
command.arg(image);
// The autopkgtest process must die with us: run it in its own
// process group and SIGTERM the group from the interrupt-time
// cleanup hook (autopkgtest cleans up its testbed on TERM).
let mut child = command
.stdin(Stdio::inherit())
.stdout(Stdio::inherit())
.stderr(Stdio::inherit())
.spawn()
.map_err(|e| format!("cannot run autopkgtest (is it installed?): {e}"))?;
let pid = child.id();
// Best-effort: without its own group, the cleanup kill below would
// only reach the autopkgtest front-end, not its virt server.
unsafe { libc::setpgid(pid as libc::pid_t, pid as libc::pid_t) };
let hook = interrupt::register_cleanup_hook(Box::new(move || {
// SAFETY: kill(2) to the spawned autopkgtest process group
unsafe { libc::kill(-(pid as libc::pid_t), libc::SIGTERM) };
}));
let status = child.wait()?;
drop(hook);
let raw_code = status.code().unwrap_or(16);
// autopkgtest's own failure classes are runtime errors here; the
// pass/fail/skip classes become the report.
if matches!(raw_code, 12 | 14 | 16 | 20) {
return Err(format!(
"autopkgtest failed (exit {raw_code}): check the artifacts in '{}'",
run_dir.display()
)
.into());
}
let outcomes = parse_summary(&summary_path, tests)?;
Ok((outcomes, raw_code))
}
/// Parse autopkgtest's `--summary` file: one line per test, `<testname>
/// PASS|FAIL|SKIP|FLAKY [reason...]`. Tolerant by design — the format is
/// stable but not formally versioned; unparseable lines are skipped, and
/// a missing file yields no outcomes (the raw exit code carries the
/// verdict).
pub(crate) fn parse_summary(
summary_path: &Path,
tests: &[Test],
) -> Result<Vec<TestOutcome>, Box<dyn std::error::Error>> {
let mut outcomes = Vec::new();
let Ok(content) = std::fs::read_to_string(summary_path) else {
return Ok(outcomes);
};
for line in content.lines() {
let mut tokens = line.splitn(3, ' ');
let Some(name) = tokens.next() else {
continue;
};
let Some(verdict) = tokens.next() else {
continue;
};
// Summary names carry the argid prefix ("<argid>-<testname>"); a
// declared test matching by suffix wins, else the token stands.
let name = name.to_string();
let name = match tests
.iter()
.find(|test| name.ends_with(&test.name) || test.name == name)
{
Some(test) => test.name.clone(),
None => name,
};
let reason = tokens.next().map(str::to_string);
let verdict = match verdict.trim() {
"PASS" => Verdict::Pass,
"FAIL" => Verdict::Fail,
"SKIP" => Verdict::Skip,
"FLAKY" => Verdict::Flaky,
_ => continue,
};
outcomes.push(TestOutcome {
name,
verdict,
reason,
duration: std::time::Duration::ZERO,
});
}
Ok(outcomes)
}
#[cfg(test)]
mod tests {
use super::*;
use crate::debian::tests_control::Test;
fn test(name: &str) -> Test {
Test {
name: name.to_string(),
script: Some(name.to_string()),
command: None,
depends: None,
restrictions: Vec::new(),
tests_directory: None,
}
}
/// The summary parser maps the four verdicts, keeps reasons and
/// resolves argid-prefixed names back to the declared tests.
#[test]
fn summary_parsing() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("summary");
std::fs::write(
&path,
"hello-test-demo-smoke PASS\n\
hello-test-demo-command1 FAIL exit code 1\n\
container SKIP required isolation not available\n\
flaky-one FLAKY passed on retry\n",
)
.unwrap();
let tests = [test("smoke"), test("command1")];
let outcomes = parse_summary(&path, &tests).unwrap();
assert_eq!(outcomes.len(), 4);
assert_eq!(outcomes[0].name, "smoke");
assert_eq!(outcomes[0].verdict, Verdict::Pass);
assert_eq!(outcomes[1].verdict, Verdict::Fail);
assert_eq!(outcomes[1].reason.as_deref(), Some("exit code 1"));
assert_eq!(outcomes[2].name, "container");
assert_eq!(outcomes[2].verdict, Verdict::Skip);
assert_eq!(outcomes[3].verdict, Verdict::Flaky);
}
/// A missing summary file parses to no outcomes: the raw exit code
/// carries the verdict.
#[test]
fn missing_summary_yields_no_outcomes() {
let dir = tempfile::tempdir().unwrap();
let outcomes = parse_summary(&dir.path().join("absent"), &[]).unwrap();
assert!(outcomes.is_empty());
}
}
src/ui.rs
+122 -23
View File
@@ -7,6 +7,7 @@ pub mod deb;
/// yes/no confirmation
pub mod prompt;
use crossterm::terminal;
#[cfg(test)]
use indicatif::ProgressDrawTarget;
use indicatif::{MultiProgress, ProgressBar, ProgressStyle};
@@ -40,34 +41,132 @@ pub(crate) fn transfer_style() -> ProgressStyle {
.progress_chars("=> ")
}
/// Create a spinner-style progress bar attached to `multi`, returning the bar
/// and a callback compatible with [`crate::ProgressCallback`]
pub fn create_progress_bar(
multi: &MultiProgress,
) -> (ProgressBar, impl Fn(&str, &str, usize, usize) + '_) {
let pb = multi.add(ProgressBar::new(0));
pb.enable_steady_tick(Duration::from_millis(50));
pb.set_style(spinner_style());
/// [`transfer_style`] with human-formatted byte counters, for the events
/// reported through [`crate::report::BuildView::byte_progress`]
pub(crate) fn byte_style() -> ProgressStyle {
ProgressStyle::default_bar()
.template(
"> {spinner:.blue} {prefix}\n {msg} [{bar:40.cyan/blue}] {bytes}/{total_bytes} ({eta})",
)
.unwrap()
.progress_chars("=> ")
}
let pb_clone = pb.clone();
let callback = move |prefix: &str, msg: &str, progress: usize, total: usize| {
let pb = &pb_clone;
if progress != 0 && total != 0 {
pb.set_style(transfer_style());
} else {
pb.set_style(spinner_style());
}
/// A standalone spinner for one-line waits outside a build view (the
/// prune size pass): braille frames over the message behind the `> `
/// prompt prefix, drawn only on terminals. Callers finish it with
/// [`ProgressBar::finish_and_clear`].
pub fn wait_spinner(msg: &str) -> ProgressBar {
let pb = ProgressBar::new_spinner();
pb.set_style(
ProgressStyle::with_template("> {spinner:.blue} {msg}").expect("spinner template"),
);
pb.set_message(msg.to_string());
pb.enable_steady_tick(Duration::from_millis(120));
pb
}
if !prefix.is_empty() {
pb.set_prefix(prefix.to_string());
}
/// Width of the terminal in columns, or `None` when it cannot be determined
pub(crate) fn terminal_width() -> Option<usize> {
terminal::size().ok().map(|(cols, _)| cols as usize)
}
pb.set_message(msg.to_string());
pb.set_length(total as u64);
pb.set_position(progress as u64);
/// Removal progress bar of `pkh prune`, in the deb live view's layout:
/// the running count behind the spinner on the first line, the byte bar
/// with its counters on the second when the plan carries sizes (`--force`
/// skips the sizing and gets the count line only). The removed path is
/// deliberately not rendered — it fights the bar on a narrow terminal,
/// and the failed-removal errors name the paths that matter. Drive it
/// with [`removal_progress`]; finish it with
/// [`ProgressBar::finish_and_clear`].
pub fn removal_bar(multi: &MultiProgress, total_bytes: u64) -> ProgressBar {
let pb = multi.add(ProgressBar::new(total_bytes));
let template = if total_bytes > 0 {
"> {spinner:.blue} {prefix}\n [{bar:24.cyan/blue}] {bytes}/{total_bytes}"
} else {
"> {spinner:.blue} {prefix}"
};
pb.set_style(
ProgressStyle::with_template(template)
.expect("removal template")
.progress_chars("=> "),
);
pb.enable_steady_tick(Duration::from_millis(120));
pb
}
(pb, callback)
/// Advance the removal bar to the `idx`-th of `count` artifact;
/// `done` bytes already freed
pub fn removal_progress(bar: &ProgressBar, idx: usize, count: usize, done: u64) {
bar.set_prefix(format!("Removing {}/{}", idx + 1, count));
if let Some(total) = bar.length()
&& total > 0
{
bar.set_position(done.min(total));
}
}
/// Terminal build view for the pull flow: one bar line attached to the
/// shared [`MultiProgress`], spinner-style while the work is unsized,
/// transfer-style once a byte or item total is known. The bar is created
/// eagerly (an empty spinner, so the line exists from the first moment)
/// and released on drop.
pub struct PullView {
multi: MultiProgress,
pb: ProgressBar,
}
impl PullView {
/// Attach a pull view to `multi`
pub fn new(multi: &MultiProgress) -> Self {
let pb = multi.add(ProgressBar::new(0));
pb.enable_steady_tick(Duration::from_millis(50));
pb.set_style(spinner_style());
Self {
multi: multi.clone(),
pb,
}
}
}
impl Drop for PullView {
fn drop(&mut self) {
self.pb.finish_and_clear();
self.multi.remove(&self.pb);
}
}
impl crate::report::BuildView for PullView {
fn message(&self, text: &str) {
self.pb.set_style(spinner_style());
self.pb.set_message(text.to_string());
}
fn progress(&self, label: &str, pos: usize, total: usize) {
if pos != 0 && total != 0 {
self.pb.set_style(transfer_style());
} else {
self.pb.set_style(spinner_style());
}
self.pb.set_message(label.to_string());
self.pb.set_length(total as u64);
self.pb.set_position(pos as u64);
}
fn byte_progress(&self, label: &str, pos: u64, total: u64) {
if pos != 0 && total != 0 {
self.pb.set_style(byte_style());
} else {
self.pb.set_style(spinner_style());
}
self.pb.set_message(label.to_string());
self.pb.set_length(total);
self.pb.set_position(pos);
}
fn is_enabled(&self) -> bool {
true
}
}
/// Label of the interactive series selector, rendered by [`prompt::select`]
src/ui/deb.rs
+52 -10
View File
@@ -46,6 +46,9 @@ struct Pipeline {
errors: Vec<String>,
last_draw: Instant,
bar_total: u64,
/// Whether the bar's counters are bytes ([`BuildView::byte_progress`])
/// rather than plain counts
bar_bytes: bool,
}
/// State shared between [`DebUi`] and its sinks
@@ -119,6 +122,7 @@ impl DebUi {
errors: Vec::new(),
last_draw: Instant::now(),
bar_total: 0,
bar_bytes: false,
}),
tee: Mutex::new(None),
log_path: Mutex::new(log_path),
@@ -195,6 +199,7 @@ impl DebUi {
st.classifier = classifier;
st.lines.clear();
st.bar_total = 0;
st.bar_bytes = false;
st.last_draw = Instant::now();
}
if self.active() {
@@ -272,7 +277,11 @@ impl crate::report::BuildView for DebUi {
self.shared.top.set_prefix(target.display.clone());
}
if target.tee_log {
let kind = if target.source_only { "build" } else { "deb" };
let kind = match target.kind {
crate::report::TargetKind::Source => "build",
crate::report::TargetKind::Binary => "deb",
crate::report::TargetKind::Test => "test",
};
self.open_log(
kind,
target.package,
@@ -305,6 +314,19 @@ impl crate::report::BuildView for DebUi {
self.shared.top.set_message(label.to_string());
}
fn byte_progress(&self, label: &str, pos: u64, total: u64) {
if !self.active() || total == 0 {
return;
}
apply_byte_progress(
&self.shared.top,
&mut self.shared.state.lock().unwrap(),
pos,
total,
);
self.shared.top.set_message(label.to_string());
}
fn sink(&self) -> Option<Arc<dyn LineSink>> {
Some(Arc::new(Sink {
shared: self.shared.clone(),
@@ -380,7 +402,8 @@ fn apply_progress(top: &ProgressBar, st: &mut Pipeline, pos: u64, total: u64) {
if total == 0 {
return;
}
if st.bar_total != total {
if st.bar_bytes || st.bar_total != total {
st.bar_bytes = false;
st.bar_total = total;
top.set_style(determinate_style());
top.set_length(total);
@@ -388,6 +411,21 @@ fn apply_progress(top: &ProgressBar, st: &mut Pipeline, pos: u64, total: u64) {
top.set_position(pos.min(total));
}
/// Update the determinate bar with byte counters (human-formatted by the
/// template), restyling it when needed
fn apply_byte_progress(top: &ProgressBar, st: &mut Pipeline, pos: u64, total: u64) {
if total == 0 {
return;
}
if !st.bar_bytes || st.bar_total != total {
st.bar_bytes = true;
st.bar_total = total;
top.set_style(byte_style());
top.set_length(total);
}
top.set_position(pos.min(total));
}
/// Push a line into the rolling pane and redraw (throttled)
///
/// When the pane is full, the oldest non-error line is dropped first so
@@ -450,7 +488,7 @@ fn drop_pane(shared: &Shared) {
/// Render the pane content with per-kind colors, ellipsizing lines that are
/// wider than the terminal so they do not overflow onto a wrapped line
fn render_pane(lines: &VecDeque<(Kind, String)>) -> String {
let max_width = terminal_width().map(|w| w.saturating_sub(PANE_PREFIX_WIDTH));
let max_width = crate::ui::terminal_width().map(|w| w.saturating_sub(PANE_PREFIX_WIDTH));
render_pane_with_width(lines, max_width)
}
@@ -477,13 +515,6 @@ fn render_pane_with_width(lines: &VecDeque<(Kind, String)>, max_width: Option<us
/// Display width of the ` │ ` prefix rendered before each pane line
const PANE_PREFIX_WIDTH: usize = 4;
/// Width of the terminal in columns, or `None` when it cannot be determined
fn terminal_width() -> Option<usize> {
crossterm::terminal::size()
.ok()
.map(|(cols, _)| cols as usize)
}
/// Ellipsize `text` to at most `max_width` display columns, keeping its head
/// and appending `…` when it does not fit
fn ellipsize(text: &str, max_width: usize) -> String {
@@ -526,6 +557,15 @@ fn determinate_style() -> ProgressStyle {
.progress_chars("=> ")
}
/// Status bar style with a determinate progress bar over a byte
/// transfer: the counters render as byte units (`{bytes}`/`{total_bytes}`)
fn byte_style() -> ProgressStyle {
ProgressStyle::default_bar()
.template("> {spinner:.blue} {prefix}\n {msg} [{bar:24.cyan/blue}] {bytes}/{total_bytes}")
.expect("valid template")
.progress_chars("=> ")
}
/// Whether stdout is a terminal
fn is_stdout_tty() -> bool {
// SAFETY: isatty only inspects the file descriptor
@@ -718,6 +758,7 @@ mod tests {
errors: Vec::new(),
last_draw: Instant::now(),
bar_total: 0,
bar_bytes: false,
}),
tee: Mutex::new(None),
log_path: Mutex::new(std::env::temp_dir().join("pkh-pane-test.log")),
@@ -760,6 +801,7 @@ mod tests {
errors: Vec::new(),
last_draw: Instant::now(),
bar_total: 0,
bar_bytes: false,
}),
tee: Mutex::new(None),
log_path: Mutex::new(std::env::temp_dir().join("pkh-interrupt-report-test.log")),
src/utils/mod.rs
+2
View File
@@ -1,2 +1,4 @@
pub mod gpg;
pub mod root;
/// Directory-tree sizing (concurrent walker) and byte formatting
pub mod size;
src/utils/size.rs
+162
View File
@@ -0,0 +1,162 @@
//! Directory-tree sizing: the concurrent walker shared by the prune
//! report and the staged-tree copy progress, plus the byte formatting
//! both display.
use std::fs;
use std::path::{Path, PathBuf};
/// Total size of every tree in `paths`, in bytes, measured concurrently:
/// the sized trees hold hundreds of thousands of files (a single kernel
/// build tree is ~100 GB of objects), and a serial walk costs tens of
/// seconds.
///
/// Directory walks never cross a mount point (`du -x`): residual chroots
/// hold bind-mounted `/proc` and overlay filesystems whose virtual files
/// (`/proc/kcore` alone reports 128 TiB) would poison the totals.
/// Unreadable entries count as zero — prune removes them all the same.
pub(crate) fn tree_sizes(paths: &[&Path]) -> Vec<u64> {
use std::collections::VecDeque;
use std::os::unix::fs::MetadataExt;
use std::sync::Mutex;
use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
// The device each tree lives on: the walk descends only into
// directories on the same device.
let root_devs: Vec<u64> = paths
.iter()
.map(|path| fs::symlink_metadata(path).map_or(0, |meta| meta.dev()))
.collect();
let totals: Vec<AtomicU64> = (0..paths.len()).map(|_| AtomicU64::new(0)).collect();
let queue: Mutex<VecDeque<(usize, PathBuf)>> = Mutex::new(
paths
.iter()
.enumerate()
.map(|(idx, path)| (idx, path.to_path_buf()))
.collect(),
);
// Every queued-but-unfinished item: the workers stop when the queue
// is empty and nothing is in flight.
let pending = AtomicUsize::new(paths.len());
let workers = std::thread::available_parallelism()
.map(|n| n.get())
.unwrap_or(4)
.min(16);
std::thread::scope(|scope| {
for _ in 0..workers {
scope.spawn(|| {
loop {
let item = queue.lock().unwrap().pop_front();
let Some((idx, path)) = item else {
if pending.load(Ordering::Acquire) == 0 {
return;
}
std::thread::yield_now();
continue;
};
let Ok(meta) = fs::symlink_metadata(&path) else {
pending.fetch_sub(1, Ordering::AcqRel);
continue;
};
if !meta.is_dir() {
totals[idx].fetch_add(meta.len(), Ordering::Relaxed);
pending.fetch_sub(1, Ordering::AcqRel);
continue;
}
let Ok(entries) = fs::read_dir(&path) else {
pending.fetch_sub(1, Ordering::AcqRel);
continue;
};
for entry in entries.flatten() {
let child = entry.path();
let Ok(meta) = fs::symlink_metadata(&child) else {
continue;
};
let same_device = meta.dev() == root_devs[idx];
if meta.is_dir() && same_device {
pending.fetch_add(1, Ordering::AcqRel);
queue.lock().unwrap().push_back((idx, child));
} else if !meta.is_dir() && same_device {
totals[idx].fetch_add(meta.len(), Ordering::Relaxed);
}
}
pending.fetch_sub(1, Ordering::AcqRel);
}
});
}
});
totals.into_iter().map(|total| total.into_inner()).collect()
}
/// Human-readable size ("512 B", "3.5 MiB", "1.2 GiB", "1.3 TiB").
pub(crate) fn format_bytes(bytes: u64) -> String {
const KIB: f64 = 1024.0;
const MIB: f64 = 1024.0 * KIB;
const GIB: f64 = 1024.0 * MIB;
const TIB: f64 = 1024.0 * GIB;
let size = bytes as f64;
if size >= TIB {
format!("{:.1} TiB", size / TIB)
} else if size >= GIB {
format!("{:.1} GiB", size / GIB)
} else if size >= MIB {
format!("{:.1} MiB", size / MIB)
} else if size >= KIB {
format!("{:.1} KiB", size / KIB)
} else {
format!("{bytes} B")
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::fs;
use tempfile::tempdir;
#[test]
fn sizes_files_directories_and_missing_roots() {
let temp = tempdir().unwrap();
let tree = temp.path().join("tree");
fs::create_dir_all(tree.join("sub")).unwrap();
fs::write(tree.join("sub/obj.o"), vec![0u8; 3000]).unwrap();
fs::write(tree.join("top"), vec![0u8; 10]).unwrap();
let file = temp.path().join("file");
fs::write(&file, vec![0u8; 5]).unwrap();
let missing = temp.path().join("missing");
let sizes = tree_sizes(&[tree.as_path(), file.as_path(), missing.as_path()]);
assert_eq!(sizes, vec![3010, 5, 0]);
}
#[test]
fn symlinks_count_as_themselves_never_their_target() {
let temp = tempdir().unwrap();
let target = temp.path().join("target");
fs::write(&target, vec![0u8; 100]).unwrap();
let link = temp.path().join("link");
std::os::unix::fs::symlink(&target, &link).unwrap();
// symlink_metadata: the link's own size (the stored target path),
// never the target's content.
assert_eq!(
tree_sizes(&[link.as_path()]),
vec![target.as_os_str().len() as u64]
);
}
#[test]
fn formats_bytes_like_the_reports_display_them() {
assert_eq!(format_bytes(0), "0 B");
assert_eq!(format_bytes(512), "512 B");
assert_eq!(format_bytes(2048), "2.0 KiB");
assert_eq!(format_bytes(5 * 1024 * 1024), "5.0 MiB");
assert_eq!(format_bytes(3 * 1024 * 1024 * 1024), "3.0 GiB");
// /proc/kcore inside a bind-mounted chroot /proc reports exactly
// this size; the mount-crossing walk keeps it out of the totals.
assert_eq!(format_bytes(140_737_477_959_680), "128.0 TiB");
}
}
tests/e2e/.gitignore
+2
View File
@@ -0,0 +1,2 @@
# Harness run output: trees, logs, results
runs/
tests/e2e/README.md
+103
View File
@@ -0,0 +1,103 @@
# pkh end-to-end package harness
Exercises pkh the way it is used in anger: real packages, the whole
workflow. For every package in `catalog.yml` it runs
```
pkh pull <name> --archive # sources from the Debian/Ubuntu archive
pkh chlog --rebuild # no-change rebuild entry (1.0-1 -> 1.0-1build1)
pkh build # .dsc + _source.changes next to the tree
pkh deb # binary packages in an ephemeral chroot
```
The catalog spans ~60 source packages: smoke-test smalls (`hello`,
`micro`, `jq`...), core utilities, server software, databases, desktop
stacks, toolchains and the weird corners — `dracut`, `opensbi`, `grub2`,
a cross `u-boot`, the kbuild-based `kbuild`, and (opt-in, hours each)
`gcc-14`, `glibc`, `rustc`, `libreoffice`, `firefox`, the Ubuntu `linux`
kernel...
## Prerequisites
- A pkh binary: the harness prefers the repo build
(`target/release/pkh`, then `target/debug/pkh`), falls back to `pkh`
on PATH; override with `--pkh PATH` or build first with `--build-pkh`.
- The pkh runtime toolchain: `mmdebstrap`, `sudo` (passwordless, for
the unshare chroot builds), `dpkg-dev`, `distro-info` data, and
network access to the Debian and Ubuntu archives.
- Python 3 only (stdlib; PyYAML is used when installed, with a built-in
fallback parser for the catalog's YAML subset).
- Disk: tens of GB under `runs/` for trees and artifacts, plus pkh's own
caches (`/var/tmp/pkh`); expect a full default-tier run to take many
hours — the tiers exist so you never have to.
## Usage
```
./run.py --list # what would run, with tiers and notes
./run.py --dry-run # the exact commands, nothing executed
./run.py # smoke + standard tiers
./run.py --tier long # the multi-hour builds
./run.py --tier all # everything
./run.py --only hello,dracut # exact names
./run.py --category database # or server, desktop, boot-firmware...
./run.py --tag cross --arch arm64 # tag/arch sweeps
./run.py --stages pull # catalog sanity: can we pull everything?
./run.py --resume runs/<id>/results.json # continue an interrupted run
```
Useful global pass-throughs: `--dist`, `--series`, `--arch`, `--cross`,
`--pocket`, `--ppa user/ppa` (deb dependencies), `--jobs N`,
`--no-ccache`, `--git` (pull through the VCS instead of the archive).
Exit codes: `0` nothing unexpected failed, `1` at least one package
failed (or xpassed under `--strict`), `2` usage/catalog error, `130`
interrupted.
## Selection: tiers, categories, tags
Entries carry a **tier**: `smoke` (minutes), `standard` (the default,
up to ~1-2h each), `long` (hours: compilers, browsers, the kernel —
only run when asked). Default selection is smoke + standard. On top of
that, filters compose: `--category`, `--tag`, `--only`, `--skip`.
An entry can be marked `xfail: <reason>` when it is known-broken; it
then reports `xfail` on failure and `xpass` on unexpected success
(xpass fails the run only under `--strict`).
## What a run produces
```
runs/<id>/
logs/<pkg>.log # full output of every stage of that package
<pkg>/<pkg>/ # the pulled source tree
<pkg>/*.dsc ... # build artifacts, next to the tree as pkh puts them
results.json # machine-readable, written progressively
results.md # the summary table
```
The run directory is per invocation; `--resume <id>/results.json`
reuses it, keeps the recorded outcomes and only reruns packages that
did not finish. Already-pulled trees are kept (`pkh pull` refuses
existing directories anyway — delete `runs/<id>/<pkg>/` to force a
re-pull), and stages that already succeeded are reused rather than
re-run, so a resumed `chlog` does not bump the version twice; change
`--dist`/`--series`/`--arch`/`--cross` and everything re-runs.
Pulls run a few at a time (`--pull-jobs`), the build pipeline is
sequential by design: one heavy build at a time keeps pkh's shared
chroot tarball, apt and ccache caches warm, so repeat runs get
dramatically faster.
Each stage gets the package's timeout budget (tier default, overridable
per entry with `timeout:`); on timeout the whole build process group is
killed and the package is marked `timeout`.
## Adding packages
Edit `catalog.yml`. Fields are documented at the top of the file;
keep **source** names as they exist in the archive — not binary names
(`make-dfsg` and `rust-ripgrep`, not `make` and `ripgrep`;
`xz-utils`, not `xz`; `postgresql-17`, not `postgresql`). The file
stays within a small YAML subset so the stdlib fallback parser can read
it: flat mappings, inline lists, no colons in values.
tests/e2e/__pycache__/run.cpython-314.pyc
BIN
View File
Binary file not shown.
tests/e2e/catalog.yml
+380
View File
@@ -0,0 +1,380 @@
# pkh end-to-end test catalog
#
# One entry per source package, exercised by run.py as:
# pkh pull -> pkh chlog --rebuild -> pkh build -> pkh deb
#
# The catalog deliberately alternates between debian and ubuntu within
# every category, so a default run exercises both archives, series
# namings and chroots.
#
# The file is plain YAML but is also parsed by a small built-in parser when
# PyYAML is unavailable: keep to the subset it supports — full-line comments,
# 'key: value' scalars, inline [a, b] lists, no nested mappings, no block
# scalars, no colons inside values.
#
# Fields (all optional except 'name'):
# category free grouping label (smoke, server, desktop, ...) [defaults.tier]
# tier smoke | standard | long — 'long' is only run when asked for
# dist debian | ubuntu [defaults.dist]
# series pin a series (default: the dist development series)
# arch pass '-a ARCH' to 'pkh deb'
# cross true -> also pass '--cross' (needs 'arch') [false]
# pocket pass '-p POCKET' to pull/deb
# ppa inline list of PPAs passed to 'pkh deb --ppa'
# timeout per-stage budget in minutes [tier default]
# xfail known-broken marker: the reason it is expected to fail
# notes free text, shown by --list and in reports
defaults:
dist: debian
tier: standard
packages:
# --- smoke: small, fast, reliable; the first line of defense -------------
- name: hello
category: smoke
tier: smoke
tags: [gnu, autotools, tiny]
notes: the canonical packaging smoke test
- name: micro
category: smoke
tier: smoke
tags: [go, editor, tiny]
notes: Go build, golang build-dependencies resolved offline
- name: jq
category: smoke
tier: smoke
tags: [c, autotools, tiny]
- name: htop
category: smoke
dist: ubuntu
tier: smoke
tags: [c, autotools, tiny]
- name: tmux
category: smoke
tier: smoke
tags: [c, autotools, tiny]
- name: tree
category: smoke
dist: ubuntu
tier: smoke
tags: [c, make, tiny]
- name: figlet
category: smoke
dist: ubuntu
tier: smoke
tags: [c, make, tiny]
- name: dash
category: smoke
tier: smoke
tags: [c, autotools, shell]
- name: file
category: smoke
tier: smoke
tags: [c, autotools]
- name: zlib
category: smoke
dist: ubuntu
tier: smoke
tags: [c, autotools, lib]
- name: xz-utils
category: smoke
tier: smoke
tags: [c, autotools]
xfail: sid carries the 5.8.1-1+deb13u1 trixie upload whose packaging fails at dh_installdocs (doxygen api docs missing) - package-side, under triage
- name: busybox
category: smoke
dist: ubuntu
tier: smoke
tags: [c, kconfig, static]
- name: dnsmasq
category: smoke
tier: smoke
tags: [c, make, network]
- name: memcached
category: smoke
dist: ubuntu
tier: smoke
tags: [c, autotools, server]
- name: kbuild
category: smoke
tier: smoke
tags: [c, kbuild, weird]
notes: the virtualbox build system; exercises the kbuild log classifier
# --- core -----------------------------------------------------------------
- name: bash
category: core
tags: [c, autotools, shell]
- name: coreutils
category: core
dist: ubuntu
tags: [c, autotools]
- name: util-linux
category: core
tags: [c, autotools]
- name: openssl
category: core
dist: ubuntu
tags: [c, perl-build, lib]
- name: vim
category: core
tags: [c, autotools, editor]
- name: neovim
category: core
dist: ubuntu
tags: [c, cmake, lua, editor]
- name: git
category: core
tags: [c, make, vcs]
- name: rust-ripgrep
category: core
dist: ubuntu
tags: [rust, cargo]
notes: debcargo-style vendored Rust build (binary name is ripgrep)
- name: make-dfsg
category: core
tier: smoke
tags: [c, autotools, tiny]
notes: the GNU make source package (binary name is make)
- name: systemd
category: core
dist: ubuntu
tags: [c, meson, init]
xfail: FTBFS against python3-lxml 6.1 - it no longer consults custom resolvers for parameter entities, so make-directive-index fails with 'Entity entities not defined' on the man pages - distro toolchain change, package-side
# --- server ---------------------------------------------------------------
- name: nginx
category: server
tags: [c, autoconf, server]
- name: apache2
category: server
dist: ubuntu
tags: [c, autotools, server]
- name: openssh
category: server
tags: [c, autotools, server]
- name: postfix
category: server
dist: ubuntu
tags: [c, make, mail, server]
- name: dovecot
category: server
tags: [c, autotools, mail, server]
- name: samba
category: server
dist: ubuntu
tags: [c, python-waf, server]
- name: bind9
category: server
tags: [c, autotools, dns, server]
- name: haproxy
category: server
dist: ubuntu
tags: [c, make, network, server]
- name: exim4
category: server
tags: [c, make, mail, server]
# --- databases --------------------------------------------------------------
- name: redis
category: database
tags: [c, make, server]
- name: sqlite3
category: database
dist: ubuntu
tags: [c, autotools]
- name: postgresql-17
category: database
tags: [c, autotools, server]
notes: long-ish but reliable; a good mid-weight benchmark
- name: leveldb
category: database
dist: ubuntu
tier: smoke
tags: [cpp, cmake]
- name: mysql-8.0
category: database
tier: long
dist: ubuntu
timeout: 240
tags: [cpp, cmake, server]
notes: Ubuntu-only source name; huge C++ build
# --- desktop ----------------------------------------------------------------
- name: gtk4
category: desktop
tags: [c, meson, gui]
- name: vlc
category: desktop
dist: ubuntu
tags: [cpp, autotools, gui]
- name: gimp
category: desktop
tags: [c, meson, gui]
- name: inkscape
category: desktop
dist: ubuntu
tags: [cpp, cmake, gui]
- name: mutter
category: desktop
tags: [c, meson, gui]
- name: pipewire
category: desktop
dist: ubuntu
tags: [c, meson, audio]
- name: plasma-desktop
category: desktop
tags: [cpp, cmake, kde, gui]
- name: gnome-control-center
category: desktop
dist: ubuntu
tags: [c, meson, gui]
- name: ffmpeg
category: desktop
tags: [c, autotools, media]
# --- toolchains ---------------------------------------------------------------
- name: binutils
category: toolchain
tags: [c, autotools]
xfail: FTBFS against glibc 2.42+ - string.h turned strstr into a macro with C2x redirect annotations and gprofng's libcollector does not survive it ('expected identifier before _Generic') - upstream fixed it after 2.45, distro toolchain change
- name: gdb
category: toolchain
dist: ubuntu
tags: [c, autotools]
xfail: FTBFS with texinfo 7.3 - texi2dvi aborts with 'pdfetex exited with bad status' building gdb.pdf - the last Launchpad rebuild predates the texinfo upgrade, distro toolchain change
- name: cmake
category: toolchain
tier: smoke
tags: [cpp, cmake, bootstrap]
# --- boot / firmware / weird ------------------------------------------------
- name: dracut
category: boot-firmware
tier: smoke
tags: [shell, make, initramfs, weird]
notes: unusual make logic; initramfs generator
- name: opensbi
category: boot-firmware
tier: smoke
tags: [c, make, firmware, weird]
notes: RISC-V firmware, cross-built inside a native build
- name: grub2
category: boot-firmware
tags: [c, autotools, bootloader, weird]
notes: multi-platform grub; heavy but dependable
xfail: sid carries the 2.06-2 branch whose control pins gcc-10, no longer in sid - the build-deps of the no-change rebuild are unsatisfiable, package-side
- name: u-boot
category: boot-firmware
dist: ubuntu
arch: arm64
cross: true
xfail: in cross builds dpkg-buildflags serves the arm64 target CFLAGS and u-boot compiles its kbuild host tools with them - cc rejects -mbranch-protection=none on amd64 - upstream cross-host-tool breakage, same under sbuild cross
tags: [c, kconfig, firmware, weird, cross]
notes: cross build for arm64; many board configs
# --- long: hours each; opt-in via --tier long or --tier all -----------------
- name: gcc-14
category: toolchain
dist: ubuntu
tier: long
timeout: 360
tags: [c, autotools, compiler]
- name: glibc
category: core
tier: long
timeout: 360
tags: [c, autotools, libc]
- name: rustc
category: toolchain
tier: long
timeout: 360
tags: [rust, compiler]
notes: bootstraps the Rust compiler; the longest single build here
- name: libreoffice
category: desktop
dist: ubuntu
tier: long
timeout: 480
tags: [cpp, make, office, gui]
- name: firefox
category: desktop
tier: long
timeout: 480
tags: [cpp, rust, browser, gui]
- name: qt6-base
category: desktop
dist: ubuntu
tier: long
timeout: 360
tags: [cpp, cmake, gui]
- name: linux
category: kernel
tier: long
dist: ubuntu
timeout: 480
tags: [c, kbuild, kernel, weird]
notes: the Ubuntu kernel; multi-hour, many flavours, kbuild quirks apply
- name: openjdk-21
category: toolchain
tier: long
dist: ubuntu
timeout: 360
tags: [java, make, compiler]
tests/e2e/run.py
Executable
+969
View File
@@ -0,0 +1,969 @@
#!/usr/bin/env python3
"""pkh end-to-end package harness.
Runs real packages from catalog.yml through the full pkh workflow:
pkh pull (from the archive, by default)
pkh chlog --rebuild (no-change rebuild entry; EDITOR is stubbed out)
pkh build (.dsc + _source.changes next to the tree)
pkh deb (binary packages in an ephemeral chroot)
Every stage is logged per package, timed, and checked for its artifacts;
results land in <work>/runs/<id>/results.json and results.md. Pulls run in
parallel (network-bound), the build pipeline runs sequentially (one heavy
build at a time, and pkh's shared chroot/apt/ccache caches stay warm).
Usage:
run.py --list
run.py --dry-run
run.py # smoke + standard tiers
run.py --tier long # the multi-hour builds
run.py --only hello,dracut # exact names
run.py --resume runs/<id>/results.json # continue an interrupted run
See README.md in this directory for the full documentation.
"""
from __future__ import annotations
import argparse
import concurrent.futures
import datetime as dt
import json
import os
import platform
import shlex
import shutil
import signal
import subprocess
import sys
import threading
import time
from dataclasses import dataclass, field
from pathlib import Path
TIERS = ("smoke", "standard", "long")
DEFAULT_TIERS = ("smoke", "standard")
TIER_TIMEOUT_MIN = {"smoke": 30, "standard": 120, "long": 360}
STAGES = ("pull", "chlog", "build", "deb")
PIPELINE_STAGES = ("chlog", "build", "deb")
TERMINAL_OUTCOMES = ("pass", "xpass", "xfail")
REPO_ROOT = Path(__file__).resolve().parents[2]
SCRIPT_DIR = Path(__file__).resolve().parent
class CatalogError(Exception):
"""Invalid catalog content or unusable options."""
@dataclass
class Entry:
"""One catalog entry: a source package and how to build it."""
name: str
category: str = "misc"
tier: str = "standard"
dist: str = "debian"
series: str = ""
arch: str = ""
cross: bool = False
pocket: str = ""
ppa: list[str] = field(default_factory=list)
timeout: float = 0.0 # minutes; 0 = the tier default
xfail: str = ""
notes: str = ""
tags: list[str] = field(default_factory=list)
def timeout_min(self) -> float:
return self.timeout if self.timeout else TIER_TIMEOUT_MIN.get(self.tier, 120)
# ---------------------------------------------------------------------------
# Catalog loading: PyYAML when available, a schema-specific fallback parser
# otherwise (the subset it accepts is documented in catalog.yml).
# ---------------------------------------------------------------------------
def _scalar(text: str):
text = text.strip()
if text.startswith("[") and text.endswith("]"):
inner = text[1:-1].strip()
return [v.strip().strip("'\"") for v in inner.split(",") if v.strip()]
if len(text) >= 2 and text[0] == text[-1] and text[0] in "'\"":
return text[1:-1]
if text in ("true", "false"):
return text == "true"
if text.isdigit():
return int(text)
return text
def _parse_mini_yaml(text: str) -> dict:
"""Parse the flat 'defaults' mapping + flat 'packages' list subset."""
doc: dict = {"defaults": {}, "packages": []}
mode = None # None | "defaults" | "packages"
item: dict | None = None
for lineno, raw in enumerate(text.splitlines(), 1):
if not raw.strip() or raw.lstrip().startswith("#"):
continue
indent = len(raw) - len(raw.lstrip(" "))
stripped = raw.strip()
if indent == 0:
if stripped in ("defaults:", "packages:"):
mode = stripped[:-1]
item = None
continue
raise CatalogError(f"catalog:{lineno}: unexpected top-level '{stripped}'")
if mode is None:
raise CatalogError(f"catalog:{lineno}: content outside a top-level key")
if mode == "defaults":
key, sep, value = stripped.partition(":")
if not sep:
raise CatalogError(f"catalog:{lineno}: expected 'key: value'")
doc["defaults"][key.strip()] = _scalar(value)
else:
if stripped.startswith("- "):
item = {}
doc["packages"].append(item)
stripped = stripped[2:].strip()
if not stripped:
continue
if item is None:
raise CatalogError(f"catalog:{lineno}: list items must start with '- '")
key, sep, value = stripped.partition(":")
if not sep:
raise CatalogError(f"catalog:{lineno}: expected 'key: value'")
item[key.strip()] = _scalar(value)
return doc
def load_catalog(path: Path) -> list[Entry]:
text = path.read_text(encoding="utf-8")
use_pyyaml = not os.environ.get("PKH_E2E_NO_PYYAML")
try:
if use_pyyaml:
import yaml # type: ignore import-not-found
doc = yaml.safe_load(text)
else:
doc = _parse_mini_yaml(text)
except CatalogError:
raise
except Exception as exc:
if use_pyyaml:
raise CatalogError(f"cannot parse {path}: {exc}") from exc
raise
defaults = doc.get("defaults") or {}
raw_entries = doc.get("packages") or []
if not raw_entries:
raise CatalogError(f"{path}: no packages listed")
entries: list[Entry] = []
seen: set[str] = set()
for i, raw in enumerate(raw_entries):
name = str(raw.get("name", "")).strip()
if not name:
raise CatalogError(f"{path}: package #{i + 1} has no name")
if name in seen:
raise CatalogError(f"{path}: duplicate package '{name}'")
seen.add(name)
entry = Entry(
name=name,
category=str(raw.get("category", "misc")),
tier=str(raw.get("tier", defaults.get("tier", "standard"))),
dist=str(raw.get("dist", defaults.get("dist", "debian"))),
series=str(raw.get("series", "")),
arch=str(raw.get("arch", "")),
cross=bool(raw.get("cross", False)),
pocket=str(raw.get("pocket", "")),
ppa=[str(p) for p in (raw.get("ppa") or [])],
timeout=float(raw.get("timeout", 0)),
xfail=str(raw.get("xfail", "")),
notes=str(raw.get("notes", "")),
tags=[str(t) for t in (raw.get("tags") or [])],
)
if entry.tier not in TIERS:
raise CatalogError(f"{path}: {name}: unknown tier '{entry.tier}'")
if entry.dist not in ("debian", "ubuntu"):
raise CatalogError(f"{path}: {name}: unknown dist '{entry.dist}'")
if entry.cross and not entry.arch:
raise CatalogError(f"{path}: {name}: cross: true needs an arch")
entries.append(entry)
return entries
# ---------------------------------------------------------------------------
# Selection and rendering
# ---------------------------------------------------------------------------
def select_entries(
entries: list[Entry], args: argparse.Namespace
) -> tuple[list[Entry], list[str]]:
"""Apply the CLI filters; returns the selection and the filtered-out names."""
tiers = (
list(TIERS) if args.tier == "all" else [t.strip() for t in args.tier.split(",")]
)
for tier in tiers:
if tier not in TIERS:
raise CatalogError(f"unknown tier '{tier}' (expected: {', '.join(TIERS)})")
selected = [e for e in entries if e.tier in tiers]
dropped = [e.name for e in entries if e.tier not in tiers]
if args.category:
wanted = {c.strip() for c in args.category.split(",") if c.strip()}
dropped += [e.name for e in selected if e.category not in wanted]
selected = [e for e in selected if e.category in wanted]
if args.tag:
wanted = {t.strip() for t in args.tag.split(",") if t.strip()}
dropped += [e.name for e in selected if not wanted.intersection(e.tags)]
selected = [e for e in selected if wanted.intersection(e.tags)]
if args.only:
names = {n.strip() for n in args.only.split(",") if n.strip()}
unknown = sorted(names - {e.name for e in entries})
if unknown:
raise CatalogError(f"--only: not in the catalog: {', '.join(unknown)}")
dropped += [e.name for e in selected if e.name not in names]
selected = [e for e in selected if e.name in names]
if args.skip:
names = {n.strip() for n in args.skip.split(",") if n.strip()}
dropped += [e.name for e in selected if e.name in names]
selected = [e for e in selected if e.name not in names]
return selected, dropped
def fmt_duration(seconds: float | None) -> str:
if seconds is None:
return "-"
seconds = int(seconds)
if seconds < 60:
return f"{seconds}s"
if seconds < 3600:
return f"{seconds // 60}m{seconds % 60:02d}s"
return f"{seconds // 3600}h{(seconds % 3600) // 60:02d}m"
def render_list(entries: list[Entry]) -> str:
lines = [
f"{'PACKAGE':<20} {'TIER':<9} {'DIST':<7} {'CATEGORY':<14} "
f"{'TAGS':<30} NOTES"
]
for e in entries:
extra = []
if e.cross:
extra.append(f"cross {e.arch}")
if e.series:
extra.append(f"series {e.series}")
note = " ".join(extra + [e.xfail or e.notes])
lines.append(
f"{e.name:<20} {e.tier:<9} {e.dist:<7} {e.category:<14} "
f"{','.join(e.tags):<30} {note}"
)
return "\n".join(lines) + f"\n\n{len(entries)} package(s)\n"
# ---------------------------------------------------------------------------
# Command construction and child-process plumbing
# ---------------------------------------------------------------------------
def stage_command(args: argparse.Namespace, entry: Entry, stage: str, pkh: Path) -> list[str]:
cmd = [str(pkh), stage]
if stage == "pull":
cmd += [entry.name, "-d", args.dist or entry.dist]
if args.series or entry.series:
cmd += ["-s", args.series or entry.series]
if args.pocket or entry.pocket:
cmd += ["-p", args.pocket or entry.pocket]
if not args.git:
cmd += ["--archive"]
elif stage == "chlog":
cmd += ["--rebuild"]
if args.series or entry.series:
cmd += ["-s", args.series or entry.series]
elif stage == "build":
cmd += ["--verbose"]
elif stage == "deb":
cmd += ["--verbose"]
if args.series or entry.series:
cmd += ["-s", args.series or entry.series]
arch = args.arch or entry.arch
if arch:
cmd += ["-a", arch]
if entry.cross or args.cross:
cmd += ["--cross"]
if args.pocket or entry.pocket:
cmd += ["-p", args.pocket or entry.pocket]
for ppa in entry.ppa + args.ppa:
cmd += ["--ppa", ppa]
if args.jobs:
cmd += ["-j", str(args.jobs)]
if args.no_ccache:
cmd += ["--no-ccache"]
return cmd
def child_env() -> dict:
"""Environment for pkh children: no editor, a deterministic identity."""
env = dict(os.environ)
env["EDITOR"] = "true" # chlog launches it after generating; nothing to edit
env.setdefault("DEBEMAIL", "pkh-e2e@invalid")
env.setdefault("DEBFULLNAME", "pkh e2e")
return env
def run_stage(
cmd: list[str], cwd: Path, log_path: Path, timeout_min: float
) -> tuple[str, float]:
"""Run one stage, logging to log_path; returns (status, seconds).
Status is 'ok', 'fail' or 'timeout'. The whole process group is killed
on timeout: builds fan out into make/compiler children, and killing only
the direct child would leave them running. On Ctrl+C the group gets a
graceful SIGTERM first, mirroring pkh's own interrupt cleanup.
"""
with log_path.open("ab") as log:
log.write(f"\n$ {shlex.join(cmd)}\n(cwd: {cwd})\n".encode())
log.flush()
start = time.monotonic()
proc = subprocess.Popen(
cmd,
cwd=cwd,
env=child_env(),
stdin=subprocess.DEVNULL,
stdout=log,
stderr=subprocess.STDOUT,
start_new_session=True,
)
try:
proc.wait(timeout=timeout_min * 60)
except subprocess.TimeoutExpired:
_kill_group(proc, signal.SIGKILL)
return "timeout", time.monotonic() - start
except KeyboardInterrupt:
# A graceful stop first: SIGTERM lets pkh run its own interrupt
# cleanup; a child ignoring it is killed after a grace period.
_kill_group(proc, signal.SIGTERM, wait=10)
_kill_group(proc, signal.SIGKILL)
raise
return ("ok" if proc.returncode == 0 else "fail"), time.monotonic() - start
def _kill_group(
proc: subprocess.Popen, sig: signal.Signals, wait: float | None = None
) -> None:
try:
os.killpg(os.getpgid(proc.pid), sig)
except ProcessLookupError:
pass
try:
proc.wait(timeout=wait)
except subprocess.TimeoutExpired:
pass
# ---------------------------------------------------------------------------
# The harness proper
# ---------------------------------------------------------------------------
class Harness:
def __init__(self, args: argparse.Namespace, entries: list[Entry], pkh: Path):
self.args = args
self.entries = entries
self.pkh = pkh
self.stages = [s.strip() for s in args.stages.split(",") if s.strip()]
self.records: dict[str, dict] = {}
self.pull_results: dict[str, tuple[str, float]] = {}
self.lock = threading.Lock()
self.run_dir = self._open_run_dir()
self.results_path = self.run_dir / "results.json"
(self.run_dir / "logs").mkdir(parents=True, exist_ok=True)
# -- layout -------------------------------------------------------------
def _open_run_dir(self) -> Path:
if self.args.resume:
try:
data = json.loads(Path(self.args.resume).read_text())
run_dir = Path(data["run_dir"])
except (OSError, KeyError, json.JSONDecodeError) as exc:
raise CatalogError(f"--resume: cannot read {self.args.resume}: {exc}")
if not run_dir.is_dir():
raise CatalogError(f"--resume: missing run directory {run_dir}")
for record in data.get("entries", []):
self.records[record["name"]] = record
return run_dir
stamp = dt.datetime.now().strftime("%Y%m%d-%H%M%S")
return SCRIPT_DIR / "runs" / f"{stamp}-{os.getpid()}"
def log_path(self, name: str) -> Path:
return self.run_dir / "logs" / f"{name}.log"
def workroot(self, entry: Entry) -> Path:
# pkh pull (run from the run root) puts the tree at
# ./<pkg>/<pkg>/: this directory is both the tree's parent and
# where pkh build / pkh deb drop their artifacts.
return self.run_dir / entry.name
def tree(self, entry: Entry) -> Path:
return self.workroot(entry) / entry.name
# -- stages ---------------------------------------------------------------
def do_pull(self, entry: Entry) -> tuple[str, float]:
# Pull from the run root: pkh pull (cwd-less) extracts to
# ./<pkg>/<pkg>/ under its cwd, so every package still gets its own
# subtree and parallel pulls never collide. A failed pull leaves a
# partial tree that pkh pull would refuse to overwrite and that a
# later --resume must not trust: wipe it before retrying and after
# the last attempt.
for attempt in range(self.args.pull_retries + 1):
if attempt:
shutil.rmtree(self.workroot(entry), ignore_errors=True)
status, seconds = run_stage(
stage_command(self.args, entry, "pull", self.pkh),
self.run_dir,
self.log_path(entry.name),
entry.timeout_min(),
)
if status == "ok":
return status, seconds
shutil.rmtree(self.workroot(entry), ignore_errors=True)
return status, seconds
def do_chlog(self, entry: Entry) -> tuple[str, float]:
return run_stage(
stage_command(self.args, entry, "chlog", self.pkh),
self.tree(entry),
self.log_path(entry.name),
entry.timeout_min(),
)
def do_build(self, entry: Entry) -> tuple[tuple[str, float], list[str]]:
record = run_stage(
stage_command(self.args, entry, "build", self.pkh),
self.tree(entry),
self.log_path(entry.name),
entry.timeout_min(),
)
artifacts = self.build_artifacts(entry)
return self._check_artifacts(entry, record, artifacts, "dsc/changes"), artifacts
def do_deb(self, entry: Entry) -> tuple[tuple[str, float], list[str]]:
record = run_stage(
stage_command(self.args, entry, "deb", self.pkh),
self.tree(entry),
self.log_path(entry.name),
entry.timeout_min(),
)
debs = self.deb_artifacts(entry)
return self._check_artifacts(entry, record, debs, ".deb"), debs
def build_artifacts(self, entry: Entry) -> list[str]:
return sorted(
p.name
for pattern in ("*.dsc", "*_source.changes")
for p in self.workroot(entry).glob(pattern)
)
def deb_artifacts(self, entry: Entry) -> list[str]:
return sorted(p.name for p in self.workroot(entry).glob("*.deb"))
def _check_artifacts(
self, entry: Entry, record: tuple[str, float], artifacts: list[str], what: str
) -> tuple[str, float]:
"""A green stage with no artifacts is a failure: log it and flip."""
if record[0] == "ok" and not artifacts:
with self.log_path(entry.name).open("ab") as log:
log.write(
f"\npkh-e2e: stage reported success but produced no {what}\n".encode()
)
return ("fail", record[1])
return record
def changelog_version(self, entry: Entry) -> str:
try:
out = subprocess.run(
["dpkg-parsechangelog", "-S", "Version"],
cwd=self.tree(entry),
capture_output=True,
text=True,
check=True,
)
return out.stdout.strip()
except Exception:
return ""
# -- per-entry pipeline ------------------------------------------------------
def run_entry(self, entry: Entry) -> dict:
"""Full pipeline for one entry; never raises."""
# A resumed run keeps what the previous record already achieved
previous = self.records.get(entry.name)
record = {
"name": entry.name,
"category": entry.category,
"tier": entry.tier,
"dist": self.args.dist or entry.dist,
"series": self.args.series or entry.series,
"arch": self.args.arch or entry.arch or platform.machine(),
"cross": entry.cross or self.args.cross,
"tags": entry.tags,
"xfail": entry.xfail,
"notes": entry.notes,
"outcome": "fail",
"version": "",
"stages": {},
"artifacts": [],
"total_seconds": 0.0,
"log": str(self.log_path(entry.name).relative_to(self.run_dir)),
}
started = time.monotonic()
try:
self._run_entry(record, entry, previous)
finally:
record["total_seconds"] = time.monotonic() - started
if entry.xfail:
record["outcome"] = "xfail" if record["outcome"] == "fail" else "xpass"
with self.lock:
self.records[entry.name] = record
self.flush()
color = {"pass": 32, "xfail": 33, "xpass": 33}.get(record["outcome"], 31)
print(
f"[{entry.name}] \033[{color}m{record['outcome']}\033[0m "
f"({fmt_duration(record['total_seconds'])})"
+ (f" — {record['error']}" if record.get("error") else "")
)
return record
def _run_entry(self, record: dict, entry: Entry, previous: dict | None) -> None:
# Stages the resumed record already completed are reused rather than
# re-run: a resumed chlog must not bump the version a second time.
# Reuse only while the build selection is unchanged.
reusable = previous is not None and all(
previous.get(key) == record.get(key)
for key in ("dist", "series", "arch", "cross")
)
prev_stages = previous.get("stages", {}) if previous else {}
if "pull" in self.stages:
if (self.tree(entry) / "debian").is_dir():
# Resumed run: the tree survived, keep it (pkh pull refuses
# existing directories anyway).
record["stages"]["pull"] = {"status": "ok", "seconds": 0.0,
"cached": True}
print(f"[{entry.name}] pull ok (tree already present)")
else:
status, seconds = self.pull_results.get(entry.name, ("fail", 0.0))
record["stages"]["pull"] = {"status": status, "seconds": seconds}
if status != "ok":
record["error"] = f"pull {status}"
return
# Already reported by the pull phase; no duplicate line.
for stage in PIPELINE_STAGES:
if stage not in self.stages:
continue
prior = prev_stages.get(stage) if reusable else None
if prior and prior.get("status") == "ok":
record["stages"][stage] = {**prior, "reused": True}
if stage == "chlog":
record["version"] = previous.get("version", "")
elif stage == "build":
record["artifacts"] += self.build_artifacts(entry)
elif stage == "deb":
record["artifacts"] += self.deb_artifacts(entry)
print(f"[{entry.name}] {stage} ok (reused)")
continue
if not (self.tree(entry) / "debian").is_dir():
record["error"] = (
f"no source tree for {stage}; include the pull stage or "
"resume the original run directory"
)
return
if stage == "chlog":
status, seconds = self.do_chlog(entry)
record["stages"]["chlog"] = {"status": status, "seconds": seconds}
if status != "ok":
record["error"] = f"chlog {status}"
return
record["version"] = self.changelog_version(entry)
print(
f"[{entry.name}] chlog ok ({fmt_duration(seconds)})"
+ (f" -> {record['version']}" if record["version"] else "")
)
elif stage == "build":
(status, seconds), artifacts = self.do_build(entry)
record["stages"]["build"] = {"status": status, "seconds": seconds}
record["artifacts"] += artifacts
if status != "ok":
record["error"] = f"build {status}"
return
print(f"[{entry.name}] build ok ({fmt_duration(seconds)})")
elif stage == "deb":
(status, seconds), debs = self.do_deb(entry)
record["stages"]["deb"] = {"status": status, "seconds": seconds}
record["artifacts"] += debs
if status != "ok":
record["error"] = f"deb {status}"
return
print(
f"[{entry.name}] deb ok ({fmt_duration(seconds)}), "
f"{len(debs)} binary package(s)"
)
record["outcome"] = "pass"
# -- run orchestration ---------------------------------------------------------
def preflight(self) -> None:
version = subprocess.run(
[str(self.pkh), "--version"], capture_output=True, text=True
)
print(f"[pkh-e2e] pkh: {self.pkh} ({version.stdout.strip() or '?'})")
if "deb" in self.stages:
missing = [
tool
for tool in ("mmdebstrap", "unshare", "sudo")
if shutil.which(tool) is None
]
if missing:
print(f"[pkh-e2e] WARNING: deb needs: {', '.join(missing)}")
if subprocess.run(["sudo", "-n", "true"], capture_output=True).returncode:
print(
"[pkh-e2e] WARNING: passwordless sudo unavailable; "
"ephemeral chroot builds will fail"
)
free_g = shutil.disk_usage(self.run_dir).free / 2**30
if free_g < 20:
print(f"[pkh-e2e] WARNING: only {free_g:.0f}G free on {self.run_dir}")
print(
f"[pkh-e2e] {len(self.entries)} package(s), stages: "
f"{'+'.join(self.stages)} -> {self.run_dir}"
)
def resume_skip(self, entry: Entry) -> bool:
previous = self.records.get(entry.name)
if not previous or previous.get("outcome") not in TERMINAL_OUTCOMES:
return False
return all(s in previous.get("stages", {}) for s in self.stages)
def pull_phase(self) -> None:
if "pull" not in self.stages:
return
todo = [
e
for e in self.entries
if not self.resume_skip(e) and not (self.tree(e) / "debian").is_dir()
]
if not todo:
return
print(
f"[pkh-e2e] pulling {len(todo)} package(s) "
f"({self.args.pull_jobs} at a time)"
)
with concurrent.futures.ThreadPoolExecutor(
max_workers=self.args.pull_jobs
) as pool:
futures = {pool.submit(self.do_pull, e): e for e in todo}
for future in concurrent.futures.as_completed(futures):
entry = futures[future]
try:
status, seconds = future.result()
except Exception as exc: # keep the run going on worker errors
status, seconds = "fail", 0.0
print(f"[{entry.name}] pull crashed: {exc}")
self.pull_results[entry.name] = (status, seconds)
print(
f"[{entry.name}] pull {'ok' if status == 'ok' else status} "
f"({fmt_duration(seconds)})"
)
def run(self) -> int:
self.preflight()
self.flush()
self.pull_phase()
failures = 0
for entry in self.entries:
if self.resume_skip(entry):
print(f"[{entry.name}] already {self.records[entry.name]['outcome']},"
" skipping")
continue
if self.args.fail_fast and failures:
print(f"[{entry.name}] skipped (fail-fast)")
continue
record = self.run_entry(entry)
if record["outcome"] in ("fail", "timeout"):
failures += 1
self.write_report()
self.print_summary()
return self.exit_code(failures)
def print_summary(self) -> None:
rows = sorted(self.records.values(), key=lambda r: r["name"])
print(f"\n=== summary ({len(rows)} recorded) ===")
print(
f"{'PACKAGE':<20} {'TIER':<9} {'RESULT':<8} "
f"{'PULL':>7} {'CHLOG':>7} {'BUILD':>8} {'DEB':>8} {'TOTAL':>8} VERSION"
)
for r in rows:
def sec(stage):
data = r["stages"].get(stage)
return fmt_duration(data["seconds"]) if data else "-"
print(
f"{r['name']:<20} {r['tier']:<9} {r['outcome']:<8} "
f"{sec('pull'):>7} {sec('chlog'):>7} {sec('build'):>8} "
f"{sec('deb'):>8} {fmt_duration(r['total_seconds']):>8} "
f"{r['version']}"
)
tally: dict[str, int] = {}
for r in rows:
tally[r["outcome"]] = tally.get(r["outcome"], 0) + 1
print("\n" + ", ".join(f"{v} {k}" for k, v in sorted(tally.items())))
def exit_code(self, failures: int) -> int:
if failures:
return 1
if self.args.strict and any(
r["outcome"] == "xpass" for r in self.records.values()
):
return 1
return 0
# -- persistence ---------------------------------------------------------------
def flush(self) -> None:
"""Persist results progressively: a killed run keeps its records."""
if not hasattr(self, "_started"):
self._started = dt.datetime.now().astimezone().isoformat()
meta = {
"run_dir": str(self.run_dir),
"pkh": str(self.pkh),
"filters": {
"tier": self.args.tier,
"category": self.args.category,
"tag": self.args.tag,
"only": self.args.only,
"skip": self.args.skip,
"stages": self.stages,
},
"host": {
"kernel": platform.release(),
"arch": platform.machine(),
"python": platform.python_version(),
},
"started": self._started,
"updated": dt.datetime.now().astimezone().isoformat(),
"entries": sorted(self.records.values(), key=lambda r: r["name"]),
}
tmp = self.results_path.with_suffix(".json.tmp")
tmp.write_text(json.dumps(meta, indent=2) + "\n")
tmp.replace(self.results_path)
def write_report(self) -> None:
rows = sorted(self.records.values(), key=lambda r: (r["outcome"], r["name"]))
lines = [
f"# pkh e2e run — {self.run_dir.name}",
"",
f"- pkh: `{self.pkh}`",
f"- started: {getattr(self, '_started', '?')}",
f"- stages: {'+'.join(self.stages)}",
"",
"| package | tier | result | pull | chlog | build | deb | total |"
" version |",
"|---|---|---|---|---|---|---|---|---|",
]
for r in rows:
def sec(stage):
data = r["stages"].get(stage)
return fmt_duration(data["seconds"]) if data else "-"
lines.append(
f"| {r['name']} | {r['tier']} | {r['outcome']} | {sec('pull')} | "
f"{sec('chlog')} | {sec('build')} | {sec('deb')} | "
f"{fmt_duration(r['total_seconds'])} | {r['version']} |"
)
(self.run_dir / "results.md").write_text("\n".join(lines) + "\n")
# ---------------------------------------------------------------------------
# Entry point
# ---------------------------------------------------------------------------
def parse_args(argv: list[str]) -> argparse.Namespace:
parser = argparse.ArgumentParser(
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
)
parser.add_argument("--catalog", type=Path, default=SCRIPT_DIR / "catalog.yml",
help="catalog file (default: catalog.yml)")
parser.add_argument("--tier", default=",".join(DEFAULT_TIERS),
help="tiers to run, comma-separated: smoke, standard, long, "
"or 'all' (default: smoke,standard)")
parser.add_argument("--category", default="",
help="comma-separated categories to keep")
parser.add_argument("--tag", default="",
help="comma-separated tags; keep packages carrying any")
parser.add_argument("--only", default="",
help="comma-separated exact package names to run")
parser.add_argument("--skip", default="",
help="comma-separated exact package names to skip")
parser.add_argument("--stages", default=",".join(STAGES),
help=f"stages to run (default: {','.join(STAGES)})")
parser.add_argument("--pkh", default="",
help="pkh binary to exercise (default: the repo build, "
"then PATH)")
parser.add_argument("--build-pkh", action="store_true",
help="run 'cargo build --release' before resolving pkh")
parser.add_argument("--git", action="store_true",
help="pull through the package VCS instead of the archive")
parser.add_argument("--dist", default="",
help="override the catalog default dist (per-entry dist wins)")
parser.add_argument("--series", default="",
help="override the series (per-entry series wins)")
parser.add_argument("--arch", default="",
help="force the 'pkh deb -a' architecture on every entry")
parser.add_argument("--cross", action="store_true",
help="pass --cross to every deb build")
parser.add_argument("--pocket", default="",
help="distribution pocket passed to pull and deb")
parser.add_argument("--ppa", action="append", default=[],
help="PPA passed to 'pkh deb --ppa' (repeatable)")
parser.add_argument("--jobs", type=int, default=0,
help="pass -j N to 'pkh deb' (default: pkh chooses)")
parser.add_argument("--no-ccache", action="store_true",
help="pass --no-ccache to 'pkh deb'")
parser.add_argument("--pull-jobs", type=int, default=4,
help="parallel pulls (default: 4)")
parser.add_argument("--pull-retries", type=int, default=1,
help="retries per failed pull (default: 1)")
parser.add_argument("--fail-fast", action="store_true",
help="stop the pipeline at the first failure")
parser.add_argument("--strict", action="store_true",
help="also fail the run on xpass")
parser.add_argument("--resume", default="",
help="results.json of a previous run: reuse its run "
"directory and skip finished packages")
parser.add_argument("--list", action="store_true",
help="print the selected packages and exit")
parser.add_argument("--dry-run", action="store_true",
help="print the plan and the exact commands, run nothing")
return parser.parse_args(argv)
def main(argv: list[str]) -> int:
args = parse_args(argv)
try:
entries = load_catalog(args.catalog)
selected, dropped = select_entries(entries, args)
stages = [s.strip() for s in args.stages.split(",") if s.strip()]
for stage in stages:
if stage not in STAGES:
raise CatalogError(
f"unknown stage '{stage}' (expected: {', '.join(STAGES)})"
)
if args.pull_jobs < 1:
raise CatalogError("--pull-jobs must be at least 1")
if args.list:
print(render_list(selected))
if dropped:
print(f"(filtered out: {len(dropped)})")
return 0
if not selected:
print("nothing selected", file=sys.stderr)
return 2
pkh = resolve_pkh(args)
if args.dry_run:
print(f"pkh: {pkh}")
print(f"selected {len(selected)} package(s), stages: {args.stages}")
for entry in selected:
where = f" {entry.series}" if entry.series else ""
print(f"\n[{entry.name}] {entry.tier}/{entry.category} "
f"({entry.dist}{where})")
for stage in stages:
cmd = stage_command(args, entry, stage, pkh)
print(f" {shlex.join(cmd)}"
f" # timeout {entry.timeout_min():.0f}m")
return 0
harness = Harness(args, selected, pkh)
return harness.run()
except CatalogError as exc:
print(f"error: {exc}", file=sys.stderr)
return 2
except KeyboardInterrupt:
print("\ninterrupted", file=sys.stderr)
return 130
def repo_source_epoch() -> float | None:
"""The newest evidence of source changes: HEAD's commit time, or the
newest mtime under src/ for uncommitted edits. None outside a git
checkout with no src/ tree."""
stamps = []
try:
out = subprocess.run(
["git", "log", "-1", "--format=%ct"],
cwd=REPO_ROOT, capture_output=True, text=True, check=True,
)
stamps.append(int(out.stdout.strip()))
except Exception:
pass
src = REPO_ROOT / "src"
if src.is_dir():
stamps.extend(path.stat().st_mtime for path in src.rglob("*.rs"))
return max(stamps) if stamps else None
def resolve_pkh(args: argparse.Namespace) -> Path:
if args.pkh:
path = Path(args.pkh).resolve()
if not path.is_file():
raise CatalogError(f"--pkh: no such file: {path}")
return path
# A repo binary older than the sources measures whatever the last
# `cargo build --release` happened to include — yesterday's code.
# Rebuild unless the caller pinned a binary or asked for it explicitly.
if not args.build_pkh:
head = repo_source_epoch()
fresh = head is not None and any(
candidate.is_file() and candidate.stat().st_mtime >= head
for candidate in (REPO_ROOT / "target/release/pkh",
REPO_ROOT / "target/debug/pkh")
)
if fresh:
return next(candidate
for candidate in (REPO_ROOT / "target/release/pkh",
REPO_ROOT / "target/debug/pkh")
if candidate.is_file()
and candidate.stat().st_mtime >= head)
print("[pkh-e2e] the repo pkh binary is older than the sources; "
"running 'cargo build --release'")
subprocess.run(["cargo", "build", "--release"], cwd=REPO_ROOT, check=True)
for candidate in (REPO_ROOT / "target/release/pkh", REPO_ROOT / "target/debug/pkh"):
if candidate.is_file():
return candidate
found = shutil.which("pkh")
if not found:
raise CatalogError(
"no pkh binary: pass --pkh, use --build-pkh, or install pkh on PATH"
)
return Path(found)
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))