The pull downloads reported through the plain count port only because
byte_progress landed on main after the branch was cut. PullView now
implements it with a byte-counter template in the transfer-style
family (same shape, {bytes}/{total_bytes} counters, human-formatted by
indicatif), downloads emit through it, and the trait doc lists the
event among the ones the pull flow emits — un-duplicating the merged
'message, progress and progress' sentence.
Retire the ProgressCallback option-of-closure, the one pre-port
reporting idiom left in the tree: pull and package_info::lookup now
take &dyn BuildView like every other core flow, and the announcements
map onto message (unsized work), progress (item counts: series probes,
git objects, extraction) and progress again for byte downloads, which
the old callback carried in the same 4-argument shape.
The terminal side swaps the (ProgressBar, closure) pair for PullView, a
BuildView implementation owning one bar of the shared MultiProgress
with the same spinner/transfer style switching; it releases the line on
drop, which the CLI triggers explicitly on both outcomes since
exit(1) skips destructors. Headless callers (tests, internal lookups)
pass &report::Quiet. git clone sideband parsing is skipped entirely
when the view is not enabled, matching the old None-callback shortcut.
A component tarball whose entries already carry the addon prefix
('www/...' in sqlite3's 'sqlite3_3.53.4.orig-www.tar.xz') extracts
to its end state, but the wrapper detection named it a wrapper and
unwrap_into then cleared the addon target - destroying the component
just extracted into it - and unwrapped the emptied directory into
itself: no 'www' at all, and the deb failed at dh_installdocs over
the missing 'www/*' docs.
dpkg-source unpacks a component into '<root>/<addon>': when the
tarball's own layout already is that, the extraction is the end
state; leave it alone.
unwrap_into pointed the component's unwrap at '<root>/<addon>' but
never created that directory: unwrap_wrapper moves the wrapper's
entries into it by rename, and a rename whose target parent is
missing fails with ENOENT - every entry, aborting the whole pull.
The multi-orig unit test exercised unwrap_wrapper directly and could
not see it (dovecot).
Also exercise the component through unwrap_into in the test, and
assert the wrapper's own layout does not leak to the root - the
shape dpkg-source -b rejects as unexpected upstream changes.
patch reports rejected hunks on stdout, not stderr: the error used
to carry only the diff path and an empty stderr, naming neither the
failing file nor the reason.
Git-archive tarballs open with a pax global header record
('pax_global_header'): GNU tar consumes it and never lists or
extracts it, but the tar crate yields it as an entry. Counting it
for the wrapper detection made every git-archive orig look
multi-rooted - linux-riscv's orig stayed nested under 'linux-6.8/'
and its .diff.gz failed to apply with 'can't find file to patch'.
Skip the record outright: unpack would no-op on it anyway, so this
only keeps it out of the wrapper detection (and the tree).
dpkg-source assembles a multi-orig source by unpacking the main orig
first and the extra components over it, each component's content at
'<root>/<addon>' ('pigeonhole' for 'pkg_1.0.orig-pigeonhole.tar.gz'),
so component files win where they overlap the main tree: process the
archives in that order, and unwrap each archive's wrapper into where
dpkg-source expects it - the root for the main orig, '<root>/<addon>'
for a component, replacing any pre-existing addon directory.
Recognizing the wrapper per archive (the one first component every
entry of THIS archive shares) rather than from the shape of the tree
root keeps every component of a multi-orig package unwrapping after
the first one filled the root: dovecot's pigeonhole sources used to
be left nested one level too deep, and dpkg-source -b then refused
the tree over unexpected upstream changes.
Three archive-side defects broke pulled trees, each surfacing as a
later build failure rather than at pull time:
- '.tar.bz2'/'.tbz2' origs (busybox) were downloaded but never
extracted: the dispatch only knew gz and xz. dpkg-source -b then
packed the missing tree as a mass of ignored deletions and the
chroot's quilt run failed on every patch. Extract bz2 through
bzip2-rs, and match every '.tar.*' of the source package so an
unsupported compression fails loudly instead of silently skipping
the extraction.
- Hard link entries (jq) aborted the whole extraction: resolve their
targets against the archive root, where tar guarantees the shared
entry was already extracted, with the same confinement checks as
regular entries.
- The 'package-version/' wrapper was detected from explicit tar
directory entries (tree has none, so its sources stayed nested one
level too deep) and relocated through a copy that flattened
top-level symlinks (htop's 'README -> README.md' became a plain
file, which dpkg-source refuses to represent). Recognize the
wrapper by the shape of the extraction result - exactly one real
directory - and unwrap it by rename, preserving links and
timestamps.
Sources listed with "3.0 (quilt)" extra components (node-jest, php-*,
...) carry one tarball per bundled module next to the main orig, named
<package>_<uver>.orig-<component>.tar.<ext>. fetch_orig_tarball picked
the single file matching ".orig.tar." — which cannot even match the
component naming — so a git pull only fetched the main orig. The later
dpkg-source -b quilt verification then failed with "can't find file to
patch" on the first patch touching a component directory.
Select the files with the existing build::changes::is_orig_tarball
helper (mirroring dpkg's \.orig(-.+)?\.tar\. strip pattern) and fetch
all of them, pristine-tar checkout first with a checksummed archive
download fallback, per tarball.
The end-to-end test now asserts every stanza-listed orig lands in the
package dir instead of just any *.orig.tar.* file, and gains a
node-jest (trixie, 24 components) regression case.
Verified live: pkh pull node-jest -d debian fetches all 15 origs of the
sid ds7 repack, and dpkg-source -b builds the debian.tar.xz and dsc
without touching the series.
Busy mirrors and CDNs routinely break bulk fetches: pooled keep-alive
connections get closed remotely ('error sending request'), downloads are
cut short (surfacing as bogus checksum mismatches), and index generations
momentarily drift from the Release file fetched moments before.
- shared client: short idle-pool timeout and TCP keepalive, and a
bounded-retry GET helper now used for index, Release, keyring and
Launchpad fetches (previously reqwest::get, which has no timeouts)
- downloads: retry the whole download, and check the content length so
truncation is reported as such instead of a checksum mismatch
- sources index: on a checksum mismatch against the Release file, retry
pinned to the exact listed generation via Debian's by-hash mechanism;
body-read errors are retried and reported per component instead of
aborting the whole lookup
- distro_info: malformed CSV rows are skipped with a warning, dates
that fail to parse become None, and all plain HTTP requests go
through a shared reqwest client with connect/total timeouts
- package_info: the Sources stanza iterator is iterative (a crafted
index with many blank stanzas overflowed the stack), stanzas missing
a Version are skipped, and failed series/pocket probes are summarized
in the final 'not found' error instead of being silently dropped
- pull: no double unwrap on the remote-derived artifact filename, an
empty series list is an error, and streaming downloads get a
per-request timeout
- deb/cross: dpkg-architecture output parsing skips unexpected lines
and its exit status is checked, as is dpkg --add-architecture
- changelog: version increments parse as u64 with checked arithmetic
(1.0-20250123123456 used to panic on the u32 parse)
Entry::unpack performs no path sanitization, so a malicious or malformed
tarball (PPA, flat repository) could write files outside the package
directory via '..' components or absolute entry paths. Refuse such
entries with an error naming the offending path.
Add a --repository flag taking the full suite URL of an external flat
repository (e.g. https://pkg.noctalia.dev/deb/resolute/), i.e. one with
no dists/ hierarchy, like apt's exact-path suites ('Suites: resolute/').
The suite name is read from the root Release file (Codename/Suite), the
sources index is fetched from the repository root as Sources.xz/gz/plain,
and package files are resolved against the URL root, ignoring the stanza
Directory field like apt does. As with PPAs, the stanza Vcs-Git is never
used for external repositories, so the source always comes from the
repository itself.
Also make the sources index parser detect compression by magic bytes
(gz/xz/plain) instead of assuming gzip, and fix extraction of archives
with './'-prefixed entries, which previously aborted and are now
extracted in place instead of being relocated.
std::fs::copy resets the destination mtime to "now", which breaks
timestamp-based build systems (autotools/gnulib). Packages like
'hello' from Debian sid ship pre-generated files alongside their
prerequisites; when the copy flattens all mtimes, make considers the
generated targets out-of-date and tries to regenerate them with tools
(e.g. gperf) that are not declared build-dependencies, failing the
build.
Restore the source modification and access times after every file copy
in pull::copy_dir_all, pull::fetch_archive_sources merge step, and
context::unshare copy_dir_recursive/ensure_available.
The Ubuntu development series (stonking) Sources.gz no longer ships a
Checksums-Sha256 field, only Checksums-Sha512 and the legacy Files
(MD5) field. The package_info parser only read Checksums-Sha256, so
the file list ended up empty and fetch_orig_tarball panicked on
Option::unwrap() when looking for the orig tarball.
- Add a ChecksumAlgo enum (Md5/Sha256/Sha512) to FileEntry, replacing
the hardcoded sha256 field, and parse the strongest available
checksum field (Sha256 > Sha512 > MD5).
- Make download_file_checksum verify against the correct algorithm
instead of always using SHA-256.
- Replace the unwrap() on the orig tarball search with a proper error
listing the available files, so future regressions fail clearly
instead of panicking.
- Add md-5 dependency for MD5 verification.
Multiple changes:
- New contexts (schroot, unshare)
- Cross-building quirks, with ephemeral contexts and repositories management
- Contexts with parents, global context manager, better lifetime handling
- Local building of binary packages
- Pull: pulling dsc files by default
- Many small bugfixes and changes
Co-authored-by: Valentin Haudiquet <valentin.haudiquet@canonical.com>
Co-committed-by: Valentin Haudiquet <valentin.haudiquet@canonical.com>
vhaudiquet