Compare commits

...
5 Commits
Author SHA1 Message Date
vhaudiquet 53078fbca1 fmt 2026-08-24 10:31:26 +02:00
vhaudiquet fbdf1b334b build/binary: native .buildinfo/.changes for binary builds (pkh deb)
Extend the metadata writers to binary-only uploads and wire them into
the 'pkh deb' flow:

- build/binary.rs generates <pkg>_<ver>_<arch>.buildinfo/.changes
  through any Context: debian/files consumption, encounter-order
  Architecture accumulation (sorted in .buildinfo like dpkg-genbuildinfo),
  sorted Binary lists, dpkg-formatted Description lines with udeb
  suffixes, Installed-Build-Depends closure over the context status DB,
  and binNMU handling (Source: pkg (prev), Binary-Only-Changes, previous
  .dsc redistribution);
- artifact digests are computed inside the context via coreutils
  (md5sum/sha1sum/sha256sum/stat) so chrooted/remote trees work;
- deb/local.rs runs the generation after 'rules binary', exports
  SOURCE_DATE_EPOCH from the changelog (reproducibility), and resolves
  vendor/profiles inside the context; deb/mod.rs retrieves the new
  artifacts alongside the debs;
- reusable helpers added: FilesList::parse/render,
  parse_changelog_entry_from_str, parse_previous_version_from_str,
  installed_build_depends_from_content.

Differential gate: same tree built with real 'dpkg-buildpackage -b' and
with the pkh flow; .changes/.buildinfo compared field-by-field modulo
machine-dependent fields, artifact checksums included.
2026-08-24 09:39:03 +02:00
vhaudiquet a3cbc87627 debian/deps: native dependency grammar and build-dep checking
Replace dpkg-checkbuilddeps with a native implementation:

- full dependency grammar: comma clauses, | alternatives, << <= = >= >>
  relations, :arch qualifiers (any/native/specific), [arch lists] and
  <profile restriction> formulas per alternative;
- restriction reduction against active build profiles and the host arch
  at parse time (reduce_restrictions semantics);
- evaluation against a parsed dpkg status database with Multi-Arch
  semantics (foreign/allowed) and versioned Provides rules (unversioned
  provides never satisfy versioned deps; versioned ones must satisfy the
  relation);
- clause simplification with implication-based deduplication, rendering
  dpkg-compatible 'unmet build dependencies/conflicts' diagnostics.

check_build_depends() consumes debian/control + CheckOpts (-A/-B/-I
equivalents). run_source_build performs the check when forced (-D
parity); source-only builds skip it entirely like dpkg-buildpackage,
and unsatisfied deps propagate as UnmetBuildDependencies -> exit 3.

Unit tests port the Dpkg_Deps.t reduction matrices; differential gate
runs 24 scenarios (alternatives, versions, arch/profile restrictions,
Multi-Arch, Provides, conflicts, -A/-B flags) against real
dpkg-checkbuilddeps comparing exit status and diagnostics.
2026-08-24 09:11:10 +02:00
vhaudiquet f3be8f8ba5 debian/version: dpkg-compatible version comparison
Implement the documented dpkg ordering algorithm (Debian Policy 5.6.1):
numeric epoch, then upstream/revision compared as alternating non-digit
and digit chunks, with '~' ordering before anything including the empty
chunk and letters before non-letters in non-digit chunks.

Adds Ord/PartialOrd for DebianVersion, a free compare() and a
later_than() convenience.

Unit tests port all vectors from dpkg's scripts/t/Dpkg_Version.t plus
Ubuntu-flavored cases (security uploads, ~ppa1 backports). Differential
gate: every vector cross-checked against real 'dpkg --compare-versions'
for <<, <=, =, >= and >>.
2026-08-23 21:57:56 +02:00
vhaudiquet 01c05f04a3 debian/arch: native architecture tables replacing dpkg-architecture
Embed dpkg's factual cputable/ostable/tupletable/abitable data and
implement tuple/triplet/multiarch lookups, wildcard matching, arch
restriction evaluation and the full DEB_BUILD_*/DEB_HOST_*/DEB_TARGET_*
environment dump natively.

build/env.rs::arch_env now delegates to the native implementation
instead of shelling out to 'dpkg-architecture -f'.

Differential gate (build/mod.rs): arch_env(Some(a)) must equal real
'dpkg-architecture -f -a a' key-for-key for every architecture listed by
'dpkg-architecture -L', plus the native case. Data tables carry upstream
attribution comments; no dpkg code was transliterated.
2026-08-23 21:53:09 +02:00
14 changed files with 4006 additions and 80 deletions
+328
View File
@@ -0,0 +1,328 @@
# Native build pipeline — replacing the `dpkg-buildpackage` shell-out
Status: **Phase 0 + Phase 1 (source builds) implemented** — see §11
Reference codebase: https://salsa.debian.org/dpkg-team/dpkg (`main` branch, analyzed 2026-08)
---
## 0. TL;DR — honest take
| Scope | Verdict | Effort | Risk |
|---|---|---|---|
| **A.** Re-implement the *orchestrator* ([`dpkg-buildpackage.pl`](https://salsa.debian.org/dpkg-team/dpkg/-/blob/main/scripts/dpkg-buildpackage.pl), 1209 lines of Perl) natively in Rust, still invoking `dpkg-source`, `debian/rules`, etc. as subprocesses | **Doable, worth it** | ~2–4 weeks | Low–medium |
| **B.** Additionally replace the cheap satellite tools natively (`dpkg-parsechangelog`, `dpkg-checkbuilddeps`, `dpkg-architecture`, `dpkg-genchanges`, `dpkg-genbuildinfo`) | Doable incrementally | +1–2 weeks each | Medium |
| **C.** Replace `dpkg-source` (tarball/diff generation, quilt integration, `.dsc` assembly) | **Not recommended** | 2–3+ months, then endless edge cases | High |
The key insight: `dpkg-buildpackage` itself is a thin (~1200-line) Perl *sequencer*. The real complexity lives in `dpkg-source` (~4000 lines of Perl across `Dpkg/Source/*`). A source build fundamentally requires `dpkg-source`-class functionality (orig tarballs, debian diffs, `.dsc` assembly), so "no dpkg tools at all" is not a realistic goal — but "no `dpkg-buildpackage`, full control of the pipeline" absolutely is, and it unlocks things pkh currently cannot do (live UI on source builds, `.changes`/`.buildinfo` generation on binary builds, unified error classification).
---
## 1. Where pkh stands today
Two divergent build paths:
### Source builds — [`build_source_package()`](../src/build/mod.rs)
Shelled out on the host:
```
dpkg-buildpackage -S -I -i -nc -d [--sign-keyid=<id> | --no-sign]
```
Problems:
- Requires `dpkg-dev` on the host.
- Output bypasses the live UI entirely ([`DebUi`](../src/ui/deb.rs) / [`LineSink`](../src/context/mod.rs)) — no phase tracking, no log classification, unlike binary builds.
- No control over failure semantics beyond exit status.
### Binary builds — [`deb::local::build()`](../src/deb/local.rs)
Already does **not** use `dpkg-buildpackage`. Hand-rolled sequence: ephemeral unshare chroot → `apt-get update` → essentials → manual `quilt push -a` → `apt-get build-dep` → `debian/rules build` → `fakeroot debian/rules binary` → retrieve `.deb`s.
What it silently skips compared to a real build:
- `debian/rules clean` before building,
- `dpkg-source --before-build/--after-build` lifecycle (it re-implements patch application crudely),
- `.buildinfo` and `.changes` generation,
- build-conflicts checking, `Rules-Requires-Root` semantics,
- `SOURCE_DATE_EPOCH` export (reproducibility).
So pkh already pays part of the "native orchestrator" cost without getting its benefits. A native implementation would unify both paths.
### Existing native assets to reuse
- Changelog header/footer parsing — [`changelog.rs`](../src/changelog.rs) (needs extension to full metadata: timestamp, urgency, trailer key/values).
- GPG key discovery — [`utils/gpg.rs`](../src/utils/gpg.rs) via `gpgme`.
- Hashing crates already present: `sha2`, `md-5` (need to add `sha1`); compression: `flate2`, `xz2`; archives: `tar`.
- Command execution through remote-capable contexts — [`ContextCommand`](../src/context/mod.rs) (local/ssh/schroot/unshare/capture).
- Live UI phases — [`ui/deb.rs`](../src/ui/deb.rs).
---
## 2. What `dpkg-buildpackage` actually is (measured)
It is a Perl script (`use v5.36`). Measured sizes on `main`:
| Component | Lines | Role |
|---|---|---|
| `scripts/dpkg-buildpackage.pl` | 1209 | Orchestrator |
| `scripts/dpkg-source.pl` | 806 | Driver; work in `Dpkg/Source/*` |
| `scripts/dpkg-genbuildinfo.pl` | 637 | `.buildinfo` generation |
| `scripts/dpkg-genchanges.pl` | 626 | `.changes` generation |
| `scripts/dpkg-architecture.pl` | 498 | Arch name/triplet tables |
| `scripts/dpkg-gensymbols.pl` | 398 | (called by rules, not by us) |
| `scripts/dpkg-checkbuilddeps.pl` | 270 | Dep checking vs status file |
| `scripts/dpkg-parsechangelog.pl` | 195 | Changelog CLI wrapper |
| `scripts/dpkg-distaddfile.pl` | 99 | Registers files in `debian/files` |
| `Dpkg/*.pm` + `Dpkg/Source/*.pm` (subset examined) | ~11,600 | Shared library code |
Total relevant Perl surface ≈ **16k LOC**, of which the orchestrator is only ~8%.
---
## 3. The exact pipeline (verified against source)
What `dpkg-buildpackage` does, in order:
1. **Config**: load `buildpackage.conf` (Dpkg::Conf), inject as leading argv.
2. **Option parsing** (~200 lines): build types `-F/-g/-G/-b/-B/-A/-S` (+ `--build=full,source,binary,any,all`), signing (`-us/-uc/-ui/-k/-p/--no-sign/--force-sign`), `-j/-J/--jobs-force`, `-r<root-cmd>`, `-R<rules>`, `-T<targets>`, `-a/-t/--target-arch`, `-P<profiles>`, `-d/-D`, `-nc/-tc`, hooks `--hook-<name>=<cmd>`, passthrough buckets for `dpkg-source` / `dpkg-genchanges` / `dpkg-genbuildinfo`.
3. **Build type → rules targets**: `binary`→`build`+`binary`; arch-dep only→`build-arch`+`binary-arch`; indep only→`build-indep`+`binary-indep`.
4. **Implied flags**: `-nc` alone implies `-b`; `-nc -S` disables build-dep checks.
5. **Environment prep**:
- `parallel=auto` default → `DEB_BUILD_OPTIONS=parallel=N` exported;
- forced jobs additionally appended to `MAKEFLAGS`;
- `DEB_BUILD_PROFILES` exported if `-P`;
- optional `.dsc` input → `dpkg-source --extract` first;
- `SOURCE_DATE_EPOCH ||= changelog timestamp || time()` (reproducible-builds.org spec);
- full env dump of `dpkg-architecture -f [-a…][-t…]` imported into the environment (all `DEB_BUILD_*`, `DEB_HOST_*`, `DEB_TARGET_*` incl. `*_OS/CPU/MULTIARCH/GNU_TYPE/ARCH_BITS/ENDIAN`);
- OpenPGP key resolution: `--sign-keyfile` > `--sign-keyid` > maintainer userid; secrets probed up-front; **UNRELEASED distribution ⇒ signing disabled** unless `--force-sign`.
6. **Pre-flight**: `Dpkg::BuildDriver->pre_check()` (rules file exists/executable); `dpkg-source --before-build .` (applies patches for quilt formats); `dpkg-checkbuilddeps [-A|-B|-I]` unless `-d` (exit 3 on unmet).
7. **Hooks** at 12 points: `preinit init preclean source build binary buildinfo changes postclean check sign done`, with `%p/%v/%s/%u/%a` substitution.
8. **Preclean**: `debian/rules clean` via BuildDriver (gain-root per `Rules-Requires-Root`).
9. **Source build** (if any SOURCE component): `dpkg-source -b .` → `.dsc` + tarballs in `..`.
10. **Binary build** (if any BINARY component): `run_build_task(build-target)` then `binary-target` through BuildDriver (RRR-aware root command; skips separate non-root `build` pass when running rootless).
11. **Metadata generation**:
- `dpkg-genbuildinfo` → `../<pkg>_<ver_noepoch>_<arch>.buildinfo` (Format 1.0; records `Installed-Build-Depends` snapshot from the dpkg status DB, build environment, checksums);
- `dpkg-genchanges` → `../<pkg>_<ver_noepoch>_<arch>.changes` (Format 1.8; aggregates `debian/files` + changelog + control).
12. **Post**: optional `-tc` clean; `dpkg-source --after-build .` (unapplies patches it applied); human summary ("full upload (original source is included)" etc. derived from the `Files` field); optional lintian-style check command.
13. **Signing cascade** (inline/clearsig via OpenPGP backend gpg|sequoia|sop):
- sign `<pkg>_<ver>.dsc` → recompute its checksums **inside `.buildinfo`**;
- sign `.buildinfo` → recompute dsc+buildinfo checksums **inside `.changes`** (rewriting both `Checksums-*` and legacy `Files` entries);
- sign `.changes`.
Exit codes matter: e.g. unsatisfied build-deps ⇒ exit 3.
---
## 4. Sub-tool inventory and replacement strategy
| Tool | Used for | Complexity to replace natively | Strategy |
|---|---|---|---|
| `dpkg-parsechangelog` | source/version/maintainer/distribution/timestamp | **Low** — documented format; crates exist (`debian-changelog`, `deb822-parser` ecosystem) | Replaced (see §11, [`metadata.rs`](../src/build/metadata.rs)) |
| `dpkg-version` compare | epoch/upstream/revision ordering | **Low** — small well-specified algorithm; crate `debversion` | **Replaced** (§11, [`debian/version.rs`](../src/debian/version.rs): `Ord`/`compare`/`later_than`) |
| `dpkg-architecture` | arch ↔ triplet tables, multiarch tuple, env dump | **Low-medium** — embed cputable/ostable/tupletable/abitable data (stable for years) | **Replaced** (§11, [`debian/arch.rs`](../src/debian/arch.rs)) |
| `dpkg-checkbuilddeps` | deps vs installed status | **Medium** — `Dpkg::Deps` grammar (alternatives, arch qualifiers, `<profiles>` restrictions, versioned Provides subtleties, Multi-Arch facts) + status-file scan | **Replaced** (§11, [`debian/deps.rs`](../src/debian/deps.rs); wired into the pipeline behind `-D`, source-only builds skip it like dpkg-buildpackage) |
| `dpkg-genbuildinfo` | `.buildinfo` | **Medium** — deb822 emit + status snapshot + checksums | Native (see §11, [`buildinfo.rs`](../src/build/buildinfo.rs)) |
| `dpkg-genchanges` | `.changes` | **Medium** — deb822 emit + `debian/files` consumption + `.deb` control extraction (ar+tar, trivial with crates) | **Replaced** for source (§11) and binary uploads (§11, [`build/binary.rs`](../src/build/binary.rs)) |
| `dpkg-genbuildinfo` (binary) | `.buildinfo` for `-b` builds | **Medium** — deb822 emit + in-context artifact hashing | **Replaced** (§11, [`build/binary.rs`](../src/build/binary.rs), wired into [`deb/local.rs`](../src/deb/local.rs)) |
| `dpkg-distaddfile`/`debian/files` protocol | build outputs registry | **Trivial** — one append-only line format | Native ([`files.rs`](../src/build/files.rs)) |
| OpenPGP signing | inline clearsign of dsc/buildinfo/changes | **Low** — `gpgme` (already a dependency) supports clearsigning | Native ([`sign.rs`](../src/build/sign.rs)) |
| `dpkg-source` | orig tarball, debian diff, patches, `.dsc` | **Very high** — V1/V2/quilt/native formats, byte-exact tar normalization, quilt bookkeeping, `--include-binaries`, hundreds of validation warnings | **Keep as subprocess** (see §6) |
| `debian/rules` execution | the actual build | N/A (foreign code) | Keep, via `ContextCommand` |
---
## 5. Feasibility detail per scope
### Scope A — native orchestrator (recommended)
The 1209-line script decomposes into clean Rust pieces:
```
src/build/
├── mod.rs // pipeline driver (the equivalent of main())
├── buildtype.rs // BUILD_SOURCE|ARCH_DEP|ARCH_INDEP bitflags + target mapping
├── metadata.rs // changelog + control resolution, version splitting
├── env.rs // SOURCE_DATE_EPOCH, DEB_BUILD_OPTIONS, arch env dump
├── buildinfo.rs // native .buildinfo writer
├── changes.rs // native .changes writer
├── files.rs // debian/files registry emulation
├── checksums.rs // md5/sha1/sha256 registry
├── control.rs // deb822 parser/writer
└── sign.rs // gpgme clearsign + post-sign checksum cascade
```
Estimated ~1.5–3 kLOC. Everything is deterministic file munging + subprocess sequencing — no daemons, no parsing of arbitrary upstream code (that's `dpkg-source`'s job, which we keep).
**pkh-specific simplifications** (legitimate because pkh controls the environment):
- Builds run as **real root** inside ephemeral unshare chroots ⇒ the entire gain-root/fakeroot matrix collapses: run `debian/rules` directly when the context is root (keep `fakeroot` fallback for host-side source builds).
- pkh's option surface is a fraction of dpkg's: `-S/-b/-B/-A/-g/-G`, `-us/-uc`, `-k`, `-j`, `-a`, `-P`, `-d/-D`, `-nc/-tc`, `-R`, `-T` cover everything pkh passes today plus obvious headroom. Hooks and `--hook-*` can be dropped initially.
- Vendor hooks (`run_vendor_hook`) are rarely used in the build path; Ubuntu/Debian differences pkh cares about are already handled via [`distro_info.yml`](../distro_info.yml).
**What this buys pkh concretely:**
1. Live UI + log classification for source builds (today's `-S` path is a black box).
2. `.changes`/`.buildinfo` for binary builds — currently missing entirely; needed for uploads/PPA submissions and lintian checks.
3. Correct `dpkg-source --before-build/--after-build` lifecycle replacing the manual `quilt push -a` hack in [`local.rs`](../src/deb/local.rs) (handles `3.0 (quilt)` properly, incl. unapply-on-exit and format detection instead of sniffing `debian/patches/series`).
4. `SOURCE_DATE_EPOCH` reproducibility for free.
5. No host `dpkg-dev` requirement for orchestration decisions; `dpkg-source` still needed inside build environments where pkh already installs packages anyway.
### Scope B — satellite tools
Incremental, each independently testable against the real tool (differential testing). Crates from the `rust-debian-*` ecosystem (maintained by Jelmer Vernooij) cover most parsing: `deb822-parser`, `debian-control`, `debian-changelog`, `debversion`. Priority order if pursued: version compare → changelog → architecture tables → checkbuilddeps.
### Scope C — `dpkg-source`
The honest numbers: `Dpkg/Source/Package/V2.pm` alone is 847 lines, V1 is 599, plus Archive/Quilt/Functions/BinaryFiles modules, GNU diff generation, and — critically — **byte-level tar normalization** (mtime/uid/gid/mode canonicalization, pax header handling) that reproducible builds depend on. Parity means being bug-compatible with dpkg against ~40k archive source packages. This is a multi-month project with a long tail, and it buys pkh almost nothing since `dpkg-source` is guaranteed present inside the very chroots pkh creates. **Do not do this.**
---
## 6. Proposed phasing
### Phase 0 — quick win (days)
Route the existing `dpkg-buildpackage -S` call through the UI capture machinery (like [`cap()`](../src/deb/local.rs) does for binary builds): phase display + `LineSink` classification. No behavior change otherwise.
### Phase 1 — native orchestrator (2–4 weeks)
Implement §5 scope A. Both entry points converge:
- `pkh build -S` → native pipeline, `dpkg-source -b` subprocess, native signing.
- `pkh build` (binary) → same pipeline skeleton inside the ephemeral context: preclean → before-build → dep check → `rules build` → `rules binary[-arch|-indep]` → native buildinfo/changes → after-build. Deletes the manual quilt logic.
Deliverables:
- `src/build/*` module tree (§5).
- Native `.buildinfo` (Format 1.0) and `.changes` (Format 1.8) writers emitting exactly the field sets observed in `dpkg-genbuildinfo.pl` / `dpkg-genchanges.pl`, in dpkg's canonical field order:
- buildinfo: `Format, Source, Binary, Architecture, Version, Binary-Only-Changes, Checksums-Md5/Shа1/Sha256, Build-Origin, Build-Architecture, Build-Kernel-Version(opt), Build-Date, Build-Path(opt), Build-Tainted-By(opt), Installed-Build-Depends, Environment`
- changes: `Format, Date, Source, Binary, Built-For-Profiles, Architecture, Version, Distribution, Urgency, Maintainer, Changed-By, Description, Changes, Checksums-Sha1/Sha256, Files`
- Post-signature checksum cascade implemented exactly as dpkg does (sign dsc → patch buildinfo checksums → sign buildinfo → patch changes `Files`+`Checksums-*` → sign changes).
- Differential test harness (§8).
### Phase 2 — satellite replacement (optional, incremental)
Swap subprocesses for native implementations, one tool at a time, gated by differential tests. Start with version-compare + changelog (already half-present in pkh).
### Explicitly out of scope
`dpkg-source` internals, `dpkg-deb` packing, apt resolution (pkh correctly delegates to `apt-get build-dep` + dose3 for explanations already).
---
## 7. Tricky details to get right (gotchas list)
1. **Version splitting**: `<epoch:>upstream<-revision>`; `.dsc`/tarball names use *upstream* portion without epoch; `.changes` name uses version **without epoch** but **with revision** (`$sversion` in the script).
2. **`-nc` implication chain**: `-nc` ⇒ binary build implied; `-nc -S` ⇒ no dep check.
3. **UNRELEASED** ⇒ auto-disable all signing (warn), `--force-sign` overrides.
4. **Signing invalidates checksums transitively** (dsc → buildinfo → changes); get the cascade order right or archive tools reject the upload.
5. **`debian/files`** is the contract between `debian/rules` (via `dh_builddeb`/`dpkg-gencontrol`/`dpkg-distaddfile`) and the changes generator: lines of `filename section priority [key=value...]`.
6. **Arch selection for names**: `arch` suffix in artifact filenames is `host-arch` for arch-dep builds, `all` for indep-only, `source` for source-only.
7. **`dpkg-source --before-build` must run even for binary-only builds** (patch application), and `--after-build` at the end — this replaces pkh's current manual quilt step and fixes `3.0 (quilt)` correctness.
8. **Environment parity**: `dpkg-architecture -f` dump must be imported wholesale (not cherry-picked) — packages test `DEB_HOST_GNU_TYPE`, `DEB_BUILD_MULTIARCH`, `DEB_BUILD_ARCH_ENDIAN`, etc. If embedding tables later, mirror the full variable set.
9. **Exit codes**: preserve dpkg conventions (3 = unmet build-deps) so wrappers/scripts behave identically.
10. **Locale**: dpkg sets `LANG=C`-ish determinism for subprocesses — pkh already does this in [`local.rs`](../src/deb/local.rs); keep for all pipeline steps.
11. **Multiline field rendering**: values starting with `\n` render as `Field:` + indented continuation lines (no inline first line, no trailing space) — this is how dpkg emits `Changes`, `Files`, `Installed-Build-Depends`, `Environment`.
12. **Artifact ordering** in `Checksums-*`/`Files` is insertion order (dsc → tarballs in dsc-field order → debs → buildinfo), not alphabetical.
---
## 8. Testing strategy
- **Differential harness**: run real `dpkg-buildpackage` and the native pipeline over a corpus (packages covering: `1.0` non-native, `3.0 (quilt)` with/without patches, native, binaries-only, indep-only, cross, RRR variants, UNRELEASED) and diff artifacts modulo timestamps/signatures.
- **Port unit cases** from dpkg's own `t/` tests for version compare, deps parsing, changelog parsing.
- **Golden-file tests** for `.changes`/`.buildinfo` writers.
- Pin the reference dpkg version in CI commentary (behavior drift across dpkg releases is the main maintenance cost).
---
## 9. Risks
| Risk | Mitigation |
|---|---|
| Behavior drift vs future dpkg releases (new fields, format bumps) | Differential tests pinned to a reference version; changes/buildinfo formats are extremely stable (1.8 / 1.0 for years) |
| Archive/upload tooling rejects our `.changes`/`.buildinfo` | Validate with `lintian` + a real PPA upload early in Phase 1 |
| License contamination | dpkg is GPL-2+. Write from documented behavior/format specs and observation, **do not transliterate Perl**; alternatively accept GPL for pkh (currently no license field in [`Cargo.toml`](../Cargo.toml) — decision needed) |
| Scope creep toward Scope C | Hard rule: `dpkg-source` stays a subprocess |
| Remote-context divergence | All tree-touching steps go through [`ContextCommand`](../src/context/mod.rs); metadata steps operate on locally-synced copies like [`changelog.rs`](../src/changelog.rs) already does |
---
## 10. Effort summary
| Item | Estimate |
|---|---|
| Phase 0 (UI capture for `-S`) | 1–2 days |
| Phase 1 (native orchestrator + buildinfo/changes/signing) | 2–4 weeks |
| Phase 2 per satellite tool | 3 days – 2 weeks each |
| Scope C (`dpkg-source`) | 2–3+ months — rejected |
Bottom line: **yes, it's doable — for the orchestrator.** Treat `dpkg-source` as a permanent subprocess dependency, and the task becomes a well-bounded, high-value refactor that also fixes real gaps in pkh's binary path.
---
## 11. Implementation status (Phase 1 — source builds)
Landed in `src/build/` (~2.8 kLOC incl. tests), wired behind the historical
entry point [`build_source_package()`](../src/build/mod.rs) so `pkh build`
now runs the native pipeline:
| Module | Role |
|---|---|
| [`control.rs`](../src/build/control.rs) | deb822 paragraph parser/writer (dpkg-compatible multiline rendering) |
| [`checksums.rs`](../src/build/checksums.rs) | md5/sha1/sha256 registry, insertion-ordered like dpkg's artifact accumulation |
| [`metadata.rs`](../src/build/metadata.rs) | version splitting/validation, full changelog entry parse (incl. binNMU `binary-only`), control info |
| [`buildtype.rs`](../src/build/buildtype.rs) | build-type bitflags + rules-target/artifact-suffix mapping |
| [`env.rs`](../src/build/env.rs) | `SOURCE_DATE_EPOCH`, `DEB_BUILD_OPTIONS`, `dpkg-architecture` env dump, vendor/profiles, sanitized `Environment` field |
| [`files.rs`](../src/build/files.rs) | `debian/files` registry (parse/atomic save) |
| [`buildinfo.rs`](../src/build/buildinfo.rs) | `.buildinfo` writer + `Installed-Build-Depends` closure over the dpkg status DB |
| [`changes.rs`](../src/build/changes.rs) | `.changes` writer (canonical field order, legacy `Files` + `Checksums-Sha1/256`) |
| [`sign.rs`](../src/build/sign.rs) | gpgme clearsigning + key-id validation |
Still delegated to subprocesses: `dpkg-source -b/--before-build/--after-build`
(by design, see §5 Scope C).
**Differential validation** — automated in
[`build/mod.rs`](../src/build/mod.rs) (`mod differential_tests`, runs by
default with `cargo test --lib`):
- a corpus of 12 synthetic fixtures covering: native/quilt/1.0 formats,
epochs, `~` pre-releases, Ubuntu/Debian series (focal, noble, jammy,
trixie, unstable), high urgency, multiple binary stanzas, one/two quilt
patches, binNMU (`binary-only=yes` + previous-entry metadata),
`(Closes: #…)` extraction, UNRELEASED no-sign, extra source-stanza fields;
- each case builds the tree twice (`cp -a` copies) — once with real
`dpkg-buildpackage -S -I -i -nc -d --no-sign`, once with the native
pipeline — then compares:
- `.dsc` payload byte-for-byte,
- `.changes` field-by-field (checksum lines of the `.buildinfo` itself
excluded),
- `.buildinfo` structure (machine-dependent fields excluded);
- real **archive packages** are pulled with pkh's own
[`pull`](../src/pull.rs) (archive download mode) and compared the same
way, via [`differential_real_archive_package()`](../src/build/mod.rs)
which takes `(package, dist, series)`:
- always-on CI tests: `hello` @ ubuntu/noble, `dosfstools` @
debian/trixie, `sl` @ ubuntu/focal;
- an additional `#[ignore]`-gated test for ad-hoc broad runs:
```text
PKH_DIFF_PACKAGES="bash coreutils curl" PKH_DIFF_DIST=ubuntu \
PKH_DIFF_SERIES=noble cargo test --lib \
differential_real_archive_packages -- --ignored
```
(requires network access).
Manual validation additionally confirmed: signed builds verify with
`gpg --verify` on all three artifacts; the patch apply/unapply lifecycle is
correct for quilt formats.
**Known divergences** (documented, all informational fields): no
`Build-Tainted-By` (vendor hook), no `dpkg-buildflags` origin tracking in the
`Environment` field, vendor default profiles approximated
(`derivative.ubuntu noudeb` for Ubuntu).
## 12. Implementation status (Phase 2 — satellite tools)
All four satellite replacements landed, each gated by differential tests
against the real tool:
| Work item | Module | Differential gate |
|---|---|---|
| WI-1 `dpkg-architecture` | [`debian/arch.rs`](../src/debian/arch.rs) (tables + lookups), wired into [`build/env.rs`](../src/build/env.rs) | `arch_env(Some(a))` equals real `dpkg-architecture -f -a a` key-for-key for **every** arch from `dpkg-architecture -L`, plus native (`build/mod.rs::diff_arch_env_*`) |
| WI-2 version compare | [`debian/version.rs`](../src/debian/version.rs) (`Ord`, `compare`, `later_than`) | all vectors from dpkg `scripts/t/Dpkg_Version.t` + Ubuntu-flavored cases, cross-checked against `dpkg --compare-versions` for `<< <= = >= >>` (`diff_version_compare_against_dpkg`) |
| WI-3 `dpkg-checkbuilddeps` | [`debian/deps.rs`](../src/debian/deps.rs) (grammar, restriction reduction, KnownFacts evaluation, `check_build_depends`) | 24 scenarios vs real `dpkg-checkbuilddeps` (alternatives, versions, arch/profile restrictions, Multi-Arch, versioned Provides, conflicts, `-A`/`-B`) comparing exit status + diagnostics (`diff_checkbuilddeps_matrix`); unit tests port the `Dpkg_Deps.t` reduction matrices. Wired into `run_source_build` behind `-D` parity: source-only builds skip the check like `dpkg-buildpackage`, unsatisfied deps exit 3 |
| WI-4 binary `.buildinfo`/`.changes` | [`build/binary.rs`](../src/build/binary.rs) (context-generic generation), wired into [`deb/local.rs`](../src/deb/local.rs); artifact retrieval extended in [`deb/mod.rs`](../src/deb/mod.rs) | same tree built with real `dpkg-buildpackage -b` and with the pkh flow (rules build/binary + native metadata through a local context): `.changes`/`.buildinfo` compared field-by-field modulo machine-dependent fields, artifact checksums included (`diff_binary_build_metadata`) |
Binary-flow notes: `SOURCE_DATE_EPOCH` is now exported to the rules
environment (reproducibility); digests are computed inside the context via
coreutils so remote/chrooted trees work; binNMU binary builds set
`Source: pkg (prev)` / `Binary-Only-Changes` and redistribute the previous
`.dsc` when present; `Architecture` is encounter-ordered in `.changes`
(sorted in `.buildinfo`, matching dpkg).
Still delegated to subprocesses: `dpkg-source` and `debian/rules` (by
design, see §5 Scope C).
+475
View File
@@ -0,0 +1,475 @@
//! Binary-build metadata generation: native `.buildinfo` / `.changes`
//! production for binary-only builds (`pkh deb`), the equivalent of
//! `dpkg-genbuildinfo -b` + `dpkg-genchanges -b`.
//!
//! All tree/database access goes through a [`Context`] so the generation can
//! run against a build tree living in a local directory, an ephemeral
//! chroot or a remote host. Artifact digests are computed inside the context
//! with coreutils (`md5sum`, `sha1sum`, `sha256sum`, `stat`), keeping the
//! flow binary-safe regardless of the transport.
use std::collections::BTreeMap;
use std::error::Error;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use crate::context::Context;
use crate::debian::{
ChecksumEntry, ControlInfo, FileChecksums, FilesList, parse_changelog_entry_from_str,
};
/// Digests of one artifact.
#[derive(Debug, Clone, Default)]
struct ArtifactHashes {
size: u64,
md5: String,
sha1: String,
sha256: String,
}
/// Options driving binary metadata generation.
#[derive(Debug, Clone)]
pub struct BinaryMetadataOptions {
/// Active build profiles (`Built-For-Profiles`).
pub profiles: Vec<String>,
/// Vendor name (`Build-Origin`).
pub vendor: String,
/// Parallel job count advertised in `DEB_BUILD_OPTIONS`.
pub parallel: usize,
/// Reproducible-builds epoch exported to the build.
pub source_date_epoch: i64,
/// Build architecture (the machine inside the build context).
pub build_arch: String,
/// Host architecture (the packages' target); equals the build
/// architecture except for cross builds.
pub host_arch: String,
}
/// Generate `<pkg>_<ver>_<arch>.buildinfo` and `.changes` for a finished
/// binary build, consuming `debian/files` from `package_dir` and the
/// artifacts sitting in `upload_dir`. Returns both paths (inside the
/// context).
///
/// Mirrors the observable behavior of `dpkg-genbuildinfo -b` and
/// `dpkg-genchanges -b`: sorted `Binary` list, encounter-order `Architecture`
/// accumulation, sorted `Description` lines formatted like dpkg, `.buildinfo`
/// registration in `debian/files`, and binary-NMU handling (`Source:
/// pkg (prev)` + previous `.dsc` redistribution when present).
pub fn generate_binary_metadata(
ctx: &Arc<Context>,
package_dir: &Path,
upload_dir: &Path,
opts: &BinaryMetadataOptions,
) -> Result<(PathBuf, PathBuf), Box<dyn Error>> {
// ------------------------------------------------------------------
// Metadata sources inside the context
// ------------------------------------------------------------------
let changelog_content = ctx.read_file(&package_dir.join("debian/changelog"))?;
let entry = parse_changelog_entry_from_str(&changelog_content)?;
let control_content = ctx.read_file(&package_dir.join("debian/control"))?;
let control = ControlInfo::parse_content(&control_content)?;
let files_content = ctx
.read_file(&package_dir.join("debian/files"))
.unwrap_or_default();
let mut files_list = FilesList::parse(&files_content)?;
// ------------------------------------------------------------------
// Collect binary artifacts registered in debian/files
// ------------------------------------------------------------------
let artifact_names: Vec<String> = files_list
.iter()
.filter(|e| matches!(e.package_type.as_deref(), Some("deb") | Some("udeb")))
.map(|e| e.filename.clone())
.collect();
if artifact_names.is_empty() {
return Err("binary build with no binary artifacts found; cannot distribute".into());
}
let mut hashes = hashes_in_context(ctx, upload_dir, &artifact_names)?;
let mut checksums = FileChecksums::new();
let mut arch_values: Vec<String> = Vec::new();
let mut arch_seen = std::collections::HashSet::new();
for name in &artifact_names {
let entry_hashes = hashes
.remove(name)
.ok_or_else(|| format!("artifact '{name}' listed in debian/files but not found"))?;
checksums.insert_entry(
name,
ChecksumEntry {
size: entry_hashes.size,
md5: entry_hashes.md5,
sha1: entry_hashes.sha1,
sha256: entry_hashes.sha256,
},
);
// Architecture accumulation in encounter order (dpkg-genchanges).
if let Some(file_entry) = files_list.get(name)
&& let Some(arch) = file_entry
.arch
.as_ref()
.or_else(|| file_entry.attrs.get("architecture"))
&& arch_seen.insert(arch.clone())
{
arch_values.push(arch.clone());
}
}
// ------------------------------------------------------------------
// Binary-NMU: redistribute the previous source when present
// ------------------------------------------------------------------
let sversion = entry.version.no_epoch();
let mut source_display = entry.source.clone();
let mut binary_only_changes = None;
if entry.binary_only
&& let Ok(prev_entry) = crate::debian::changelog::parse_previous_version_from_str(
&ctx.read_file(&package_dir.join("debian/changelog"))?,
)
&& let Some(prev) = prev_entry
{
source_display = format!("{} ({})", entry.source, prev);
binary_only_changes = Some(format!(
"{}\n\n -- {} <{}> {}",
entry.changes_field, entry.maintainer_name, entry.maintainer_email, entry.date_raw
));
let prev_version = crate::debian::DebianVersion::parse(&prev)?;
let dsc_name = format!("{}_{}.dsc", entry.source, prev_version.no_epoch());
let dsc_path = upload_dir.join(&dsc_name);
if ctx.exists(&dsc_path)? {
include_dsc_artifacts(ctx, upload_dir, &dsc_name, &mut checksums)?;
}
}
// ------------------------------------------------------------------
// Binary package names and descriptions
// ------------------------------------------------------------------
let mut binaries: Vec<String> = Vec::new();
for name in &artifact_names {
if let Some(e) = files_list.get(name)
&& let Some(pkg) = &e.package
&& !binaries.contains(pkg)
{
binaries.push(pkg.clone());
}
}
binaries.sort();
// Description lines: first line of each binary stanza's Description,
// formatted exactly like dpkg-genchanges, sorted.
let mut descriptions = Vec::new();
for stanza in &control.binaries {
let Some(pkg) = stanza.get("Package") else {
continue;
};
if !binaries.contains(&pkg.to_string()) {
continue;
}
let summary = stanza
.get("Description")
.unwrap_or("no description available")
.lines()
.next()
.unwrap_or("no description available");
// Package-Type overrides the artifact-derived type (deb default).
let pkg_type = stanza
.get("Package-Type")
.map(str::to_string)
.unwrap_or_else(|| {
files_list
.iter()
.find(|f| f.package.as_deref() == Some(pkg))
.and_then(|f| f.package_type.clone())
.unwrap_or_else(|| "deb".to_string())
});
descriptions.push(crate::build::changes::format_description(
pkg, &pkg_type, summary,
));
}
descriptions.sort();
// ------------------------------------------------------------------
// Installed-Build-Depends closure over the context status database
// ------------------------------------------------------------------
let status_content = ctx
.read_file(Path::new("/var/lib/dpkg/status"))
.unwrap_or_default();
let bd_fields = [
control.source.get("Build-Depends").unwrap_or(""),
control.source.get("Build-Depends-Arch").unwrap_or(""),
control.source.get("Build-Depends-Indep").unwrap_or(""),
];
let installed_build_depends =
crate::build::buildinfo::installed_build_depends_from_content(&status_content, &bd_fields)?;
// ------------------------------------------------------------------
// .buildinfo generation, then registration in debian/files
// ------------------------------------------------------------------
let pipeline_env = pipeline_environment(opts);
let environment = crate::build::env::buildinfo_environment(&pipeline_env);
// dpkg-genbuildinfo sorts the accumulated architecture values, while
// dpkg-genchanges keeps encounter order.
let mut buildinfo_arch_values = arch_values.clone();
buildinfo_arch_values.sort();
let buildinfo_name = format!("{}_{}_{}.buildinfo", entry.source, sversion, opts.host_arch);
let buildinfo_doc =
crate::build::buildinfo::render_buildinfo(&crate::build::buildinfo::BuildInfoInput {
source: source_display.clone(),
binaries: binaries.clone(),
architecture: buildinfo_arch_values.join(" "),
version: entry.version.full(),
binary_only_changes: binary_only_changes.clone(),
build_origin: opts.vendor.clone(),
build_architecture: opts.build_arch.clone(),
build_date: chrono::Local::now().to_rfc2822(),
checksums: checksums.clone(),
installed_build_depends,
environment,
});
let buildinfo_path = upload_dir.join(&buildinfo_name);
ctx.write_file(
&buildinfo_path,
&crate::debian::control::write_paragraph(&buildinfo_doc),
)?;
// Register the .buildinfo in debian/files, like dpkg-genbuildinfo does,
// so the .changes distributes it.
files_list.add(crate::debian::FilesEntry::new(
&buildinfo_name,
control.section(),
control.priority(),
));
ctx.write_file(&package_dir.join("debian/files"), &files_list.render())?;
// Hash the freshly written .buildinfo inside the context.
let buildinfo_hashes =
hashes_in_context(ctx, upload_dir, std::slice::from_ref(&buildinfo_name))?;
if let Some(h) = buildinfo_hashes.get(&buildinfo_name) {
checksums.insert_entry(
&buildinfo_name,
ChecksumEntry {
size: h.size,
md5: h.md5.clone(),
sha1: h.sha1.clone(),
sha256: h.sha256.clone(),
},
);
}
// ------------------------------------------------------------------
// .changes generation
// ------------------------------------------------------------------
let changes_name = format!("{}_{}_{}.changes", entry.source, sversion, opts.host_arch);
let changed_by = format!("{} <{}>", entry.maintainer_name, entry.maintainer_email);
let changes_doc = crate::build::changes::render_changes(&crate::build::changes::ChangesInput {
date: entry.date_raw.clone(),
source: source_display,
binaries,
built_for_profiles: opts.profiles.clone(),
architecture: arch_values.join(" "),
version: entry.version.full(),
distribution: entry.distribution.clone(),
urgency: entry.urgency.clone(),
maintainer: control.source.get("Maintainer").map(str::to_string),
changed_by: Some(changed_by),
descriptions,
closes: entry.closes.clone(),
changes_field: entry.changes_field.clone(),
checksums,
files_list,
});
let changes_path = upload_dir.join(&changes_name);
ctx.write_file(
&changes_path,
&crate::debian::control::write_paragraph(&changes_doc),
)?;
Ok((buildinfo_path, changes_path))
}
/// Environment exported to the build steps; recorded (filtered) in the
/// `.buildinfo` `Environment` field.
fn pipeline_environment(opts: &BinaryMetadataOptions) -> BTreeMap<String, String> {
let mut env = BTreeMap::new();
env.insert(
"SOURCE_DATE_EPOCH".to_string(),
opts.source_date_epoch.to_string(),
);
env.insert(
"DEB_BUILD_OPTIONS".to_string(),
format!("parallel={}", opts.parallel),
);
if !opts.profiles.is_empty() {
env.insert("DEB_BUILD_PROFILES".to_string(), opts.profiles.join(","));
}
env
}
/// Compute md5/sha1/sha256 digests and sizes for the named files inside the
/// context directory `dir`, using coreutils.
fn hashes_in_context(
ctx: &Arc<Context>,
dir: &Path,
names: &[String],
) -> Result<BTreeMap<String, ArtifactHashes>, Box<dyn Error>> {
let mut out: BTreeMap<String, ArtifactHashes> = names
.iter()
.map(|n| (n.clone(), ArtifactHashes::default()))
.collect();
// Sizes.
let output = ctx
.command("stat")
.current_dir(dir)
.arg("-c")
.arg("%s %n")
.args(names)
.output()?;
let stdout = String::from_utf8_lossy(&output.stdout);
for line in stdout.lines() {
let Some((size, name)) = line.trim().split_once(' ') else {
continue;
};
if let Some(slot) = out.get_mut(name) {
slot.size = size.parse().unwrap_or(0);
}
}
// Digests.
for (tool, field) in [
("md5sum", 0usize),
("sha1sum", 1usize),
("sha256sum", 2usize),
] {
let output = ctx
.command(tool)
.current_dir(dir)
.args(names)
.output()
.map_err(|e| format!("failed to run '{tool}' inside the build context: {e}"))?;
if !output.status.success() {
return Err(format!(
"'{tool}' failed inside the build context: {}",
String::from_utf8_lossy(&output.stderr).trim()
)
.into());
}
let stdout = String::from_utf8_lossy(&output.stdout);
for line in stdout.lines() {
let Some((digest, name)) = line.trim().split_once(" ") else {
continue;
};
let name = name.trim_start_matches('*');
if let Some(slot) = out.get_mut(name) {
match field {
0 => slot.md5 = digest.to_string(),
1 => slot.sha1 = digest.to_string(),
_ => slot.sha256 = digest.to_string(),
}
}
}
}
Ok(out)
}
/// Pull the `.dsc` checksums (and its referenced tarballs) into the
/// checksum registry, mirroring how binary-NMU uploads redistribute the
/// previous source.
fn include_dsc_artifacts(
ctx: &Arc<Context>,
upload_dir: &Path,
dsc_name: &str,
checksums: &mut FileChecksums,
) -> Result<(), Box<dyn Error>> {
let dsc_content = ctx.read_file(&upload_dir.join(dsc_name))?;
let para = crate::debian::control::parse_paragraphs(&dsc_content)
.into_iter()
.next()
.ok_or_else(|| format!("'{dsc_name}' is empty"))?;
let mut names: Vec<String> = Vec::new();
let mut partials: BTreeMap<String, PartialDscChecksums> = BTreeMap::new();
for field in ["Checksums-Sha1", "Checksums-Sha256"] {
if let Some(value) = para.get(field) {
for line in value.lines() {
let tokens: Vec<&str> = line.split_whitespace().collect();
if tokens.len() != 3 {
continue;
}
let slot = partials.entry(tokens[2].to_string()).or_default();
if field == "Checksums-Sha1" {
slot.sha1 = Some(tokens[0].to_string());
} else {
slot.sha256 = Some(tokens[0].to_string());
}
slot.size = tokens[1].parse().ok().or(slot.size);
}
}
}
if let Some(files_value) = para.get("Files") {
for line in files_value.lines() {
let tokens: Vec<&str> = line.split_whitespace().collect();
if tokens.len() >= 3 {
let slot = partials.entry(tokens[2].to_string()).or_default();
slot.md5 = Some(tokens[0].to_string());
slot.size = tokens[1].parse().ok().or(slot.size);
}
}
}
for field in ["Checksums-Sha1", "Checksums-Sha256"] {
if let Some(value) = para.get(field) {
for line in value.lines() {
if let Some(name) = line.split_whitespace().nth(2) {
names.push(name.to_string());
}
}
}
}
// The .dsc itself is hashed fresh (it may be signed/rewritten); the
// tarballs reuse the .dsc-recorded digests, like dpkg-genchanges does.
let dsc_hashes =
hashes_in_context(ctx, upload_dir, std::slice::from_ref(&dsc_name.to_string()))?;
if let Some(h) = dsc_hashes.get(dsc_name) {
checksums.insert_entry(
dsc_name,
ChecksumEntry {
size: h.size,
md5: h.md5.clone(),
sha1: h.sha1.clone(),
sha256: h.sha256.clone(),
},
);
}
for name in &names {
if name == dsc_name {
continue;
}
let p = &partials[name];
checksums.insert_entry(
name,
ChecksumEntry {
size: p.size.unwrap_or(0),
md5: p.md5.clone().unwrap_or_default(),
sha1: p.sha1.clone().unwrap_or_default(),
sha256: p.sha256.clone().unwrap_or_default(),
},
);
}
Ok(())
}
/// Partially-known checksums taken from a `.dsc` checksum field.
#[derive(Debug, Default)]
struct PartialDscChecksums {
size: Option<u64>,
md5: Option<String>,
sha1: Option<String>,
sha256: Option<String>,
}
+13 -8
View File
@@ -28,13 +28,6 @@ struct StatusDb {
} }
impl StatusDb { impl StatusDb {
/// Parse a dpkg status file (e.g. `/var/lib/dpkg/status`).
fn load(path: &Path) -> Result<StatusDb, Box<dyn std::error::Error>> {
let content = std::fs::read_to_string(path)
.map_err(|e| format!("cannot read status file '{}': {}", path.display(), e))?;
Ok(Self::from_str(&content))
}
fn from_str(content: &str) -> StatusDb { fn from_str(content: &str) -> StatusDb {
let mut db = StatusDb::default(); let mut db = StatusDb::default();
for para in parse_paragraphs(content) { for para in parse_paragraphs(content) {
@@ -132,7 +125,19 @@ pub fn installed_build_depends(
status_path: &Path, status_path: &Path,
build_depends_fields: &[&str], build_depends_fields: &[&str],
) -> Result<String, Box<dyn std::error::Error>> { ) -> Result<String, Box<dyn std::error::Error>> {
let db = StatusDb::load(status_path)?; let content = std::fs::read_to_string(status_path)
.map_err(|e| format!("cannot read status file '{}': {}", status_path.display(), e))?;
installed_build_depends_from_content(&content, build_depends_fields).map_err(|e| e.into())
}
/// Compute the `Installed-Build-Depends` value from the textual content of a
/// dpkg status database (used when the database lives in another context,
/// e.g. inside a chroot).
pub fn installed_build_depends_from_content(
status_content: &str,
build_depends_fields: &[&str],
) -> Result<String, String> {
let db = StatusDb::from_str(status_content);
let mut work: VecDeque<String> = VecDeque::new(); let mut work: VecDeque<String> = VecDeque::new();
for name in &db.essential { for name in &db.essential {
+6 -33
View File
@@ -1,10 +1,9 @@
//! Build environment setup: `SOURCE_DATE_EPOCH`, `DEB_BUILD_OPTIONS`, //! Build environment setup: `SOURCE_DATE_EPOCH`, `DEB_BUILD_OPTIONS`,
//! architecture variables (via `dpkg-architecture`) and the sanitized //! architecture variables (native `dpkg-architecture` equivalent) and the
//! environment recorded in `.buildinfo` files. //! sanitized environment recorded in `.buildinfo` files.
use std::collections::BTreeMap; use std::collections::BTreeMap;
use std::path::Path; use std::path::Path;
use std::process::Command;
/// Number of parallel jobs to advertise in `DEB_BUILD_OPTIONS`. /// Number of parallel jobs to advertise in `DEB_BUILD_OPTIONS`.
pub fn num_parallel() -> usize { pub fn num_parallel() -> usize {
@@ -41,41 +40,15 @@ pub fn build_env(
env env
} }
/// Import the full architecture variable set by running /// Import the full architecture variable set, computed natively by
/// `dpkg-architecture -f [-a <host-arch>]` and parsing its `KEY=VALUE` dump. /// [`crate::debian::arch`] (the equivalent of `dpkg-architecture -f
/// [-a <host-arch>]`).
/// ///
/// This exports all `DEB_BUILD_*`, `DEB_HOST_*` and `DEB_TARGET_*` variables /// This exports all `DEB_BUILD_*`, `DEB_HOST_*` and `DEB_TARGET_*` variables
/// (`*_ARCH`, `*_OS`, `*_CPU`, `*_MULTIARCH`, `*_GNU_TYPE`, ...), exactly as /// (`*_ARCH`, `*_OS`, `*_CPU`, `*_MULTIARCH`, `*_GNU_TYPE`, ...), exactly as
/// `dpkg-buildpackage` does. /// `dpkg-buildpackage` does.
pub fn arch_env(host_arch: Option<&str>) -> Result<BTreeMap<String, String>, String> { pub fn arch_env(host_arch: Option<&str>) -> Result<BTreeMap<String, String>, String> {
let mut cmd = Command::new("dpkg-architecture"); crate::debian::arch::arch_env(host_arch)
cmd.arg("-f");
if let Some(arch) = host_arch {
cmd.args(["--host-arch", arch]);
}
let output = cmd.output().map_err(|e| {
format!(
"failed to run 'dpkg-architecture': {}. Is 'dpkg-dev' installed?",
e
)
})?;
if !output.status.success() {
return Err(format!(
"dpkg-architecture failed with status {}: {}",
output.status,
String::from_utf8_lossy(&output.stderr).trim()
));
}
let mut env = BTreeMap::new();
for line in String::from_utf8_lossy(&output.stdout).lines() {
if let Some((key, value)) = line.split_once('=') {
env.insert(key.to_string(), value.to_string());
}
}
Ok(env)
} }
/// Read the current vendor name from `/etc/dpkg/origins/default` /// Read the current vendor name from `/etc/dpkg/origins/default`
+393 -1
View File
@@ -6,6 +6,7 @@
//! source-tree work (tarballs, diffs, patches) to `dpkg-source` as a //! source-tree work (tarballs, diffs, patches) to `dpkg-source` as a
//! subprocess. //! subprocess.
pub mod binary;
pub mod buildinfo; pub mod buildinfo;
pub mod buildtype; pub mod buildtype;
pub mod changes; pub mod changes;
@@ -28,6 +29,10 @@ pub struct SourceBuildOptions {
pub sign_keyid: Option<String>, pub sign_keyid: Option<String>,
/// Sign even for an UNRELEASED changelog (`--force-sign`). /// Sign even for an UNRELEASED changelog (`--force-sign`).
pub force_sign: bool, pub force_sign: bool,
/// Force build-dependency checking even though this is a source-only
/// build (`-D`). `dpkg-buildpackage` skips `dpkg-checkbuilddeps`
/// entirely for source-only builds unless forced.
pub force_dep_check: bool,
} }
/// Artifacts produced by a successful source build. /// Artifacts produced by a successful source build.
@@ -198,6 +203,28 @@ pub fn run_source_build(
&["-I", "-i", "--before-build", "."], &["-I", "-i", "--before-build", "."],
&pipeline_env, &pipeline_env,
)?; )?;
// Build-dependency check (native dpkg-checkbuilddeps equivalent).
// dpkg-buildpackage skips it entirely for source-only builds unless
// forced with -D; unsatisfied dependencies abort with exit status 3.
if opts.force_dep_check {
let check_opts = crate::debian::deps::CheckOpts {
host_arch: arch_vars
.get("DEB_HOST_ARCH")
.cloned()
.unwrap_or_else(|| crate::debian::arch::native().unwrap_or_default()),
build_profiles: profiles.clone(),
..Default::default()
};
let report = crate::debian::deps::check_build_depends(&ctrl, &check_opts)?;
if !report.is_ok() {
eprintln!("{}", report.message());
return Err(Box::new(crate::debian::deps::UnmetBuildDependencies(
report,
)));
}
}
run_command(cwd, "dpkg-source", &["-I", "-i", "-b", "."], &pipeline_env)?; run_command(cwd, "dpkg-source", &["-I", "-i", "-b", "."], &pipeline_env)?;
if !dsc_path.exists() { if !dsc_path.exists() {
@@ -747,7 +774,7 @@ mod differential_tests {
if matches!(key, "Checksums-Sha1" | "Checksums-Sha256" | "Files") { if matches!(key, "Checksums-Sha1" | "Checksums-Sha256" | "Files") {
let without_buildinfo = |v: &str| -> String { let without_buildinfo = |v: &str| -> String {
v.lines() v.lines()
.filter(|l| !l.trim_end().ends_with("_source.buildinfo")) .filter(|l| !l.trim_end().ends_with(".buildinfo"))
.collect::<Vec<_>>() .collect::<Vec<_>>()
.join("\n") .join("\n")
}; };
@@ -859,6 +886,371 @@ mod differential_tests {
differential_on_tree(&tree); differential_on_tree(&tree);
} }
/// Differential check of [`crate::debian::arch::arch_env`] against real
/// `dpkg-architecture -f -a <arch>` for one architecture.
fn diff_arch_env_one(arch: Option<&str>) {
let mut cmd = Command::new("dpkg-architecture");
cmd.arg("-f");
if let Some(a) = arch {
cmd.args(["-a", a]);
}
let output = cmd
.output()
.expect("run dpkg-architecture (is dpkg-dev installed?)");
assert!(
output.status.success(),
"dpkg-architecture -f {arch:?} failed: {}",
String::from_utf8_lossy(&output.stderr)
);
let mut expected = BTreeMap::new();
for line in String::from_utf8_lossy(&output.stdout).lines() {
if let Some((key, value)) = line.split_once('=') {
expected.insert(key.to_string(), value.to_string());
}
}
let ours = crate::debian::arch::arch_env(arch)
.unwrap_or_else(|e| panic!("native arch_env({arch:?}) failed: {e}"));
assert_eq!(
ours, expected,
"arch_env({arch:?}) differs from dpkg-architecture"
);
}
/// Every architecture known to the local dpkg must produce an identical
/// environment dump (`dpkg-architecture -L`).
#[test]
fn diff_arch_env_all_known_arches() {
let output = Command::new("dpkg-architecture")
.arg("-L")
.output()
.expect("run dpkg-architecture -L (is dpkg-dev installed?)");
assert!(output.status.success());
for arch in String::from_utf8_lossy(&output.stdout).lines() {
let arch = arch.trim();
if arch.is_empty() {
continue;
}
diff_arch_env_one(Some(arch));
}
}
/// Native (no explicit host architecture) must match too.
#[test]
fn diff_arch_env_native() {
diff_arch_env_one(None);
}
/// Differential check of [`crate::debian::deps::check_build_depends`]
/// against real `dpkg-checkbuilddeps` on one fixture: exit status and
/// reported unmet/conflict lists must match.
fn diff_checkbuilddeps_case(control: &str, status: &str, args: &[&str]) {
let dir = tempfile::tempdir().expect("tempdir");
std::fs::write(dir.path().join("control"), control).expect("write control");
let admindir = dir.path().join("admin");
fs::create_dir_all(&admindir).expect("mkdir admindir");
fs::write(admindir.join("status"), status).expect("write status");
// Real tool. Profiles are always pinned via -P so the comparison is
// independent of the local vendor defaults; -I skips the vendor
// builtin dependencies (build-essential:native), matching the
// native checker which knows no builtins. All options must precede
// the control-file operand (POSIX-style option parsing).
let output = Command::new("dpkg-checkbuilddeps")
.current_dir(dir.path())
.arg("--admindir")
.arg(&admindir)
.args(args)
.arg("-I")
.arg("control")
.output()
.expect("run dpkg-checkbuilddeps (is dpkg-dev installed?)");
let real_exit = output.status.code().unwrap_or(-1);
let real_msg = String::from_utf8_lossy(&output.stderr)
.lines()
.filter_map(|l| l.split_once("error: ").map(|(_, m)| m.trim()))
.collect::<Vec<_>>()
.join("\n");
// Native checker with equivalent options.
let mut profiles: Vec<String> = Vec::new();
let mut ignore_arch = false;
let mut ignore_indep = false;
let mut i = 0;
while i < args.len() {
match args[i] {
"-A" => ignore_arch = true,
"-B" => ignore_indep = true,
"-P" => {
i += 1;
profiles = args
.get(i)
.map(|p| p.split(',').map(str::to_string).collect())
.unwrap_or_default();
}
_ => {}
}
i += 1;
}
let opts = crate::debian::deps::CheckOpts {
host_arch: crate::debian::arch::native().unwrap_or_else(|_| "amd64".into()),
build_profiles: profiles,
ignore_arch,
ignore_indep,
ignore_builtin: true,
admindir: admindir.clone(),
};
let control_info =
crate::debian::ControlInfo::parse_content(control).expect("parse control");
let report = crate::debian::deps::check_build_depends(&control_info, &opts)
.expect("native parse failure");
let ours_exit = if report.is_ok() { 0 } else { 1 };
assert_eq!(
ours_exit, real_exit,
"exit status mismatch for {control:?} {args:?}"
);
assert_eq!(
report.message(),
real_msg,
"diagnostics mismatch for {control:?} {args:?}"
);
}
/// Matrix of dependency-checking scenarios validated against the real
/// tool: alternatives, version relations, arch/profile restrictions,
/// conflicts and `-A`/`-B`/`-P` flag handling.
#[test]
fn diff_checkbuilddeps_matrix() {
const STATUS: &str = "\
Package: libc6
Status: install ok installed
Version: 2.39-0ubuntu8
Architecture: amd64
Package: libfoo-dev
Status: install ok installed
Version: 1.2-3
Architecture: amd64
Package: ma-foreign-pkg
Status: install ok installed
Version: 1.0
Architecture: i386
Multi-Arch: foreign
Package: provider
Status: install ok installed
Version: 5.0
Architecture: amd64
Provides: virtual-thing (= 2.0), plain-virtual
";
const HEAD: &str = "Source: t\nMaintainer: a <a@b.c>\n";
const TAIL: &str = "\nPackage: t\nArchitecture: any\nDescription: x\n y\n";
let case = |bd: &str, bc: &str, args: &[&str]| {
let mut control = String::from(HEAD);
if !bd.is_empty() {
control.push_str(&format!("Build-Depends: {bd}\n"));
}
if !bc.is_empty() {
control.push_str(&format!("Build-Conflicts: {bc}\n"));
}
control.push_str(TAIL);
diff_checkbuilddeps_case(&control, STATUS, args);
};
// Satisfied / unsatisfied basics.
case("libc6 (>= 1)", "", &["-P", "cross"]);
case("missing-abc", "", &["-P", "cross"]);
case("libc6 (>> 999)", "", &["-P", "cross"]);
// Alternatives.
case("missing-a | libc6", "", &["-P", "cross"]);
case("missing-a | missing-b", "", &["-P", "cross"]);
// Architecture restrictions (host is the native arch).
case("missing-abc [!amd64]", "", &["-P", "cross"]);
case("missing-abc [amd64]", "", &["-P", "cross"]);
// Profile restrictions.
case("missing-abc <stage1>", "", &["-P", "stage1"]);
case("missing-abc <stage1>", "", &["-P", "cross"]);
case("missing-abc <!stage1>", "", &["-P", "stage1"]);
// Multi-Arch foreign satisfies unqualified deps.
case("ma-foreign-pkg", "", &["-P", "cross"]);
// Provides: versioned provide satisfying / not satisfying.
case("virtual-thing (>= 1.0)", "", &["-P", "cross"]);
case("virtual-thing (>= 3.0)", "", &["-P", "cross"]);
case("plain-virtual", "", &["-P", "cross"]);
case("plain-virtual (>= 1.0)", "", &["-P", "cross"]);
// Conflicts.
case("", "libc6 (<< 1)", &["-P", "cross"]);
case("", "libc6", &["-P", "cross"]);
case("", "missing-abc", &["-P", "cross"]);
// -A/-B field handling.
let control_ab = format!(
"{HEAD}Build-Depends: libc6\nBuild-Depends-Arch: missing-arch-dep\nBuild-Depends-Indep: missing-indep-dep\n{TAIL}"
);
diff_checkbuilddeps_case(&control_ab, STATUS, &["-P", "cross"]);
diff_checkbuilddeps_case(&control_ab, STATUS, &["-A", "-P", "cross"]);
diff_checkbuilddeps_case(&control_ab, STATUS, &["-B", "-P", "cross"]);
// Combined unmet + conflict reporting in one run.
case(
"missing-one, libc6 (>> 999)",
"libfoo-dev",
&["-P", "cross"],
);
}
/// Differential check of [`crate::debian::version`] against real
/// `dpkg --compare-versions` over every ported dpkg test vector and
/// every relation operator.
#[test]
fn diff_version_compare_against_dpkg() {
let vectors = crate::debian::version::test_vectors::COMPARE;
assert!(!vectors.is_empty());
for (a, b, expected) in vectors {
let va =
crate::debian::DebianVersion::parse(a).unwrap_or_else(|e| panic!("parse {a}: {e}"));
let vb =
crate::debian::DebianVersion::parse(b).unwrap_or_else(|e| panic!("parse {b}: {e}"));
let ours = match va.cmp(&vb) {
std::cmp::Ordering::Less => -1,
std::cmp::Ordering::Equal => 0,
std::cmp::Ordering::Greater => 1,
};
assert_eq!(ours, *expected, "native compare: {a} cmp {b}");
// Cross-check the relation operators against the real tool.
for (op, holds) in [
("<<", *expected < 0),
("<=", *expected <= 0),
("=", *expected == 0),
(">=", *expected >= 0),
(">>", *expected > 0),
] {
let output = Command::new("dpkg")
.args(["--compare-versions", "--", a, op, b])
.status()
.expect("run dpkg --compare-versions");
assert_eq!(
output.success(),
holds,
"dpkg --compare-versions -- {a} {op} {b}"
);
}
}
}
/// Differential check of the binary-build metadata generation against
/// real `dpkg-buildpackage -b`: both sides build the same tree (rules
/// driving dpkg-gencontrol/dpkg-deb directly, no debhelper needed),
/// then the produced `.changes`/`.buildinfo` are compared field by
/// field modulo machine-dependent values.
#[test]
fn diff_binary_build_metadata() {
const NAME: &str = "pkh-diff-m";
let control = format!(
"Source: {NAME}\nSection: utils\nPriority: optional\nMaintainer: {MAINTAINER}\nBuild-Depends: libc6\n\n\
Package: {NAME}\nArchitecture: any\nDescription: test package main\n long description\n\n\
Package: {NAME}-u\nPackage-Type: udeb\nArchitecture: all\nDescription: test udeb\n short\n"
);
let changelog = format!(
"{NAME} (1.0-1) unstable; urgency=medium\n\n * Binary build test.\n\n -- {MAINTAINER} {DATE}\n"
);
let rules = format!(
"#!/usr/bin/make -f\nV = $(shell dpkg-parsechangelog -S Version)\nA = $(shell dpkg-architecture -qDEB_HOST_ARCH)\n\nbuild:\n\tmkdir -p debian/tmp/usr/bin\n\tprintf '#!/bin/sh\\necho hi\\n' > debian/tmp/usr/bin/hello\n\tchmod 755 debian/tmp/usr/bin/hello\n\ttouch $@\n\nbinary: build\n\trm -rf debian/{NAME} debian/{NAME}-u\n\tmkdir -p debian/{NAME}/usr/bin debian/{NAME}/DEBIAN\n\tcp -r debian/tmp/. debian/{NAME}/\n\tdpkg-gencontrol -p{NAME} -Pdebian/{NAME}\n\tdpkg-deb --build debian/{NAME} ..\n\tmkdir -p debian/{NAME}-u/usr/share debian/{NAME}-u/DEBIAN\n\techo data > debian/{NAME}-u/usr/share/data.txt\n\tdpkg-gencontrol -p{NAME}-u -Pdebian/{NAME}-u\n\tdpkg-deb --build debian/{NAME}-u ..\n\tmv ../{NAME}-u_$(V)_all.deb ../{NAME}-u_$(V)_all.udeb\n\nclean:\n\trm -rf debian/tmp debian/{NAME} debian/{NAME}-u build-stamp debian/files debian/*.substvars\n\n.PHONY: build binary clean\n"
);
let write_tree = |root: &Path| {
fs::create_dir_all(root.join(format!("{NAME}/debian/source"))).expect("mkdir tree");
let tree = root.join(NAME);
fs::write(tree.join("debian/control"), &control).expect("write control");
fs::write(tree.join("debian/changelog"), &changelog).expect("write changelog");
fs::write(tree.join("debian/source/format"), "3.0 (native)\n").expect("write format");
fs::write(tree.join("debian/rules"), &rules).expect("write rules");
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
fs::set_permissions(tree.join("debian/rules"), fs::Permissions::from_mode(0o755))
.expect("chmod rules");
}
tree
};
let base = tempfile::tempdir().expect("tempdir");
let golden_root = base.path().join("golden");
let ours_root = base.path().join("ours");
fs::create_dir_all(&golden_root).expect("mkdir golden");
fs::create_dir_all(&ours_root).expect("mkdir ours");
let golden_tree = write_tree(&golden_root);
let ours_tree = write_tree(&ours_root);
// Golden side: real dpkg-buildpackage binary build.
let status = Command::new("dpkg-buildpackage")
.current_dir(&golden_tree)
.args(["-b", "-d", "--no-sign"])
.status()
.expect("run dpkg-buildpackage (is dpkg-dev installed?)");
assert!(status.success(), "golden dpkg-buildpackage -b failed");
// Ours: emulate the pkh deb flow (rules build + rules binary with a
// dpkg-buildpackage-like environment), then run the native metadata
// generation through a local context. dpkg-buildpackage runs the
// rules targets directly by default (missing Rules-Requires-Root is
// treated as 'no'), so no fakeroot wrapper here either.
let entry =
crate::debian::parse_changelog_entry_from_str(&changelog).expect("parse changelog");
let vendor = env::current_vendor();
let profiles = env::resolve_build_profiles(&[], &vendor);
let parallel = env::num_parallel();
let build_env_vars: Vec<(String, String)> = [
("LANG".to_string(), "C".to_string()),
(
"DEB_BUILD_OPTIONS".to_string(),
format!("parallel={parallel}"),
),
("SOURCE_DATE_EPOCH".to_string(), entry.timestamp.to_string()),
]
.into_iter()
.collect();
for target in ["build", "binary"] {
let status = Command::new("debian/rules")
.current_dir(&ours_tree)
.envs(build_env_vars.clone())
.arg(target)
.status()
.expect("run rules target");
assert!(status.success(), "debian/rules {target} failed");
}
let ctx = std::sync::Arc::new(crate::context::Context::new(
crate::context::ContextConfig::Local,
));
let native_arch = crate::debian::arch::native().unwrap_or_else(|_| "amd64".into());
let opts = crate::build::binary::BinaryMetadataOptions {
profiles,
vendor,
parallel,
source_date_epoch: entry.timestamp,
build_arch: native_arch.clone(),
host_arch: native_arch,
};
crate::build::binary::generate_binary_metadata(&ctx, &ours_tree, &ours_root, &opts)
.expect("native binary metadata generation failed");
// Compare artifacts.
assert_changes_equivalent(
&golden_root.join(format!("{NAME}_1.0-1_amd64.changes")),
&ours_root.join(format!("{NAME}_1.0-1_amd64.changes")),
);
assert_buildinfo_equivalent(
&golden_root.join(format!("{NAME}_1.0-1_amd64.buildinfo")),
&ours_root.join(format!("{NAME}_1.0-1_amd64.buildinfo")),
);
}
#[test] #[test]
fn diff_native_minimal() { fn diff_native_minimal() {
differential_case(&FixtureSpec::new("pkh-diff-a", "1.0-1", "unstable")); differential_case(&FixtureSpec::new("pkh-diff-a", "1.0-1", "unstable"));
+96
View File
@@ -227,6 +227,17 @@ pub async fn build(
.to_str() .to_str()
.ok_or("Invalid package directory path")?; .ok_or("Invalid package directory path")?;
// Reproducibility: export SOURCE_DATE_EPOCH from the changelog entry,
// like dpkg-buildpackage does.
match ctx.read_file(&package_dir.join("debian/changelog")) {
Ok(content) => {
if let Ok(entry) = crate::debian::parse_changelog_entry_from_str(&content) {
env.insert("SOURCE_DATE_EPOCH".to_string(), entry.timestamp.to_string());
}
}
Err(e) => log::debug!("cannot read changelog for SOURCE_DATE_EPOCH: {}", e),
}
// Apply quilt patches if the package provides a patch series // Apply quilt patches if the package provides a patch series
apply_quilt_patches(package_dir_str, &env, ctx.clone(), &ui, &sink)?; apply_quilt_patches(package_dir_str, &env, ctx.clone(), &ui, &sink)?;
@@ -318,6 +329,91 @@ pub async fn build(
); );
} }
// Generate the upload metadata (.buildinfo + .changes) natively, the
// equivalent of dpkg-genbuildinfo -b + dpkg-genchanges -b, consuming
// debian/files produced by the build. Failures are logged but do not
// discard the produced binaries.
if let Err(e) = generate_upload_metadata(package_dir_str, build_root, arch, cross, &env, &ctx) {
warn!("failed to generate .buildinfo/.changes: {}", e);
}
Ok(())
}
/// Generate `.buildinfo` and `.changes` for the finished binary build,
/// inside the build context.
fn generate_upload_metadata(
package_dir: &str,
build_root: &str,
arch: &str,
cross: bool,
env: &HashMap<String, String>,
ctx: &Arc<Context>,
) -> Result<(), Box<dyn Error>> {
use std::path::Path;
let changelog_path = Path::new(package_dir).join("debian/changelog");
let changelog_content = ctx.read_file(&changelog_path)?;
let entry = crate::debian::parse_changelog_entry_from_str(&changelog_content)?;
// Build architecture: the machine inside the build context.
let build_arch = ctx
.command("dpkg")
.arg("--print-architecture")
.output()
.ok()
.filter(|o| o.status.success())
.map(|o| String::from_utf8_lossy(&o.stdout).trim().to_string())
.filter(|s| !s.is_empty())
.unwrap_or_else(crate::get_current_arch);
let host_arch = if cross {
arch.to_string()
} else {
build_arch.clone()
};
// Vendor resolution inside the context (falls back to the host view).
let vendor = ctx
.read_file(Path::new("/etc/dpkg/origins/default"))
.ok()
.and_then(|content| {
for line in content.lines() {
if let Some(v) = line.strip_prefix("Vendor:") {
let v = v.trim();
if !v.is_empty() {
return Some(v.to_string());
}
}
}
None
})
.unwrap_or_else(crate::build::env::current_vendor);
let profiles = crate::build::env::resolve_build_profiles(&[], &vendor);
let source_date_epoch = env
.get("SOURCE_DATE_EPOCH")
.and_then(|v| v.parse::<i64>().ok())
.unwrap_or(entry.timestamp);
let opts = crate::build::binary::BinaryMetadataOptions {
profiles,
vendor,
parallel: crate::build::env::num_parallel(),
source_date_epoch,
build_arch,
host_arch,
};
let (buildinfo, changes) = crate::build::binary::generate_binary_metadata(
ctx,
Path::new(package_dir),
Path::new(build_root),
&opts,
)?;
log::info!(
"generated upload metadata: {} and {}",
buildinfo.display(),
changes.display()
);
Ok(()) Ok(())
} }
+10 -2
View File
@@ -165,14 +165,22 @@ async fn build_binary_package_impl(
} }
} }
// Retrieve produced .deb files // Retrieve produced artifacts (.deb files plus the upload metadata
// (.buildinfo/.changes) generated natively after the build)
if let Some(u) = ui { if let Some(u) = ui {
u.phase(Phase::RetrievingArtifacts); u.phase(Phase::RetrievingArtifacts);
} }
let remote_files = build_ctx.list_files(Path::new(&build_root))?; let remote_files = build_ctx.list_files(Path::new(&build_root))?;
let deb_files: Vec<PathBuf> = remote_files let deb_files: Vec<PathBuf> = remote_files
.into_iter() .into_iter()
.filter(|f| f.extension().is_some_and(|ext| ext == "deb")) .filter(|f| {
f.extension().is_some_and(|ext| {
matches!(
ext.to_str(),
Some("deb") | Some("buildinfo") | Some("changes")
)
})
})
.collect(); .collect();
let total_debs = deb_files.len(); let total_debs = deb_files.len();
+1027
View File
File diff suppressed because it is too large Load Diff
+26 -23
View File
@@ -46,7 +46,15 @@ pub fn parse_changelog_entry(path: &Path) -> Result<ChangelogEntry, Box<dyn std:
e e
) )
})?; })?;
parse_changelog_entry_from_str(&content)
}
/// Parse the most recent changelog entry from its textual content. `origin`
/// is used in error messages only.
pub fn parse_changelog_entry_from_str(
content: &str,
) -> Result<ChangelogEntry, Box<dyn std::error::Error>> {
let origin = "changelog";
let mut lines = content.lines().peekable(); let mut lines = content.lines().peekable();
// --- Header line: `package (version) distributions; urgency=medium[, key=value]` // --- Header line: `package (version) distributions; urgency=medium[, key=value]`
@@ -55,18 +63,14 @@ pub fn parse_changelog_entry(path: &Path) -> Result<ChangelogEntry, Box<dyn std:
Some(l) if l.trim().is_empty() => continue, Some(l) if l.trim().is_empty() => continue,
Some(l) => break l.trim_end(), Some(l) => break l.trim_end(),
None => { None => {
return Err(format!("changelog '{}' is empty", path.display()).into()); return Err(format!("changelog '{origin}' is empty").into());
} }
} }
}; };
let open = header.find('(').ok_or_else(|| { let open = header
format!( .find('(')
"invalid changelog header in '{}': {}", .ok_or_else(|| format!("invalid changelog header in '{origin}': {header}"))?;
path.display(),
header
)
})?;
let close = header[open..] let close = header[open..]
.find(')') .find(')')
.ok_or_else(|| format!("unbalanced parenthesis in changelog header '{}'", header))?; .ok_or_else(|| format!("unbalanced parenthesis in changelog header '{}'", header))?;
@@ -123,9 +127,8 @@ pub fn parse_changelog_entry(path: &Path) -> Result<ChangelogEntry, Box<dyn std:
let trailer_line = trailer.ok_or_else(|| { let trailer_line = trailer.ok_or_else(|| {
format!( format!(
"no maintainer trailer found in '{}': expected a line of the form \ "no maintainer trailer found in '{origin}': expected a line of the form \
' -- Name <email> Date'", ' -- Name <email> Date'"
path.display()
) )
})?; })?;
@@ -147,14 +150,7 @@ pub fn parse_changelog_entry(path: &Path) -> Result<ChangelogEntry, Box<dyn std:
let date_raw = trailer_body[gt + 1..].trim().to_string(); let date_raw = trailer_body[gt + 1..].trim().to_string();
let timestamp = DateTime::parse_from_rfc2822(&date_raw) let timestamp = DateTime::parse_from_rfc2822(&date_raw)
.map_err(|e| { .map_err(|e| format!("cannot parse changelog date '{date_raw}' in '{origin}': {e}"))?
format!(
"cannot parse changelog date '{}' in '{}': {}",
date_raw,
path.display(),
e
)
})?
.timestamp(); .timestamp();
// Changes field value (leading `\n` marks it as a pre-wrapped multiline // Changes field value (leading `\n` marks it as a pre-wrapped multiline
@@ -220,7 +216,14 @@ fn find_closes(body_lines: &[String]) -> Option<String> {
pub fn parse_previous_version(path: &Path) -> Result<Option<String>, Box<dyn std::error::Error>> { pub fn parse_previous_version(path: &Path) -> Result<Option<String>, Box<dyn std::error::Error>> {
let content = std::fs::read_to_string(path) let content = std::fs::read_to_string(path)
.map_err(|e| format!("failed to read changelog '{}': {}", path.display(), e))?; .map_err(|e| format!("failed to read changelog '{}': {}", path.display(), e))?;
parse_previous_version_from_str(&content)
}
/// Return the version of the *previous* changelog entry from the textual
/// content of a changelog file.
pub fn parse_previous_version_from_str(
content: &str,
) -> Result<Option<String>, Box<dyn std::error::Error>> {
let mut seen_first = false; let mut seen_first = false;
for line in content.lines() { for line in content.lines() {
let line = line.trim_end(); let line = line.trim_end();
@@ -229,12 +232,12 @@ pub fn parse_previous_version(path: &Path) -> Result<Option<String>, Box<dyn std
seen_first = true; seen_first = true;
continue; continue;
} }
let open = line.find('(').ok_or_else(|| { let open = line
format!("invalid changelog header in '{}': {}", path.display(), line) .find('(')
})?; .ok_or_else(|| format!("invalid changelog header: {line}"))?;
let close = line[open..] let close = line[open..]
.find(')') .find(')')
.ok_or_else(|| format!("unbalanced parenthesis in changelog header '{}'", line))?; .ok_or_else(|| format!("unbalanced parenthesis in changelog header '{line}'"))?;
return Ok(Some(line[open + 1..open + close].to_string())); return Ok(Some(line[open + 1..open + close].to_string()));
} }
} }
+1312
View File
File diff suppressed because it is too large Load Diff
+28 -10
View File
@@ -114,29 +114,30 @@ impl FilesList {
/// Load `debian/files`. A missing file yields an empty registry. /// Load `debian/files`. A missing file yields an empty registry.
pub fn load(path: &Path) -> Result<FilesList, Box<dyn std::error::Error>> { pub fn load(path: &Path) -> Result<FilesList, Box<dyn std::error::Error>> {
let mut list = FilesList::new();
let content = match std::fs::read_to_string(path) { let content = match std::fs::read_to_string(path) {
Ok(c) => c, Ok(c) => c,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(list), Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(FilesList::new()),
Err(e) => { Err(e) => {
return Err(format!("cannot read '{}': {}", path.display(), e).into()); return Err(format!("cannot read '{}': {}", path.display(), e).into());
} }
}; };
FilesList::parse(&content).map_err(|e| format!("in '{}': {}", path.display(), e).into())
}
/// Parse a `debian/files` registry from its textual content
/// (`filename section priority [key=value...]` lines).
pub fn parse(content: &str) -> Result<FilesList, String> {
let mut list = FilesList::new();
for line in content.lines() { for line in content.lines() {
if line.trim().is_empty() { if line.trim().is_empty() {
continue; continue;
} }
let tokens: Vec<&str> = line.split_whitespace().collect(); let tokens: Vec<&str> = line.split_whitespace().collect();
if tokens.len() < 3 { if tokens.len() < 3 {
return Err(format!("badly formed line in '{}': {}", path.display(), line).into()); return Err(format!("badly formed line: {line}"));
} }
let mut entry = parse_filename(tokens[0]).ok_or_else(|| { let mut entry = parse_filename(tokens[0])
format!( .ok_or_else(|| format!("badly formed file name: {}", tokens[0]))?;
"badly formed file name in '{}': {}",
path.display(),
tokens[0]
)
})?;
entry.section = tokens[1].to_string(); entry.section = tokens[1].to_string();
entry.priority = tokens[2].to_string(); entry.priority = tokens[2].to_string();
for attr in &tokens[3..] { for attr in &tokens[3..] {
@@ -181,6 +182,23 @@ impl FilesList {
self.files.is_empty() self.files.is_empty()
} }
/// Render the registry to its textual `debian/files` representation.
pub fn render(&self) -> String {
let mut out = String::new();
for entry in self.iter() {
out.push_str(&entry.filename);
out.push(' ');
out.push_str(&entry.section);
out.push(' ');
out.push_str(&entry.priority);
for (k, v) in &entry.attrs {
out.push_str(&format!(" {k}={v}"));
}
out.push('\n');
}
out
}
/// Save atomically: write `<path>.new` then rename over `path`, like /// Save atomically: write `<path>.new` then rename over `path`, like
/// dpkg does. /// dpkg does.
pub fn save_atomic(&self, path: &Path) -> Result<(), Box<dyn std::error::Error>> { pub fn save_atomic(&self, path: &Path) -> Result<(), Box<dyn std::error::Error>> {
+9 -2
View File
@@ -3,19 +3,26 @@
//! These components are independent from any build orchestration and can be //! These components are independent from any build orchestration and can be
//! used by any pkh submodule (or external consumers of the library): //! used by any pkh submodule (or external consumers of the library):
//! //!
//! - [`arch`]: Debian architecture tables and lookups (dpkg-architecture)
//! - [`control`]: deb822 paragraph parsing/writing and `debian/control` //! - [`control`]: deb822 paragraph parsing/writing and `debian/control`
//! - [`checksums`]: file checksum registry (`Dpkg::Checksums` equivalent) //! - [`checksums`]: file checksum registry (`Dpkg::Checksums` equivalent)
//! - [`deps`]: dependency grammar and evaluation (dpkg-checkbuilddeps)
//! - [`files`]: `debian/files` artifact registry (`Dpkg::Dist::Files`) //! - [`files`]: `debian/files` artifact registry (`Dpkg::Dist::Files`)
//! - [`version`]: Debian version splitting/validation //! - [`version`]: Debian version splitting/validation/comparison
//! - [`changelog`]: `debian/changelog` entry parsing //! - [`changelog`]: `debian/changelog` entry parsing
pub mod arch;
pub mod changelog; pub mod changelog;
pub mod checksums; pub mod checksums;
pub mod control; pub mod control;
pub mod deps;
pub mod files; pub mod files;
pub mod version; pub mod version;
pub use changelog::{ChangelogEntry, parse_changelog_entry}; pub use changelog::{
ChangelogEntry, parse_changelog_entry, parse_changelog_entry_from_str,
parse_previous_version_from_str,
};
pub use checksums::{Entry as ChecksumEntry, FileChecksums}; pub use checksums::{Entry as ChecksumEntry, FileChecksums};
pub use control::{ControlInfo, Paragraph, parse_paragraphs, write_paragraph}; pub use control::{ControlInfo, Paragraph, parse_paragraphs, write_paragraph};
pub use files::{FilesEntry, FilesList}; pub use files::{FilesEntry, FilesList};
+276 -1
View File
@@ -1,4 +1,4 @@
//! Debian version handling: splitting and validation of //! Debian version handling: splitting, validation and ordering of
//! `[epoch:]upstream[-revision]` version strings. //! `[epoch:]upstream[-revision]` version strings.
/// A Debian version, split into its `[epoch:]upstream[-revision]` parts. /// A Debian version, split into its `[epoch:]upstream[-revision]` parts.
@@ -83,6 +83,217 @@ impl DebianVersion {
None => self.upstream.clone(), None => self.upstream.clone(),
} }
} }
/// Convenience predicate: whether this version orders strictly later
/// than `other`.
pub fn later_than(&self, other: &DebianVersion) -> bool {
self > other
}
}
/// Compare two versions according to dpkg's ordering algorithm
/// (Debian Policy §5.6.1 / `dpkg(1)`):
///
/// - the epoch compares numerically (a missing epoch counts as `0`),
/// - then the upstream version and the Debian revision compare by
/// alternating non-digit and digit chunks, from left to right,
/// - in non-digit chunks letters sort earlier than non-letters, and `~`
/// sorts before anything, including the end of the chunk,
/// - digit chunks compare numerically (leading zeroes are irrelevant; an
/// empty digit chunk counts as `0`, so a missing revision equals `0`).
pub fn compare(a: &DebianVersion, b: &DebianVersion) -> std::cmp::Ordering {
a.epoch
.unwrap_or(0)
.cmp(&b.epoch.unwrap_or(0))
.then_with(|| verrevcmp(a.upstream.as_bytes(), b.upstream.as_bytes()))
.then_with(|| {
verrevcmp(
a.debian_revision.as_deref().unwrap_or("").as_bytes(),
b.debian_revision.as_deref().unwrap_or("").as_bytes(),
)
})
}
/// Sort weight of a character inside a non-digit chunk: `~` sorts before the
/// end of the chunk, letters before non-letters, everything else by ASCII
/// order.
fn char_order(c: u8) -> i32 {
if c == b'~' {
-1
} else if c.is_ascii_alphabetic() {
i32::from(c)
} else {
i32::from(c) + 256
}
}
/// Compare the upstream/revision part of two versions by alternating
/// non-digit and digit chunks.
fn verrevcmp(mut a: &[u8], mut b: &[u8]) -> std::cmp::Ordering {
use std::cmp::Ordering;
while !a.is_empty() || !b.is_empty() {
let mut first_diff: i32 = 0;
// Non-digit chunks: compare by character weight. A chunk boundary
// (end of string or start of a digit run) weighs 0, which sorts
// after `~` (-1) and before every real character.
while (!a.is_empty() && !a[0].is_ascii_digit()) || (!b.is_empty() && !b[0].is_ascii_digit())
{
let ac = if !a.is_empty() && !a[0].is_ascii_digit() {
char_order(a[0])
} else {
0
};
let bc = if !b.is_empty() && !b[0].is_ascii_digit() {
char_order(b[0])
} else {
0
};
if ac != bc {
return ac.cmp(&bc);
}
// Reaching here means both sides carried equal real characters.
a = &a[1..];
b = &b[1..];
}
// Digit chunks: strip leading zeroes, then the number whose
// remaining digit run is longer is larger; otherwise the first
// differing digit decides.
while !a.is_empty() && a[0] == b'0' {
a = &a[1..];
}
while !b.is_empty() && b[0] == b'0' {
b = &b[1..];
}
while !a.is_empty() && !b.is_empty() && a[0].is_ascii_digit() && b[0].is_ascii_digit() {
if first_diff == 0 {
first_diff = i32::from(a[0]) - i32::from(b[0]);
}
a = &a[1..];
b = &b[1..];
}
if !a.is_empty() && a[0].is_ascii_digit() {
return Ordering::Greater;
}
if !b.is_empty() && b[0].is_ascii_digit() {
return Ordering::Less;
}
if first_diff != 0 {
return first_diff.cmp(&0);
}
}
Ordering::Equal
}
impl PartialOrd for DebianVersion {
fn partial_cmp(&self, other: &Self) -> Option<std::cmp::Ordering> {
Some(self.cmp(other))
}
}
impl Ord for DebianVersion {
fn cmp(&self, other: &Self) -> std::cmp::Ordering {
compare(self, other)
}
}
/// Test vectors ported from dpkg's `scripts/t/Dpkg_Version.t` (`__DATA__`
/// section): `(version_a, version_b, expected_cmp)` with `-1/0/1`. Shared
/// with the differential tests against real `dpkg --compare-versions`.
#[cfg(test)]
pub(crate) mod test_vectors {
/// `(a, b, cmp)` triples.
pub(crate) const COMPARE: &[(&str, &str, i32)] = &[
("1.0-1", "2.0-2", -1),
("2.2~rc-4", "2.2-1", -1),
("2.2-1", "2.2~rc-4", 1),
("1.0000-1", "1.0-1", 0),
("1", "0:1", 0),
("0", "0:0-0", 0),
("2:2.5", "1:7.5", 1),
("1:0foo", "0foo", 1),
("0:0foo", "0foo", 0),
("0foo", "0foo", 0),
("0foo-0", "0foo", 0),
("0foo", "0foo-0", 0),
("0foo", "0fo", 1),
("0foo-0", "0foo+", -1),
("0foo~1", "0foo", -1),
("0foo~foo+Bar", "0foo~foo+bar", -1),
("0foo~~", "0foo~", -1),
("1~", "1", -1),
(
"12345+that-really-is-some-ver-0",
"12345+that-really-is-some-ver-10",
-1,
),
("0foo-0", "0foo-01", -1),
("0foo.bar", "0foobar", 1),
("0foo.bar", "0foo1bar", 1),
("0foo.bar", "0foo0bar", 1),
("0foo1bar-1", "0foobar-1", -1),
("0foo2.0", "0foo2", 1),
("0foo2.0.0", "0foo2.10.0", -1),
("0foo2.0", "0foo2.0.0", -1),
("0foo2.0", "0foo2.10", -1),
("0foo2.1", "0foo2.10", -1),
("1.09", "1.9", 0),
("1.0.8+nmu1", "1.0.8", 1),
("3.11", "3.10+nmu1", 1),
("0.9j-20080306-4", "0.9i-20070324-2", 1),
("1.2.0~b7-1", "1.2.0~b6-1", 1),
("1.011-1", "1.06-2", 1),
("0.0.9+dfsg1-1", "0.0.8+dfsg1-3", 1),
("4.6.99+svn6582-1", "4.6.99+svn6496-1", 1),
("53", "52", 1),
("0.9.9~pre122-1", "0.9.9~pre111-1", 1),
("2:2.3.2-2+lenny2", "2:2.3.2-2", 1),
("1:3.8.1-1", "3.8.GA-1", 1),
("1.0.1+gpl-1", "1.0.1-2", 1),
("1a", "1000a", -1),
];
/// Unsorted lists with their expected order under dpkg comparison.
pub(crate) const SORTED: &[(&[&str], &[&str])] = &[
(
&[
"4:4-4",
"5.0abc",
"0.0-0.0alpha0",
"10.100.1-1",
"0~999.999zeta",
"0:1.0-0",
],
&[
"0~999.999zeta",
"0.0-0.0alpha0",
"0:1.0-0",
"5.0abc",
"10.100.1-1",
"4:4-4",
],
),
(
&[
"4",
"5.0abc",
"0.0alpha0",
"10.100.1",
"0~999.999zeta",
"1.0",
],
&[
"0~999.999zeta",
"0.0alpha0",
"1.0",
"4",
"5.0abc",
"10.100.1",
],
),
];
} }
#[cfg(test)] #[cfg(test)]
@@ -117,4 +328,68 @@ mod tests {
assert!(DebianVersion::parse("1.0").is_ok()); assert!(DebianVersion::parse("1.0").is_ok());
assert!(DebianVersion::parse("1.0~rc1-2").is_ok()); assert!(DebianVersion::parse("1.0~rc1-2").is_ok());
} }
fn cmp_sign(a: &DebianVersion, b: &DebianVersion) -> i32 {
match a.cmp(b) {
std::cmp::Ordering::Less => -1,
std::cmp::Ordering::Equal => 0,
std::cmp::Ordering::Greater => 1,
}
}
/// All vectors from dpkg's own `Dpkg_Version.t` must pass.
#[test]
fn comparison_dpkg_vectors() {
for (a, b, expected) in test_vectors::COMPARE {
let va = DebianVersion::parse(a).unwrap_or_else(|e| panic!("parse {a}: {e}"));
let vb = DebianVersion::parse(b).unwrap_or_else(|e| panic!("parse {b}: {e}"));
assert_eq!(
cmp_sign(&va, &vb),
*expected,
"{a} cmp {b} must be {expected}"
);
// Ordering is antisymmetric.
assert_eq!(cmp_sign(&vb, &va), -*expected, "{b} cmp {a}");
}
}
#[test]
fn sorting_dpkg_vectors() {
for (unsorted, expected) in test_vectors::SORTED {
let mut versions: Vec<DebianVersion> = unsorted
.iter()
.map(|v| DebianVersion::parse(v).unwrap())
.collect();
versions.sort();
let rendered: Vec<String> = versions.iter().map(DebianVersion::full).collect();
let expected: Vec<String> = expected.iter().map(|s| s.to_string()).collect();
assert_eq!(rendered, expected);
}
}
/// Ubuntu-flavored cases: security updates, backports, PPA versions.
#[test]
fn comparison_ubuntu_flavored() {
let cases: &[(&str, &str, i32)] = &[
// Security update on top of a release upload.
("1.0-0ubuntu1", "1.0-0ubuntu1.22.04.1", -1),
// PPA/backports pre-releases sort before the real upload.
("1.0-0ubuntu1~ppa1", "1.0-0ubuntu1", -1),
("1.0~bpo22.04.1", "1.0", -1),
// Series-specific uploads.
("2.3-1ubuntu3.22.04.2", "2.3-1ubuntu3", 1),
("1:2.0.4-0ubuntu1", "1:2.0.4-0ubuntu1.1", -1),
];
for (a, b, expected) in cases {
let va = DebianVersion::parse(a).unwrap();
let vb = DebianVersion::parse(b).unwrap();
assert_eq!(cmp_sign(&va, &vb), *expected, "{a} cmp {b}");
}
// later_than convenience.
let old = DebianVersion::parse("1.0-0ubuntu1").unwrap();
let new = DebianVersion::parse("1.0-0ubuntu1.22.04.1").unwrap();
assert!(new.later_than(&old));
assert!(!old.later_than(&old));
}
} }
+7
View File
@@ -252,6 +252,13 @@ fn main() {
let cwd = current_dir_or_exit(); let cwd = current_dir_or_exit();
if let Err(e) = pkh::build::build_source_package(Some(&cwd)) { if let Err(e) = pkh::build::build_source_package(Some(&cwd)) {
error!("{}", e); error!("{}", e);
// Unmet build dependencies/conflicts exit with status 3,
// like dpkg-buildpackage does.
if e.downcast_ref::<pkh::debian::deps::UnmetBuildDependencies>()
.is_some()
{
std::process::exit(3);
}
std::process::exit(1); std::process::exit(1);
} }
} }