Compare commits

..
5 Commits
Author SHA1 Message Date
vhaudiquet 53078fbca1 fmt 2026-08-24 10:31:26 +02:00
vhaudiquet fbdf1b334b build/binary: native .buildinfo/.changes for binary builds (pkh deb)
Extend the metadata writers to binary-only uploads and wire them into
the 'pkh deb' flow:

- build/binary.rs generates <pkg>_<ver>_<arch>.buildinfo/.changes
  through any Context: debian/files consumption, encounter-order
  Architecture accumulation (sorted in .buildinfo like dpkg-genbuildinfo),
  sorted Binary lists, dpkg-formatted Description lines with udeb
  suffixes, Installed-Build-Depends closure over the context status DB,
  and binNMU handling (Source: pkg (prev), Binary-Only-Changes, previous
  .dsc redistribution);
- artifact digests are computed inside the context via coreutils
  (md5sum/sha1sum/sha256sum/stat) so chrooted/remote trees work;
- deb/local.rs runs the generation after 'rules binary', exports
  SOURCE_DATE_EPOCH from the changelog (reproducibility), and resolves
  vendor/profiles inside the context; deb/mod.rs retrieves the new
  artifacts alongside the debs;
- reusable helpers added: FilesList::parse/render,
  parse_changelog_entry_from_str, parse_previous_version_from_str,
  installed_build_depends_from_content.

Differential gate: same tree built with real 'dpkg-buildpackage -b' and
with the pkh flow; .changes/.buildinfo compared field-by-field modulo
machine-dependent fields, artifact checksums included.
2026-08-24 09:39:03 +02:00
vhaudiquet a3cbc87627 debian/deps: native dependency grammar and build-dep checking
Replace dpkg-checkbuilddeps with a native implementation:

- full dependency grammar: comma clauses, | alternatives, << <= = >= >>
  relations, :arch qualifiers (any/native/specific), [arch lists] and
  <profile restriction> formulas per alternative;
- restriction reduction against active build profiles and the host arch
  at parse time (reduce_restrictions semantics);
- evaluation against a parsed dpkg status database with Multi-Arch
  semantics (foreign/allowed) and versioned Provides rules (unversioned
  provides never satisfy versioned deps; versioned ones must satisfy the
  relation);
- clause simplification with implication-based deduplication, rendering
  dpkg-compatible 'unmet build dependencies/conflicts' diagnostics.

check_build_depends() consumes debian/control + CheckOpts (-A/-B/-I
equivalents). run_source_build performs the check when forced (-D
parity); source-only builds skip it entirely like dpkg-buildpackage,
and unsatisfied deps propagate as UnmetBuildDependencies -> exit 3.

Unit tests port the Dpkg_Deps.t reduction matrices; differential gate
runs 24 scenarios (alternatives, versions, arch/profile restrictions,
Multi-Arch, Provides, conflicts, -A/-B flags) against real
dpkg-checkbuilddeps comparing exit status and diagnostics.
2026-08-24 09:11:10 +02:00
vhaudiquet f3be8f8ba5 debian/version: dpkg-compatible version comparison
Implement the documented dpkg ordering algorithm (Debian Policy 5.6.1):
numeric epoch, then upstream/revision compared as alternating non-digit
and digit chunks, with '~' ordering before anything including the empty
chunk and letters before non-letters in non-digit chunks.

Adds Ord/PartialOrd for DebianVersion, a free compare() and a
later_than() convenience.

Unit tests port all vectors from dpkg's scripts/t/Dpkg_Version.t plus
Ubuntu-flavored cases (security uploads, ~ppa1 backports). Differential
gate: every vector cross-checked against real 'dpkg --compare-versions'
for <<, <=, =, >= and >>.
2026-08-23 21:57:56 +02:00
vhaudiquet 01c05f04a3 debian/arch: native architecture tables replacing dpkg-architecture
Embed dpkg's factual cputable/ostable/tupletable/abitable data and
implement tuple/triplet/multiarch lookups, wildcard matching, arch
restriction evaluation and the full DEB_BUILD_*/DEB_HOST_*/DEB_TARGET_*
environment dump natively.

build/env.rs::arch_env now delegates to the native implementation
instead of shelling out to 'dpkg-architecture -f'.

Differential gate (build/mod.rs): arch_env(Some(a)) must equal real
'dpkg-architecture -f -a a' key-for-key for every architecture listed by
'dpkg-architecture -L', plus the native case. Data tables carry upstream
attribution comments; no dpkg code was transliterated.
2026-08-23 21:53:09 +02:00
8 changed files with 378 additions and 360 deletions
+328
View File
@@ -0,0 +1,328 @@
# Native build pipeline — replacing the `dpkg-buildpackage` shell-out
Status: **Phase 0 + Phase 1 (source builds) implemented** — see §11
Reference codebase: https://salsa.debian.org/dpkg-team/dpkg (`main` branch, analyzed 2026-08)
---
## 0. TL;DR — honest take
| Scope | Verdict | Effort | Risk |
|---|---|---|---|
| **A.** Re-implement the *orchestrator* ([`dpkg-buildpackage.pl`](https://salsa.debian.org/dpkg-team/dpkg/-/blob/main/scripts/dpkg-buildpackage.pl), 1209 lines of Perl) natively in Rust, still invoking `dpkg-source`, `debian/rules`, etc. as subprocesses | **Doable, worth it** | ~2–4 weeks | Low–medium |
| **B.** Additionally replace the cheap satellite tools natively (`dpkg-parsechangelog`, `dpkg-checkbuilddeps`, `dpkg-architecture`, `dpkg-genchanges`, `dpkg-genbuildinfo`) | Doable incrementally | +1–2 weeks each | Medium |
| **C.** Replace `dpkg-source` (tarball/diff generation, quilt integration, `.dsc` assembly) | **Not recommended** | 2–3+ months, then endless edge cases | High |
The key insight: `dpkg-buildpackage` itself is a thin (~1200-line) Perl *sequencer*. The real complexity lives in `dpkg-source` (~4000 lines of Perl across `Dpkg/Source/*`). A source build fundamentally requires `dpkg-source`-class functionality (orig tarballs, debian diffs, `.dsc` assembly), so "no dpkg tools at all" is not a realistic goal — but "no `dpkg-buildpackage`, full control of the pipeline" absolutely is, and it unlocks things pkh currently cannot do (live UI on source builds, `.changes`/`.buildinfo` generation on binary builds, unified error classification).
---
## 1. Where pkh stands today
Two divergent build paths:
### Source builds — [`build_source_package()`](../src/build/mod.rs)
Shelled out on the host:
```
dpkg-buildpackage -S -I -i -nc -d [--sign-keyid=<id> | --no-sign]
```
Problems:
- Requires `dpkg-dev` on the host.
- Output bypasses the live UI entirely ([`DebUi`](../src/ui/deb.rs) / [`LineSink`](../src/context/mod.rs)) — no phase tracking, no log classification, unlike binary builds.
- No control over failure semantics beyond exit status.
### Binary builds — [`deb::local::build()`](../src/deb/local.rs)
Already does **not** use `dpkg-buildpackage`. Hand-rolled sequence: ephemeral unshare chroot → `apt-get update` → essentials → manual `quilt push -a` → `apt-get build-dep` → `debian/rules build` → `fakeroot debian/rules binary` → retrieve `.deb`s.
What it silently skips compared to a real build:
- `debian/rules clean` before building,
- `dpkg-source --before-build/--after-build` lifecycle (it re-implements patch application crudely),
- `.buildinfo` and `.changes` generation,
- build-conflicts checking, `Rules-Requires-Root` semantics,
- `SOURCE_DATE_EPOCH` export (reproducibility).
So pkh already pays part of the "native orchestrator" cost without getting its benefits. A native implementation would unify both paths.
### Existing native assets to reuse
- Changelog header/footer parsing — [`changelog.rs`](../src/changelog.rs) (needs extension to full metadata: timestamp, urgency, trailer key/values).
- GPG key discovery — [`utils/gpg.rs`](../src/utils/gpg.rs) via `gpgme`.
- Hashing crates already present: `sha2`, `md-5` (need to add `sha1`); compression: `flate2`, `xz2`; archives: `tar`.
- Command execution through remote-capable contexts — [`ContextCommand`](../src/context/mod.rs) (local/ssh/schroot/unshare/capture).
- Live UI phases — [`ui/deb.rs`](../src/ui/deb.rs).
---
## 2. What `dpkg-buildpackage` actually is (measured)
It is a Perl script (`use v5.36`). Measured sizes on `main`:
| Component | Lines | Role |
|---|---|---|
| `scripts/dpkg-buildpackage.pl` | 1209 | Orchestrator |
| `scripts/dpkg-source.pl` | 806 | Driver; work in `Dpkg/Source/*` |
| `scripts/dpkg-genbuildinfo.pl` | 637 | `.buildinfo` generation |
| `scripts/dpkg-genchanges.pl` | 626 | `.changes` generation |
| `scripts/dpkg-architecture.pl` | 498 | Arch name/triplet tables |
| `scripts/dpkg-gensymbols.pl` | 398 | (called by rules, not by us) |
| `scripts/dpkg-checkbuilddeps.pl` | 270 | Dep checking vs status file |
| `scripts/dpkg-parsechangelog.pl` | 195 | Changelog CLI wrapper |
| `scripts/dpkg-distaddfile.pl` | 99 | Registers files in `debian/files` |
| `Dpkg/*.pm` + `Dpkg/Source/*.pm` (subset examined) | ~11,600 | Shared library code |
Total relevant Perl surface ≈ **16k LOC**, of which the orchestrator is only ~8%.
---
## 3. The exact pipeline (verified against source)
What `dpkg-buildpackage` does, in order:
1. **Config**: load `buildpackage.conf` (Dpkg::Conf), inject as leading argv.
2. **Option parsing** (~200 lines): build types `-F/-g/-G/-b/-B/-A/-S` (+ `--build=full,source,binary,any,all`), signing (`-us/-uc/-ui/-k/-p/--no-sign/--force-sign`), `-j/-J/--jobs-force`, `-r<root-cmd>`, `-R<rules>`, `-T<targets>`, `-a/-t/--target-arch`, `-P<profiles>`, `-d/-D`, `-nc/-tc`, hooks `--hook-<name>=<cmd>`, passthrough buckets for `dpkg-source` / `dpkg-genchanges` / `dpkg-genbuildinfo`.
3. **Build type → rules targets**: `binary`→`build`+`binary`; arch-dep only→`build-arch`+`binary-arch`; indep only→`build-indep`+`binary-indep`.
4. **Implied flags**: `-nc` alone implies `-b`; `-nc -S` disables build-dep checks.
5. **Environment prep**:
- `parallel=auto` default → `DEB_BUILD_OPTIONS=parallel=N` exported;
- forced jobs additionally appended to `MAKEFLAGS`;
- `DEB_BUILD_PROFILES` exported if `-P`;
- optional `.dsc` input → `dpkg-source --extract` first;
- `SOURCE_DATE_EPOCH ||= changelog timestamp || time()` (reproducible-builds.org spec);
- full env dump of `dpkg-architecture -f [-a…][-t…]` imported into the environment (all `DEB_BUILD_*`, `DEB_HOST_*`, `DEB_TARGET_*` incl. `*_OS/CPU/MULTIARCH/GNU_TYPE/ARCH_BITS/ENDIAN`);
- OpenPGP key resolution: `--sign-keyfile` > `--sign-keyid` > maintainer userid; secrets probed up-front; **UNRELEASED distribution ⇒ signing disabled** unless `--force-sign`.
6. **Pre-flight**: `Dpkg::BuildDriver->pre_check()` (rules file exists/executable); `dpkg-source --before-build .` (applies patches for quilt formats); `dpkg-checkbuilddeps [-A|-B|-I]` unless `-d` (exit 3 on unmet).
7. **Hooks** at 12 points: `preinit init preclean source build binary buildinfo changes postclean check sign done`, with `%p/%v/%s/%u/%a` substitution.
8. **Preclean**: `debian/rules clean` via BuildDriver (gain-root per `Rules-Requires-Root`).
9. **Source build** (if any SOURCE component): `dpkg-source -b .` → `.dsc` + tarballs in `..`.
10. **Binary build** (if any BINARY component): `run_build_task(build-target)` then `binary-target` through BuildDriver (RRR-aware root command; skips separate non-root `build` pass when running rootless).
11. **Metadata generation**:
- `dpkg-genbuildinfo` → `../<pkg>_<ver_noepoch>_<arch>.buildinfo` (Format 1.0; records `Installed-Build-Depends` snapshot from the dpkg status DB, build environment, checksums);
- `dpkg-genchanges` → `../<pkg>_<ver_noepoch>_<arch>.changes` (Format 1.8; aggregates `debian/files` + changelog + control).
12. **Post**: optional `-tc` clean; `dpkg-source --after-build .` (unapplies patches it applied); human summary ("full upload (original source is included)" etc. derived from the `Files` field); optional lintian-style check command.
13. **Signing cascade** (inline/clearsig via OpenPGP backend gpg|sequoia|sop):
- sign `<pkg>_<ver>.dsc` → recompute its checksums **inside `.buildinfo`**;
- sign `.buildinfo` → recompute dsc+buildinfo checksums **inside `.changes`** (rewriting both `Checksums-*` and legacy `Files` entries);
- sign `.changes`.
Exit codes matter: e.g. unsatisfied build-deps ⇒ exit 3.
---
## 4. Sub-tool inventory and replacement strategy
| Tool | Used for | Complexity to replace natively | Strategy |
|---|---|---|---|
| `dpkg-parsechangelog` | source/version/maintainer/distribution/timestamp | **Low** — documented format; crates exist (`debian-changelog`, `deb822-parser` ecosystem) | Replaced (see §11, [`metadata.rs`](../src/build/metadata.rs)) |
| `dpkg-version` compare | epoch/upstream/revision ordering | **Low** — small well-specified algorithm; crate `debversion` | **Replaced** (§11, [`debian/version.rs`](../src/debian/version.rs): `Ord`/`compare`/`later_than`) |
| `dpkg-architecture` | arch ↔ triplet tables, multiarch tuple, env dump | **Low-medium** — embed cputable/ostable/tupletable/abitable data (stable for years) | **Replaced** (§11, [`debian/arch.rs`](../src/debian/arch.rs)) |
| `dpkg-checkbuilddeps` | deps vs installed status | **Medium** — `Dpkg::Deps` grammar (alternatives, arch qualifiers, `<profiles>` restrictions, versioned Provides subtleties, Multi-Arch facts) + status-file scan | **Replaced** (§11, [`debian/deps.rs`](../src/debian/deps.rs); wired into the pipeline behind `-D`, source-only builds skip it like dpkg-buildpackage) |
| `dpkg-genbuildinfo` | `.buildinfo` | **Medium** — deb822 emit + status snapshot + checksums | Native (see §11, [`buildinfo.rs`](../src/build/buildinfo.rs)) |
| `dpkg-genchanges` | `.changes` | **Medium** — deb822 emit + `debian/files` consumption + `.deb` control extraction (ar+tar, trivial with crates) | **Replaced** for source (§11) and binary uploads (§11, [`build/binary.rs`](../src/build/binary.rs)) |
| `dpkg-genbuildinfo` (binary) | `.buildinfo` for `-b` builds | **Medium** — deb822 emit + in-context artifact hashing | **Replaced** (§11, [`build/binary.rs`](../src/build/binary.rs), wired into [`deb/local.rs`](../src/deb/local.rs)) |
| `dpkg-distaddfile`/`debian/files` protocol | build outputs registry | **Trivial** — one append-only line format | Native ([`files.rs`](../src/build/files.rs)) |
| OpenPGP signing | inline clearsign of dsc/buildinfo/changes | **Low** — `gpgme` (already a dependency) supports clearsigning | Native ([`sign.rs`](../src/build/sign.rs)) |
| `dpkg-source` | orig tarball, debian diff, patches, `.dsc` | **Very high** — V1/V2/quilt/native formats, byte-exact tar normalization, quilt bookkeeping, `--include-binaries`, hundreds of validation warnings | **Keep as subprocess** (see §6) |
| `debian/rules` execution | the actual build | N/A (foreign code) | Keep, via `ContextCommand` |
---
## 5. Feasibility detail per scope
### Scope A — native orchestrator (recommended)
The 1209-line script decomposes into clean Rust pieces:
```
src/build/
├── mod.rs // pipeline driver (the equivalent of main())
├── buildtype.rs // BUILD_SOURCE|ARCH_DEP|ARCH_INDEP bitflags + target mapping
├── metadata.rs // changelog + control resolution, version splitting
├── env.rs // SOURCE_DATE_EPOCH, DEB_BUILD_OPTIONS, arch env dump
├── buildinfo.rs // native .buildinfo writer
├── changes.rs // native .changes writer
├── files.rs // debian/files registry emulation
├── checksums.rs // md5/sha1/sha256 registry
├── control.rs // deb822 parser/writer
└── sign.rs // gpgme clearsign + post-sign checksum cascade
```
Estimated ~1.5–3 kLOC. Everything is deterministic file munging + subprocess sequencing — no daemons, no parsing of arbitrary upstream code (that's `dpkg-source`'s job, which we keep).
**pkh-specific simplifications** (legitimate because pkh controls the environment):
- Builds run as **real root** inside ephemeral unshare chroots ⇒ the entire gain-root/fakeroot matrix collapses: run `debian/rules` directly when the context is root (keep `fakeroot` fallback for host-side source builds).
- pkh's option surface is a fraction of dpkg's: `-S/-b/-B/-A/-g/-G`, `-us/-uc`, `-k`, `-j`, `-a`, `-P`, `-d/-D`, `-nc/-tc`, `-R`, `-T` cover everything pkh passes today plus obvious headroom. Hooks and `--hook-*` can be dropped initially.
- Vendor hooks (`run_vendor_hook`) are rarely used in the build path; Ubuntu/Debian differences pkh cares about are already handled via [`distro_info.yml`](../distro_info.yml).
**What this buys pkh concretely:**
1. Live UI + log classification for source builds (today's `-S` path is a black box).
2. `.changes`/`.buildinfo` for binary builds — currently missing entirely; needed for uploads/PPA submissions and lintian checks.
3. Correct `dpkg-source --before-build/--after-build` lifecycle replacing the manual `quilt push -a` hack in [`local.rs`](../src/deb/local.rs) (handles `3.0 (quilt)` properly, incl. unapply-on-exit and format detection instead of sniffing `debian/patches/series`).
4. `SOURCE_DATE_EPOCH` reproducibility for free.
5. No host `dpkg-dev` requirement for orchestration decisions; `dpkg-source` still needed inside build environments where pkh already installs packages anyway.
### Scope B — satellite tools
Incremental, each independently testable against the real tool (differential testing). Crates from the `rust-debian-*` ecosystem (maintained by Jelmer Vernooij) cover most parsing: `deb822-parser`, `debian-control`, `debian-changelog`, `debversion`. Priority order if pursued: version compare → changelog → architecture tables → checkbuilddeps.
### Scope C — `dpkg-source`
The honest numbers: `Dpkg/Source/Package/V2.pm` alone is 847 lines, V1 is 599, plus Archive/Quilt/Functions/BinaryFiles modules, GNU diff generation, and — critically — **byte-level tar normalization** (mtime/uid/gid/mode canonicalization, pax header handling) that reproducible builds depend on. Parity means being bug-compatible with dpkg against ~40k archive source packages. This is a multi-month project with a long tail, and it buys pkh almost nothing since `dpkg-source` is guaranteed present inside the very chroots pkh creates. **Do not do this.**
---
## 6. Proposed phasing
### Phase 0 — quick win (days)
Route the existing `dpkg-buildpackage -S` call through the UI capture machinery (like [`cap()`](../src/deb/local.rs) does for binary builds): phase display + `LineSink` classification. No behavior change otherwise.
### Phase 1 — native orchestrator (2–4 weeks)
Implement §5 scope A. Both entry points converge:
- `pkh build -S` → native pipeline, `dpkg-source -b` subprocess, native signing.
- `pkh build` (binary) → same pipeline skeleton inside the ephemeral context: preclean → before-build → dep check → `rules build` → `rules binary[-arch|-indep]` → native buildinfo/changes → after-build. Deletes the manual quilt logic.
Deliverables:
- `src/build/*` module tree (§5).
- Native `.buildinfo` (Format 1.0) and `.changes` (Format 1.8) writers emitting exactly the field sets observed in `dpkg-genbuildinfo.pl` / `dpkg-genchanges.pl`, in dpkg's canonical field order:
- buildinfo: `Format, Source, Binary, Architecture, Version, Binary-Only-Changes, Checksums-Md5/Shа1/Sha256, Build-Origin, Build-Architecture, Build-Kernel-Version(opt), Build-Date, Build-Path(opt), Build-Tainted-By(opt), Installed-Build-Depends, Environment`
- changes: `Format, Date, Source, Binary, Built-For-Profiles, Architecture, Version, Distribution, Urgency, Maintainer, Changed-By, Description, Changes, Checksums-Sha1/Sha256, Files`
- Post-signature checksum cascade implemented exactly as dpkg does (sign dsc → patch buildinfo checksums → sign buildinfo → patch changes `Files`+`Checksums-*` → sign changes).
- Differential test harness (§8).
### Phase 2 — satellite replacement (optional, incremental)
Swap subprocesses for native implementations, one tool at a time, gated by differential tests. Start with version-compare + changelog (already half-present in pkh).
### Explicitly out of scope
`dpkg-source` internals, `dpkg-deb` packing, apt resolution (pkh correctly delegates to `apt-get build-dep` + dose3 for explanations already).
---
## 7. Tricky details to get right (gotchas list)
1. **Version splitting**: `<epoch:>upstream<-revision>`; `.dsc`/tarball names use *upstream* portion without epoch; `.changes` name uses version **without epoch** but **with revision** (`$sversion` in the script).
2. **`-nc` implication chain**: `-nc` ⇒ binary build implied; `-nc -S` ⇒ no dep check.
3. **UNRELEASED** ⇒ auto-disable all signing (warn), `--force-sign` overrides.
4. **Signing invalidates checksums transitively** (dsc → buildinfo → changes); get the cascade order right or archive tools reject the upload.
5. **`debian/files`** is the contract between `debian/rules` (via `dh_builddeb`/`dpkg-gencontrol`/`dpkg-distaddfile`) and the changes generator: lines of `filename section priority [key=value...]`.
6. **Arch selection for names**: `arch` suffix in artifact filenames is `host-arch` for arch-dep builds, `all` for indep-only, `source` for source-only.
7. **`dpkg-source --before-build` must run even for binary-only builds** (patch application), and `--after-build` at the end — this replaces pkh's current manual quilt step and fixes `3.0 (quilt)` correctness.
8. **Environment parity**: `dpkg-architecture -f` dump must be imported wholesale (not cherry-picked) — packages test `DEB_HOST_GNU_TYPE`, `DEB_BUILD_MULTIARCH`, `DEB_BUILD_ARCH_ENDIAN`, etc. If embedding tables later, mirror the full variable set.
9. **Exit codes**: preserve dpkg conventions (3 = unmet build-deps) so wrappers/scripts behave identically.
10. **Locale**: dpkg sets `LANG=C`-ish determinism for subprocesses — pkh already does this in [`local.rs`](../src/deb/local.rs); keep for all pipeline steps.
11. **Multiline field rendering**: values starting with `\n` render as `Field:` + indented continuation lines (no inline first line, no trailing space) — this is how dpkg emits `Changes`, `Files`, `Installed-Build-Depends`, `Environment`.
12. **Artifact ordering** in `Checksums-*`/`Files` is insertion order (dsc → tarballs in dsc-field order → debs → buildinfo), not alphabetical.
---
## 8. Testing strategy
- **Differential harness**: run real `dpkg-buildpackage` and the native pipeline over a corpus (packages covering: `1.0` non-native, `3.0 (quilt)` with/without patches, native, binaries-only, indep-only, cross, RRR variants, UNRELEASED) and diff artifacts modulo timestamps/signatures.
- **Port unit cases** from dpkg's own `t/` tests for version compare, deps parsing, changelog parsing.
- **Golden-file tests** for `.changes`/`.buildinfo` writers.
- Pin the reference dpkg version in CI commentary (behavior drift across dpkg releases is the main maintenance cost).
---
## 9. Risks
| Risk | Mitigation |
|---|---|
| Behavior drift vs future dpkg releases (new fields, format bumps) | Differential tests pinned to a reference version; changes/buildinfo formats are extremely stable (1.8 / 1.0 for years) |
| Archive/upload tooling rejects our `.changes`/`.buildinfo` | Validate with `lintian` + a real PPA upload early in Phase 1 |
| License contamination | dpkg is GPL-2+. Write from documented behavior/format specs and observation, **do not transliterate Perl**; alternatively accept GPL for pkh (currently no license field in [`Cargo.toml`](../Cargo.toml) — decision needed) |
| Scope creep toward Scope C | Hard rule: `dpkg-source` stays a subprocess |
| Remote-context divergence | All tree-touching steps go through [`ContextCommand`](../src/context/mod.rs); metadata steps operate on locally-synced copies like [`changelog.rs`](../src/changelog.rs) already does |
---
## 10. Effort summary
| Item | Estimate |
|---|---|
| Phase 0 (UI capture for `-S`) | 1–2 days |
| Phase 1 (native orchestrator + buildinfo/changes/signing) | 2–4 weeks |
| Phase 2 per satellite tool | 3 days – 2 weeks each |
| Scope C (`dpkg-source`) | 2–3+ months — rejected |
Bottom line: **yes, it's doable — for the orchestrator.** Treat `dpkg-source` as a permanent subprocess dependency, and the task becomes a well-bounded, high-value refactor that also fixes real gaps in pkh's binary path.
---
## 11. Implementation status (Phase 1 — source builds)
Landed in `src/build/` (~2.8 kLOC incl. tests), wired behind the historical
entry point [`build_source_package()`](../src/build/mod.rs) so `pkh build`
now runs the native pipeline:
| Module | Role |
|---|---|
| [`control.rs`](../src/build/control.rs) | deb822 paragraph parser/writer (dpkg-compatible multiline rendering) |
| [`checksums.rs`](../src/build/checksums.rs) | md5/sha1/sha256 registry, insertion-ordered like dpkg's artifact accumulation |
| [`metadata.rs`](../src/build/metadata.rs) | version splitting/validation, full changelog entry parse (incl. binNMU `binary-only`), control info |
| [`buildtype.rs`](../src/build/buildtype.rs) | build-type bitflags + rules-target/artifact-suffix mapping |
| [`env.rs`](../src/build/env.rs) | `SOURCE_DATE_EPOCH`, `DEB_BUILD_OPTIONS`, `dpkg-architecture` env dump, vendor/profiles, sanitized `Environment` field |
| [`files.rs`](../src/build/files.rs) | `debian/files` registry (parse/atomic save) |
| [`buildinfo.rs`](../src/build/buildinfo.rs) | `.buildinfo` writer + `Installed-Build-Depends` closure over the dpkg status DB |
| [`changes.rs`](../src/build/changes.rs) | `.changes` writer (canonical field order, legacy `Files` + `Checksums-Sha1/256`) |
| [`sign.rs`](../src/build/sign.rs) | gpgme clearsigning + key-id validation |
Still delegated to subprocesses: `dpkg-source -b/--before-build/--after-build`
(by design, see §5 Scope C).
**Differential validation** — automated in
[`build/mod.rs`](../src/build/mod.rs) (`mod differential_tests`, runs by
default with `cargo test --lib`):
- a corpus of 12 synthetic fixtures covering: native/quilt/1.0 formats,
epochs, `~` pre-releases, Ubuntu/Debian series (focal, noble, jammy,
trixie, unstable), high urgency, multiple binary stanzas, one/two quilt
patches, binNMU (`binary-only=yes` + previous-entry metadata),
`(Closes: #…)` extraction, UNRELEASED no-sign, extra source-stanza fields;
- each case builds the tree twice (`cp -a` copies) — once with real
`dpkg-buildpackage -S -I -i -nc -d --no-sign`, once with the native
pipeline — then compares:
- `.dsc` payload byte-for-byte,
- `.changes` field-by-field (checksum lines of the `.buildinfo` itself
excluded),
- `.buildinfo` structure (machine-dependent fields excluded);
- real **archive packages** are pulled with pkh's own
[`pull`](../src/pull.rs) (archive download mode) and compared the same
way, via [`differential_real_archive_package()`](../src/build/mod.rs)
which takes `(package, dist, series)`:
- always-on CI tests: `hello` @ ubuntu/noble, `dosfstools` @
debian/trixie, `sl` @ ubuntu/focal;
- an additional `#[ignore]`-gated test for ad-hoc broad runs:
```text
PKH_DIFF_PACKAGES="bash coreutils curl" PKH_DIFF_DIST=ubuntu \
PKH_DIFF_SERIES=noble cargo test --lib \
differential_real_archive_packages -- --ignored
```
(requires network access).
Manual validation additionally confirmed: signed builds verify with
`gpg --verify` on all three artifacts; the patch apply/unapply lifecycle is
correct for quilt formats.
**Known divergences** (documented, all informational fields): no
`Build-Tainted-By` (vendor hook), no `dpkg-buildflags` origin tracking in the
`Environment` field, vendor default profiles approximated
(`derivative.ubuntu noudeb` for Ubuntu).
## 12. Implementation status (Phase 2 — satellite tools)
All four satellite replacements landed, each gated by differential tests
against the real tool:
| Work item | Module | Differential gate |
|---|---|---|
| WI-1 `dpkg-architecture` | [`debian/arch.rs`](../src/debian/arch.rs) (tables + lookups), wired into [`build/env.rs`](../src/build/env.rs) | `arch_env(Some(a))` equals real `dpkg-architecture -f -a a` key-for-key for **every** arch from `dpkg-architecture -L`, plus native (`build/mod.rs::diff_arch_env_*`) |
| WI-2 version compare | [`debian/version.rs`](../src/debian/version.rs) (`Ord`, `compare`, `later_than`) | all vectors from dpkg `scripts/t/Dpkg_Version.t` + Ubuntu-flavored cases, cross-checked against `dpkg --compare-versions` for `<< <= = >= >>` (`diff_version_compare_against_dpkg`) |
| WI-3 `dpkg-checkbuilddeps` | [`debian/deps.rs`](../src/debian/deps.rs) (grammar, restriction reduction, KnownFacts evaluation, `check_build_depends`) | 24 scenarios vs real `dpkg-checkbuilddeps` (alternatives, versions, arch/profile restrictions, Multi-Arch, versioned Provides, conflicts, `-A`/`-B`) comparing exit status + diagnostics (`diff_checkbuilddeps_matrix`); unit tests port the `Dpkg_Deps.t` reduction matrices. Wired into `run_source_build` behind `-D` parity: source-only builds skip the check like `dpkg-buildpackage`, unsatisfied deps exit 3 |
| WI-4 binary `.buildinfo`/`.changes` | [`build/binary.rs`](../src/build/binary.rs) (context-generic generation), wired into [`deb/local.rs`](../src/deb/local.rs); artifact retrieval extended in [`deb/mod.rs`](../src/deb/mod.rs) | same tree built with real `dpkg-buildpackage -b` and with the pkh flow (rules build/binary + native metadata through a local context): `.changes`/`.buildinfo` compared field-by-field modulo machine-dependent fields, artifact checksums included (`diff_binary_build_metadata`) |
Binary-flow notes: `SOURCE_DATE_EPOCH` is now exported to the rules
environment (reproducibility); digests are computed inside the context via
coreutils so remote/chrooted trees work; binNMU binary builds set
`Source: pkg (prev)` / `Binary-Only-Changes` and redistribute the previous
`.dsc` when present; `Architecture` is encounter-ordered in `.changes`
(sorted in `.buildinfo`, matching dpkg).
Still delegated to subprocesses: `dpkg-source` and `debian/rules` (by
design, see §5 Scope C).
+2 -10
View File
@@ -12,26 +12,20 @@ pub fn num_parallel() -> usize {
.unwrap_or(1) .unwrap_or(1)
} }
/// Compute the environment variables exported before running any build step. /// Compute the environment variables exported by `dpkg-buildpackage` before
/// running any build step.
/// ///
/// Mirrors dpkg behavior: /// Mirrors dpkg behavior:
/// - `SOURCE_DATE_EPOCH` from the changelog entry timestamp /// - `SOURCE_DATE_EPOCH` from the changelog entry timestamp
/// (<https://reproducible-builds.org/specs/source-date-epoch/>), /// (<https://reproducible-builds.org/specs/source-date-epoch/>),
/// - `DEB_BUILD_OPTIONS=parallel=N` (auto-detected job count), /// - `DEB_BUILD_OPTIONS=parallel=N` (auto-detected job count),
/// - `DEB_BUILD_PROFILES` when non-default profiles are requested. /// - `DEB_BUILD_PROFILES` when non-default profiles are requested.
///
/// The locale is pinned to `C` (`LC_ALL`, which takes precedence over any
/// inherited session setting, plus `LANG`) so build tools emit deterministic,
/// English diagnostics — required for reliable log classification and
/// reproducible builds.
pub fn build_env( pub fn build_env(
source_date_epoch: i64, source_date_epoch: i64,
parallel: usize, parallel: usize,
build_profiles: &[String], build_profiles: &[String],
) -> BTreeMap<String, String> { ) -> BTreeMap<String, String> {
let mut env = BTreeMap::new(); let mut env = BTreeMap::new();
env.insert("LANG".to_string(), "C".to_string());
env.insert("LC_ALL".to_string(), "C".to_string());
env.insert( env.insert(
"SOURCE_DATE_EPOCH".to_string(), "SOURCE_DATE_EPOCH".to_string(),
source_date_epoch.to_string(), source_date_epoch.to_string(),
@@ -263,8 +257,6 @@ mod tests {
#[test] #[test]
fn build_env_values() { fn build_env_values() {
let env = build_env(1787392800, 16, &[]); let env = build_env(1787392800, 16, &[]);
assert_eq!(env.get("LANG").unwrap(), "C");
assert_eq!(env.get("LC_ALL").unwrap(), "C");
assert_eq!(env.get("SOURCE_DATE_EPOCH").unwrap(), "1787392800"); assert_eq!(env.get("SOURCE_DATE_EPOCH").unwrap(), "1787392800");
assert_eq!(env.get("DEB_BUILD_OPTIONS").unwrap(), "parallel=16"); assert_eq!(env.get("DEB_BUILD_OPTIONS").unwrap(), "parallel=16");
assert!(!env.contains_key("DEB_BUILD_PROFILES")); assert!(!env.contains_key("DEB_BUILD_PROFILES"));
+24 -135
View File
@@ -15,16 +15,11 @@ pub mod env;
use std::collections::{BTreeMap, HashMap}; use std::collections::{BTreeMap, HashMap};
use std::error::Error; use std::error::Error;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use std::process::{Command, Stdio}; use std::process::Command;
use std::sync::Arc;
use crate::context::capture::pump;
use crate::context::{LineSink, Stream};
use crate::debian::{ use crate::debian::{
ChecksumEntry, ControlInfo, FileChecksums, FilesEntry, FilesList, parse_paragraphs, ChecksumEntry, ControlInfo, FileChecksums, FilesEntry, FilesList, parse_paragraphs,
}; };
use crate::ui::deb::DebUi;
use crate::ui::logfmt::{DpkgSourceClassifier, GenericClassifier};
/// Options for a native source-package build. /// Options for a native source-package build.
#[derive(Debug, Clone, Default)] #[derive(Debug, Clone, Default)]
@@ -57,46 +52,11 @@ pub struct SourceBuildOutput {
/// Build a Debian source package (to a .dsc) using the native pipeline. /// Build a Debian source package (to a .dsc) using the native pipeline.
/// ///
/// When `ui` is set, subprocess output is captured into a live view (status /// Keeps the historical pkh entry-point signature; see [`run_source_build`]
/// bar + rolling pane) and tee'd to a log file; on failure the view prints a /// for the configurable version.
/// summary of the last captured errors. Without a UI, commands inherit the pub fn build_source_package(cwd: Option<&Path>) -> Result<(), Box<dyn Error>> {
/// terminal as before.
pub fn build_source_package(
cwd: Option<&Path>,
ui: Option<Arc<DebUi>>,
) -> Result<(), Box<dyn Error>> {
let cwd = cwd.unwrap_or_else(|| Path::new(".")); let cwd = cwd.unwrap_or_else(|| Path::new("."));
let output = match run_source_build(cwd, &SourceBuildOptions::default(), ui.clone()) { let output = run_source_build(cwd, &SourceBuildOptions::default())?;
Ok(output) => output,
Err(e) => {
if let Some(u) = &ui {
u.finish_failure();
}
return Err(e);
}
};
// Artifact listing in dpkg order: dsc → tarballs → buildinfo → changes.
let mut artifacts = Vec::with_capacity(3 + output.tarballs.len());
artifacts.push(output.dsc.clone());
artifacts.extend(output.tarballs.iter().cloned());
artifacts.push(output.buildinfo.clone());
artifacts.push(output.changes.clone());
// The live view lists the artifacts itself when it renders; otherwise
// (verbose mode or non-TTY stdout) print them as plain lines.
let listed = match &ui {
Some(u) if u.is_enabled() => {
u.finish_success(&artifacts, u.elapsed());
true
}
_ => false,
};
if !listed {
for artifact in &artifacts {
println!(" {}", crate::ui::display_path(artifact));
}
}
if output.signed { if output.signed {
println!("Package built and signed successfully!"); println!("Package built and signed successfully!");
@@ -121,9 +81,7 @@ pub fn build_source_package(
pub fn run_source_build( pub fn run_source_build(
cwd: &Path, cwd: &Path,
opts: &SourceBuildOptions, opts: &SourceBuildOptions,
ui: Option<Arc<DebUi>>,
) -> Result<SourceBuildOutput, Box<dyn Error>> { ) -> Result<SourceBuildOutput, Box<dyn Error>> {
let sink: Option<Arc<dyn LineSink>> = ui.as_ref().map(|u| u.sink());
// ------------------------------------------------------------------ // ------------------------------------------------------------------
// 1. Sanity checks // 1. Sanity checks
// ------------------------------------------------------------------ // ------------------------------------------------------------------
@@ -158,9 +116,9 @@ pub fn run_source_build(
let entry = crate::debian::parse_changelog_entry(&changelog_path)?; let entry = crate::debian::parse_changelog_entry(&changelog_path)?;
let ctrl = ControlInfo::parse(&control_path)?; let ctrl = ControlInfo::parse(&control_path)?;
if let Some(u) = &ui { log::info!("source package {}", entry.source);
u.set_build_target(&entry.source, &entry.version.full(), &entry.distribution); log::info!("source version {}", entry.version.full());
} log::info!("source distribution {}", entry.distribution);
// binNMU builds reference the *previous* (source) version in their // binNMU builds reference the *previous* (source) version in their
// artifact metadata, like dpkg-genchanges/genbuildinfo do. // artifact metadata, like dpkg-genchanges/genbuildinfo do.
@@ -209,19 +167,19 @@ pub fn run_source_build(
if signing_key.is_none() { if signing_key.is_none() {
match crate::utils::gpg::find_signing_key_for_email(&entry.maintainer_email) { match crate::utils::gpg::find_signing_key_for_email(&entry.maintainer_email) {
Ok(Some(key)) => { Ok(Some(key)) => {
log::info!("Using GPG key {} for signing", key); log::info!("using GPG key {} for signing", key);
signing_key = Some(key); signing_key = Some(key);
} }
Ok(None) => { Ok(None) => {
log::warn!( log::warn!(
"No GPG secret key found for {} <{}>, building without signing", "no GPG secret key found for {} <{}>, building without signing",
entry.maintainer_name, entry.maintainer_name,
entry.maintainer_email entry.maintainer_email
); );
} }
Err(e) => { Err(e) => {
log::warn!( log::warn!(
"Failed to check for GPG key: {}, building without signing", "failed to check for GPG key: {}, building without signing",
e e
); );
} }
@@ -230,7 +188,7 @@ pub fn run_source_build(
let do_sign = match &signing_key { let do_sign = match &signing_key {
None => false, None => false,
Some(_) if entry.distribution == "UNRELEASED" && !opts.force_sign => { Some(_) if entry.distribution == "UNRELEASED" && !opts.force_sign => {
log::warn!("Not signing UNRELEASED build; use force_sign to override"); log::warn!("not signing UNRELEASED build; use force_sign to override");
false false
} }
Some(_) => true, Some(_) => true,
@@ -239,24 +197,17 @@ pub fn run_source_build(
// ------------------------------------------------------------------ // ------------------------------------------------------------------
// 5. dpkg-source lifecycle: before-build + source build // 5. dpkg-source lifecycle: before-build + source build
// ------------------------------------------------------------------ // ------------------------------------------------------------------
if let Some(u) = &ui {
u.phase_custom("Applying patches", Box::new(DpkgSourceClassifier::new()));
}
run_command( run_command(
cwd, cwd,
"dpkg-source", "dpkg-source",
&["-I", "-i", "--before-build", "."], &["-I", "-i", "--before-build", "."],
&pipeline_env, &pipeline_env,
sink.as_ref(),
)?; )?;
// Build-dependency check (native dpkg-checkbuilddeps equivalent). // Build-dependency check (native dpkg-checkbuilddeps equivalent).
// dpkg-buildpackage skips it entirely for source-only builds unless // dpkg-buildpackage skips it entirely for source-only builds unless
// forced with -D; unsatisfied dependencies abort with exit status 3. // forced with -D; unsatisfied dependencies abort with exit status 3.
if opts.force_dep_check { if opts.force_dep_check {
if let Some(u) = &ui {
u.progress_message("Checking build dependencies");
}
let check_opts = crate::debian::deps::CheckOpts { let check_opts = crate::debian::deps::CheckOpts {
host_arch: arch_vars host_arch: arch_vars
.get("DEB_HOST_ARCH") .get("DEB_HOST_ARCH")
@@ -274,19 +225,7 @@ pub fn run_source_build(
} }
} }
if let Some(u) = &ui { run_command(cwd, "dpkg-source", &["-I", "-i", "-b", "."], &pipeline_env)?;
u.phase_custom(
"Building source package",
Box::new(DpkgSourceClassifier::new()),
);
}
run_command(
cwd,
"dpkg-source",
&["-I", "-i", "-b", "."],
&pipeline_env,
sink.as_ref(),
)?;
if !dsc_path.exists() { if !dsc_path.exists() {
return Err(format!( return Err(format!(
@@ -320,9 +259,6 @@ pub fn run_source_build(
// ------------------------------------------------------------------ // ------------------------------------------------------------------
// 6. .buildinfo generation (native dpkg-genbuildinfo equivalent) // 6. .buildinfo generation (native dpkg-genbuildinfo equivalent)
// ------------------------------------------------------------------ // ------------------------------------------------------------------
if let Some(u) = &ui {
u.progress_message("Generating .buildinfo");
}
let mut checksums = FileChecksums::new(); let mut checksums = FileChecksums::new();
checksums.add_file_as(&ref_dsc_path, &ref_dsc_name)?; checksums.add_file_as(&ref_dsc_path, &ref_dsc_name)?;
@@ -368,9 +304,6 @@ pub fn run_source_build(
// ------------------------------------------------------------------ // ------------------------------------------------------------------
// 7. .changes generation (native dpkg-genchanges equivalent) // 7. .changes generation (native dpkg-genchanges equivalent)
// ------------------------------------------------------------------ // ------------------------------------------------------------------
if let Some(u) = &ui {
u.progress_message("Generating .changes");
}
// Pull the tarball checksums out of the referenced .dsc so they are // Pull the tarball checksums out of the referenced .dsc so they are
// distributed through the .changes like dpkg-genchanges does, in the // distributed through the .changes like dpkg-genchanges does, in the
// order the .dsc itself lists them. // order the .dsc itself lists them.
@@ -483,15 +416,11 @@ pub fn run_source_build(
// ------------------------------------------------------------------ // ------------------------------------------------------------------
// 8. dpkg-source after-build (unapplies quilt patches it applied) // 8. dpkg-source after-build (unapplies quilt patches it applied)
// ------------------------------------------------------------------ // ------------------------------------------------------------------
if let Some(u) = &ui {
u.phase_custom("Restoring patches", Box::new(DpkgSourceClassifier::new()));
}
run_command( run_command(
cwd, cwd,
"dpkg-source", "dpkg-source",
&["-I", "-i", "--after-build", "."], &["-I", "-i", "--after-build", "."],
&pipeline_env, &pipeline_env,
sink.as_ref(),
)?; )?;
// ------------------------------------------------------------------ // ------------------------------------------------------------------
@@ -501,11 +430,7 @@ pub fn run_source_build(
if let Some(keyid) = signing_key.filter(|_| do_sign) { if let Some(keyid) = signing_key.filter(|_| do_sign) {
crate::utils::gpg::validate_key_id(&keyid)?; crate::utils::gpg::validate_key_id(&keyid)?;
if let Some(u) = &ui { println!("signfile {}", dsc_name);
u.phase_custom("Signing artifacts", Box::new(GenericClassifier::new()));
}
log::info!("Signing {}", dsc_name);
crate::utils::gpg::clearsign_file(&dsc_path, &keyid)?; crate::utils::gpg::clearsign_file(&dsc_path, &keyid)?;
// The freshly built .dsc changed: refresh its checksums inside the // The freshly built .dsc changed: refresh its checksums inside the
// .buildinfo. For binary-only builds the metadata references the // .buildinfo. For binary-only builds the metadata references the
@@ -516,13 +441,13 @@ pub fn run_source_build(
} }
buildinfo::save_buildinfo(&buildinfo_path, &render_buildinfo_doc(&checksums))?; buildinfo::save_buildinfo(&buildinfo_path, &render_buildinfo_doc(&checksums))?;
log::info!("Signing {}", buildinfo_name); println!("signfile {}", buildinfo_name);
crate::utils::gpg::clearsign_file(&buildinfo_path, &keyid)?; crate::utils::gpg::clearsign_file(&buildinfo_path, &keyid)?;
// Both .dsc and .buildinfo changed: refresh the .changes. // Both .dsc and .buildinfo changed: refresh the .changes.
checksums.add_file_as(&buildinfo_path, &buildinfo_name)?; checksums.add_file_as(&buildinfo_path, &buildinfo_name)?;
changes::save_changes(&changes_path, &render_changes_doc(&checksums))?; changes::save_changes(&changes_path, &render_changes_doc(&checksums))?;
log::info!("Signing {}", changes_name); println!("signfile {}", changes_name);
crate::utils::gpg::clearsign_file(&changes_path, &keyid)?; crate::utils::gpg::clearsign_file(&changes_path, &keyid)?;
signed = true; signed = true;
@@ -547,17 +472,13 @@ struct PartialChecksum {
} }
/// Run a build command in `cwd` with extra environment variables layered on /// Run a build command in `cwd` with extra environment variables layered on
/// top of the inherited environment. /// top of the inherited environment, with stdio attached to the terminal.
///
/// When `sink` is set, stdout/stderr are piped and every line is forwarded to
/// it (live view + tee log); otherwise stdio is inherited from the terminal.
/// Returns an error on non-zero exit status. /// Returns an error on non-zero exit status.
fn run_command( fn run_command(
cwd: &Path, cwd: &Path,
program: &str, program: &str,
args: &[&str], args: &[&str],
env: &BTreeMap<String, String>, env: &BTreeMap<String, String>,
sink: Option<&Arc<dyn LineSink>>,
) -> Result<(), Box<dyn Error>> { ) -> Result<(), Box<dyn Error>> {
log::debug!( log::debug!(
"running: {} {} (in {})", "running: {} {} (in {})",
@@ -565,44 +486,12 @@ fn run_command(
args.join(" "), args.join(" "),
cwd.display() cwd.display()
); );
let status = Command::new(program)
let mut cmd = Command::new(program); .current_dir(cwd)
cmd.current_dir(cwd).envs(env).args(args); .envs(env)
.args(args)
let status = match sink { .status()
None => cmd .map_err(|e| format!("failed to run '{}': {}", program, e))?;
.status()
.map_err(|e| format!("failed to run '{}': {}", program, e))?,
Some(sink) => {
cmd.stdout(Stdio::piped()).stderr(Stdio::piped());
let mut child = cmd
.spawn()
.map_err(|e| format!("failed to run '{}': {}", program, e))?;
let stdout = child.stdout.take();
let stderr = child.stderr.take();
// One reader thread per stream; interleaving across streams is
// approximate (channel arrival order), acceptable for display.
let out_sink = sink.clone();
let err_sink = sink.clone();
let out_thread = std::thread::spawn(move || {
if let Some(out) = stdout {
pump(out, Stream::Stdout, &*out_sink);
}
});
let err_thread = std::thread::spawn(move || {
if let Some(err) = stderr {
pump(err, Stream::Stderr, &*err_sink);
}
});
let _ = out_thread.join();
let _ = err_thread.join();
child
.wait()
.map_err(|e| format!("failed to wait for '{}': {}", program, e))?
}
};
if !status.success() { if !status.success() {
return Err(format!( return Err(format!(
@@ -963,7 +852,7 @@ mod differential_tests {
let ours_tree = ours_root.join(&tree_name); let ours_tree = ours_root.join(&tree_name);
run_dpkg(&golden_tree); run_dpkg(&golden_tree);
run_source_build(&ours_tree, &SourceBuildOptions::default(), None) run_source_build(&ours_tree, &SourceBuildOptions::default())
.expect("native source pipeline failed"); .expect("native source pipeline failed");
let entry = let entry =
+1 -1
View File
@@ -1,5 +1,5 @@
mod api; mod api;
pub(crate) mod capture; mod capture;
mod local; mod local;
mod manager; mod manager;
mod schroot; mod schroot;
+3 -20
View File
@@ -58,11 +58,7 @@ fn main() {
.arg(arg!(--backport "This changelog is for a backport entry").required(false)) .arg(arg!(--backport "This changelog is for a backport entry").required(false))
.arg(arg!(-v --version <version> "Target version").required(false)), .arg(arg!(-v --version <version> "Target version").required(false)),
) )
.subcommand( .subcommand(Command::new("build").about("Build the source package (into a .dsc)"))
Command::new("build")
.about("Build the source package (into a .dsc)")
.arg(arg!(--verbose "Show raw tool output instead of the live build view").required(false)),
)
.subcommand( .subcommand(
Command::new("deb") Command::new("deb")
.about("Build the source package into binary package (.deb)") .about("Build the source package into binary package (.deb)")
@@ -252,22 +248,9 @@ fn main() {
std::process::exit(1); std::process::exit(1);
}); });
} }
Some(("build", sub_matches)) => { Some(("build", _sub_matches)) => {
let cwd = current_dir_or_exit(); let cwd = current_dir_or_exit();
let verbose = sub_matches if let Err(e) = pkh::build::build_source_package(Some(&cwd)) {
.get_one::<bool>("verbose")
.copied()
.unwrap_or(false);
// Live build view: disabled by --verbose or when stdout is not a
// terminal (DebUi handles the non-TTY case itself)
let ui = if verbose {
None
} else {
Some(std::sync::Arc::new(pkh::ui::deb::DebUi::new(&multi)))
};
if let Err(e) = pkh::build::build_source_package(Some(&cwd), ui) {
error!("{}", e); error!("{}", e);
// Unmet build dependencies/conflicts exit with status 3, // Unmet build dependencies/conflicts exit with status 3,
// like dpkg-buildpackage does. // like dpkg-buildpackage does.
-19
View File
@@ -13,27 +13,8 @@ use crossterm::{
}; };
use indicatif::{MultiProgress, ProgressBar, ProgressStyle}; use indicatif::{MultiProgress, ProgressBar, ProgressStyle};
use std::io::{self, Write}; use std::io::{self, Write};
use std::path::Path;
use std::time::Duration; use std::time::Duration;
/// Render a path for terminal display: relative to the current working
/// directory when the target lives inside it or directly next to it
/// (`../name`, the usual layout of build artifacts), absolute otherwise.
pub fn display_path(path: &Path) -> String {
let Ok(cwd) = std::env::current_dir() else {
return path.display().to_string();
};
if let Ok(rel) = path.strip_prefix(&cwd) {
return rel.display().to_string();
}
if let Some(parent) = cwd.parent()
&& let Ok(rel) = path.strip_prefix(parent)
{
return format!("../{}", rel.display());
}
path.display().to_string()
}
/// Create a spinner-style progress bar attached to `multi`, returning the bar /// Create a spinner-style progress bar attached to `multi`, returning the bar
/// and a callback compatible with [`crate::ProgressCallback`] /// and a callback compatible with [`crate::ProgressCallback`]
pub fn create_progress_bar( pub fn create_progress_bar(
+20 -45
View File
@@ -1,6 +1,6 @@
//! Live build view (`pkh deb`, `pkh build`): a status bar with the current //! Live UI for `pkh deb`: a status bar with the current build phase on top
//! build phase on top and a rolling pane of rewritten log lines below //! and a rolling pane of rewritten log lines below ("a terminal in the
//! ("a terminal in the terminal"). //! terminal").
//! //!
//! Subprocess output is captured through a [`LineSink`] implementation, //! Subprocess output is captured through a [`LineSink`] implementation,
//! rewritten by classifiers ([`crate::ui::logfmt`]) and rendered in place //! rewritten by classifiers ([`crate::ui::logfmt`]) and rendered in place
@@ -127,7 +127,7 @@ struct Shared {
started: Instant, started: Instant,
} }
/// Live build view for `pkh deb` / `pkh build` /// Live build view for `pkh deb`
/// ///
/// Create one per build (disabled automatically when stdout is not a TTY or /// Create one per build (disabled automatically when stdout is not a TTY or
/// when the user requests verbose output), pass it down as /// when the user requests verbose output), pass it down as
@@ -202,35 +202,20 @@ impl DebUi {
ui ui
} }
/// Identify the binary package being built; names the log file and the /// Identify the package being built; names the log file and the status bar
/// status bar
pub fn set_target(&self, package: &str, version: &str, series: &str, arch: &str) { pub fn set_target(&self, package: &str, version: &str, series: &str, arch: &str) {
if self.shared.enabled { if self.shared.enabled {
self.shared.top.set_prefix(format!( self.shared.top.set_prefix(format!(
"Building {package} ({version}) for {series}/{arch}" "Building {package} ({version}) for {series}/{arch}"
)); ));
} }
self.open_log("deb", package, version, &format!("for {series}/{arch}"));
}
/// Identify the source package being built; names the log file // Rename the log file to include the package identity (best-effort),
/// (`build-<package>-<version>-<timestamp>.log`) and the status bar // then open it so subsequent captured lines are tee'd.
pub fn set_build_target(&self, package: &str, version: &str, distribution: &str) {
if self.shared.enabled {
self.shared.top.set_prefix(format!(
"Building source package {package} ({version}) for {distribution}"
));
}
self.open_log("build", package, version, &format!("for {distribution}"));
}
/// Rename the placeholder log file to include the build identity
/// (best-effort), then open it so subsequent captured lines are tee'd
fn open_log(&self, kind: &str, package: &str, version: &str, detail: &str) {
let old_path = self.shared.log_path.lock().unwrap().clone(); let old_path = self.shared.log_path.lock().unwrap().clone();
let log_path = match old_path.parent() { let log_path = match old_path.parent() {
Some(dir) => dir.join(format!( Some(dir) => dir.join(format!(
"{kind}-{package}-{version}-{}.log", "deb-{package}-{version}-{}.log",
self.shared.timestamp self.shared.timestamp
)), )),
None => old_path.clone(), None => old_path.clone(),
@@ -246,7 +231,11 @@ impl DebUi {
Ok(mut file) => { Ok(mut file) => {
let _ = writeln!( let _ = writeln!(
file, file,
"# pkh {kind} {package} ({version}) {detail} started {}", "# pkh deb {} ({}) for {}/{} started {}",
package,
version,
series,
arch,
chrono::Utc::now().to_rfc3339() chrono::Utc::now().to_rfc3339()
); );
*self.shared.tee.lock().unwrap() = Some(file); *self.shared.tee.lock().unwrap() = Some(file);
@@ -263,18 +252,12 @@ impl DebUi {
/// Switch to a phase, installing its default classifier /// Switch to a phase, installing its default classifier
pub fn phase(&self, phase: Phase) { pub fn phase(&self, phase: Phase) {
self.phase_custom(phase.label(), default_classifier(phase)); self.phase_with(phase, default_classifier(phase));
} }
/// Switch to a phase with a custom classifier (e.g. quilt with a known /// Switch to a phase with a custom classifier (e.g. quilt with a known
/// patch count) /// patch count)
pub fn phase_with(&self, phase: Phase, classifier: Box<dyn Classifier>) { pub fn phase_with(&self, phase: Phase, classifier: Box<dyn Classifier>) {
self.phase_custom(phase.label(), classifier);
}
/// Switch to an arbitrary status label with a custom classifier; used by
/// flows whose phases are not part of [`Phase`] (e.g. source builds)
pub fn phase_custom(&self, label: &str, classifier: Box<dyn Classifier>) {
{ {
let mut st = self.shared.state.lock().unwrap(); let mut st = self.shared.state.lock().unwrap();
st.classifier = classifier; st.classifier = classifier;
@@ -284,7 +267,7 @@ impl DebUi {
} }
if self.shared.enabled { if self.shared.enabled {
self.shared.top.set_style(spinner_style()); self.shared.top.set_style(spinner_style());
self.shared.top.set_message(label.to_string()); self.shared.top.set_message(phase.label());
self.shared.pane.set_message(""); self.shared.pane.set_message("");
} }
} }
@@ -316,12 +299,6 @@ impl DebUi {
self.shared.enabled && !self.shared.suspended.load(Ordering::SeqCst) self.shared.enabled && !self.shared.suspended.load(Ordering::SeqCst)
} }
/// Whether the widget renders at all (false on non-TTY stdout); callers
/// use this to fall back to plain-line summaries
pub fn is_enabled(&self) -> bool {
self.shared.enabled
}
/// Obtain a sink feeding this view; pass it to `ContextCommand::capture` /// Obtain a sink feeding this view; pass it to `ContextCommand::capture`
pub fn sink(self: &Arc<Self>) -> Arc<dyn LineSink> { pub fn sink(self: &Arc<Self>) -> Arc<dyn LineSink> {
Arc::new(Sink { Arc::new(Sink {
@@ -348,15 +325,14 @@ impl DebUi {
self.shared.pane.finish_and_clear(); self.shared.pane.finish_and_clear();
} }
/// Clear the widget and print a success summary with the artifacts, /// Clear the widget and print a success summary with the artifacts
/// rendered relative to the working directory when possible
pub fn finish_success(&self, artifacts: &[PathBuf], elapsed: Duration) { pub fn finish_success(&self, artifacts: &[PathBuf], elapsed: Duration) {
self.suspend(); self.suspend();
if self.shared.enabled && !artifacts.is_empty() { if self.shared.enabled && !artifacts.is_empty() {
println!("Built in {}s:", elapsed.as_secs());
for artifact in artifacts { for artifact in artifacts {
println!(" {}", crate::ui::display_path(artifact)); println!(" → {}", artifact.display());
} }
println!(" ✔ Built in {}s", elapsed.as_secs());
} }
} }
@@ -518,13 +494,12 @@ fn is_stdout_tty() -> bool {
unsafe { libc::isatty(libc::STDOUT_FILENO) == 1 } unsafe { libc::isatty(libc::STDOUT_FILENO) == 1 }
} }
/// Default (placeholder) log file path for a given timestamp; renamed by /// Default log file path for a given timestamp
/// [`DebUi::set_target`] / [`DebUi::set_build_target`] once the target is known
fn default_log_path(timestamp: &str) -> PathBuf { fn default_log_path(timestamp: &str) -> PathBuf {
let dir = ProjectDirs::from("com", "pkh", "pkh") let dir = ProjectDirs::from("com", "pkh", "pkh")
.map(|dirs| dirs.cache_dir().join("logs")) .map(|dirs| dirs.cache_dir().join("logs"))
.unwrap_or_else(std::env::temp_dir); .unwrap_or_else(std::env::temp_dir);
dir.join(format!("pkh-{timestamp}.log")) dir.join(format!("deb-{timestamp}.log"))
} }
static SIGINT_LOG_PATH: Mutex<Option<PathBuf>> = Mutex::new(None); static SIGINT_LOG_PATH: Mutex<Option<PathBuf>> = Mutex::new(None);
-130
View File
@@ -282,54 +282,6 @@ impl Classifier for MakeClassifier {
} }
} }
/// Classifier for `dpkg-source` output (source-build phases)
///
/// The build pipeline pins `LC_ALL=C`, so dpkg-source emits stable English
/// messages prefixed with `info:` / `warning:` / `error:`; the prefix is
/// stripped and the severity drives the pane color. Raw `tar:` diagnostics
/// emitted while repacking tarballs are surfaced too.
#[derive(Default)]
pub struct DpkgSourceClassifier {}
impl DpkgSourceClassifier {
/// Create a new classifier
pub fn new() -> Self {
Self::default()
}
}
impl Classifier for DpkgSourceClassifier {
fn feed(&mut self, _stream: Stream, line: &str) -> Vec<Action> {
const PREFIX: &str = "dpkg-source: ";
let rest = line.strip_prefix(PREFIX).unwrap_or(line);
if let Some(rest) = rest.strip_prefix("info: ") {
vec![Action::Shown(truncate(rest))]
} else if let Some(rest) = rest.strip_prefix("warning: ") {
vec![Action::Warning(truncate(rest))]
} else if let Some(rest) = rest.strip_prefix("error: ") {
vec![Action::Error(truncate(rest))]
} else if let Some(tar) = rest.strip_prefix("tar: ") {
// Diagnostics from the tarball repacking subprocess; warnings
// about unknown header keywords are benign, real failures are not.
let lower = tar.to_lowercase();
if ["error", "cannot", "failed", "exited"]
.iter()
.any(|m| lower.contains(m))
{
vec![Action::Error(truncate(tar))]
} else {
vec![Action::Warning(truncate(tar))]
}
} else if line == PREFIX.trim_end() || rest.is_empty() {
vec![Action::Hidden]
} else {
// Unprefixed output from a foreign subprocess: keep it visible
vec![Action::Shown(truncate(line))]
}
}
}
/// Classifier for `mmdebstrap` output (chroot tarball creation) /// Classifier for `mmdebstrap` output (chroot tarball creation)
/// ///
/// mmdebstrap prefixes its own messages with `I:` / `W:` / `E:`; everything /// mmdebstrap prefixes its own messages with `I:` / `W:` / `E:`; everything
@@ -547,88 +499,6 @@ mod tests {
); );
} }
#[test]
fn test_dpkg_source_severity_prefixes() {
let mut c = DpkgSourceClassifier::new();
assert_eq!(
feed_one(
&mut c,
"dpkg-source: info: using patch list from debian/patches/series"
),
vec![Action::Shown(
"using patch list from debian/patches/series".to_string()
)]
);
assert_eq!(
feed_one(
&mut c,
"dpkg-source: info: applying patch debian/patches/reproducible.patch"
),
vec![Action::Shown(
"applying patch debian/patches/reproducible.patch".to_string()
)]
);
assert_eq!(
feed_one(
&mut c,
"dpkg-source: info: building hello in ../hello_2.10-5.dsc"
),
vec![Action::Shown(
"building hello in ../hello_2.10-5.dsc".to_string()
)]
);
assert_eq!(
feed_one(
&mut c,
"dpkg-source: warning: upstream signing key but no upstream signature"
),
vec![Action::Warning(
"upstream signing key but no upstream signature".to_string()
)]
);
assert_eq!(
feed_one(
&mut c,
"dpkg-source: error: unrepresentable changes to source"
),
vec![Action::Error(
"unrepresentable changes to source".to_string()
)]
);
}
#[test]
fn test_dpkg_source_tar_and_unknown_lines() {
let mut c = DpkgSourceClassifier::new();
// Benign tar header-keyword warnings stay yellow
assert_eq!(
feed_one(
&mut c,
"tar: Ignoring unknown extended header keyword 'SCHILY.xattr.user.foo'"
),
vec![Action::Warning(
"Ignoring unknown extended header keyword 'SCHILY.xattr.user.foo'".to_string()
)]
);
// Real tar failures are errors
assert_eq!(
feed_one(
&mut c,
"tar: ../hello_2.10.orig.tar.xz: Cannot open: No such file or directory"
),
vec![Action::Error(
"../hello_2.10.orig.tar.xz: Cannot open: No such file or directory".to_string()
)]
);
// Unprefixed foreign output stays visible
assert_eq!(
feed_one(&mut c, "gpgv: Signature made Tue 01 Jan 2026"),
vec![Action::Shown(
"gpgv: Signature made Tue 01 Jan 2026".to_string()
)]
);
}
#[test] #[test]
fn test_truncate_long_lines() { fn test_truncate_long_lines() {
let long = "x".repeat(300); let long = "x".repeat(300);