pkh/plans/native-build.md

Native build pipeline — replacing the dpkg-buildpackage shell-out

Status: Phase 0 + Phase 1 (source builds) implemented — see §11 Reference codebase: https://salsa.debian.org/dpkg-team/dpkg (main branch, analyzed 2026-08)

---

0. TL;DR — honest take

Scope	Verdict	Effort	Risk
A. Re-implement the orchestrator (dpkg-buildpackage.pl, 1209 lines of Perl) natively in Rust, still invoking dpkg-source, debian/rules, etc. as subprocesses	Doable, worth it	~2–4 weeks	Low–medium
B. Additionally replace the cheap satellite tools natively (dpkg-parsechangelog, dpkg-checkbuilddeps, dpkg-architecture, dpkg-genchanges, dpkg-genbuildinfo)	Doable incrementally	+1–2 weeks each	Medium
C. Replace dpkg-source (tarball/diff generation, quilt integration, .dsc assembly)	Not recommended	2–3+ months, then endless edge cases	High

The key insight: dpkg-buildpackage itself is a thin (~1200-line) Perl sequencer. The real complexity lives in dpkg-source (~4000 lines of Perl across Dpkg/Source/*). A source build fundamentally requires dpkg-source-class functionality (orig tarballs, debian diffs, .dsc assembly), so "no dpkg tools at all" is not a realistic goal — but "no dpkg-buildpackage, full control of the pipeline" absolutely is, and it unlocks things pkh currently cannot do (live UI on source builds, .changes/.buildinfo generation on binary builds, unified error classification).

---

1. Where pkh stands today

Two divergent build paths:

Source builds — build_source_package()

Shelled out on the host:

dpkg-buildpackage -S -I -i -nc -d [--sign-keyid=<id> | --no-sign]

Problems:

• Requires dpkg-dev on the host.
• Output bypasses the live UI entirely (DebUi / LineSink) — no phase tracking, no log classification, unlike binary builds.
• No control over failure semantics beyond exit status.

Binary builds — deb::local::build()

Already does not use dpkg-buildpackage. Hand-rolled sequence: ephemeral unshare chroot → apt-get update → essentials → manual quilt push -a → apt-get build-dep → debian/rules build → fakeroot debian/rules binary → retrieve .debs. What it silently skips compared to a real build:

• debian/rules clean before building,
• dpkg-source --before-build/--after-build lifecycle (it re-implements patch application crudely),
• .buildinfo and .changes generation,
• build-conflicts checking, Rules-Requires-Root semantics,
• SOURCE_DATE_EPOCH export (reproducibility).

So pkh already pays part of the "native orchestrator" cost without getting its benefits. A native implementation would unify both paths.

Existing native assets to reuse

• Changelog header/footer parsing — changelog.rs (needs extension to full metadata: timestamp, urgency, trailer key/values).
• GPG key discovery — utils/gpg.rs via gpgme.
• Hashing crates already present: sha2, md-5 (need to add sha1); compression: flate2, xz2; archives: tar.
• Command execution through remote-capable contexts — ContextCommand (local/ssh/schroot/unshare/capture).
• Live UI phases — ui/deb.rs.

---

2. What dpkg-buildpackage actually is (measured)

It is a Perl script (use v5.36). Measured sizes on main:

Component	Lines	Role
scripts/dpkg-buildpackage.pl	1209	Orchestrator
scripts/dpkg-source.pl	806	Driver; work in Dpkg/Source/*
scripts/dpkg-genbuildinfo.pl	637	.buildinfo generation
scripts/dpkg-genchanges.pl	626	.changes generation
scripts/dpkg-architecture.pl	498	Arch name/triplet tables
scripts/dpkg-gensymbols.pl	398	(called by rules, not by us)
scripts/dpkg-checkbuilddeps.pl	270	Dep checking vs status file
scripts/dpkg-parsechangelog.pl	195	Changelog CLI wrapper
scripts/dpkg-distaddfile.pl	99	Registers files in debian/files
Dpkg/*.pm + Dpkg/Source/*.pm (subset examined)	~11,600	Shared library code

Total relevant Perl surface ≈ 16k LOC, of which the orchestrator is only ~8%.

---

3. The exact pipeline (verified against source)

What dpkg-buildpackage does, in order:

1. Config: load buildpackage.conf (Dpkg::Conf), inject as leading argv.
2. Option parsing (~200 lines): build types -F/-g/-G/-b/-B/-A/-S (+ --build=full,source,binary,any,all), signing (-us/-uc/-ui/-k/-p/--no-sign/--force-sign), -j/-J/--jobs-force, -r<root-cmd>, -R<rules>, -T<targets>, -a/-t/--target-arch, -P<profiles>, -d/-D, -nc/-tc, hooks --hook-<name>=<cmd>, passthrough buckets for dpkg-source / dpkg-genchanges / dpkg-genbuildinfo.
3. Build type → rules targets: binary→build+binary; arch-dep only→build-arch+binary-arch; indep only→build-indep+binary-indep.
4. Implied flags: -nc alone implies -b; -nc -S disables build-dep checks.
5. Environment prep:
   • parallel=auto default → DEB_BUILD_OPTIONS=parallel=N exported;
   • forced jobs additionally appended to MAKEFLAGS;
   • DEB_BUILD_PROFILES exported if -P;
   • optional .dsc input → dpkg-source --extract first;
   • SOURCE_DATE_EPOCH ||= changelog timestamp || time() (reproducible-builds.org spec);
   • full env dump of dpkg-architecture -f [-a…][-t…] imported into the environment (all DEB_BUILD_*, DEB_HOST_*, DEB_TARGET_* incl. *_OS/CPU/MULTIARCH/GNU_TYPE/ARCH_BITS/ENDIAN);
   • OpenPGP key resolution: --sign-keyfile > --sign-keyid > maintainer userid; secrets probed up-front; UNRELEASED distribution ⇒ signing disabled unless --force-sign.
6. Pre-flight: Dpkg::BuildDriver->pre_check() (rules file exists/executable); dpkg-source --before-build . (applies patches for quilt formats); dpkg-checkbuilddeps [-A|-B|-I] unless -d (exit 3 on unmet).
7. Hooks at 12 points: preinit init preclean source build binary buildinfo changes postclean check sign done, with %p/%v/%s/%u/%a substitution.
8. Preclean: debian/rules clean via BuildDriver (gain-root per Rules-Requires-Root).
9. Source build (if any SOURCE component): dpkg-source -b . → .dsc + tarballs in ...
10. Binary build (if any BINARY component): run_build_task(build-target) then binary-target through BuildDriver (RRR-aware root command; skips separate non-root build pass when running rootless).
11. Metadata generation:
    • dpkg-genbuildinfo → ../<pkg>_<ver_noepoch>_<arch>.buildinfo (Format 1.0; records Installed-Build-Depends snapshot from the dpkg status DB, build environment, checksums);
    • dpkg-genchanges → ../<pkg>_<ver_noepoch>_<arch>.changes (Format 1.8; aggregates debian/files + changelog + control).
12. Post: optional -tc clean; dpkg-source --after-build . (unapplies patches it applied); human summary ("full upload (original source is included)" etc. derived from the Files field); optional lintian-style check command.
13. Signing cascade (inline/clearsig via OpenPGP backend gpg|sequoia|sop):
    • sign <pkg>_<ver>.dsc → recompute its checksums inside .buildinfo;
    • sign .buildinfo → recompute dsc+buildinfo checksums inside .changes (rewriting both Checksums-* and legacy Files entries);
    • sign .changes.

Exit codes matter: e.g. unsatisfied build-deps ⇒ exit 3.

---

4. Sub-tool inventory and replacement strategy

Tool	Used for	Complexity to replace natively	Strategy
dpkg-parsechangelog	source/version/maintainer/distribution/timestamp	Low — documented format; crates exist (debian-changelog, deb822-parser ecosystem)	Replaced (see §11, metadata.rs)
dpkg-version compare	epoch/upstream/revision ordering	Low — small well-specified algorithm; crate debversion	Replaced (§11, debian/version.rs: Ord/compare/later_than)
dpkg-architecture	arch ↔ triplet tables, multiarch tuple, env dump	Low-medium — embed cputable/ostable/tupletable/abitable data (stable for years)	Replaced (§11, debian/arch.rs)
dpkg-checkbuilddeps	deps vs installed status	Medium — Dpkg::Deps grammar (alternatives, arch qualifiers, <profiles> restrictions, versioned Provides subtleties, Multi-Arch facts) + status-file scan	Replaced (§11, debian/deps.rs; wired into the pipeline behind -D, source-only builds skip it like dpkg-buildpackage)
dpkg-genbuildinfo	.buildinfo	Medium — deb822 emit + status snapshot + checksums	Native (see §11, buildinfo.rs)
dpkg-genchanges	.changes	Medium — deb822 emit + debian/files consumption + .deb control extraction (ar+tar, trivial with crates)	Replaced for source (§11) and binary uploads (§11, build/binary.rs)
dpkg-genbuildinfo (binary)	.buildinfo for -b builds	Medium — deb822 emit + in-context artifact hashing	Replaced (§11, build/binary.rs, wired into deb/local.rs)
dpkg-distaddfile/debian/files protocol	build outputs registry	Trivial — one append-only line format	Native (files.rs)
OpenPGP signing	inline clearsign of dsc/buildinfo/changes	Low — gpgme (already a dependency) supports clearsigning	Native (sign.rs)
dpkg-source	orig tarball, debian diff, patches, .dsc	Very high — V1/V2/quilt/native formats, byte-exact tar normalization, quilt bookkeeping, --include-binaries, hundreds of validation warnings	Keep as subprocess (see §6)
debian/rules execution	the actual build	N/A (foreign code)	Keep, via ContextCommand

---

5. Feasibility detail per scope

Scope A — native orchestrator (recommended)

The 1209-line script decomposes into clean Rust pieces:

src/build/
├── mod.rs          // pipeline driver (the equivalent of main())
├── buildtype.rs    // BUILD_SOURCE|ARCH_DEP|ARCH_INDEP bitflags + target mapping
├── metadata.rs     // changelog + control resolution, version splitting
├── env.rs          // SOURCE_DATE_EPOCH, DEB_BUILD_OPTIONS, arch env dump
├── buildinfo.rs    // native .buildinfo writer
├── changes.rs      // native .changes writer
├── files.rs        // debian/files registry emulation
├── checksums.rs    // md5/sha1/sha256 registry
├── control.rs      // deb822 parser/writer
└── sign.rs         // gpgme clearsign + post-sign checksum cascade

Estimated ~1.5–3 kLOC. Everything is deterministic file munging + subprocess sequencing — no daemons, no parsing of arbitrary upstream code (that's dpkg-source's job, which we keep).

pkh-specific simplifications (legitimate because pkh controls the environment):

• Builds run as real root inside ephemeral unshare chroots ⇒ the entire gain-root/fakeroot matrix collapses: run debian/rules directly when the context is root (keep fakeroot fallback for host-side source builds).
• pkh's option surface is a fraction of dpkg's: -S/-b/-B/-A/-g/-G, -us/-uc, -k, -j, -a, -P, -d/-D, -nc/-tc, -R, -T cover everything pkh passes today plus obvious headroom. Hooks and --hook-* can be dropped initially.
• Vendor hooks (run_vendor_hook) are rarely used in the build path; Ubuntu/Debian differences pkh cares about are already handled via distro_info.yml.

What this buys pkh concretely:

1. Live UI + log classification for source builds (today's -S path is a black box).
2. .changes/.buildinfo for binary builds — currently missing entirely; needed for uploads/PPA submissions and lintian checks.
3. Correct dpkg-source --before-build/--after-build lifecycle replacing the manual quilt push -a hack in local.rs (handles 3.0 (quilt) properly, incl. unapply-on-exit and format detection instead of sniffing debian/patches/series).
4. SOURCE_DATE_EPOCH reproducibility for free.
5. No host dpkg-dev requirement for orchestration decisions; dpkg-source still needed inside build environments where pkh already installs packages anyway.

Scope B — satellite tools

Incremental, each independently testable against the real tool (differential testing). Crates from the rust-debian-* ecosystem (maintained by Jelmer Vernooij) cover most parsing: deb822-parser, debian-control, debian-changelog, debversion. Priority order if pursued: version compare → changelog → architecture tables → checkbuilddeps.

Scope C — dpkg-source

The honest numbers: Dpkg/Source/Package/V2.pm alone is 847 lines, V1 is 599, plus Archive/Quilt/Functions/BinaryFiles modules, GNU diff generation, and — critically — byte-level tar normalization (mtime/uid/gid/mode canonicalization, pax header handling) that reproducible builds depend on. Parity means being bug-compatible with dpkg against ~40k archive source packages. This is a multi-month project with a long tail, and it buys pkh almost nothing since dpkg-source is guaranteed present inside the very chroots pkh creates. Do not do this.

---

6. Proposed phasing

Phase 0 — quick win (days)

Route the existing dpkg-buildpackage -S call through the UI capture machinery (like cap() does for binary builds): phase display + LineSink classification. No behavior change otherwise.

Phase 1 — native orchestrator (2–4 weeks)

Implement §5 scope A. Both entry points converge:

• pkh build -S → native pipeline, dpkg-source -b subprocess, native signing.
• pkh build (binary) → same pipeline skeleton inside the ephemeral context: preclean → before-build → dep check → rules build → rules binary[-arch|-indep] → native buildinfo/changes → after-build. Deletes the manual quilt logic.

Deliverables:

• src/build/* module tree (§5).
• Native .buildinfo (Format 1.0) and .changes (Format 1.8) writers emitting exactly the field sets observed in dpkg-genbuildinfo.pl / dpkg-genchanges.pl, in dpkg's canonical field order:
  • buildinfo: Format, Source, Binary, Architecture, Version, Binary-Only-Changes, Checksums-Md5/Shа1/Sha256, Build-Origin, Build-Architecture, Build-Kernel-Version(opt), Build-Date, Build-Path(opt), Build-Tainted-By(opt), Installed-Build-Depends, Environment
  • changes: Format, Date, Source, Binary, Built-For-Profiles, Architecture, Version, Distribution, Urgency, Maintainer, Changed-By, Description, Changes, Checksums-Sha1/Sha256, Files
• Post-signature checksum cascade implemented exactly as dpkg does (sign dsc → patch buildinfo checksums → sign buildinfo → patch changes Files+Checksums-* → sign changes).
• Differential test harness (§8).

Phase 2 — satellite replacement (optional, incremental)

Swap subprocesses for native implementations, one tool at a time, gated by differential tests. Start with version-compare + changelog (already half-present in pkh).

Explicitly out of scope

dpkg-source internals, dpkg-deb packing, apt resolution (pkh correctly delegates to apt-get build-dep + dose3 for explanations already).

---

7. Tricky details to get right (gotchas list)

1. Version splitting: <epoch:>upstream<-revision>; .dsc/tarball names use upstream portion without epoch; .changes name uses version without epoch but with revision ($sversion in the script).
2. -nc implication chain: -nc ⇒ binary build implied; -nc -S ⇒ no dep check.
3. UNRELEASED ⇒ auto-disable all signing (warn), --force-sign overrides.
4. Signing invalidates checksums transitively (dsc → buildinfo → changes); get the cascade order right or archive tools reject the upload.
5. debian/files is the contract between debian/rules (via dh_builddeb/dpkg-gencontrol/dpkg-distaddfile) and the changes generator: lines of filename section priority [key=value...].
6. Arch selection for names: arch suffix in artifact filenames is host-arch for arch-dep builds, all for indep-only, source for source-only.
7. dpkg-source --before-build must run even for binary-only builds (patch application), and --after-build at the end — this replaces pkh's current manual quilt step and fixes 3.0 (quilt) correctness.
8. Environment parity: dpkg-architecture -f dump must be imported wholesale (not cherry-picked) — packages test DEB_HOST_GNU_TYPE, DEB_BUILD_MULTIARCH, DEB_BUILD_ARCH_ENDIAN, etc. If embedding tables later, mirror the full variable set.
9. Exit codes: preserve dpkg conventions (3 = unmet build-deps) so wrappers/scripts behave identically.
10. Locale: dpkg sets LANG=C-ish determinism for subprocesses — pkh already does this in local.rs; keep for all pipeline steps.
11. Multiline field rendering: values starting with \n render as Field: + indented continuation lines (no inline first line, no trailing space) — this is how dpkg emits Changes, Files, Installed-Build-Depends, Environment.
12. Artifact ordering in Checksums-*/Files is insertion order (dsc → tarballs in dsc-field order → debs → buildinfo), not alphabetical.

---

8. Testing strategy

• Differential harness: run real dpkg-buildpackage and the native pipeline over a corpus (packages covering: 1.0 non-native, 3.0 (quilt) with/without patches, native, binaries-only, indep-only, cross, RRR variants, UNRELEASED) and diff artifacts modulo timestamps/signatures.
• Port unit cases from dpkg's own t/ tests for version compare, deps parsing, changelog parsing.
• Golden-file tests for .changes/.buildinfo writers.
• Pin the reference dpkg version in CI commentary (behavior drift across dpkg releases is the main maintenance cost).

---

9. Risks

Risk	Mitigation
Behavior drift vs future dpkg releases (new fields, format bumps)	Differential tests pinned to a reference version; changes/buildinfo formats are extremely stable (1.8 / 1.0 for years)
Archive/upload tooling rejects our .changes/.buildinfo	Validate with lintian + a real PPA upload early in Phase 1
License contamination	dpkg is GPL-2+. Write from documented behavior/format specs and observation, do not transliterate Perl; alternatively accept GPL for pkh (currently no license field in Cargo.toml — decision needed)
Scope creep toward Scope C	Hard rule: dpkg-source stays a subprocess
Remote-context divergence	All tree-touching steps go through ContextCommand; metadata steps operate on locally-synced copies like changelog.rs already does

---

10. Effort summary

Item	Estimate
Phase 0 (UI capture for -S)	1–2 days
Phase 1 (native orchestrator + buildinfo/changes/signing)	2–4 weeks
Phase 2 per satellite tool	3 days – 2 weeks each
Scope C (dpkg-source)	2–3+ months — rejected

Bottom line: yes, it's doable — for the orchestrator. Treat dpkg-source as a permanent subprocess dependency, and the task becomes a well-bounded, high-value refactor that also fixes real gaps in pkh's binary path.

---

11. Implementation status (Phase 1 — source builds)

Landed in src/build/ (~2.8 kLOC incl. tests), wired behind the historical entry point build_source_package() so pkh build now runs the native pipeline:

Module	Role
control.rs	deb822 paragraph parser/writer (dpkg-compatible multiline rendering)
checksums.rs	md5/sha1/sha256 registry, insertion-ordered like dpkg's artifact accumulation
metadata.rs	version splitting/validation, full changelog entry parse (incl. binNMU binary-only), control info
buildtype.rs	build-type bitflags + rules-target/artifact-suffix mapping
env.rs	SOURCE_DATE_EPOCH, DEB_BUILD_OPTIONS, dpkg-architecture env dump, vendor/profiles, sanitized Environment field
files.rs	debian/files registry (parse/atomic save)
buildinfo.rs	.buildinfo writer + Installed-Build-Depends closure over the dpkg status DB
changes.rs	.changes writer (canonical field order, legacy Files + Checksums-Sha1/256)
sign.rs	gpgme clearsigning + key-id validation

Still delegated to subprocesses: dpkg-source -b/--before-build/--after-build (by design, see §5 Scope C).

Differential validation — automated in build/mod.rs (mod differential_tests, runs by default with cargo test --lib):

• a corpus of 12 synthetic fixtures covering: native/quilt/1.0 formats, epochs, ~ pre-releases, Ubuntu/Debian series (focal, noble, jammy, trixie, unstable), high urgency, multiple binary stanzas, one/two quilt patches, binNMU (binary-only=yes + previous-entry metadata), (Closes: #…) extraction, UNRELEASED no-sign, extra source-stanza fields;
• each case builds the tree twice (cp -a copies) — once with real dpkg-buildpackage -S -I -i -nc -d --no-sign, once with the native pipeline — then compares:
  • .dsc payload byte-for-byte,
  • .changes field-by-field (checksum lines of the .buildinfo itself excluded),
  • .buildinfo structure (machine-dependent fields excluded);
• real archive packages are pulled with pkh's own pull (archive download mode) and compared the same way, via differential_real_archive_package() which takes (package, dist, series):
  • always-on CI tests: hello @ ubuntu/noble, dosfstools @ debian/trixie, sl @ ubuntu/focal;
  • an additional #[ignore]-gated test for ad-hoc broad runs:

    PKH_DIFF_PACKAGES="bash coreutils curl" PKH_DIFF_DIST=ubuntu \
    PKH_DIFF_SERIES=noble cargo test --lib \
        differential_real_archive_packages -- --ignored
    

  (requires network access).

Manual validation additionally confirmed: signed builds verify with gpg --verify on all three artifacts; the patch apply/unapply lifecycle is correct for quilt formats.

Known divergences (documented, all informational fields): no Build-Tainted-By (vendor hook), no dpkg-buildflags origin tracking in the Environment field, vendor default profiles approximated (derivative.ubuntu noudeb for Ubuntu).

12. Implementation status (Phase 2 — satellite tools)

All four satellite replacements landed, each gated by differential tests against the real tool:

Work item	Module	Differential gate
WI-1 dpkg-architecture	debian/arch.rs (tables + lookups), wired into build/env.rs	arch_env(Some(a)) equals real dpkg-architecture -f -a a key-for-key for every arch from dpkg-architecture -L, plus native (build/mod.rs::diff_arch_env_*)
WI-2 version compare	debian/version.rs (Ord, compare, later_than)	all vectors from dpkg scripts/t/Dpkg_Version.t + Ubuntu-flavored cases, cross-checked against dpkg --compare-versions for << <= = >= >> (diff_version_compare_against_dpkg)
WI-3 dpkg-checkbuilddeps	debian/deps.rs (grammar, restriction reduction, KnownFacts evaluation, check_build_depends)	24 scenarios vs real dpkg-checkbuilddeps (alternatives, versions, arch/profile restrictions, Multi-Arch, versioned Provides, conflicts, -A/-B) comparing exit status + diagnostics (diff_checkbuilddeps_matrix); unit tests port the Dpkg_Deps.t reduction matrices. Wired into run_source_build behind -D parity: source-only builds skip the check like dpkg-buildpackage, unsatisfied deps exit 3
WI-4 binary .buildinfo/.changes	build/binary.rs (context-generic generation), wired into deb/local.rs; artifact retrieval extended in deb/mod.rs	same tree built with real dpkg-buildpackage -b and with the pkh flow (rules build/binary + native metadata through a local context): .changes/.buildinfo compared field-by-field modulo machine-dependent fields, artifact checksums included (diff_binary_build_metadata)

Binary-flow notes: SOURCE_DATE_EPOCH is now exported to the rules environment (reproducibility); digests are computed inside the context via coreutils so remote/chrooted trees work; binNMU binary builds set Source: pkg (prev) / Binary-Only-Changes and redistribute the previous .dsc when present; Architecture is encounter-ordered in .changes (sorted in .buildinfo, matching dpkg).

Still delegated to subprocesses: dpkg-source and debian/rules (by design, see §5 Scope C).