pkh/plans/native-build.md

# Native build pipeline — replacing the `dpkg-buildpackage` shell-out

Status: **Phase 0 + Phase 1 (source builds) implemented** — see §11
Reference codebase: https://salsa.debian.org/dpkg-team/dpkg (`main` branch, analyzed 2026-08)

---

## 0. TL;DR — honest take

| Scope | Verdict | Effort | Risk |
|---|---|---|---|
| **A.** Re-implement the *orchestrator* ([`dpkg-buildpackage.pl`](https://salsa.debian.org/dpkg-team/dpkg/-/blob/main/scripts/dpkg-buildpackage.pl), 1209 lines of Perl) natively in Rust, still invoking `dpkg-source`, `debian/rules`, etc. as subprocesses | **Doable, worth it** | ~2–4 weeks | Low–medium |
| **B.** Additionally replace the cheap satellite tools natively (`dpkg-parsechangelog`, `dpkg-checkbuilddeps`, `dpkg-architecture`, `dpkg-genchanges`, `dpkg-genbuildinfo`) | Doable incrementally | +1–2 weeks each | Medium |
| **C.** Replace `dpkg-source` (tarball/diff generation, quilt integration, `.dsc` assembly) | **Not recommended** | 2–3+ months, then endless edge cases | High |

The key insight: `dpkg-buildpackage` itself is a thin (~1200-line) Perl *sequencer*. The real complexity lives in `dpkg-source` (~4000 lines of Perl across `Dpkg/Source/*`). A source build fundamentally requires `dpkg-source`-class functionality (orig tarballs, debian diffs, `.dsc` assembly), so "no dpkg tools at all" is not a realistic goal — but "no `dpkg-buildpackage`, full control of the pipeline" absolutely is, and it unlocks things pkh currently cannot do (live UI on source builds, `.changes`/`.buildinfo` generation on binary builds, unified error classification).

---

## 1. Where pkh stands today

Two divergent build paths:

### Source builds — [`build_source_package()`](../src/build/mod.rs)
Shelled out on the host:
```
dpkg-buildpackage -S -I -i -nc -d [--sign-keyid=<id> | --no-sign]
```
Problems:
- Requires `dpkg-dev` on the host.
- Output bypasses the live UI entirely ([`DebUi`](../src/ui/deb.rs) / [`LineSink`](../src/context/mod.rs)) — no phase tracking, no log classification, unlike binary builds.
- No control over failure semantics beyond exit status.

### Binary builds — [`deb::local::build()`](../src/deb/local.rs)
Already does **not** use `dpkg-buildpackage`. Hand-rolled sequence: ephemeral unshare chroot → `apt-get update` → essentials → manual `quilt push -a` → `apt-get build-dep` → `debian/rules build` → `fakeroot debian/rules binary` → retrieve `.deb`s.
What it silently skips compared to a real build:
- `debian/rules clean` before building,
- `dpkg-source --before-build/--after-build` lifecycle (it re-implements patch application crudely),
- `.buildinfo` and `.changes` generation,
- build-conflicts checking, `Rules-Requires-Root` semantics,
- `SOURCE_DATE_EPOCH` export (reproducibility).

So pkh already pays part of the "native orchestrator" cost without getting its benefits. A native implementation would unify both paths.

### Existing native assets to reuse
- Changelog header/footer parsing — [`changelog.rs`](../src/changelog.rs) (needs extension to full metadata: timestamp, urgency, trailer key/values).
- GPG key discovery — [`utils/gpg.rs`](../src/utils/gpg.rs) via `gpgme`.
- Hashing crates already present: `sha2`, `md-5` (need to add `sha1`); compression: `flate2`, `xz2`; archives: `tar`.
- Command execution through remote-capable contexts — [`ContextCommand`](../src/context/mod.rs) (local/ssh/schroot/unshare/capture).
- Live UI phases — [`ui/deb.rs`](../src/ui/deb.rs).

---

## 2. What `dpkg-buildpackage` actually is (measured)

It is a Perl script (`use v5.36`). Measured sizes on `main`:

| Component | Lines | Role |
|---|---|---|
| `scripts/dpkg-buildpackage.pl` | 1209 | Orchestrator |
| `scripts/dpkg-source.pl` | 806 | Driver; work in `Dpkg/Source/*` |
| `scripts/dpkg-genbuildinfo.pl` | 637 | `.buildinfo` generation |
| `scripts/dpkg-genchanges.pl` | 626 | `.changes` generation |
| `scripts/dpkg-architecture.pl` | 498 | Arch name/triplet tables |
| `scripts/dpkg-gensymbols.pl` | 398 | (called by rules, not by us) |
| `scripts/dpkg-checkbuilddeps.pl` | 270 | Dep checking vs status file |
| `scripts/dpkg-parsechangelog.pl` | 195 | Changelog CLI wrapper |
| `scripts/dpkg-distaddfile.pl` | 99 | Registers files in `debian/files` |
| `Dpkg/*.pm` + `Dpkg/Source/*.pm` (subset examined) | ~11,600 | Shared library code |

Total relevant Perl surface ≈ **16k LOC**, of which the orchestrator is only ~8%.

---

## 3. The exact pipeline (verified against source)

What `dpkg-buildpackage` does, in order:

1. **Config**: load `buildpackage.conf` (Dpkg::Conf), inject as leading argv.
2. **Option parsing** (~200 lines): build types `-F/-g/-G/-b/-B/-A/-S` (+ `--build=full,source,binary,any,all`), signing (`-us/-uc/-ui/-k/-p/--no-sign/--force-sign`), `-j/-J/--jobs-force`, `-r<root-cmd>`, `-R<rules>`, `-T<targets>`, `-a/-t/--target-arch`, `-P<profiles>`, `-d/-D`, `-nc/-tc`, hooks `--hook-<name>=<cmd>`, passthrough buckets for `dpkg-source` / `dpkg-genchanges` / `dpkg-genbuildinfo`.
3. **Build type → rules targets**: `binary`→`build`+`binary`; arch-dep only→`build-arch`+`binary-arch`; indep only→`build-indep`+`binary-indep`.
4. **Implied flags**: `-nc` alone implies `-b`; `-nc -S` disables build-dep checks.
5. **Environment prep**:
   - `parallel=auto` default → `DEB_BUILD_OPTIONS=parallel=N` exported;
   - forced jobs additionally appended to `MAKEFLAGS`;
   - `DEB_BUILD_PROFILES` exported if `-P`;
   - optional `.dsc` input → `dpkg-source --extract` first;
   - `SOURCE_DATE_EPOCH ||= changelog timestamp || time()` (reproducible-builds.org spec);
   - full env dump of `dpkg-architecture -f [-a…][-t…]` imported into the environment (all `DEB_BUILD_*`, `DEB_HOST_*`, `DEB_TARGET_*` incl. `*_OS/CPU/MULTIARCH/GNU_TYPE/ARCH_BITS/ENDIAN`);
   - OpenPGP key resolution: `--sign-keyfile` > `--sign-keyid` > maintainer userid; secrets probed up-front; **UNRELEASED distribution ⇒ signing disabled** unless `--force-sign`.
6. **Pre-flight**: `Dpkg::BuildDriver->pre_check()` (rules file exists/executable); `dpkg-source --before-build .` (applies patches for quilt formats); `dpkg-checkbuilddeps [-A|-B|-I]` unless `-d` (exit 3 on unmet).
7. **Hooks** at 12 points: `preinit init preclean source build binary buildinfo changes postclean check sign done`, with `%p/%v/%s/%u/%a` substitution.
8. **Preclean**: `debian/rules clean` via BuildDriver (gain-root per `Rules-Requires-Root`).
9. **Source build** (if any SOURCE component): `dpkg-source -b .` → `.dsc` + tarballs in `..`.
10. **Binary build** (if any BINARY component): `run_build_task(build-target)` then `binary-target` through BuildDriver (RRR-aware root command; skips separate non-root `build` pass when running rootless).
11. **Metadata generation**:
    - `dpkg-genbuildinfo` → `../<pkg>_<ver_noepoch>_<arch>.buildinfo` (Format 1.0; records `Installed-Build-Depends` snapshot from the dpkg status DB, build environment, checksums);
    - `dpkg-genchanges` → `../<pkg>_<ver_noepoch>_<arch>.changes` (Format 1.8; aggregates `debian/files` + changelog + control).
12. **Post**: optional `-tc` clean; `dpkg-source --after-build .` (unapplies patches it applied); human summary ("full upload (original source is included)" etc. derived from the `Files` field); optional lintian-style check command.
13. **Signing cascade** (inline/clearsig via OpenPGP backend gpg|sequoia|sop):
    - sign `<pkg>_<ver>.dsc` → recompute its checksums **inside `.buildinfo`**;
    - sign `.buildinfo` → recompute dsc+buildinfo checksums **inside `.changes`** (rewriting both `Checksums-*` and legacy `Files` entries);
    - sign `.changes`.

Exit codes matter: e.g. unsatisfied build-deps ⇒ exit 3.

---

## 4. Sub-tool inventory and replacement strategy

| Tool | Used for | Complexity to replace natively | Strategy |
|---|---|---|---|
| `dpkg-parsechangelog` | source/version/maintainer/distribution/timestamp | **Low** — documented format; crates exist (`debian-changelog`, `deb822-parser` ecosystem) | Replaced (see §11, [`metadata.rs`](../src/build/metadata.rs)) |
| `dpkg-version` compare | epoch/upstream/revision ordering | **Low** — small well-specified algorithm; crate `debversion` | Splitting/validation replaced; ordering still Phase 2 |
| `dpkg-architecture` | arch ↔ triplet tables, multiarch tuple, env dump | **Low-medium** — embed cputable/ostable/tupletable/abitable data (stable for years) | **Replaced** (§11, [`debian/arch.rs`](../src/debian/arch.rs)) |
| `dpkg-checkbuilddeps` | deps vs installed status | **Medium** — `Dpkg::Deps` grammar (alternatives, arch qualifiers, `<profiles>` restrictions, versioned Provides subtleties, Multi-Arch facts) + status-file scan | Phase 2; keep `apt-get build-dep`/subprocess until then |
| `dpkg-genbuildinfo` | `.buildinfo` | **Medium** — deb822 emit + status snapshot + checksums | Native (see §11, [`buildinfo.rs`](../src/build/buildinfo.rs)) |
| `dpkg-genchanges` | `.changes` | **Medium** — deb822 emit + `debian/files` consumption + `.deb` control extraction (ar+tar, trivial with crates) | Native for source uploads (§11, [`changes.rs`](../src/build/changes.rs)); binary aggregation next |
| `dpkg-distaddfile`/`debian/files` protocol | build outputs registry | **Trivial** — one append-only line format | Native ([`files.rs`](../src/build/files.rs)) |
| OpenPGP signing | inline clearsign of dsc/buildinfo/changes | **Low** — `gpgme` (already a dependency) supports clearsigning | Native ([`sign.rs`](../src/build/sign.rs)) |
| `dpkg-source` | orig tarball, debian diff, patches, `.dsc` | **Very high** — V1/V2/quilt/native formats, byte-exact tar normalization, quilt bookkeeping, `--include-binaries`, hundreds of validation warnings | **Keep as subprocess** (see §6) |
| `debian/rules` execution | the actual build | N/A (foreign code) | Keep, via `ContextCommand` |

---

## 5. Feasibility detail per scope

### Scope A — native orchestrator (recommended)

The 1209-line script decomposes into clean Rust pieces:

```
src/build/
├── mod.rs          // pipeline driver (the equivalent of main())
├── buildtype.rs    // BUILD_SOURCE|ARCH_DEP|ARCH_INDEP bitflags + target mapping
├── metadata.rs     // changelog + control resolution, version splitting
├── env.rs          // SOURCE_DATE_EPOCH, DEB_BUILD_OPTIONS, arch env dump
├── buildinfo.rs    // native .buildinfo writer
├── changes.rs      // native .changes writer
├── files.rs        // debian/files registry emulation
├── checksums.rs    // md5/sha1/sha256 registry
├── control.rs      // deb822 parser/writer
└── sign.rs         // gpgme clearsign + post-sign checksum cascade
```

Estimated ~1.5–3 kLOC. Everything is deterministic file munging + subprocess sequencing — no daemons, no parsing of arbitrary upstream code (that's `dpkg-source`'s job, which we keep).

**pkh-specific simplifications** (legitimate because pkh controls the environment):
- Builds run as **real root** inside ephemeral unshare chroots ⇒ the entire gain-root/fakeroot matrix collapses: run `debian/rules` directly when the context is root (keep `fakeroot` fallback for host-side source builds).
- pkh's option surface is a fraction of dpkg's: `-S/-b/-B/-A/-g/-G`, `-us/-uc`, `-k`, `-j`, `-a`, `-P`, `-d/-D`, `-nc/-tc`, `-R`, `-T` cover everything pkh passes today plus obvious headroom. Hooks and `--hook-*` can be dropped initially.
- Vendor hooks (`run_vendor_hook`) are rarely used in the build path; Ubuntu/Debian differences pkh cares about are already handled via [`distro_info.yml`](../distro_info.yml).

**What this buys pkh concretely:**
1. Live UI + log classification for source builds (today's `-S` path is a black box).
2. `.changes`/`.buildinfo` for binary builds — currently missing entirely; needed for uploads/PPA submissions and lintian checks.
3. Correct `dpkg-source --before-build/--after-build` lifecycle replacing the manual `quilt push -a` hack in [`local.rs`](../src/deb/local.rs) (handles `3.0 (quilt)` properly, incl. unapply-on-exit and format detection instead of sniffing `debian/patches/series`).
4. `SOURCE_DATE_EPOCH` reproducibility for free.
5. No host `dpkg-dev` requirement for orchestration decisions; `dpkg-source` still needed inside build environments where pkh already installs packages anyway.

### Scope B — satellite tools

Incremental, each independently testable against the real tool (differential testing). Crates from the `rust-debian-*` ecosystem (maintained by Jelmer Vernooij) cover most parsing: `deb822-parser`, `debian-control`, `debian-changelog`, `debversion`. Priority order if pursued: version compare → changelog → architecture tables → checkbuilddeps.

### Scope C — `dpkg-source`

The honest numbers: `Dpkg/Source/Package/V2.pm` alone is 847 lines, V1 is 599, plus Archive/Quilt/Functions/BinaryFiles modules, GNU diff generation, and — critically — **byte-level tar normalization** (mtime/uid/gid/mode canonicalization, pax header handling) that reproducible builds depend on. Parity means being bug-compatible with dpkg against ~40k archive source packages. This is a multi-month project with a long tail, and it buys pkh almost nothing since `dpkg-source` is guaranteed present inside the very chroots pkh creates. **Do not do this.**

---

## 6. Proposed phasing

### Phase 0 — quick win (days)
Route the existing `dpkg-buildpackage -S` call through the UI capture machinery (like [`cap()`](../src/deb/local.rs) does for binary builds): phase display + `LineSink` classification. No behavior change otherwise.

### Phase 1 — native orchestrator (2–4 weeks)
Implement §5 scope A. Both entry points converge:
- `pkh build -S` → native pipeline, `dpkg-source -b` subprocess, native signing.
- `pkh build` (binary) → same pipeline skeleton inside the ephemeral context: preclean → before-build → dep check → `rules build` → `rules binary[-arch|-indep]` → native buildinfo/changes → after-build. Deletes the manual quilt logic.

Deliverables:
- `src/build/*` module tree (§5).
- Native `.buildinfo` (Format 1.0) and `.changes` (Format 1.8) writers emitting exactly the field sets observed in `dpkg-genbuildinfo.pl` / `dpkg-genchanges.pl`, in dpkg's canonical field order:
  - buildinfo: `Format, Source, Binary, Architecture, Version, Binary-Only-Changes, Checksums-Md5/Shа1/Sha256, Build-Origin, Build-Architecture, Build-Kernel-Version(opt), Build-Date, Build-Path(opt), Build-Tainted-By(opt), Installed-Build-Depends, Environment`
  - changes: `Format, Date, Source, Binary, Built-For-Profiles, Architecture, Version, Distribution, Urgency, Maintainer, Changed-By, Description, Changes, Checksums-Sha1/Sha256, Files`
- Post-signature checksum cascade implemented exactly as dpkg does (sign dsc → patch buildinfo checksums → sign buildinfo → patch changes `Files`+`Checksums-*` → sign changes).
- Differential test harness (§8).

### Phase 2 — satellite replacement (optional, incremental)
Swap subprocesses for native implementations, one tool at a time, gated by differential tests. Start with version-compare + changelog (already half-present in pkh).

### Explicitly out of scope
`dpkg-source` internals, `dpkg-deb` packing, apt resolution (pkh correctly delegates to `apt-get build-dep` + dose3 for explanations already).

---

## 7. Tricky details to get right (gotchas list)

1. **Version splitting**: `<epoch:>upstream<-revision>`; `.dsc`/tarball names use *upstream* portion without epoch; `.changes` name uses version **without epoch** but **with revision** (`$sversion` in the script).
2. **`-nc` implication chain**: `-nc` ⇒ binary build implied; `-nc -S` ⇒ no dep check.
3. **UNRELEASED** ⇒ auto-disable all signing (warn), `--force-sign` overrides.
4. **Signing invalidates checksums transitively** (dsc → buildinfo → changes); get the cascade order right or archive tools reject the upload.
5. **`debian/files`** is the contract between `debian/rules` (via `dh_builddeb`/`dpkg-gencontrol`/`dpkg-distaddfile`) and the changes generator: lines of `filename section priority [key=value...]`.
6. **Arch selection for names**: `arch` suffix in artifact filenames is `host-arch` for arch-dep builds, `all` for indep-only, `source` for source-only.
7. **`dpkg-source --before-build` must run even for binary-only builds** (patch application), and `--after-build` at the end — this replaces pkh's current manual quilt step and fixes `3.0 (quilt)` correctness.
8. **Environment parity**: `dpkg-architecture -f` dump must be imported wholesale (not cherry-picked) — packages test `DEB_HOST_GNU_TYPE`, `DEB_BUILD_MULTIARCH`, `DEB_BUILD_ARCH_ENDIAN`, etc. If embedding tables later, mirror the full variable set.
9. **Exit codes**: preserve dpkg conventions (3 = unmet build-deps) so wrappers/scripts behave identically.
10. **Locale**: dpkg sets `LANG=C`-ish determinism for subprocesses — pkh already does this in [`local.rs`](../src/deb/local.rs); keep for all pipeline steps.
11. **Multiline field rendering**: values starting with `\n` render as `Field:` + indented continuation lines (no inline first line, no trailing space) — this is how dpkg emits `Changes`, `Files`, `Installed-Build-Depends`, `Environment`.
12. **Artifact ordering** in `Checksums-*`/`Files` is insertion order (dsc → tarballs in dsc-field order → debs → buildinfo), not alphabetical.

---

## 8. Testing strategy

- **Differential harness**: run real `dpkg-buildpackage` and the native pipeline over a corpus (packages covering: `1.0` non-native, `3.0 (quilt)` with/without patches, native, binaries-only, indep-only, cross, RRR variants, UNRELEASED) and diff artifacts modulo timestamps/signatures.
- **Port unit cases** from dpkg's own `t/` tests for version compare, deps parsing, changelog parsing.
- **Golden-file tests** for `.changes`/`.buildinfo` writers.
- Pin the reference dpkg version in CI commentary (behavior drift across dpkg releases is the main maintenance cost).

---

## 9. Risks

| Risk | Mitigation |
|---|---|
| Behavior drift vs future dpkg releases (new fields, format bumps) | Differential tests pinned to a reference version; changes/buildinfo formats are extremely stable (1.8 / 1.0 for years) |
| Archive/upload tooling rejects our `.changes`/`.buildinfo` | Validate with `lintian` + a real PPA upload early in Phase 1 |
| License contamination | dpkg is GPL-2+. Write from documented behavior/format specs and observation, **do not transliterate Perl**; alternatively accept GPL for pkh (currently no license field in [`Cargo.toml`](../Cargo.toml) — decision needed) |
| Scope creep toward Scope C | Hard rule: `dpkg-source` stays a subprocess |
| Remote-context divergence | All tree-touching steps go through [`ContextCommand`](../src/context/mod.rs); metadata steps operate on locally-synced copies like [`changelog.rs`](../src/changelog.rs) already does |

---

## 10. Effort summary

| Item | Estimate |
|---|---|
| Phase 0 (UI capture for `-S`) | 1–2 days |
| Phase 1 (native orchestrator + buildinfo/changes/signing) | 2–4 weeks |
| Phase 2 per satellite tool | 3 days – 2 weeks each |
| Scope C (`dpkg-source`) | 2–3+ months — rejected |

Bottom line: **yes, it's doable — for the orchestrator.** Treat `dpkg-source` as a permanent subprocess dependency, and the task becomes a well-bounded, high-value refactor that also fixes real gaps in pkh's binary path.

---

## 11. Implementation status (Phase 1 — source builds)

Landed in `src/build/` (~2.8 kLOC incl. tests), wired behind the historical
entry point [`build_source_package()`](../src/build/mod.rs) so `pkh build`
now runs the native pipeline:

| Module | Role |
|---|---|
| [`control.rs`](../src/build/control.rs) | deb822 paragraph parser/writer (dpkg-compatible multiline rendering) |
| [`checksums.rs`](../src/build/checksums.rs) | md5/sha1/sha256 registry, insertion-ordered like dpkg's artifact accumulation |
| [`metadata.rs`](../src/build/metadata.rs) | version splitting/validation, full changelog entry parse (incl. binNMU `binary-only`), control info |
| [`buildtype.rs`](../src/build/buildtype.rs) | build-type bitflags + rules-target/artifact-suffix mapping |
| [`env.rs`](../src/build/env.rs) | `SOURCE_DATE_EPOCH`, `DEB_BUILD_OPTIONS`, `dpkg-architecture` env dump, vendor/profiles, sanitized `Environment` field |
| [`files.rs`](../src/build/files.rs) | `debian/files` registry (parse/atomic save) |
| [`buildinfo.rs`](../src/build/buildinfo.rs) | `.buildinfo` writer + `Installed-Build-Depends` closure over the dpkg status DB |
| [`changes.rs`](../src/build/changes.rs) | `.changes` writer (canonical field order, legacy `Files` + `Checksums-Sha1/256`) |
| [`sign.rs`](../src/build/sign.rs) | gpgme clearsigning + key-id validation |

Still delegated to subprocesses: `dpkg-source -b/--before-build/--after-build`
(by design, see §5 Scope C).

**Differential validation** — automated in
[`build/mod.rs`](../src/build/mod.rs) (`mod differential_tests`, runs by
default with `cargo test --lib`):

- a corpus of 12 synthetic fixtures covering: native/quilt/1.0 formats,
  epochs, `~` pre-releases, Ubuntu/Debian series (focal, noble, jammy,
  trixie, unstable), high urgency, multiple binary stanzas, one/two quilt
  patches, binNMU (`binary-only=yes` + previous-entry metadata),
  `(Closes: #…)` extraction, UNRELEASED no-sign, extra source-stanza fields;
- each case builds the tree twice (`cp -a` copies) — once with real
  `dpkg-buildpackage -S -I -i -nc -d --no-sign`, once with the native
  pipeline — then compares:
  - `.dsc` payload byte-for-byte,
  - `.changes` field-by-field (checksum lines of the `.buildinfo` itself
    excluded),
  - `.buildinfo` structure (machine-dependent fields excluded);
- real **archive packages** are pulled with pkh's own
  [`pull`](../src/pull.rs) (archive download mode) and compared the same
  way, via [`differential_real_archive_package()`](../src/build/mod.rs)
  which takes `(package, dist, series)`:
  - always-on CI tests: `hello` @ ubuntu/noble, `dosfstools` @
    debian/trixie, `sl` @ ubuntu/focal;
  - an additional `#[ignore]`-gated test for ad-hoc broad runs:
    ```text
    PKH_DIFF_PACKAGES="bash coreutils curl" PKH_DIFF_DIST=ubuntu \
    PKH_DIFF_SERIES=noble cargo test --lib \
        differential_real_archive_packages -- --ignored
    ```
  (requires network access).

Manual validation additionally confirmed: signed builds verify with
`gpg --verify` on all three artifacts; the patch apply/unapply lifecycle is
correct for quilt formats.

**Known divergences** (documented, all informational fields): no
`Build-Tainted-By` (vendor hook), no `dpkg-buildflags` origin tracking in the
`Environment` field, vendor default profiles approximated
(`derivative.ubuntu noudeb` for Ubuntu).

**Next steps**: binary-build adoption of the same pipeline inside ephemeral
contexts (`.changes`/`.buildinfo` generation for `pkh deb`, replacing the
manual quilt step), then Phase 2 satellite replacement.