vhaudiquet
b246b30eac
Plain-speech pass: drop the em-dash asides and mid-sentence colons,
reword the boldface digest constructions ("What --incremental buys
is ..."), and replace the fluffy intro ("aims at wrapping ... while
being smarter at integrating all workflows") with what pkh actually
does. No content changes.
264 lines
11 KiB
Markdown
264 lines
11 KiB
Markdown
# pkh
|
|
|
|
`pkh` is a packaging helper for Debian/Ubuntu packages.
|
|
|
|
|
|
|
|
## Installation
|
|
|
|
From crates.io:
|
|
|
|
```
|
|
cargo install pkh
|
|
```
|
|
|
|
Or build from source (the same system packages are needed either way):
|
|
|
|
```
|
|
sudo apt install pkg-config libssl-dev libgpg-error-dev libgpgme-dev
|
|
git clone https://git.vhaudiquet.fr/vhaudiquet/pkh.git
|
|
cd pkh
|
|
cargo install --path .
|
|
```
|
|
|
|
At runtime pkh shells out to the Debian packaging toolchain (git,
|
|
dpkg-dev, quilt, mmdebstrap, lintian, pristine-tar, autopkgtest, ...).
|
|
Install the ones your workflows use, or build the classic snap from
|
|
`snap/snapcraft.yaml` (`snapcraft pack`), which carries them.
|
|
|
|
## Usage and features
|
|
|
|
### Basic concepts
|
|
|
|
`pkh` wraps the different Debian tools and workflows
|
|
(`git`, `git-ubuntu`, `pull-debian-sources`, `pull-lp-sources`, `pull-ppa-sources`,
|
|
`dch`, `dpkg-buildpackage`, `sbuild`, `dpkg-source`, `quilt`, ...) into
|
|
one tool with the same interface for everything, which also handles each
|
|
workflow's specifics.
|
|
|
|
The same options work across subcommands (with very few command-specific options):
|
|
```
|
|
Options:
|
|
-s, --series <series> Target package distribution series
|
|
-d, --dist <dist> Target package distribution (debian, ubuntu)
|
|
-v, --version <version> Target package version
|
|
-a, --arch <arch> Target architecture (amd64, arm64, riscv64, ...)
|
|
-p, --pocket <pocket> Target distribution pocket (updates, security, proposed, ...)
|
|
--ppa <ppa> Do the action in/for a specific PPA
|
|
```
|
|
|
|
Commands and workflows include:
|
|
```
|
|
Commands:
|
|
new Scaffold a new Debian source package (buildable right away)
|
|
pull Pull a source package from the archive or git
|
|
chlog Auto-generate changelog entry, editing it, committing it afterwards
|
|
build Build the source package (into a .dsc)
|
|
put Upload the built source package to a PPA
|
|
deb Build the source package into binary package (.deb)
|
|
test Run the package's DEP-8 as-installed tests (autopkgtests)
|
|
lint Lint the package (lintian wrapper + pkh-native checks)
|
|
prune Prune residual pkh build artifacts and caches
|
|
help Print this message or the help of the given subcommand(s)
|
|
|
|
Options:
|
|
-h, --help Print help
|
|
-V, --version Print version
|
|
```
|
|
|
|
### Examples
|
|
|
|
A typical workflow to patch an Ubuntu package `hello` on the development release could be:
|
|
```
|
|
# Obtain the package source
|
|
git ubuntu clone hello
|
|
cd hello
|
|
# Apply the patch to the package
|
|
...
|
|
dpkg-source --commit
|
|
# Increment version number
|
|
dch
|
|
# Test that the patch builds
|
|
git ubuntu export-orig
|
|
dpkg-buildpackage -S -I -i -nc -d
|
|
sbuild --dist resolute --arch amd64 ../hello_xxx.dsc
|
|
# Upload the package to a ppa
|
|
dput ppa:user/hello_xxx ../hello_xxx_source.changes
|
|
# Commit the changes to git
|
|
git add debian/patches/xxx.patch
|
|
git commit -m "Applied patch xxx"
|
|
git add debian/changelog
|
|
git commit -m "changelog"
|
|
git checkout -b xxx
|
|
git push xxx user-fork
|
|
```
|
|
|
|
That is a lot of different tools and operations. With pkh, the same workflow:
|
|
```
|
|
# Obtain the package source (and orig tarball)
|
|
pkh pull hello # needs -d ubuntu if you are not running Ubuntu
|
|
# Apply the patch to the package
|
|
...
|
|
git add debian/patches/xxx.patch
|
|
git commit -m "Applied patch xxx"
|
|
pkh chlog
|
|
git add debian/changelog
|
|
git commit -m "d/changelog"
|
|
# Test that the package builds
|
|
pkh build
|
|
pkh deb
|
|
# Upload the package to a ppa
|
|
pkh put --ppa user/hello_xxx
|
|
# Push the commits to your fork
|
|
git push xxx user-fork
|
|
```
|
|
|
|
`pkh chlog` edits `debian/changelog` by default; point it at another file
|
|
with `--file <path>` (`-f`).
|
|
|
|
`pkh chlog` walks the commit history back to the previous changelog entry.
|
|
Without a version tag, that search is capped at the last 500 commits
|
|
(`--limit <n>`, `0` = unlimited): beyond the cap, the entry simply starts
|
|
at the oldest commit searched, which keeps the run fast on large histories.
|
|
|
|
### Incremental builds (build records)
|
|
|
|
Every `pkh deb` build records its metadata under
|
|
`/var/tmp/pkh/sessions/<target>/<id>/`: the manifest (identity, fork
|
|
lineage, outcome, the sha of the chroot tarball it used) and a full
|
|
`log/build.log` tee of the build output. The chroot itself is per-run
|
|
scratch, extracted fresh from the sha-verified mmdebstrap tarball cache
|
|
on every run and removed at the end, plain and `--incremental` alike,
|
|
so records stay small. `--incremental` keeps the record's staged tree
|
|
(`tree-upper/`), the overlay upperdir where the build's writes land,
|
|
holding the compiled objects.
|
|
|
|
Fresh chroots share their downloads. Every chroot's
|
|
`/var/cache/apt/archives` is a bind mount over a shared apt download
|
|
cache at `/var/tmp/pkh/apt-cache/<series>-<arch>/`, which the installs
|
|
of one run fill and the next runs reuse (apt is configured to keep the
|
|
downloaded packages, and an `apt-get autoclean` at the end of each run
|
|
drops the debs the archive no longer serves). `pkh prune` removes the
|
|
caches untouched for a week.
|
|
|
|
Fresh chroots share their compiles too. Every run's chroot
|
|
gets a shared ccache (at `/var/tmp/pkh/ccache/<target>/`, mounted at
|
|
`/var/cache/pkh-ccache`) wired in through a compiler PATH shim that
|
|
resolves `gcc`, `g++`, `cc` and `c++` to ccache, with content-keyed
|
|
cache checks (`CCACHE_COMPILERCHECK=content`), so cache hits survive
|
|
the fresh toolchain of a new chroot. The cache is capped at 10G per
|
|
target (override with `PKH_CCACHE_MAXSIZE`); `pkh deb --no-ccache`
|
|
bypasses the shim and cache entirely, an escape hatch for rules that
|
|
mishandle compiler wrappers (kbuild-style flavour configs).
|
|
|
|
`--incremental` forks a copy of a previous record and never modifies
|
|
the base, so a failed run cannot destroy the last-good state it was
|
|
built from. The staged tree is carried only when the base's recorded
|
|
tarball sha still matches the cache (the toolchain did not move) and the
|
|
changelog version is unchanged; otherwise the fork builds from a fresh
|
|
tree:
|
|
|
|
```
|
|
pkh deb # plain run: fresh chroot, fresh tree; the
|
|
# record keeps only metadata and the log
|
|
# (nothing is kept even on failure or
|
|
# interrupt)
|
|
pkh deb --incremental # fork the newest successful record of the
|
|
# target (among the last five carrying a staged
|
|
# tree) and keep this run's staged tree, for
|
|
# the next fork to carry
|
|
pkh deb --incremental <id>
|
|
# fork that exact record (see `pkh deb list`);
|
|
# a failed record is fine if its staged tree is
|
|
pkh deb list # the records of this tree: id, outcome,
|
|
# version, fork lineage, staged-tree state,
|
|
# dep-install and build times, age
|
|
pkh deb log [-f] [<id>] # print a record's build log; -f follows a
|
|
# live build
|
|
pkh deb profile [<id>] # a record's per-phase timing profile: how
|
|
# long every step took (chroot bootstrap,
|
|
# apt phases, patch application, build-user
|
|
# setup, build and binary steps, artifact
|
|
# retrieval), with their shares of the total
|
|
pkh deb --no-ccache # build without the shared compile cache
|
|
```
|
|
|
|
Each target keeps at most five staged trees: when a sixth is retained,
|
|
the oldest degrades to a metadata-only record (manifest and log stay),
|
|
and `pkh prune` removes records untouched for a week. `pkh deb
|
|
--incremental` never forks a record built for a different
|
|
series/architecture.
|
|
|
|
Packages whose `debian/rules` keeps an input-untracked stamp cache (the
|
|
kernels' `debian/stamps`) declare it in the quirks data
|
|
(`resume_clear`). Forked builds drop the cache, so the wrapper steps,
|
|
including the kernel's flavour config export, re-run over the
|
|
incremental inner builds and a config edit is never silently ignored.
|
|
A build killed hard (OOM) can leave its chroot behind; the next run of
|
|
that target sweeps it (unmounting and escalating through sudo when
|
|
needed), and `pkh prune` collects the rest.
|
|
|
|
On a pinned runner, a machine that must resolve every package from one
|
|
mirror (typically a caching proxy), `pkh deb --mirror <url>` builds the
|
|
chroot and resolves every package from that base URL. The mirror is
|
|
passed to mmdebstrap at bootstrap, so the chroot tarball's apt sources
|
|
carry it and every apt inside the build follows. The mirror must serve
|
|
every suite and architecture the build resolves (the security pocket
|
|
and, for `--cross` builds, the target architecture's archive included);
|
|
PPAs and other third-party sources are left alone. The tarball cache is
|
|
keyed per mirror, so pinned and unpinned builds keep separate tarballs
|
|
(and `--incremental` records never fork across a mirror change).
|
|
|
|
### Running the DEP-8 tests
|
|
|
|
`pkh test` runs the package's as-installed tests (`debian/tests/control`,
|
|
declared by `Testsuite: autopkgtest` in `debian/control`) through a
|
|
native runner on a pkh testbed, and reports one verdict per test:
|
|
|
|
```
|
|
pkh test # test this tree: builds it first if needed
|
|
pkh test --mode local # on the current context, no isolation
|
|
pkh test --mode vm --image img.qcow2 # in a VM, via autopkgtest
|
|
pkh test --json --fail-on skip # CI-shaped
|
|
pkh test --list-tests # what does this package test?
|
|
```
|
|
|
|
By default the tests run in an ephemeral unshare chroot bootstrapped
|
|
from the same cached tarballs as `pkh deb`, against the binaries pkh deb
|
|
just built next to the tree (stale or missing output is rebuilt first;
|
|
`--no-build` refuses, `--debs` overrides). Pockets and PPAs resolve test
|
|
dependencies like in `pkh deb` (`-p`, `--ppa`), and `--test-name`,
|
|
`--skip-test`, `--shell-fail` and `--setup-commands` cover the day-to-day
|
|
debugging loop. Tests needing a real machine (`isolation-machine`,
|
|
`needs-reboot`) run only in `--mode vm`, which execs the installed
|
|
`autopkgtest` with its qemu runner.
|
|
|
|
Exit codes: 0 all tests passed (skips and flaky allowed), 1 at least one
|
|
test failed (or a `--fail-on` trigger), 2 runtime errors (no declared
|
|
tests, no usable testbed, ...).
|
|
|
|
## End-to-end package harness
|
|
|
|
`tests/e2e/` exercises pkh the way it is used in anger: a catalog of ~60
|
|
real source packages (from `hello` and `micro` to `dracut`, `grub2` and
|
|
the kernel) run through the full `pull` → `chlog --rebuild` → `build` →
|
|
`deb` workflow, each stage timed, logged and checked for artifacts:
|
|
|
|
```
|
|
tests/e2e/run.py --list # what would run
|
|
tests/e2e/run.py # smoke + standard tiers
|
|
tests/e2e/run.py --tier long # the multi-hour builds
|
|
```
|
|
|
|
See [tests/e2e/README.md](tests/e2e/README.md) for prerequisites,
|
|
filters (`--only`, `--category`, `--tag`), resume support and the run
|
|
output layout.
|
|
|
|
## Future improvement ideas
|
|
|
|
- pull: try to fetch the correct git branch for series on Debian
|
|
- deb: asynchronous build, detachable and monitorable
|
|
- put: allow uploads to Debian or Ubuntu archives
|
|
- pull: cache Sources.gz files to improve speed
|
|
- pull: 'pkh pull' in a package tree should git pull and re-fetch orig tgz
|