vhaudiquet
93943dcce0
The per-target retention kept the newest five staged trees however large they are: five carried kernel trees are three-digit GiB, most of it compiled objects whose resume value does not scale with their size. A staged tree larger than the retention cap (a user setting, see below) is now dropped at the end of its own build — the record degrades to metadata-only and the next build re-stages from scratch — and the retention sweep also degrades older oversized trees, covering records kept by older pkh versions and interrupted builds. The cap of 0 disables the size rule; the count cap is unchanged. The cap is the first setting of a user configuration file, ~/.config/pkh/config.json (the directory already holds contexts.json): missing file and missing keys mean defaults, a corrupt file is backed up and the defaults apply, and PKH_STAGED_MAXSIZE overrides the file. The default is 20G, the scale where a carried kernel tree stops earning its disk. Every retained tree's size is measured once at retention (the same concurrent, mount-crossing-safe walker the prune report uses) and stored in the record manifest, so sweeps reuse it instead of re-walking the trees on every build.
278 lines
11 KiB
Markdown
278 lines
11 KiB
Markdown
# pkh
|
|
|
|
`pkh` is a packaging helper for Debian/Ubuntu packages.
|
|
|
|
|
|
|
|
## Installation
|
|
|
|
From crates.io:
|
|
|
|
```
|
|
cargo install pkh
|
|
```
|
|
|
|
Or build from source (the same system packages are needed either way):
|
|
|
|
```
|
|
sudo apt install pkg-config libssl-dev libgpg-error-dev libgpgme-dev
|
|
git clone https://git.vhaudiquet.fr/vhaudiquet/pkh.git
|
|
cd pkh
|
|
cargo install --path .
|
|
```
|
|
|
|
At runtime pkh shells out to the Debian packaging toolchain (git,
|
|
dpkg-dev, quilt, mmdebstrap, lintian, pristine-tar, autopkgtest, ...).
|
|
Install the ones your workflows use, or build the classic snap from
|
|
`snap/snapcraft.yaml` (`snapcraft pack`), which carries them.
|
|
|
|
## Usage and features
|
|
|
|
### Basic concepts
|
|
|
|
`pkh` wraps the different Debian tools and workflows
|
|
(`git`, `git-ubuntu`, `pull-debian-sources`, `pull-lp-sources`, `pull-ppa-sources`,
|
|
`dch`, `dpkg-buildpackage`, `sbuild`, `dpkg-source`, `quilt`, ...) into
|
|
one tool with the same interface for everything, which also handles each
|
|
workflow's specifics.
|
|
|
|
The same options work across subcommands (with very few command-specific options):
|
|
```
|
|
Options:
|
|
-s, --series <series> Target package distribution series
|
|
-d, --dist <dist> Target package distribution (debian, ubuntu)
|
|
-v, --version <version> Target package version
|
|
-a, --arch <arch> Target architecture (amd64, arm64, riscv64, ...)
|
|
-p, --pocket <pocket> Target distribution pocket (updates, security, proposed, ...)
|
|
--ppa <ppa> Do the action in/for a specific PPA
|
|
```
|
|
|
|
Commands and workflows include:
|
|
```
|
|
Commands:
|
|
new Scaffold a new Debian source package (buildable right away)
|
|
pull Pull a source package from the archive or git
|
|
chlog Auto-generate changelog entry, editing it, committing it afterwards
|
|
build Build the source package (into a .dsc)
|
|
put Upload the built source package to a PPA
|
|
deb Build the source package into binary package (.deb)
|
|
test Run the package's DEP-8 as-installed tests (autopkgtests)
|
|
lint Lint the package (lintian wrapper + pkh-native checks)
|
|
prune Prune residual pkh build artifacts and caches
|
|
help Print this message or the help of the given subcommand(s)
|
|
|
|
Options:
|
|
-h, --help Print help
|
|
-V, --version Print version
|
|
```
|
|
|
|
### Examples
|
|
|
|
A typical workflow to patch an Ubuntu package `hello` on the development release could be:
|
|
```
|
|
# Obtain the package source
|
|
git ubuntu clone hello
|
|
cd hello
|
|
# Apply the patch to the package
|
|
...
|
|
dpkg-source --commit
|
|
# Increment version number
|
|
dch
|
|
# Test that the patch builds
|
|
git ubuntu export-orig
|
|
dpkg-buildpackage -S -I -i -nc -d
|
|
sbuild --dist resolute --arch amd64 ../hello_xxx.dsc
|
|
# Upload the package to a ppa
|
|
dput ppa:user/hello_xxx ../hello_xxx_source.changes
|
|
# Commit the changes to git
|
|
git add debian/patches/xxx.patch
|
|
git commit -m "Applied patch xxx"
|
|
git add debian/changelog
|
|
git commit -m "changelog"
|
|
git checkout -b xxx
|
|
git push xxx user-fork
|
|
```
|
|
|
|
That is a lot of different tools and operations. With pkh, the same workflow:
|
|
```
|
|
# Obtain the package source (and orig tarball)
|
|
pkh pull hello # needs -d ubuntu if you are not running Ubuntu
|
|
# Apply the patch to the package
|
|
...
|
|
git add debian/patches/xxx.patch
|
|
git commit -m "Applied patch xxx"
|
|
pkh chlog
|
|
git add debian/changelog
|
|
git commit -m "d/changelog"
|
|
# Test that the package builds
|
|
pkh build
|
|
pkh deb
|
|
# Upload the package to a ppa
|
|
pkh put --ppa user/hello_xxx
|
|
# Push the commits to your fork
|
|
git push xxx user-fork
|
|
```
|
|
|
|
`pkh chlog` edits `debian/changelog` by default; point it at another file
|
|
with `--file <path>` (`-f`).
|
|
|
|
`pkh chlog` walks the commit history back to the previous changelog entry.
|
|
Without a version tag, that search is capped at the last 500 commits
|
|
(`--limit <n>`, `0` = unlimited): beyond the cap, the entry simply starts
|
|
at the oldest commit searched, which keeps the run fast on large histories.
|
|
|
|
### Incremental builds (build records)
|
|
|
|
Every `pkh deb` build records its metadata under
|
|
`/var/tmp/pkh/sessions/<target>/<id>/`: the manifest (identity, fork
|
|
lineage, outcome, the sha of the chroot tarball it used) and a full
|
|
`log/build.log` tee of the build output. The chroot itself is per-run
|
|
scratch, extracted fresh from the sha-verified mmdebstrap tarball cache
|
|
on every run and removed at the end, plain and `--incremental` alike,
|
|
so records stay small. `--incremental` keeps the record's staged tree
|
|
(`tree-upper/`), the overlay upperdir where the build's writes land,
|
|
holding the compiled objects.
|
|
|
|
Fresh chroots share their downloads. Every chroot's
|
|
`/var/cache/apt/archives` is a bind mount over a shared apt download
|
|
cache at `/var/tmp/pkh/apt-cache/<series>-<arch>/`, which the installs
|
|
of one run fill and the next runs reuse (apt is configured to keep the
|
|
downloaded packages, and an `apt-get autoclean` at the end of each run
|
|
drops the debs the archive no longer serves). `pkh prune` removes the
|
|
caches untouched for a week.
|
|
|
|
Fresh chroots share their compiles too. Every run's chroot
|
|
gets a shared ccache (at `/var/tmp/pkh/ccache/<target>/`, mounted at
|
|
`/var/cache/pkh-ccache`) wired in through a compiler PATH shim that
|
|
resolves `gcc`, `g++`, `cc` and `c++` to ccache, with content-keyed
|
|
cache checks (`CCACHE_COMPILERCHECK=content`), so cache hits survive
|
|
the fresh toolchain of a new chroot. The cache is capped at 10G per
|
|
target (override with `PKH_CCACHE_MAXSIZE`); `pkh deb --no-ccache`
|
|
bypasses the shim and cache entirely, an escape hatch for rules that
|
|
mishandle compiler wrappers (kbuild-style flavour configs).
|
|
|
|
`--incremental` forks a copy of a previous record and never modifies
|
|
the base, so a failed run cannot destroy the last-good state it was
|
|
built from. The staged tree is carried only when the base's recorded
|
|
tarball sha still matches the cache (the toolchain did not move) and the
|
|
changelog version is unchanged; otherwise the fork builds from a fresh
|
|
tree:
|
|
|
|
```
|
|
pkh deb # plain run: fresh chroot, fresh tree; the
|
|
# record keeps only metadata and the log
|
|
# (nothing is kept even on failure or
|
|
# interrupt)
|
|
pkh deb --incremental # fork the newest successful record of the
|
|
# target (among the last five carrying a staged
|
|
# tree) and keep this run's staged tree, for
|
|
# the next fork to carry
|
|
pkh deb --incremental <id>
|
|
# fork that exact record (see `pkh deb list`);
|
|
# a failed record is fine if its staged tree is
|
|
pkh deb list # the records of this tree: id, outcome,
|
|
# version, fork lineage, staged-tree state,
|
|
# dep-install and build times, age
|
|
pkh deb log [-f] [<id>] # print a record's build log; -f follows a
|
|
# live build
|
|
pkh deb profile [<id>] # a record's per-phase timing profile: how
|
|
# long every step took (chroot bootstrap,
|
|
# apt phases, patch application, build-user
|
|
# setup, build and binary steps, artifact
|
|
# retrieval), with their shares of the total
|
|
pkh deb --no-ccache # build without the shared compile cache
|
|
```
|
|
|
|
Each target keeps at most five staged trees: when a sixth is retained,
|
|
the oldest degrades to a metadata-only record (manifest and log stay).
|
|
A staged tree larger than 20 GiB is never kept at all — a tree that
|
|
size costs more disk than the resume it buys, so its record degrades
|
|
too and the next build re-stages from scratch. The cap is a setting:
|
|
|
|
```json
|
|
// ~/.config/pkh/config.json (missing = defaults; keys may be omitted)
|
|
{ "staged_maxsize": "20G" }
|
|
```
|
|
|
|
`"0"` disables the size rule entirely, and `PKH_STAGED_MAXSIZE`
|
|
overrides the file when set. `pkh prune` removes records older than a
|
|
week. `pkh prune --dry-run` previews everything it would remove,
|
|
grouped by kind with sizes; `--all` ignores the age and removes
|
|
everything a live build does not hold, recent records and staged trees
|
|
included; `-f/--force` skips the size pass and the confirmation
|
|
prompt, removing immediately. `pkh deb --incremental` never forks a
|
|
record built for a different series/architecture.
|
|
|
|
Packages whose `debian/rules` keeps an input-untracked stamp cache (the
|
|
kernels' `debian/stamps`) declare it in the quirks data
|
|
(`resume_clear`). Forked builds drop the cache, so the wrapper steps,
|
|
including the kernel's flavour config export, re-run over the
|
|
incremental inner builds and a config edit is never silently ignored.
|
|
A build killed hard (OOM) can leave its chroot behind; the next run of
|
|
that target sweeps it (unmounting and escalating through sudo when
|
|
needed), and `pkh prune` collects the rest.
|
|
|
|
On a pinned runner, a machine that must resolve every package from one
|
|
mirror (typically a caching proxy), `pkh deb --mirror <url>` builds the
|
|
chroot and resolves every package from that base URL. The mirror is
|
|
passed to mmdebstrap at bootstrap, so the chroot tarball's apt sources
|
|
carry it and every apt inside the build follows. The mirror must serve
|
|
every suite and architecture the build resolves (the security pocket
|
|
and, for `--cross` builds, the target architecture's archive included);
|
|
PPAs and other third-party sources are left alone. The tarball cache is
|
|
keyed per mirror, so pinned and unpinned builds keep separate tarballs
|
|
(and `--incremental` records never fork across a mirror change).
|
|
|
|
### Running the DEP-8 tests
|
|
|
|
`pkh test` runs the package's as-installed tests (`debian/tests/control`,
|
|
declared by `Testsuite: autopkgtest` in `debian/control`) through a
|
|
native runner on a pkh testbed, and reports one verdict per test:
|
|
|
|
```
|
|
pkh test # test this tree: builds it first if needed
|
|
pkh test --mode local # on the current context, no isolation
|
|
pkh test --mode vm --image img.qcow2 # in a VM, via autopkgtest
|
|
pkh test --json --fail-on skip # CI-shaped
|
|
pkh test --list-tests # what does this package test?
|
|
```
|
|
|
|
By default the tests run in an ephemeral unshare chroot bootstrapped
|
|
from the same cached tarballs as `pkh deb`, against the binaries pkh deb
|
|
just built next to the tree (stale or missing output is rebuilt first;
|
|
`--no-build` refuses, `--debs` overrides). Pockets and PPAs resolve test
|
|
dependencies like in `pkh deb` (`-p`, `--ppa`), and `--test-name`,
|
|
`--skip-test`, `--shell-fail` and `--setup-commands` cover the day-to-day
|
|
debugging loop. Tests needing a real machine (`isolation-machine`,
|
|
`needs-reboot`) run only in `--mode vm`, which execs the installed
|
|
`autopkgtest` with its qemu runner.
|
|
|
|
Exit codes: 0 all tests passed (skips and flaky allowed), 1 at least one
|
|
test failed (or a `--fail-on` trigger), 2 runtime errors (no declared
|
|
tests, no usable testbed, ...).
|
|
|
|
## End-to-end package harness
|
|
|
|
`tests/e2e/` exercises pkh the way it is used in anger: a catalog of ~60
|
|
real source packages (from `hello` and `micro` to `dracut`, `grub2` and
|
|
the kernel) run through the full `pull` → `chlog --rebuild` → `build` →
|
|
`deb` workflow, each stage timed, logged and checked for artifacts:
|
|
|
|
```
|
|
tests/e2e/run.py --list # what would run
|
|
tests/e2e/run.py # smoke + standard tiers
|
|
tests/e2e/run.py --tier long # the multi-hour builds
|
|
```
|
|
|
|
See [tests/e2e/README.md](tests/e2e/README.md) for prerequisites,
|
|
filters (`--only`, `--category`, `--tag`), resume support and the run
|
|
output layout.
|
|
|
|
## Future improvement ideas
|
|
|
|
- pull: try to fetch the correct git branch for series on Debian
|
|
- deb: asynchronous build, detachable and monitorable
|
|
- put: allow uploads to Debian or Ubuntu archives
|
|
- pull: cache Sources.gz files to improve speed
|
|
- pull: 'pkh pull' in a package tree should git pull and re-fetch orig tgz
|