vhaudiquet
dd2634abe3
Records carried a full chroot each, and space turned out to be the hard constraint: five records per target multiplied by a multi-GB chroot does not fit. The chroot is now per-run scratch — every build extracts a fresh one from the sha-verified mmdebstrap tarball cache into its record's chroot/ dir and removes it at the end (success, failure and interrupt included); resume_chroot, the .pkh-ready probe and chroot_is_usable are gone, and crashed-run leftovers are swept when the next run of that identity starts. What a record keeps instead: manifest.json (renamed, now with the tarball sha it used, a kept_upper flag and a tree-only journal — the environment phases rerun in every fresh chroot), a log/build.log full tee of the build output (kept for EVERY build; plain runs become metadata-only records, so keep-iff-incremental now applies to the staged tree alone), and — for --incremental runs — tree-upper/ with its host-files.list. The fork gate carries a base's staged tree only while its recorded tarball sha still matches the cache and the changelog version is unchanged; a fork candidate is any record with a staged tree. The keep-last-5 cap now applies to staged trees: a sixth retention degrades the oldest record to metadata-only instead of deleting it. New `pkh deb log [-f] [<id>]` prints (or follows) a record's build log, and `pkh deb list` shows the staged-tree state and plain-run records too. README rewritten around the shrunk model.
211 lines
8.1 KiB
Markdown
211 lines
8.1 KiB
Markdown
# pkh
|
|
|
|
`pkh` is a packaging helper for Debian/Ubuntu packages.
|
|
|
|
|
|
|
|
## Installation
|
|
|
|
From crates.io:
|
|
|
|
```
|
|
cargo install pkh
|
|
```
|
|
|
|
Or build from source (the same system packages are needed either way):
|
|
|
|
```
|
|
sudo apt install pkg-config libssl-dev libgpg-error-dev libgpgme-dev
|
|
git clone https://git.vhaudiquet.fr/vhaudiquet/pkh.git
|
|
cd pkh
|
|
cargo install --path .
|
|
```
|
|
|
|
At runtime pkh shells out to the Debian packaging toolchain (git,
|
|
dpkg-dev, quilt, mmdebstrap, lintian, pristine-tar, autopkgtest, ...):
|
|
install the ones your workflows use, or build the classic snap from
|
|
`snap/snapcraft.yaml` (`snapcraft pack`), which carries them.
|
|
|
|
## Usage and features
|
|
|
|
### Basic concepts
|
|
|
|
`pkh` aims at wrapping the different debian tools and workflows
|
|
(`git`, `git-ubuntu`, `pull-debian-sources`, `pull-lp-sources`, `pull-ppa-sources`,
|
|
`dch`, `dpkg-buildpackage`, `sbuild`, `dpkg-source`, `quilt`, ...) into one tool, that
|
|
would have the same interface for everything, while being smarter at integrating all workflows.
|
|
|
|
Thus, `pkh` uses similar options for all subcommands (with very few command-specific options):
|
|
```
|
|
Options:
|
|
-s, --series <series> Target package distribution series
|
|
-d, --dist <dist> Target package distribution (debian, ubuntu)
|
|
-v, --version <version> Target package version
|
|
-a, --arch <arch> Target architecture (amd64, arm64, riscv64, ...)
|
|
-p, --pocket <pocket> Target distribution pocket (updates, security, proposed, ...)
|
|
--ppa <ppa> Do the action in/for a specific PPA
|
|
```
|
|
|
|
Commands and workflows include:
|
|
```
|
|
Commands:
|
|
new Scaffold a new Debian source package (buildable right away)
|
|
pull Pull a source package from the archive or git
|
|
chlog Auto-generate changelog entry, editing it, committing it afterwards
|
|
build Build the source package (into a .dsc)
|
|
put Upload the built source package to a PPA
|
|
deb Build the source package into binary package (.deb)
|
|
test Run the package's DEP-8 as-installed tests (autopkgtests)
|
|
lint Lint the package (lintian wrapper + pkh-native checks)
|
|
prune Prune residual pkh build artifacts and caches
|
|
help Print this message or the help of the given subcommand(s)
|
|
|
|
Options:
|
|
-h, --help Print help
|
|
-V, --version Print version
|
|
```
|
|
|
|
### Examples
|
|
|
|
A typical workflow to patch an Ubuntu package `hello` on the development release could be:
|
|
```
|
|
# Obtain the package source
|
|
git ubuntu clone hello
|
|
cd hello
|
|
# Apply the patch to the package
|
|
...
|
|
dpkg-source --commit
|
|
# Increment version number
|
|
dch
|
|
# Test that the patch builds
|
|
git ubuntu export-orig
|
|
dpkg-buildpackage -S -I -i -nc -d
|
|
sbuild --dist resolute --arch amd64 ../hello_xxx.dsc
|
|
# Upload the package to a ppa
|
|
dput ppa:user/hello_xxx ../hello_xxx_source.changes
|
|
# Commit the changes to git
|
|
git add debian/patches/xxx.patch
|
|
git commit -m "Applied patch xxx"
|
|
git add debian/changelog
|
|
git commit -m "changelog"
|
|
git checkout -b xxx
|
|
git push xxx user-fork
|
|
```
|
|
|
|
That is a lot of different tools and operations. With pkh, the same workflow:
|
|
```
|
|
# Obtain the package source (and orig tarball)
|
|
pkh pull hello # needs -d ubuntu if you are not running Ubuntu
|
|
# Apply the patch to the package
|
|
...
|
|
git add debian/patches/xxx.patch
|
|
git commit -m "Applied patch xxx"
|
|
pkh chlog
|
|
git add debian/changelog
|
|
git commit -m "d/changelog"
|
|
# Test that the package builds
|
|
pkh build
|
|
pkh deb
|
|
# Upload the package to a ppa
|
|
pkh put --ppa user/hello_xxx
|
|
# Push the commits to your fork
|
|
git push xxx user-fork
|
|
```
|
|
|
|
`pkh chlog` edits `debian/changelog` by default; point it at another file
|
|
with `--file <path>` (`-f`).
|
|
|
|
`pkh chlog` walks the commit history back to the previous changelog entry.
|
|
Without a version tag, that search is capped at the last 500 commits
|
|
(`--limit <n>`, `0` = unlimited): beyond the cap, the entry simply starts
|
|
at the oldest commit searched, which keeps the run fast on large histories.
|
|
|
|
### Incremental builds (build records)
|
|
|
|
Every `pkh deb` build records its metadata under
|
|
`/var/tmp/pkh/sessions/<target>/<id>/`: the manifest (identity, fork
|
|
lineage, outcome, the sha of the chroot tarball it used) and a full
|
|
`log/build.log` tee of the build output. The chroot itself is per-run
|
|
scratch — extracted fresh from the sha-verified mmdebstrap tarball cache
|
|
on every run and removed at the end, plain and `--incremental` alike —
|
|
so records stay small. What `--incremental` buys is the record's
|
|
**staged tree** (`tree-upper/`): the overlay upperdir where the build's
|
|
writes land, holding the compiled objects.
|
|
|
|
`--incremental` forks a previous record — a copy, the base is never
|
|
modified — so a failed run can never destroy the last-good state it was
|
|
built from. The staged tree is carried only when the base's recorded
|
|
tarball sha still matches the cache (the toolchain did not move) and the
|
|
changelog version is unchanged; otherwise the fork builds from a fresh
|
|
tree:
|
|
|
|
```
|
|
pkh deb # plain run: fresh chroot, fresh tree; the
|
|
# record keeps only metadata and the log
|
|
# (nothing is kept even on failure or
|
|
# interrupt)
|
|
pkh deb --incremental # fork the newest successful record of the
|
|
# target (among the last five carrying a staged
|
|
# tree) and keep this run's staged tree, for
|
|
# the next fork to carry
|
|
pkh deb --incremental <id>
|
|
# fork that exact record (see `pkh deb list`);
|
|
# a failed record is fine if its staged tree is
|
|
pkh deb list # the records of this tree: id, outcome,
|
|
# version, fork lineage, staged-tree state, age
|
|
pkh deb log [-f] [<id>] # print a record's build log; -f follows a
|
|
# live build
|
|
pkh prune # garbage-collect records (7-day retention)
|
|
```
|
|
|
|
Each target keeps at most five staged trees: when a sixth is retained,
|
|
the oldest degrades to a metadata-only record (manifest and log stay),
|
|
and `pkh prune` removes records untouched for a week. `pkh deb
|
|
--incremental` never forks a record built for a different
|
|
series/architecture.
|
|
|
|
Packages whose `debian/rules` keeps an input-untracked stamp cache (the
|
|
kernels' `debian/stamps`) declare it in the quirks data
|
|
(`resume_clear`): forked builds drop the cache so the wrapper steps —
|
|
including the kernel's flavour config export — re-run over the
|
|
incremental inner builds, and a config edit is never silently ignored.
|
|
A build killed hard (OOM) can leave its chroot behind; the next run of
|
|
that target sweeps it (unmounting and escalating through sudo when
|
|
needed), and `pkh prune` collects the rest.
|
|
|
|
### Running the DEP-8 tests
|
|
|
|
`pkh test` runs the package's as-installed tests (`debian/tests/control`,
|
|
declared by `Testsuite: autopkgtest` in `debian/control`) through a
|
|
native runner on a pkh testbed, and reports one verdict per test:
|
|
|
|
```
|
|
pkh test # test this tree: builds it first if needed
|
|
pkh test --mode local # on the current context, no isolation
|
|
pkh test --mode vm --image img.qcow2 # in a VM, via autopkgtest
|
|
pkh test --json --fail-on skip # CI-shaped
|
|
pkh test --list-tests # what does this package test?
|
|
```
|
|
|
|
By default the tests run in an ephemeral unshare chroot bootstrapped
|
|
from the same cached tarballs as `pkh deb`, against the binaries pkh deb
|
|
just built next to the tree (stale or missing output is rebuilt first;
|
|
`--no-build` refuses, `--debs` overrides). Pockets and PPAs resolve test
|
|
dependencies like in `pkh deb` (`-p`, `--ppa`), and `--test-name`,
|
|
`--skip-test`, `--shell-fail` and `--setup-commands` cover the day-to-day
|
|
debugging loop. Tests needing a real machine (`isolation-machine`,
|
|
`needs-reboot`) run only in `--mode vm`, which execs the installed
|
|
`autopkgtest` with its qemu runner.
|
|
|
|
Exit codes: 0 all tests passed (skips and flaky allowed), 1 at least one
|
|
test failed (or a `--fail-on` trigger), 2 runtime errors (no declared
|
|
tests, no usable testbed, ...).
|
|
|
|
## Future improvement ideas
|
|
|
|
- pull: try to fetch the correct git branch for series on Debian
|
|
- deb: asynchronous build, detachable and monitorable
|
|
- put: allow uploads to Debian or Ubuntu archives
|
|
- pull: cache Sources.gz files to improve speed
|
|
- pull: 'pkh pull' in a package tree should git pull and re-fetch orig tgz
|