vhaudiquet/pkh
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

deb: shrink records to upper, journal and log; keep metadata for every build

Browse Source 
Records carried a full chroot each, and space turned out to be the hard
constraint: five records per target multiplied by a multi-GB chroot does
not fit. The chroot is now per-run scratch — every build extracts a
fresh one from the sha-verified mmdebstrap tarball cache into its
record's chroot/ dir and removes it at the end (success, failure and
interrupt included); resume_chroot, the .pkh-ready probe and
chroot_is_usable are gone, and crashed-run leftovers are swept when the
next run of that identity starts.

What a record keeps instead: manifest.json (renamed, now with the
tarball sha it used, a kept_upper flag and a tree-only journal — the
environment phases rerun in every fresh chroot), a log/build.log full
tee of the build output (kept for EVERY build; plain runs become
metadata-only records, so keep-iff-incremental now applies to the
staged tree alone), and — for --incremental runs — tree-upper/ with its
host-files.list. The fork gate carries a base's staged tree only while
its recorded tarball sha still matches the cache and the changelog
version is unchanged; a fork candidate is any record with a staged tree.

The keep-last-5 cap now applies to staged trees: a sixth retention
degrades the oldest record to metadata-only instead of deleting it.
New `pkh deb log [-f] [<id>]` prints (or follows) a record's build log,
and `pkh deb list` shows the staged-tree state and plain-run records
too. README rewritten around the shrunk model.
This commit is contained in:
vhaudiquet
2026-09-29 14:00:34 +02:00
parent 81f0d52f85
commit dd2634abe3
7 changed files with 1287 additions and 946 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+35 -23
View File
@@ -122,44 +122,56 @@ at the oldest commit searched, which keeps the run fast on large histories.
### Incremental builds (build records)
Every `pkh deb` build runs in its own immutable **record** under
`/var/tmp/pkh/sessions/<target>/<id>/`: the bootstrapped chroot, the
installed build dependencies and the build artifacts of the staged
tree. `--incremental` forks a previous record — a full copy, the base
is never modified — so a failed run can never destroy the last-good
environment it was built from:
Every `pkh deb` build records its metadata under
`/var/tmp/pkh/sessions/<target>/<id>/`: the manifest (identity, fork
lineage, outcome, the sha of the chroot tarball it used) and a full
`log/build.log` tee of the build output. The chroot itself is per-run
scratch — extracted fresh from the sha-verified mmdebstrap tarball cache
on every run and removed at the end, plain and `--incremental` alike —
so records stay small. What `--incremental` buys is the record's
**staged tree** (`tree-upper/`): the overlay upperdir where the build's
writes land, holding the compiled objects.
`--incremental` forks a previous record — a copy, the base is never
modified — so a failed run can never destroy the last-good state it was
built from. The staged tree is carried only when the base's recorded
tarball sha still matches the cache (the toolchain did not move) and the
changelog version is unchanged; otherwise the fork builds from a fresh
tree:
```
pkh deb # fully ephemeral: nothing is kept, even on
# failure or interrupt
pkh deb # plain run: fresh chroot, fresh tree; the
# record keeps only metadata and the log
# (nothing is kept even on failure or
# interrupt)
pkh deb --incremental # fork the newest successful record of the
# target (last five usable): the chroot, build
# deps and, at the same version, the objects
# are reused; only what changed is recompiled.
# The result is always kept (success or
# failure) for the next fork
# target (among the last five carrying a staged
# tree) and keep this run's staged tree, for
# the next fork to carry
pkh deb --incremental <id>
# fork that exact record (see `pkh deb list`);
# a failed record is fine if its environment is
# a failed record is fine if its staged tree is
pkh deb list # the records of this tree: id, outcome,
# version, fork lineage, age
# version, fork lineage, staged-tree state, age
pkh deb log [-f] [<id>] # print a record's build log; -f follows a
# live build
pkh prune # garbage-collect records (7-day retention)
```
Each target (series/arch) keeps its last five records — older ones are
pruned as new ones are created, and `pkh prune` removes those untouched
for a week. `pkh deb --incremental` never forks a record built for a
different series/architecture, and a plain `pkh deb` still never
reuses anything: it builds in a throwaway record that is discarded on
success, failure and interrupt alike.
Each target keeps at most five staged trees: when a sixth is retained,
the oldest degrades to a metadata-only record (manifest and log stay),
and `pkh prune` removes records untouched for a week. `pkh deb
--incremental` never forks a record built for a different
series/architecture.
Packages whose `debian/rules` keeps an input-untracked stamp cache (the
kernels' `debian/stamps`) declare it in the quirks data
(`resume_clear`): forked builds drop the cache so the wrapper steps —
including the kernel's flavour config export — re-run over the
incremental inner builds, and a config edit is never silently ignored.
If a build died hard (OOM kill), the next fork unmounts the leftover
state first (escalating through sudo when needed).
A build killed hard (OOM) can leave its chroot behind; the next run of
that target sweeps it (unmounting and escalating through sudo when
needed), and `pkh prune` collects the rest.
### Running the DEP-8 tests
src/deb/ephemeral.rs
+176 -157
View File
@@ -13,7 +13,7 @@ use tar::Archive;
use xz2::read::XzDecoder;
// ---------------------------------------------------------------------------
// Interrupt-time chroot cleanup
// Interrupt-time cleanup
//
// On Ctrl-C, the watchdog in `crate::interrupt` runs the hook registered in
// [`EphemeralContextGuard::new_with_context`] right before exiting — the
@@ -22,34 +22,61 @@ use xz2::read::XzDecoder;
// /proc and any overlayfs mounts.
// ---------------------------------------------------------------------------
/// Interrupt-time teardown of a build tree. When `keep_tree` is set the
/// tree itself is left in place after unmounting (an `--incremental`
/// record: the interrupt is a pause, not a discard); otherwise this is
/// the historical full cleanup. For a plain `pkh deb` the `path` is the
/// whole throwaway record root, so the record is removed — the record
/// model's keep-iff-incremental policy applies to interrupts too.
/// Interrupt-time teardown of an anonymous temp chroot (no build record):
/// unmount everything under it, then remove it — a plain `rm -rf` with a
/// retry loop, best-effort. See [`sigint_cleanup_record`] for records.
fn sigint_cleanup_chroot(chroot_path: &Path) {
let is_root = unsafe { libc::geteuid() } == 0;
unmount_under_logged(chroot_path);
remove_with_retries(chroot_path, is_root);
}
/// Interrupt-time teardown of a build record: unmount everything under
/// the record root, remove the per-run state (the chroot and the overlay
/// scratch workdir) and — unless `keep_upper` is set — the staged tree
/// and its host-tree snapshot. The manifest and the build log always
/// stay: every build leaves a metadata record, the interrupt included.
///
/// Unlike [`EphemeralContextGuard::drop`], this deliberately does NOT go
/// through the context manager, the ephemeral context's driver (whose
/// `cleanup()` unmounts the tracked overlays) or the base context's command
/// builder: interrupt-time hooks must be self-contained, and those
/// machineries may be mid-mutation on the interrupted thread. Instead it
/// only reads /proc/mounts and spawns umount/rm directly.
///
/// On a full cleanup it removes the tree regardless of the build result:
/// the build was aborted, and leaving a still-mounted chroot behind is
/// exactly the leak this hook exists to prevent.
/// `cleanup()` unmounts the tracked overlays) or the base context's
/// command builder: interrupt-time hooks must be self-contained, and
/// those machineries may be mid-mutation on the interrupted thread.
/// Instead it only reads /proc/mounts and spawns umount/rm directly.
///
/// Best-effort by design: if a child process still holds a mount busy or
/// privilege escalation is unavailable, individual steps fail; failures are
/// logged (pointing at `pkh prune` for the leftovers) and never panic.
fn sigint_cleanup_chroot(chroot_path: &Path, keep_tree: bool) {
/// privilege escalation is unavailable, individual steps fail; failures
/// are logged (pointing at `pkh prune` for the leftovers) and never
/// panic.
fn sigint_cleanup_record(record_root: &Path, keep_upper: bool) {
let is_root = unsafe { libc::geteuid() } == 0;
unmount_under_logged(record_root);
// Per-run state first: the chroot holds the /proc bind mount and the
// device nodes, the workdir holds root-owned overlay scratch.
for dir in ["chroot", "tree-upper-work"] {
remove_with_retries(&record_root.join(dir), is_root);
}
if keep_upper {
log::debug!(
"Interrupted build record kept its staged tree at {} (fork it \
with `pkh deb --incremental`)",
record_root.display()
);
} else {
for path in ["tree-upper", "host-files.list"] {
remove_with_retries(&record_root.join(path), is_root);
}
}
}
// Unmount children before parents: /proc/mounts lists mounts roughly in
// creation order, so walk it in reverse
let mounts = host_mounts_under(chroot_path);
for mount_point in mounts.into_iter().rev() {
/// Unmount every mount at or under `base`, deepest first, logging the
/// failures (interrupt-time: leftover mounts are `pkh prune`'s to
/// collect).
fn unmount_under_logged(base: &Path) {
let is_root = unsafe { libc::geteuid() } == 0;
// Unmount children before parents: /proc/mounts lists mounts roughly
// in creation order, so walk it in reverse.
for mount_point in host_mounts_under(base).into_iter().rev() {
if unmount_path(&mount_point, is_root) {
log::debug!(
"Unmounted {} during interrupt cleanup",
@@ -63,19 +90,13 @@ fn sigint_cleanup_chroot(chroot_path: &Path, keep_tree: bool) {
);
}
}
if keep_tree {
log::debug!(
"Interrupted build record kept at {} (fork it with `pkh deb --incremental`)",
chroot_path.display()
);
return;
}
// Remove the chroot tree itself (tolerates a missing directory). A
// child the Ctrl+C interrupted may still be finishing its writeout —
// dpkg defers SIGINT until it reaches a safe state — so retry while rm
// reports the tree non-empty instead of leaving it half-removed.
/// Privileged `rm -rf` with a retry loop: a child the Ctrl+C interrupted
/// may still be finishing its writeout — dpkg defers SIGINT until it
/// reaches a safe state — so retry while rm reports the tree non-empty
/// instead of leaving it half-removed. A missing path succeeds at once.
fn remove_with_retries(path: &Path, is_root: bool) {
const RETRIES: usize = 10;
const RETRY_DELAY: std::time::Duration = std::time::Duration::from_millis(300);
let mut last = None;
@@ -86,7 +107,7 @@ fn sigint_cleanup_chroot(chroot_path: &Path, keep_tree: bool) {
last = Some(
privileged_command("rm", is_root)
.arg("-rf")
.arg(chroot_path)
.arg(path)
.status(),
);
if matches!(&last, Some(Ok(status)) if status.success()) {
@@ -95,22 +116,19 @@ fn sigint_cleanup_chroot(chroot_path: &Path, keep_tree: bool) {
}
match last {
Some(Ok(status)) if status.success() => {
log::debug!(
"Removed chroot {} during interrupt cleanup",
chroot_path.display()
);
log::debug!("Removed {} during interrupt cleanup", path.display());
}
Some(Ok(status)) => {
log::error!(
"Failed to remove chroot {} during interrupt cleanup \
"Failed to remove {} during interrupt cleanup \
(rm exited with {status}); run `pkh prune`",
chroot_path.display()
path.display()
);
}
Some(Err(e)) => {
log::error!(
"Failed to run rm for chroot {} during interrupt cleanup: {e}; run `pkh prune`",
chroot_path.display()
"Failed to run rm for {} during interrupt cleanup: {e}; run `pkh prune`",
path.display()
);
}
None => unreachable!("at least one rm attempt ran"),
@@ -305,10 +323,11 @@ fn unescape_mount_field(field: &str) -> String {
/// for building packages with unshare permissions.
///
/// In session mode ([`EphemeralContextGuard::new_for_session`]) the chroot
/// lives inside a [`Session`] record: a forked record's chroot is reused
/// when the caller verified it, and the keep policy is strict — the record
/// is kept exactly for `--incremental` runs (`persist_record`), discarded
/// for plain ones whatever the outcome.
/// lives inside a [`Session`] record's `chroot/` directory — extracted
/// fresh from the tarball cache on every run (nothing chroot-shaped is
/// persisted) and removed at the end of the run. The record itself is
/// always kept as metadata and log; its staged tree is retained exactly
/// for `--incremental` runs (`keep_upper`).
pub struct EphemeralContextGuard {
/// The ephemeral build context this guard created (an unshare context
/// bound to the chroot, parented on the base context). Held explicitly so
@@ -327,8 +346,9 @@ pub struct EphemeralContextGuard {
cleanup_hook: Option<CleanupHookGuard>,
/// The build record this guard works in, when any.
session: Option<Session>,
/// Whether the record is kept after the build (`pkh deb --incremental`).
persist_record: bool,
/// Whether the record's staged tree is retained after the build
/// (`pkh deb --incremental`).
keep_upper: bool,
}
impl EphemeralContextGuard {
@@ -345,7 +365,7 @@ impl EphemeralContextGuard {
base_ctx: Arc<Context>,
view: &dyn BuildView,
) -> Result<Self, Box<dyn Error>> {
Self::new_internal(None, series, arch, base_ctx, view, false, false).await
Self::new_internal(None, series, arch, base_ctx, view, false).await
}
/// Create the ephemeral build context inside `session`'s chroot.
@@ -357,31 +377,18 @@ impl EphemeralContextGuard {
/// * `arch` - Optional target architecture, as for
/// [`EphemeralContextGuard::new_with_context`]
/// * `base_ctx` - The base context to use for chroot work
/// * `reuse_chroot` - Reuse the forked record's bootstrapped chroot
/// instead of re-extracting the tarball; the caller must only set
/// this after the record's chroot passed the integrity probe
/// * `persist_record` - Keep the record after the build
/// (`pkh deb --incremental`); a plain run's record is discarded
/// whatever the outcome
/// * `keep_upper` - Retain the record's staged tree after the build
/// (`pkh deb --incremental`); a plain run's record keeps metadata
/// and log only
pub async fn new_for_session(
session: Session,
series: &str,
arch: Option<&str>,
base_ctx: Arc<Context>,
view: &dyn BuildView,
reuse_chroot: bool,
persist_record: bool,
keep_upper: bool,
) -> Result<Self, Box<dyn Error>> {
Self::new_internal(
Some(session),
series,
arch,
base_ctx,
view,
reuse_chroot,
persist_record,
)
.await
Self::new_internal(Some(session), series, arch, base_ctx, view, keep_upper).await
}
async fn new_internal(
@@ -390,8 +397,7 @@ impl EphemeralContextGuard {
arch: Option<&str>,
base_ctx: Arc<Context>,
view: &dyn BuildView,
reuse_chroot: bool,
persist_record: bool,
keep_upper: bool,
) -> Result<Self, Box<dyn Error>> {
// Save the globally-installed context so Drop can restore exactly
// this handle: concurrent builds install their own ephemeral
@@ -399,8 +405,8 @@ impl EphemeralContextGuard {
// before this guard swapped anything in.
let previous_context = context::current();
// In session mode the chroot lives inside the record root (and is
// reused when the caller says so); otherwise it is a fresh temp dir.
// In session mode the chroot lives inside the record root, as
// per-run scratch; otherwise it is a fresh temp dir.
let chroot_path = match &session {
Some(s) => {
let path = s.chroot_path();
@@ -424,22 +430,23 @@ impl EphemeralContextGuard {
// base context: the hook must be self-contained (stored path +
// direct umount/rm subprocesses) and cannot go through `base_ctx`.
// For remote or nested bases the chroot lives elsewhere, and
// leftovers stay handled by `pkh prune` as before. An
// `--incremental` record is kept on interrupt (a pause, not a
// discard); a plain run's whole record root and a plain temp chroot
// are removed.
// leftovers stay handled by `pkh prune` as before.
//
// Session mode tears the record's per-run state down (chroot and
// overlay workdir) and keeps its staged tree exactly for
// `--incremental`; a plain temp chroot is removed entirely.
let cleanup_hook = if matches!(base_ctx.config, ContextConfig::Local) {
Some(crate::interrupt::register_cleanup_hook(Box::new({
// Session mode tears the whole record root down (the
// manifest is part of the record); plain temp chroots
// only ever had the chroot dir.
let cleanup_path = match &session {
Some(s) => s.root().to_path_buf(),
None => chroot_path.clone(),
let hook: Box<dyn Fn() + Send> = match session.as_ref().map(|s| s.root().to_path_buf())
{
Some(record_root) => {
Box::new(move || sigint_cleanup_record(&record_root, keep_upper))
}
None => {
let chroot_path = chroot_path.clone();
Box::new(move || sigint_cleanup_chroot(&chroot_path))
}
};
let keep_tree = persist_record;
move || sigint_cleanup_chroot(&cleanup_path, keep_tree)
})))
Some(crate::interrupt::register_cleanup_hook(hook))
} else {
log::debug!(
"Base context is not local; skipping interrupt-time cleanup registration for {}",
@@ -448,17 +455,10 @@ impl EphemeralContextGuard {
None
};
// Download and extract the chroot tarball (or reuse the record's
// prepared chroot)
if let Err(e) = Self::download_and_extract_chroot(
series,
arch,
&chroot_path,
base_ctx.clone(),
view,
session.as_ref(),
reuse_chroot,
)
// Download and extract the chroot tarball — always fresh; the
// record's chroot directory is per-run scratch.
if let Err(e) =
Self::download_and_extract_chroot(series, arch, &chroot_path, base_ctx.clone(), view)
.await
{
// On a Ctrl+C the interrupt watchdog owns the tree: keep the
@@ -476,6 +476,17 @@ impl EphemeralContextGuard {
return Err(e);
}
// Record the tarball digest this environment came from: the fork
// gate of a later `--incremental` run compares it against the
// cache to decide whether the record's staged tree is still
// reusable.
if let Some(s) = &session
&& let Ok(tarball) = chroot_tarball_path(series, arch)
&& let Ok(sha) = session::sha256_file(&tarball)
{
s.set_tarball_sha256(&sha);
}
// Switch to an ephemeral context to build the package in the chroot.
// The parent is the base context itself (the one that bootstrapped
// the chroot), wired through `with_parent` instead of a config-name
@@ -502,7 +513,7 @@ impl EphemeralContextGuard {
base_ctx,
cleanup_hook,
session,
persist_record,
keep_upper,
})
}
@@ -522,25 +533,10 @@ impl EphemeralContextGuard {
chroot_path: &PathBuf,
ctx: Arc<context::Context>,
view: &dyn BuildView,
session: Option<&Session>,
reuse_chroot: bool,
) -> Result<(), Box<dyn Error>> {
// Record fork: the base's chroot passed the caller's checks —
// skip the tarball extraction entirely, the environment (apt
// lists, installed build-deps) is still in there.
if let Some(s) = session {
if reuse_chroot && s.chroot_is_usable() {
log::info!(
"Reusing the prepared chroot of record {} (bootstrap skipped)",
s.manifest().id
);
Self::finalize_chroot(chroot_path, ctx, view)?;
return Ok(());
}
// Not reusing (fresh record or failed probe): never extract
// over stale content.
// The chroot is per-run scratch: nothing is reused, so never
// extract over stale content.
clear_dir(chroot_path)?;
}
// Clone ctx for use in create_device_nodes after download_chroot_tarball consumes it
let ctx_for_devices = ctx.clone();
@@ -619,33 +615,13 @@ impl EphemeralContextGuard {
// Create device nodes in the chroot and bind-mount /proc
Self::finalize_chroot(chroot_path, ctx_for_devices, view)?;
// Record the bootstrap in the record: marker file (the integrity
// probe) and the tarball the chroot came from.
if let Some(s) = session {
std::fs::write(chroot_path.join(".pkh-ready"), b"")?;
let metadata = fs::metadata(&tarball_path)?;
let tarball_sha256 = session::sha256_file(&tarball_path)
.map_err(|e| format!("cannot hash the chroot tarball: {e}"))?;
s.set_chroot(session::ChrootInfo {
tarball: tarball_filename,
tarball_size: metadata.len(),
tarball_mtime_secs: metadata
.modified()
.ok()
.and_then(|t| t.duration_since(std::time::UNIX_EPOCH).ok())
.map(|d| d.as_secs())
.unwrap_or(0),
tarball_sha256,
ready: true,
});
log::debug!("Record {} chroot is ready", s.manifest().id);
}
// The bootstrap's tarball digest is recorded by the caller (the
// record's fork gate compares it against the cache later).
Ok(())
}
/// Device nodes and the /proc bind mount for a (possibly reused)
/// chroot: idempotent finalization, safe to run on every attempt.
/// Device nodes and the /proc bind mount for a fresh chroot:
/// idempotent finalization, safe to run on every run.
fn finalize_chroot(
chroot_path: &Path,
ctx: Arc<context::Context>,
@@ -660,8 +636,8 @@ impl EphemeralContextGuard {
// Bind mount /proc from host into chroot (before entering unshare namespace)
// This allows /proc to work in containers where mounting inside unshare fails.
// A reused chroot may still carry the mount from the previous
// attempt: only mount when it is not already there.
// A chroot reused within the same run may still carry the mount:
// only mount when it is not already there.
if is_mountpoint(&chroot_path.join("proc")) {
log::debug!("/proc is already bind-mounted in the chroot");
} else {
@@ -908,6 +884,11 @@ impl Drop for EphemeralContextGuard {
// drained, forgetting is a harmless no-op).
if crate::interrupt::interrupted() {
context::manager().set_current_ephemeral(self.previous_context.clone());
// The record keeps its metadata: mark the aborted build as
// failed before the hook tears the per-run state down.
if let Some(session) = &self.session {
session.set_outcome(session::OUTCOME_FAILED);
}
if let Some(hook) = self.cleanup_hook.take() {
std::mem::forget(hook);
}
@@ -942,11 +923,11 @@ impl Drop for EphemeralContextGuard {
// touches it.
context::manager().set_current_ephemeral(self.previous_context.clone());
// Session mode: the tree is a build record. The /proc bind mount
// must come down in every case (a kept record has to be
// unmount-clean for the next fork); the record itself follows the
// keep policy — kept exactly for `--incremental` runs, discarded
// for plain ones whatever the outcome.
// Session mode: the record's per-run chroot comes down in every
// case (nothing chroot-shaped is persisted — success, failure and
// interrupt all end the run), then the record is retained:
// metadata and log always, the staged tree exactly for
// `--incremental` runs.
if let Some(session) = &self.session {
let proc_path = self.chroot_path.join("proc");
let is_root = crate::utils::root::is_root().unwrap_or(false);
@@ -960,7 +941,8 @@ impl Drop for EphemeralContextGuard {
.status()
};
session.dispose_after_build(self.persist_record, self.build_succeeded);
session.remove_run_state();
session.retain_upper(self.keep_upper, self.build_succeeded);
return;
}
@@ -1058,7 +1040,7 @@ mod chroot_cleanup_tests {
fn sigint_cleanup_of_missing_chroot_is_a_noop() {
let dir = tempfile::tempdir().unwrap();
let missing = dir.path().join("no-such-chroot");
sigint_cleanup_chroot(&missing, false);
sigint_cleanup_chroot(&missing);
assert!(!missing.exists());
}
@@ -1081,24 +1063,61 @@ mod chroot_cleanup_tests {
std::fs::create_dir_all(chroot.join("rootfs")).unwrap();
std::fs::write(chroot.join("rootfs").join("file.txt"), "data").unwrap();
sigint_cleanup_chroot(&chroot, false);
sigint_cleanup_chroot(&chroot);
assert!(!chroot.exists());
}
/// The interrupt teardown of a kept tree (an `--incremental` record)
/// unmounts and leaves the tree in place — the pause, not the discard.
/// The record interrupt teardown removes the per-run state and keeps
/// the manifest and log; an `--incremental` record keeps its staged
/// tree, a plain one loses it.
#[test]
fn sigint_cleanup_keeps_the_tree_when_asked() {
fn sigint_record_teardown_keeps_metadata_and_drops_the_chroot() {
let is_root = unsafe { libc::geteuid() } == 0;
if !is_root
&& !privileged_command("true", false)
.status()
.is_ok_and(|s| s.success())
{
return;
}
let dir = tempfile::tempdir().unwrap();
let record_root = dir.path().join("record");
std::fs::create_dir_all(record_root.join("chroot/rootfs")).unwrap();
std::fs::write(record_root.join("session.json"), "{}").unwrap();
let incremental = dir.path().join("incremental");
let plain = dir.path().join("plain");
for root in [&incremental, &plain] {
std::fs::create_dir_all(root.join("chroot/rootfs")).unwrap();
std::fs::create_dir_all(root.join("tree-upper/src")).unwrap();
std::fs::create_dir_all(root.join("log")).unwrap();
std::fs::write(root.join("manifest.json"), "{}").unwrap();
std::fs::write(root.join("log/build.log"), "output\n").unwrap();
std::fs::write(root.join("host-files.list"), "f\t1\t1\ta\n").unwrap();
}
sigint_cleanup_chroot(&record_root, true);
sigint_cleanup_record(&incremental, true);
sigint_cleanup_record(&plain, false);
assert!(record_root.join("chroot/rootfs").exists());
assert!(record_root.join("session.json").is_file());
for root in [&incremental, &plain] {
assert!(!root.join("chroot").exists(), "the chroot is per-run");
assert!(root.join("manifest.json").is_file(), "metadata stays");
assert!(root.join("log/build.log").is_file(), "the log stays");
}
assert!(
incremental.join("tree-upper/src").exists(),
"an interrupted incremental run keeps its staged tree"
);
assert!(
incremental.join("host-files.list").is_file(),
"the snapshot stays with the staged tree"
);
assert!(
!plain.join("tree-upper").exists(),
"an interrupted plain run drops its staged tree"
);
assert!(
!plain.join("host-files.list").exists(),
"the snapshot goes with the staged tree"
);
}
/// clear_dir removes content a plain `rm -rf` cannot: a root-owned
src/deb/local.rs
+25 -57
View File
@@ -27,35 +27,12 @@ fn cap<'a>(
cmd
}
/// The session journal stamps this build works with: the input stamps of
/// the patch and build-dependency phases, computed from the host tree and
/// the resolved selectors.
struct JournalStamps {
patches: Option<String>,
control: Option<String>,
}
/// Compute the journal stamps for a session build; `None` without one.
fn journal_stamps(
session: &session::Session,
arch: &str,
series: &str,
pocket: Option<&str>,
cross: bool,
ppa: &[String],
inject_packages: &[String],
) -> JournalStamps {
/// The input stamp of the patch phase, computed from the host tree: the
/// only tree phase the record journal tracks (the environment phases
/// rerun in every fresh chroot and are not recorded).
fn patch_stamp(session: &session::Session) -> Option<String> {
let host_tree = PathBuf::from(session.manifest().host_tree);
let selectors = format!(
"{arch}\n{cross}\n{series}\n{}\n{}\n{}",
pocket.unwrap_or(""),
ppa.join(","),
inject_packages.join(",")
);
JournalStamps {
patches: session::patches_stamp(&host_tree),
control: session::control_stamp(&host_tree, &selectors),
}
session::patches_stamp(&host_tree)
}
#[allow(clippy::too_many_arguments)]
@@ -74,13 +51,12 @@ pub async fn build(
view: &dyn BuildView,
jobs: Option<usize>,
session: Option<&session::Session>,
resume: bool,
build_resume: bool,
) -> Result<Vec<PathBuf>, Box<dyn Error>> {
let sink: Option<Arc<dyn LineSink>> = view.sink();
// Journal stamps for a session build; recording is a no-op without one.
let stamps =
session.map(|s| journal_stamps(s, arch, series, pocket, cross, ppa, inject_packages));
// Journal stamp for a record build; recording is a no-op without one.
let stamps = session.and_then(patch_stamp);
// Environment
let mut env = HashMap::<String, String>::new();
@@ -245,9 +221,6 @@ pub async fn build(
or re-run with RUST_LOG=debug for more details."
.into());
}
if let Some(s) = session {
s.record_phase("apt_update", None);
}
// Install essential packages
log::debug!("Installing essential packages for local build...");
@@ -272,9 +245,6 @@ pub async fn build(
if !status.success() {
return Err("Could not install essential packages for the build".into());
}
if let Some(s) = session {
s.record_phase("essentials", None);
}
// The package directory was resolved by the caller (the staged copy of
// the tree the user pointed at, or the name-pattern search fallback)
@@ -294,10 +264,11 @@ pub async fn build(
}
// Apply quilt patches if the package provides a patch series. On a
// resumed session the previous attempt's patches may still be applied:
// pop them first when the patch tree changed since.
let patch_stamp = stamps.as_ref().and_then(|s| s.patches.clone());
let pop_patches = resume && {
// build resuming over a carried upperdir the previous attempt's
// patches may still be applied: pop them first when the patch tree
// changed since.
let patch_stamp = stamps.clone();
let pop_patches = build_resume && {
let previous = session
.and_then(|s| s.manifest().phases.get("patches").map(|p| p.stamp.clone()))
.unwrap_or(None);
@@ -333,20 +304,17 @@ pub async fn build(
view,
&sink,
)?;
if let Some(s) = session {
s.record_phase(
"build_deps",
stamps.as_ref().and_then(|st| st.control.clone()),
);
}
// (The build-dependency phase reruns in every fresh chroot: nothing
// to journal.)
// A resumed build continues in the previous attempt's tree. Packages
// whose debian/rules keeps an input-untracked stamp cache (quirk:
// `resume_clear`, e.g. the kernels' `debian/stamps`, whose flavour
// config rule exports .config with no config prerequisite) would build
// from the previous attempt's configuration: drop the listed caches so
// the wrapper steps re-run over the incremental inner builds.
if resume {
// A build resuming over a carried upperdir continues in the previous
// attempt's tree. Packages whose `debian/rules` keeps an
// input-untracked stamp cache (quirk: `resume_clear`, e.g. the
// kernels' `debian/stamps`, whose flavour config rule exports .config
// with no config prerequisite) would build from the previous
// attempt's configuration: drop the listed caches so the wrapper
// steps re-run over the incremental inner builds.
if build_resume {
for rel in crate::quirks::get_resume_clear_paths(package, series) {
let path = Path::new(package_dir_str).join(&rel);
if !ctx.exists(&path).unwrap_or(false) {
@@ -383,10 +351,10 @@ pub async fn build(
s.record_phase("build", None);
}
// On a resumed session the previous attempt's debian/files would make
// On a resumed build the previous attempt's debian/files would make
// the artifact collection below surface its outputs: only the new
// attempt's registration may remain.
if resume {
if build_resume {
let _ = ctx
.command("rm")
.arg("-f")
src/deb/mod.rs
+195 -106
View File
@@ -119,36 +119,87 @@ pub struct DebBuildOptions<'a> {
/// Explicit build context; defaults to the current context.
pub ctx: Option<Arc<Context>>,
/// Incremental request (`pkh deb --incremental [<id>]`): which
/// record to fork as the base of this build, and whether this
/// build's own record is kept. Defaults to a plain throwaway run.
/// record to fork as the base of this build's staged tree, and
/// whether this build's own staged tree is retained. Defaults to a
/// plain run.
pub incremental: Incremental,
/// Where build events (phases, progress, outcome) are reported.
pub view: &'a dyn BuildView,
}
/// The `--incremental` request of a `pkh deb` build: which record the
/// build forks its environment from, and whether its own record is
/// kept afterwards.
/// build forks its staged tree from, and whether its own staged tree is
/// retained afterwards (the record's metadata and log are kept for
/// every build either way).
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub enum Incremental {
/// A plain run (`pkh deb` without `--incremental`): build from
/// scratch in a throwaway record, discarded on success, failure
/// and interrupt alike.
/// scratch in a record whose staged tree is dropped at the end —
/// metadata and log only remain.
#[default]
Off,
/// `--incremental`: fork the auto-selected base — the newest
/// successful record of this identity among the last five usable —
/// and always keep the new record. With no usable candidate the
/// build runs from scratch and its record becomes the identity's
/// first base.
/// successful record of this identity among the last five carrying
/// a staged tree — and retain the new record's staged tree. With no
/// candidate the build runs from scratch and its record becomes
/// the identity's first base.
Auto,
/// `--incremental <id>`: fork that exact record (unique id or
/// prefix, restricted to the records of this host tree; a FAILED
/// record is a valid base — its environment may be fine, and the
/// user pinned it deliberately) and always keep the new record.
/// record is a valid base — the user pinned it deliberately) and
/// retain the new record's staged tree.
Id(String),
}
/// A view delegating every event to `inner` while teeing all captured
/// subprocess lines into a build record's log through [`RecordLogSink`]:
/// everything downstream (the chroot bootstrap, the build phases) asks
/// the view for its sink, so one wrapper tees the whole run.
struct RecordLogView<'a> {
/// The caller's view.
inner: &'a dyn BuildView,
/// The record log sink, forwarding to the inner view's sink too.
log: Arc<session::RecordLogSink>,
}
impl BuildView for RecordLogView<'_> {
fn target(&self, target: crate::report::BuildTarget) {
self.inner.target(target);
}
fn phase(&self, name: &str, classifier: Box<dyn Classifier>) {
self.inner.phase(name, classifier);
}
fn message(&self, text: &str) {
self.inner.message(text);
}
fn progress(&self, label: &str, pos: usize, total: usize) {
self.inner.progress(label, pos, total);
}
fn sink(&self) -> Option<Arc<dyn crate::context::LineSink>> {
Some(self.log.clone())
}
fn finish_success(&self, artifacts: &[PathBuf]) {
self.inner.finish_success(artifacts);
}
fn finish_failure(&self) {
self.inner.finish_failure();
}
fn suspend(&self) {
self.inner.suspend();
}
fn is_enabled(&self) -> bool {
self.inner.is_enabled()
}
}
impl Default for DebBuildOptions<'_> {
fn default() -> Self {
static QUIET: crate::report::Quiet = crate::report::Quiet;
@@ -281,7 +332,7 @@ async fn build_binary_package_impl(
cross,
};
let mut persist_record = false;
let mut keep_upper = false;
let session = if session_enabled {
match resolve_record(
incremental,
@@ -291,8 +342,8 @@ async fn build_binary_package_impl(
series,
chroot_arch,
) {
Ok((record, persist)) => {
persist_record = persist;
Ok((record, keep)) => {
keep_upper = keep;
Some(record)
}
Err(e) => {
@@ -303,9 +354,26 @@ async fn build_binary_package_impl(
} else {
None
};
// Only a forked record carries a bootstrapped chroot at this point;
// fresh ones (plain runs and first `--incremental` runs) extract it.
let resume_chroot = session.as_ref().is_some_and(|s| s.chroot_is_usable());
// Tee every captured build line into the record's log: the wrapper
// view hands the combined sink to everything downstream (the chroot
// bootstrap, the build phases), so the log is a full tee whatever
// renders the output. A record-less build (non-local base context)
// keeps the plain view.
let record_log: Option<Result<Arc<session::RecordLogSink>, std::io::Error>> = session
.as_ref()
.map(|s| session::RecordLogSink::new(s, opts.view.sink()).map(Arc::new));
let view: &dyn BuildView = match record_log {
Some(Ok(log)) => &RecordLogView {
inner: opts.view,
log,
},
Some(Err(e)) => {
log::warn!("Cannot open the build record's log: {e}");
opts.view
}
None => opts.view,
};
let mut guard = if let Some(s) = session.clone() {
Some(
@@ -315,8 +383,7 @@ async fn build_binary_package_impl(
chroot_arch,
base_ctx.clone(),
view,
resume_chroot,
persist_record,
keep_upper,
)
.await?,
)
@@ -454,7 +521,7 @@ async fn build_binary_package_impl(
view,
*jobs,
session.as_ref(),
resume_chroot,
build_resume,
)
.await?
}
@@ -486,19 +553,17 @@ async fn build_binary_package_impl(
// the chroot) inherit the terminal, so they must not fight the widget.
view.suspend();
// Mark the build as successful for the guard's drop, which disposes of
// the record per the keep policy: an `--incremental` record is kept
// with its outcome recorded, a plain run's throwaway record is
// always discarded.
// Mark the build as successful for the guard's drop, which tears the
// per-run chroot down and retains the record: metadata and log for
// every build, the staged tree exactly for `--incremental` runs.
if result.is_ok()
&& let Some(g) = &mut guard
{
g.mark_build_successful();
}
// A failed `--incremental` run keeps its record for a later fork:
// record the outcome (discarded plain runs need none).
// Every record's outcome is written — plain runs leave a metadata
// record too.
if result.is_err()
&& persist_record
&& let Some(s) = &session
{
s.set_outcome(session::OUTCOME_FAILED);
@@ -518,10 +583,9 @@ fn resolve_record(
series: &str,
chroot_arch: Option<&str>,
) -> Result<(session::Session, bool), Box<dyn Error>> {
let expected = ephemeral::chroot_tarball_path(series, chroot_arch).ok();
resolve_record_in(
&session::sessions_root(),
expected.as_deref(),
&session::cache_tarball_sha256(series, chroot_arch),
incremental,
host_tree,
identity,
@@ -530,30 +594,22 @@ fn resolve_record(
}
/// The testable core of [`resolve_record`] over an explicit sessions root
/// and bootstrap tarball.
/// and cached-tarball digest (the fork gate's reference).
///
/// Every local build runs in its own record: [`Incremental::Off`] gets a
/// brand-new throwaway one, [`Incremental::Auto`] and
/// [`Incremental::Id`] fork the selected base. The returned flag says
/// whether the record is kept after the build — exactly the
/// `--incremental` runs keep theirs.
///
/// A base must be usable: its chroot passes the integrity probe and was
/// bootstrapped from the tarball this build would use (`expected_tarball`
/// is `None` when the path cannot be resolved — no record is usable
/// then, which degrades an `--incremental` run to a fresh build).
/// brand-new one, [`Incremental::Auto`] and [`Incremental::Id`] fork the
/// selected base. The returned flag says whether the record's staged
/// tree is retained after the build — exactly the `--incremental` runs
/// retain theirs. Metadata and the log are kept for every build either
/// way.
fn resolve_record_in(
records_root: &Path,
expected_tarball: Option<&Path>,
cache_tarball_sha256: &str,
incremental: &Incremental,
host_tree: &Path,
identity: &session::SessionIdentity,
version: &str,
) -> Result<(session::Session, bool), Box<dyn Error>> {
let usable = |s: &session::Session| {
s.chroot_is_usable()
&& expected_tarball.is_some_and(|t| session::tarball_matches_path(s, t))
};
match incremental {
Incremental::Off => Ok((
session::new_record_in(records_root, host_tree, identity, version)?,
@@ -561,14 +617,12 @@ fn resolve_record_in(
)),
Incremental::Auto => {
// A record locked by a live build cannot be a base: its
// chroot may be mid-mutation (apt always runs, deps land
// inside it), and the fork would copy that state.
// staged tree may be mid-mutation by the running build.
let candidates: Vec<session::Session> =
session::identity_records_in(records_root, identity)
session::fork_candidates_in(records_root, identity)
.into_iter()
.filter(|s| usable(s))
.filter(|s| !session::lock_is_held(&s.lock_path()).unwrap_or(true))
.take(session::KEEP_RECORDS)
.take(session::KEEP_UPPERS)
.collect();
for base in session::rank_base_candidates(candidates) {
let why = if base.manifest().last_outcome == session::OUTCOME_SUCCESS {
@@ -576,7 +630,14 @@ fn resolve_record_in(
} else {
"no successful record among the last usable ones; the newest record"
};
match session::fork_record_in(records_root, &base, host_tree, identity, version) {
match session::fork_record_in(
records_root,
&base,
host_tree,
identity,
version,
cache_tarball_sha256,
) {
Ok(record) => {
log::info!(
"Base of the incremental build: record {} ({why})",
@@ -591,8 +652,8 @@ fn resolve_record_in(
}
}
log::info!(
"No usable record of {identity} to fork; building from scratch — \
this run's record becomes the target's first base"
"No record of {identity} carrying a staged tree to fork; building \
from scratch — this run's record becomes the target's first base"
);
Ok((
session::new_record_in(records_root, host_tree, identity, version)?,
@@ -603,11 +664,10 @@ fn resolve_record_in(
let mut matches: Vec<session::Session> = session::by_id_prefix_in(records_root, id)
.into_iter()
.filter(|s| s.manifest().host_tree == host_tree.to_string_lossy())
.filter(|s| usable(s))
.collect();
match matches.len() {
0 => Err(format!(
"no usable build record matches id '{id}' for this tree \
"no build record matches id '{id}' for this tree \
(see `pkh deb list`)"
)
.into()),
@@ -618,7 +678,14 @@ fn resolve_record_in(
base.manifest().id
);
Ok((
session::fork_record_in(records_root, &base, host_tree, identity, version)?,
session::fork_record_in(
records_root,
&base,
host_tree,
identity,
version,
cache_tarball_sha256,
)?,
true,
))
}
@@ -1354,30 +1421,37 @@ mod tests {
// -------------------------------------------------------------------
use crate::deb::session::record_fixtures::{
record_with_environment, sessions_fixture, test_identity,
metadata_record, record_with_upper, sessions_fixture, test_identity,
};
use crate::deb::session::{self, SessionIdentity};
/// A plain run resolves to a brand-new, lineage-free record that the
/// caller will discard — including when no record exists at all yet.
/// The sha of the fixture's cached tarball (the fork gate's
/// reference).
fn cache_sha(tarball: &std::path::Path) -> String {
session::sha256_file(tarball).unwrap()
}
/// A plain run resolves to a brand-new, lineage-free record whose
/// staged tree will be dropped — including when no record exists at
/// all yet.
#[test]
fn resolve_plain_run_builds_a_fresh_discardable_record() {
let (root, tarball) = sessions_fixture();
let identity = test_identity();
let host_tree = root.path().join("tree");
let (record, persist) = resolve_record_in(
let (record, keep_upper) = resolve_record_in(
root.path(),
Some(&tarball),
&cache_sha(&tarball),
&Incremental::Off,
&host_tree,
&identity,
"1.0",
)
.expect("a plain run always resolves to a fresh record");
assert!(!persist, "a plain run discards its record");
assert!(!keep_upper, "a plain run drops its staged tree");
assert_eq!(record.manifest().base, None);
assert!(!record.chroot_is_usable());
assert!(!record.has_upper());
}
/// The user scenario, end to end over the record layout: record A
@@ -1388,7 +1462,7 @@ mod tests {
let (root, tarball) = sessions_fixture();
let identity = test_identity();
let host_tree = root.path().join("tree");
let a = record_with_environment(
let a = record_with_upper(
root.path(),
&tarball,
&host_tree,
@@ -1397,56 +1471,67 @@ mod tests {
Some(session::OUTCOME_SUCCESS),
);
let a_id = a.manifest().id;
let b = session::fork_record_in(root.path(), &a, &host_tree, &identity, "1.0").unwrap();
let b = session::fork_record_in(
root.path(),
&a,
&host_tree,
&identity,
"1.0",
&cache_sha(&tarball),
)
.unwrap();
let b_id = b.manifest().id;
b.set_outcome(session::OUTCOME_FAILED);
drop((a, b));
let (record, persist) = resolve_record_in(
let (record, keep_upper) = resolve_record_in(
root.path(),
Some(&tarball),
&cache_sha(&tarball),
&Incremental::Auto,
&host_tree,
&identity,
"1.0",
)
.expect("the auto selection must fork a usable record");
assert!(persist);
.expect("the auto selection must fork a candidate record");
assert!(keep_upper);
assert_eq!(record.manifest().base.as_deref(), Some(a_id.as_str()));
assert_ne!(record.manifest().id, b_id);
// The successful base's gate passes: its staged tree is carried.
assert!(record.has_upper());
}
/// A first `--incremental` run with no candidate builds completely
/// fresh and persists its record — it becomes the identity's first
/// base.
/// fresh and retains its staged tree — it becomes the identity's
/// first base.
#[test]
fn resolve_auto_without_candidates_builds_fresh_and_persists() {
fn resolve_auto_without_candidates_builds_fresh_and_retains() {
let (root, tarball) = sessions_fixture();
let identity = test_identity();
let host_tree = root.path().join("tree");
let (record, persist) = resolve_record_in(
let (record, keep_upper) = resolve_record_in(
root.path(),
Some(&tarball),
&cache_sha(&tarball),
&Incremental::Auto,
&host_tree,
&identity,
"1.0",
)
.expect("no candidates must degrade to a fresh build");
assert!(persist, "the first record becomes a base");
assert!(keep_upper, "the first record becomes a base");
assert_eq!(record.manifest().base, None);
}
/// Non-usable records never make it into the selection: a newer
/// successful record with a broken chroot loses to an older usable
/// failed one.
/// Records without a staged tree (plain runs, metadata-only ones)
/// never make it into the selection: a newer successful
/// metadata-only record loses to an older failed one carrying a
/// staged tree.
#[test]
fn resolve_auto_skips_non_usable_records() {
fn resolve_auto_skips_records_without_a_staged_tree() {
let (root, tarball) = sessions_fixture();
let identity = test_identity();
let host_tree = root.path().join("tree");
let usable = record_with_environment(
let carrier = record_with_upper(
root.path(),
&tarball,
&host_tree,
@@ -1454,33 +1539,37 @@ mod tests {
"1.0",
Some(session::OUTCOME_FAILED),
);
let usable_id = usable.manifest().id;
drop(usable);
// Newer, successful, but its chroot never finished bootstrapping.
let broken = session::new_record_in(root.path(), &host_tree, &identity, "1.0").unwrap();
broken.set_outcome(session::OUTCOME_SUCCESS);
drop(broken);
let carrier_id = carrier.manifest().id;
drop(carrier);
// Newer and successful, but a plain run: no staged tree.
drop(metadata_record(
root.path(),
&host_tree,
&identity,
"1.0",
session::OUTCOME_SUCCESS,
));
let (record, _) = resolve_record_in(
root.path(),
Some(&tarball),
&cache_sha(&tarball),
&Incremental::Auto,
&host_tree,
&identity,
"1.0",
)
.expect("the usable record is still a base");
assert_eq!(record.manifest().base.as_deref(), Some(usable_id.as_str()));
.expect("the record carrying a staged tree is still the base");
assert_eq!(record.manifest().base.as_deref(), Some(carrier_id.as_str()));
}
/// `--incremental <id>` may pin a FAILED record — its environment may
/// be fine and the user chose it deliberately.
/// `--incremental <id>` may pin a FAILED record — the user chose it
/// deliberately and its staged tree may be perfectly fine.
#[test]
fn resolve_pinned_id_forks_a_failed_record() {
let (root, tarball) = sessions_fixture();
let identity = test_identity();
let host_tree = root.path().join("tree");
let failed = record_with_environment(
let failed = record_with_upper(
root.path(),
&tarball,
&host_tree,
@@ -1491,17 +1580,18 @@ mod tests {
let failed_id = failed.manifest().id;
drop(failed);
let (record, persist) = resolve_record_in(
let (record, keep_upper) = resolve_record_in(
root.path(),
Some(&tarball),
&cache_sha(&tarball),
&Incremental::Id(failed_id.clone()),
&host_tree,
&identity,
"1.0",
)
.expect("a pinned failed record is a valid base");
assert!(persist);
assert!(keep_upper);
assert_eq!(record.manifest().base.as_deref(), Some(failed_id.as_str()));
assert!(record.has_upper(), "the gate passes: sha and version match");
}
/// A pinned id is restricted to the records of this host tree: an id
@@ -1512,7 +1602,7 @@ mod tests {
let identity = test_identity();
let host_tree = root.path().join("tree");
let other_tree = root.path().join("elsewhere");
let foreign = record_with_environment(
let foreign = record_with_upper(
root.path(),
&tarball,
&other_tree,
@@ -1525,7 +1615,7 @@ mod tests {
let err = resolve_record_in(
root.path(),
Some(&tarball),
&cache_sha(&tarball),
&Incremental::Id(foreign_id),
&host_tree,
&identity,
@@ -1533,8 +1623,7 @@ mod tests {
)
.expect_err("records of other trees must not be forked by id");
assert!(
err.to_string()
.contains("no usable build record matches id"),
err.to_string().contains("no build record matches id"),
"wrong error: {err}"
);
}
@@ -1545,7 +1634,7 @@ mod tests {
let (root, tarball) = sessions_fixture();
let identity = test_identity();
let host_tree = root.path().join("tree");
drop(record_with_environment(
drop(record_with_upper(
root.path(),
&tarball,
&host_tree,
@@ -1553,7 +1642,7 @@ mod tests {
"1.0",
None,
));
drop(record_with_environment(
drop(record_with_upper(
root.path(),
&tarball,
&host_tree,
@@ -1567,7 +1656,7 @@ mod tests {
let err = resolve_record_in(
root.path(),
Some(&tarball),
&cache_sha(&tarball),
&Incremental::Id(prefix),
&host_tree,
&identity,
@@ -1591,8 +1680,8 @@ mod tests {
.collect()
}
/// A pinned record of another identity is refused: its environment
/// was bootstrapped for a different target.
/// A pinned record of another identity is refused: its staged tree
/// was built for a different target.
#[test]
fn resolve_pinned_id_refuses_foreign_identities() {
let (root, tarball) = sessions_fixture();
@@ -1602,7 +1691,7 @@ mod tests {
..identity.clone()
};
let host_tree = root.path().join("tree");
let foreign = record_with_environment(
let foreign = record_with_upper(
root.path(),
&tarball,
&host_tree,
@@ -1615,7 +1704,7 @@ mod tests {
let err = resolve_record_in(
root.path(),
Some(&tarball),
&cache_sha(&tarball),
&Incremental::Id(foreign_id),
&host_tree,
&identity,
src/deb/session.rs
+791 -574
View File
File diff suppressed because it is too large Load Diff
src/main.rs
+38 -3
View File
@@ -371,15 +371,22 @@ fn main() {
.num_args(0..=1)
.require_equals(false)
.default_missing_value("")
.help("Incremental build: fork a previous build record of this target and keep the result")
.long_help("Incremental build: fork a previous build record of this target — its prepared chroot and installed build dependencies are copied into a new record, and at the same changelog version its build artifacts too (make recompiles only what changed).\nWithout a value, the newest successful record of the target (among the last five usable) is the base; with an ID (as shown by 'pkh deb list'), that exact record — a failed one is fine if its environment is.\nEvery --incremental run keeps its record, success or failure, so the next one can fork it; a plain 'pkh deb' is fully ephemeral (nothing is kept, even on failure or interrupt)."),
.help("Incremental build: fork a previous build record of this target and keep its staged tree")
.long_help("Incremental build: fork a previous build record of this target — at an unchanged changelog version and while the chroot tarball is unchanged, the base's staged tree is copied into a new record, so make recompiles only what changed.\nWithout a value, the newest successful record of the target (among the last five carrying a staged tree) is the base; with an ID (as shown by 'pkh deb list'), that exact record — a failed one is fine if its staged tree is.\nEvery --incremental run keeps its staged tree, success or failure, so the next one can fork it; a plain 'pkh deb' keeps only metadata and its build log (the chroot is per-run and always discarded)."),
)
.arg(arg!(--verbose "Show raw tool output instead of the live build view").required(false)
.long_help("Show raw tool output instead of the live build view.\nAlso implied by RUST_LOG=debug for pkh's own logs."))
.subcommand(
Command::new("list")
.about("List the build records recorded from this tree")
.long_about("List the build records recorded from this tree, with their outcome, version, fork lineage and the id to pass to 'pkh deb --incremental <id>'.\nRead-only: never builds or removes anything (only the last five records per target are kept, 'pkh prune' removes the ones older than a week)."),
.long_about("List the build records recorded from this tree (plain runs too, as metadata records), with their outcome, version, fork lineage, whether their staged tree is kept, and the id to pass to 'pkh deb --incremental <id>'.\nRead-only: never builds or removes anything (at most five staged trees per target are kept, 'pkh prune' removes records older than a week)."),
)
.subcommand(
Command::new("log")
.about("Print a recorded build's log")
.long_about("Print the build log of a recorded build: the full tee of its output.\nWithout an ID, the newest record of this tree; with an ID (as shown by 'pkh deb list'), that exact record.\n--follow keeps reading a live build's log as it is written and exits once the build's record reaches a terminal state.")
.arg(arg!([id] "Record id (default: the newest record of this tree)").required(false))
.arg(arg!(-f --follow "Keep reading the log as the build writes it")),
)
)
.subcommand(
@@ -896,6 +903,34 @@ fn main() {
return;
}
// `pkh deb log`: print a recorded build's log (optionally
// following a live build); read-only too.
if let Some(("log", log_matches)) = sub_matches.subcommand() {
let tree = cwd.canonicalize().unwrap_or(cwd.clone());
let id = log_matches.get_one::<String>("id").map(String::as_str);
let follow = log_matches.get_flag("follow");
let record = match pkh::deb::session::record_for_log(
&pkh::deb::session::sessions_root(),
&tree,
id,
) {
Ok(record) => record,
Err(e) => {
error!("{e}");
std::process::exit(1);
}
};
let mut out = std::io::stdout();
if let Err(e) = pkh::deb::session::print_record_log(&record, follow, &mut out) {
error!(
"cannot print the build log of {}: {e}",
record.root().display()
);
std::process::exit(1);
}
return;
}
// Ctrl+C during the build must say what happened and release the
// ephemeral chroot instead of dying on the default disposition.
// The live view (when enabled) registers its own reporter on top
src/prune.rs
+6 -5
View File
@@ -98,7 +98,8 @@ enum Artifact {
Tarball(PathBuf),
/// A build log file under `<cache>/logs`.
LogFile(PathBuf),
/// A build record root (chroot + journal), one per build.
/// A build record root (manifest + log, and the staged tree of
/// `--incremental` runs), one per build.
Session(PathBuf),
/// A `pkh test` run directory (report artifacts).
TestRun(PathBuf),
@@ -184,7 +185,7 @@ fn discover_sessions(sessions_dir: &Path, all: bool) -> Vec<Artifact> {
} else if is_stale_path(&path, all) {
// A record root only ever holds directories; a plain
// file inside a slug directory is a leftover of the old
// flat layout (its session.json / host-tree snapshot).
// flat layout (its manifest / host-tree snapshot).
artifacts.push(Artifact::Session(path));
}
}
@@ -240,7 +241,7 @@ fn is_prunable_session(path: &Path, all: bool) -> bool {
if all {
return true;
}
let last_used = fs::read_to_string(path.join("session.json"))
let last_used = fs::read_to_string(path.join("manifest.json"))
.ok()
.and_then(|content| serde_json::from_str::<serde_json::Value>(&content).ok())
.and_then(|value| {
@@ -1277,7 +1278,7 @@ none /tmp/other proc rw 0 0
fn write_session_manifest(session_dir: &Path, last_used: &str) {
fs::create_dir_all(session_dir).unwrap();
fs::write(
session_dir.join("session.json"),
session_dir.join("manifest.json"),
format!(r#"{{ "last_used": "{last_used}" }}"#),
)
.unwrap();
@@ -1299,7 +1300,7 @@ none /tmp/other proc rw 0 0
&(chrono::Utc::now() - chrono::Duration::hours(1)).to_rfc3339(),
);
fs::create_dir_all(&corrupt).unwrap();
fs::write(corrupt.join("session.json"), "not json").unwrap();
fs::write(corrupt.join("manifest.json"), "not json").unwrap();
let report = prune_in_roots(
temp.path(),