vhaudiquet/pkh
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

deb: shrink records to upper, journal and log; keep metadata for every build

Browse Source 
Records carried a full chroot each, and space turned out to be the hard
constraint: five records per target multiplied by a multi-GB chroot does
not fit. The chroot is now per-run scratch — every build extracts a
fresh one from the sha-verified mmdebstrap tarball cache into its
record's chroot/ dir and removes it at the end (success, failure and
interrupt included); resume_chroot, the .pkh-ready probe and
chroot_is_usable are gone, and crashed-run leftovers are swept when the
next run of that identity starts.

What a record keeps instead: manifest.json (renamed, now with the
tarball sha it used, a kept_upper flag and a tree-only journal — the
environment phases rerun in every fresh chroot), a log/build.log full
tee of the build output (kept for EVERY build; plain runs become
metadata-only records, so keep-iff-incremental now applies to the
staged tree alone), and — for --incremental runs — tree-upper/ with its
host-files.list. The fork gate carries a base's staged tree only while
its recorded tarball sha still matches the cache and the changelog
version is unchanged; a fork candidate is any record with a staged tree.

The keep-last-5 cap now applies to staged trees: a sixth retention
degrades the oldest record to metadata-only instead of deleting it.
New `pkh deb log [-f] [<id>]` prints (or follows) a record's build log,
and `pkh deb list` shows the staged-tree state and plain-run records
too. README rewritten around the shrunk model.
This commit is contained in:
vhaudiquet
2026-09-29 14:00:34 +02:00
parent 81f0d52f85
commit dd2634abe3
7 changed files with 1287 additions and 946 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/deb/ephemeral.rs
+180 -161
View File
@@ -13,7 +13,7 @@ use tar::Archive;
use xz2::read::XzDecoder;
// ---------------------------------------------------------------------------
// Interrupt-time chroot cleanup
// Interrupt-time cleanup
//
// On Ctrl-C, the watchdog in `crate::interrupt` runs the hook registered in
// [`EphemeralContextGuard::new_with_context`] right before exiting — the
@@ -22,34 +22,61 @@ use xz2::read::XzDecoder;
// /proc and any overlayfs mounts.
// ---------------------------------------------------------------------------
/// Interrupt-time teardown of a build tree. When `keep_tree` is set the
/// tree itself is left in place after unmounting (an `--incremental`
/// record: the interrupt is a pause, not a discard); otherwise this is
/// the historical full cleanup. For a plain `pkh deb` the `path` is the
/// whole throwaway record root, so the record is removed — the record
/// model's keep-iff-incremental policy applies to interrupts too.
/// Interrupt-time teardown of an anonymous temp chroot (no build record):
/// unmount everything under it, then remove it — a plain `rm -rf` with a
/// retry loop, best-effort. See [`sigint_cleanup_record`] for records.
fn sigint_cleanup_chroot(chroot_path: &Path) {
let is_root = unsafe { libc::geteuid() } == 0;
unmount_under_logged(chroot_path);
remove_with_retries(chroot_path, is_root);
}
/// Interrupt-time teardown of a build record: unmount everything under
/// the record root, remove the per-run state (the chroot and the overlay
/// scratch workdir) and — unless `keep_upper` is set — the staged tree
/// and its host-tree snapshot. The manifest and the build log always
/// stay: every build leaves a metadata record, the interrupt included.
///
/// Unlike [`EphemeralContextGuard::drop`], this deliberately does NOT go
/// through the context manager, the ephemeral context's driver (whose
/// `cleanup()` unmounts the tracked overlays) or the base context's command
/// builder: interrupt-time hooks must be self-contained, and those
/// machineries may be mid-mutation on the interrupted thread. Instead it
/// only reads /proc/mounts and spawns umount/rm directly.
///
/// On a full cleanup it removes the tree regardless of the build result:
/// the build was aborted, and leaving a still-mounted chroot behind is
/// exactly the leak this hook exists to prevent.
/// `cleanup()` unmounts the tracked overlays) or the base context's
/// command builder: interrupt-time hooks must be self-contained, and
/// those machineries may be mid-mutation on the interrupted thread.
/// Instead it only reads /proc/mounts and spawns umount/rm directly.
///
/// Best-effort by design: if a child process still holds a mount busy or
/// privilege escalation is unavailable, individual steps fail; failures are
/// logged (pointing at `pkh prune` for the leftovers) and never panic.
fn sigint_cleanup_chroot(chroot_path: &Path, keep_tree: bool) {
/// privilege escalation is unavailable, individual steps fail; failures
/// are logged (pointing at `pkh prune` for the leftovers) and never
/// panic.
fn sigint_cleanup_record(record_root: &Path, keep_upper: bool) {
let is_root = unsafe { libc::geteuid() } == 0;
unmount_under_logged(record_root);
// Per-run state first: the chroot holds the /proc bind mount and the
// device nodes, the workdir holds root-owned overlay scratch.
for dir in ["chroot", "tree-upper-work"] {
remove_with_retries(&record_root.join(dir), is_root);
}
if keep_upper {
log::debug!(
"Interrupted build record kept its staged tree at {} (fork it \
with `pkh deb --incremental`)",
record_root.display()
);
} else {
for path in ["tree-upper", "host-files.list"] {
remove_with_retries(&record_root.join(path), is_root);
}
}
}
// Unmount children before parents: /proc/mounts lists mounts roughly in
// creation order, so walk it in reverse
let mounts = host_mounts_under(chroot_path);
for mount_point in mounts.into_iter().rev() {
/// Unmount every mount at or under `base`, deepest first, logging the
/// failures (interrupt-time: leftover mounts are `pkh prune`'s to
/// collect).
fn unmount_under_logged(base: &Path) {
let is_root = unsafe { libc::geteuid() } == 0;
// Unmount children before parents: /proc/mounts lists mounts roughly
// in creation order, so walk it in reverse.
for mount_point in host_mounts_under(base).into_iter().rev() {
if unmount_path(&mount_point, is_root) {
log::debug!(
"Unmounted {} during interrupt cleanup",
@@ -63,19 +90,13 @@ fn sigint_cleanup_chroot(chroot_path: &Path, keep_tree: bool) {
);
}
}
}
if keep_tree {
log::debug!(
"Interrupted build record kept at {} (fork it with `pkh deb --incremental`)",
chroot_path.display()
);
return;
}
// Remove the chroot tree itself (tolerates a missing directory). A
// child the Ctrl+C interrupted may still be finishing its writeout —
// dpkg defers SIGINT until it reaches a safe state — so retry while rm
// reports the tree non-empty instead of leaving it half-removed.
/// Privileged `rm -rf` with a retry loop: a child the Ctrl+C interrupted
/// may still be finishing its writeout — dpkg defers SIGINT until it
/// reaches a safe state — so retry while rm reports the tree non-empty
/// instead of leaving it half-removed. A missing path succeeds at once.
fn remove_with_retries(path: &Path, is_root: bool) {
const RETRIES: usize = 10;
const RETRY_DELAY: std::time::Duration = std::time::Duration::from_millis(300);
let mut last = None;
@@ -86,7 +107,7 @@ fn sigint_cleanup_chroot(chroot_path: &Path, keep_tree: bool) {
last = Some(
privileged_command("rm", is_root)
.arg("-rf")
.arg(chroot_path)
.arg(path)
.status(),
);
if matches!(&last, Some(Ok(status)) if status.success()) {
@@ -95,22 +116,19 @@ fn sigint_cleanup_chroot(chroot_path: &Path, keep_tree: bool) {
}
match last {
Some(Ok(status)) if status.success() => {
log::debug!(
"Removed chroot {} during interrupt cleanup",
chroot_path.display()
);
log::debug!("Removed {} during interrupt cleanup", path.display());
}
Some(Ok(status)) => {
log::error!(
"Failed to remove chroot {} during interrupt cleanup \
"Failed to remove {} during interrupt cleanup \
(rm exited with {status}); run `pkh prune`",
chroot_path.display()
path.display()
);
}
Some(Err(e)) => {
log::error!(
"Failed to run rm for chroot {} during interrupt cleanup: {e}; run `pkh prune`",
chroot_path.display()
"Failed to run rm for {} during interrupt cleanup: {e}; run `pkh prune`",
path.display()
);
}
None => unreachable!("at least one rm attempt ran"),
@@ -305,10 +323,11 @@ fn unescape_mount_field(field: &str) -> String {
/// for building packages with unshare permissions.
///
/// In session mode ([`EphemeralContextGuard::new_for_session`]) the chroot
/// lives inside a [`Session`] record: a forked record's chroot is reused
/// when the caller verified it, and the keep policy is strict — the record
/// is kept exactly for `--incremental` runs (`persist_record`), discarded
/// for plain ones whatever the outcome.
/// lives inside a [`Session`] record's `chroot/` directory — extracted
/// fresh from the tarball cache on every run (nothing chroot-shaped is
/// persisted) and removed at the end of the run. The record itself is
/// always kept as metadata and log; its staged tree is retained exactly
/// for `--incremental` runs (`keep_upper`).
pub struct EphemeralContextGuard {
/// The ephemeral build context this guard created (an unshare context
/// bound to the chroot, parented on the base context). Held explicitly so
@@ -327,8 +346,9 @@ pub struct EphemeralContextGuard {
cleanup_hook: Option<CleanupHookGuard>,
/// The build record this guard works in, when any.
session: Option<Session>,
/// Whether the record is kept after the build (`pkh deb --incremental`).
persist_record: bool,
/// Whether the record's staged tree is retained after the build
/// (`pkh deb --incremental`).
keep_upper: bool,
}
impl EphemeralContextGuard {
@@ -345,7 +365,7 @@ impl EphemeralContextGuard {
base_ctx: Arc<Context>,
view: &dyn BuildView,
) -> Result<Self, Box<dyn Error>> {
Self::new_internal(None, series, arch, base_ctx, view, false, false).await
Self::new_internal(None, series, arch, base_ctx, view, false).await
}
/// Create the ephemeral build context inside `session`'s chroot.
@@ -357,31 +377,18 @@ impl EphemeralContextGuard {
/// * `arch` - Optional target architecture, as for
/// [`EphemeralContextGuard::new_with_context`]
/// * `base_ctx` - The base context to use for chroot work
/// * `reuse_chroot` - Reuse the forked record's bootstrapped chroot
/// instead of re-extracting the tarball; the caller must only set
/// this after the record's chroot passed the integrity probe
/// * `persist_record` - Keep the record after the build
/// (`pkh deb --incremental`); a plain run's record is discarded
/// whatever the outcome
/// * `keep_upper` - Retain the record's staged tree after the build
/// (`pkh deb --incremental`); a plain run's record keeps metadata
/// and log only
pub async fn new_for_session(
session: Session,
series: &str,
arch: Option<&str>,
base_ctx: Arc<Context>,
view: &dyn BuildView,
reuse_chroot: bool,
persist_record: bool,
keep_upper: bool,
) -> Result<Self, Box<dyn Error>> {
Self::new_internal(
Some(session),
series,
arch,
base_ctx,
view,
reuse_chroot,
persist_record,
)
.await
Self::new_internal(Some(session), series, arch, base_ctx, view, keep_upper).await
}
async fn new_internal(
@@ -390,8 +397,7 @@ impl EphemeralContextGuard {
arch: Option<&str>,
base_ctx: Arc<Context>,
view: &dyn BuildView,
reuse_chroot: bool,
persist_record: bool,
keep_upper: bool,
) -> Result<Self, Box<dyn Error>> {
// Save the globally-installed context so Drop can restore exactly
// this handle: concurrent builds install their own ephemeral
@@ -399,8 +405,8 @@ impl EphemeralContextGuard {
// before this guard swapped anything in.
let previous_context = context::current();
// In session mode the chroot lives inside the record root (and is
// reused when the caller says so); otherwise it is a fresh temp dir.
// In session mode the chroot lives inside the record root, as
// per-run scratch; otherwise it is a fresh temp dir.
let chroot_path = match &session {
Some(s) => {
let path = s.chroot_path();
@@ -424,22 +430,23 @@ impl EphemeralContextGuard {
// base context: the hook must be self-contained (stored path +
// direct umount/rm subprocesses) and cannot go through `base_ctx`.
// For remote or nested bases the chroot lives elsewhere, and
// leftovers stay handled by `pkh prune` as before. An
// `--incremental` record is kept on interrupt (a pause, not a
// discard); a plain run's whole record root and a plain temp chroot
// are removed.
// leftovers stay handled by `pkh prune` as before.
//
// Session mode tears the record's per-run state down (chroot and
// overlay workdir) and keeps its staged tree exactly for
// `--incremental`; a plain temp chroot is removed entirely.
let cleanup_hook = if matches!(base_ctx.config, ContextConfig::Local) {
Some(crate::interrupt::register_cleanup_hook(Box::new({
// Session mode tears the whole record root down (the
// manifest is part of the record); plain temp chroots
// only ever had the chroot dir.
let cleanup_path = match &session {
Some(s) => s.root().to_path_buf(),
None => chroot_path.clone(),
};
let keep_tree = persist_record;
move || sigint_cleanup_chroot(&cleanup_path, keep_tree)
})))
let hook: Box<dyn Fn() + Send> = match session.as_ref().map(|s| s.root().to_path_buf())
{
Some(record_root) => {
Box::new(move || sigint_cleanup_record(&record_root, keep_upper))
}
None => {
let chroot_path = chroot_path.clone();
Box::new(move || sigint_cleanup_chroot(&chroot_path))
}
};
Some(crate::interrupt::register_cleanup_hook(hook))
} else {
log::debug!(
"Base context is not local; skipping interrupt-time cleanup registration for {}",
@@ -448,18 +455,11 @@ impl EphemeralContextGuard {
None
};
// Download and extract the chroot tarball (or reuse the record's
// prepared chroot)
if let Err(e) = Self::download_and_extract_chroot(
series,
arch,
&chroot_path,
base_ctx.clone(),
view,
session.as_ref(),
reuse_chroot,
)
.await
// Download and extract the chroot tarball — always fresh; the
// record's chroot directory is per-run scratch.
if let Err(e) =
Self::download_and_extract_chroot(series, arch, &chroot_path, base_ctx.clone(), view)
.await
{
// On a Ctrl+C the interrupt watchdog owns the tree: keep the
// hook registered (forgetting the guard) so it removes the
@@ -476,6 +476,17 @@ impl EphemeralContextGuard {
return Err(e);
}
// Record the tarball digest this environment came from: the fork
// gate of a later `--incremental` run compares it against the
// cache to decide whether the record's staged tree is still
// reusable.
if let Some(s) = &session
&& let Ok(tarball) = chroot_tarball_path(series, arch)
&& let Ok(sha) = session::sha256_file(&tarball)
{
s.set_tarball_sha256(&sha);
}
// Switch to an ephemeral context to build the package in the chroot.
// The parent is the base context itself (the one that bootstrapped
// the chroot), wired through `with_parent` instead of a config-name
@@ -502,7 +513,7 @@ impl EphemeralContextGuard {
base_ctx,
cleanup_hook,
session,
persist_record,
keep_upper,
})
}
@@ -522,25 +533,10 @@ impl EphemeralContextGuard {
chroot_path: &PathBuf,
ctx: Arc<context::Context>,
view: &dyn BuildView,
session: Option<&Session>,
reuse_chroot: bool,
) -> Result<(), Box<dyn Error>> {
// Record fork: the base's chroot passed the caller's checks —
// skip the tarball extraction entirely, the environment (apt
// lists, installed build-deps) is still in there.
if let Some(s) = session {
if reuse_chroot && s.chroot_is_usable() {
log::info!(
"Reusing the prepared chroot of record {} (bootstrap skipped)",
s.manifest().id
);
Self::finalize_chroot(chroot_path, ctx, view)?;
return Ok(());
}
// Not reusing (fresh record or failed probe): never extract
// over stale content.
clear_dir(chroot_path)?;
}
// The chroot is per-run scratch: nothing is reused, so never
// extract over stale content.
clear_dir(chroot_path)?;
// Clone ctx for use in create_device_nodes after download_chroot_tarball consumes it
let ctx_for_devices = ctx.clone();
@@ -619,33 +615,13 @@ impl EphemeralContextGuard {
// Create device nodes in the chroot and bind-mount /proc
Self::finalize_chroot(chroot_path, ctx_for_devices, view)?;
// Record the bootstrap in the record: marker file (the integrity
// probe) and the tarball the chroot came from.
if let Some(s) = session {
std::fs::write(chroot_path.join(".pkh-ready"), b"")?;
let metadata = fs::metadata(&tarball_path)?;
let tarball_sha256 = session::sha256_file(&tarball_path)
.map_err(|e| format!("cannot hash the chroot tarball: {e}"))?;
s.set_chroot(session::ChrootInfo {
tarball: tarball_filename,
tarball_size: metadata.len(),
tarball_mtime_secs: metadata
.modified()
.ok()
.and_then(|t| t.duration_since(std::time::UNIX_EPOCH).ok())
.map(|d| d.as_secs())
.unwrap_or(0),
tarball_sha256,
ready: true,
});
log::debug!("Record {} chroot is ready", s.manifest().id);
}
// The bootstrap's tarball digest is recorded by the caller (the
// record's fork gate compares it against the cache later).
Ok(())
}
/// Device nodes and the /proc bind mount for a (possibly reused)
/// chroot: idempotent finalization, safe to run on every attempt.
/// Device nodes and the /proc bind mount for a fresh chroot:
/// idempotent finalization, safe to run on every run.
fn finalize_chroot(
chroot_path: &Path,
ctx: Arc<context::Context>,
@@ -660,8 +636,8 @@ impl EphemeralContextGuard {
// Bind mount /proc from host into chroot (before entering unshare namespace)
// This allows /proc to work in containers where mounting inside unshare fails.
// A reused chroot may still carry the mount from the previous
// attempt: only mount when it is not already there.
// A chroot reused within the same run may still carry the mount:
// only mount when it is not already there.
if is_mountpoint(&chroot_path.join("proc")) {
log::debug!("/proc is already bind-mounted in the chroot");
} else {
@@ -908,6 +884,11 @@ impl Drop for EphemeralContextGuard {
// drained, forgetting is a harmless no-op).
if crate::interrupt::interrupted() {
context::manager().set_current_ephemeral(self.previous_context.clone());
// The record keeps its metadata: mark the aborted build as
// failed before the hook tears the per-run state down.
if let Some(session) = &self.session {
session.set_outcome(session::OUTCOME_FAILED);
}
if let Some(hook) = self.cleanup_hook.take() {
std::mem::forget(hook);
}
@@ -942,11 +923,11 @@ impl Drop for EphemeralContextGuard {
// touches it.
context::manager().set_current_ephemeral(self.previous_context.clone());
// Session mode: the tree is a build record. The /proc bind mount
// must come down in every case (a kept record has to be
// unmount-clean for the next fork); the record itself follows the
// keep policy — kept exactly for `--incremental` runs, discarded
// for plain ones whatever the outcome.
// Session mode: the record's per-run chroot comes down in every
// case (nothing chroot-shaped is persisted — success, failure and
// interrupt all end the run), then the record is retained:
// metadata and log always, the staged tree exactly for
// `--incremental` runs.
if let Some(session) = &self.session {
let proc_path = self.chroot_path.join("proc");
let is_root = crate::utils::root::is_root().unwrap_or(false);
@@ -960,7 +941,8 @@ impl Drop for EphemeralContextGuard {
.status()
};
session.dispose_after_build(self.persist_record, self.build_succeeded);
session.remove_run_state();
session.retain_upper(self.keep_upper, self.build_succeeded);
return;
}
@@ -1058,7 +1040,7 @@ mod chroot_cleanup_tests {
fn sigint_cleanup_of_missing_chroot_is_a_noop() {
let dir = tempfile::tempdir().unwrap();
let missing = dir.path().join("no-such-chroot");
sigint_cleanup_chroot(&missing, false);
sigint_cleanup_chroot(&missing);
assert!(!missing.exists());
}
@@ -1081,24 +1063,61 @@ mod chroot_cleanup_tests {
std::fs::create_dir_all(chroot.join("rootfs")).unwrap();
std::fs::write(chroot.join("rootfs").join("file.txt"), "data").unwrap();
sigint_cleanup_chroot(&chroot, false);
sigint_cleanup_chroot(&chroot);
assert!(!chroot.exists());
}
/// The interrupt teardown of a kept tree (an `--incremental` record)
/// unmounts and leaves the tree in place — the pause, not the discard.
/// The record interrupt teardown removes the per-run state and keeps
/// the manifest and log; an `--incremental` record keeps its staged
/// tree, a plain one loses it.
#[test]
fn sigint_cleanup_keeps_the_tree_when_asked() {
fn sigint_record_teardown_keeps_metadata_and_drops_the_chroot() {
let is_root = unsafe { libc::geteuid() } == 0;
if !is_root
&& !privileged_command("true", false)
.status()
.is_ok_and(|s| s.success())
{
return;
}
let dir = tempfile::tempdir().unwrap();
let record_root = dir.path().join("record");
std::fs::create_dir_all(record_root.join("chroot/rootfs")).unwrap();
std::fs::write(record_root.join("session.json"), "{}").unwrap();
let incremental = dir.path().join("incremental");
let plain = dir.path().join("plain");
for root in [&incremental, &plain] {
std::fs::create_dir_all(root.join("chroot/rootfs")).unwrap();
std::fs::create_dir_all(root.join("tree-upper/src")).unwrap();
std::fs::create_dir_all(root.join("log")).unwrap();
std::fs::write(root.join("manifest.json"), "{}").unwrap();
std::fs::write(root.join("log/build.log"), "output\n").unwrap();
std::fs::write(root.join("host-files.list"), "f\t1\t1\ta\n").unwrap();
}
sigint_cleanup_chroot(&record_root, true);
sigint_cleanup_record(&incremental, true);
sigint_cleanup_record(&plain, false);
assert!(record_root.join("chroot/rootfs").exists());
assert!(record_root.join("session.json").is_file());
for root in [&incremental, &plain] {
assert!(!root.join("chroot").exists(), "the chroot is per-run");
assert!(root.join("manifest.json").is_file(), "metadata stays");
assert!(root.join("log/build.log").is_file(), "the log stays");
}
assert!(
incremental.join("tree-upper/src").exists(),
"an interrupted incremental run keeps its staged tree"
);
assert!(
incremental.join("host-files.list").is_file(),
"the snapshot stays with the staged tree"
);
assert!(
!plain.join("tree-upper").exists(),
"an interrupted plain run drops its staged tree"
);
assert!(
!plain.join("host-files.list").exists(),
"the snapshot goes with the staged tree"
);
}
/// clear_dir removes content a plain `rm -rf` cannot: a root-owned
src/deb/local.rs
+25 -57
View File
@@ -27,35 +27,12 @@ fn cap<'a>(
cmd
}
/// The session journal stamps this build works with: the input stamps of
/// the patch and build-dependency phases, computed from the host tree and
/// the resolved selectors.
struct JournalStamps {
patches: Option<String>,
control: Option<String>,
}
/// Compute the journal stamps for a session build; `None` without one.
fn journal_stamps(
session: &session::Session,
arch: &str,
series: &str,
pocket: Option<&str>,
cross: bool,
ppa: &[String],
inject_packages: &[String],
) -> JournalStamps {
/// The input stamp of the patch phase, computed from the host tree: the
/// only tree phase the record journal tracks (the environment phases
/// rerun in every fresh chroot and are not recorded).
fn patch_stamp(session: &session::Session) -> Option<String> {
let host_tree = PathBuf::from(session.manifest().host_tree);
let selectors = format!(
"{arch}\n{cross}\n{series}\n{}\n{}\n{}",
pocket.unwrap_or(""),
ppa.join(","),
inject_packages.join(",")
);
JournalStamps {
patches: session::patches_stamp(&host_tree),
control: session::control_stamp(&host_tree, &selectors),
}
session::patches_stamp(&host_tree)
}
#[allow(clippy::too_many_arguments)]
@@ -74,13 +51,12 @@ pub async fn build(
view: &dyn BuildView,
jobs: Option<usize>,
session: Option<&session::Session>,
resume: bool,
build_resume: bool,
) -> Result<Vec<PathBuf>, Box<dyn Error>> {
let sink: Option<Arc<dyn LineSink>> = view.sink();
// Journal stamps for a session build; recording is a no-op without one.
let stamps =
session.map(|s| journal_stamps(s, arch, series, pocket, cross, ppa, inject_packages));
// Journal stamp for a record build; recording is a no-op without one.
let stamps = session.and_then(patch_stamp);
// Environment
let mut env = HashMap::<String, String>::new();
@@ -245,9 +221,6 @@ pub async fn build(
or re-run with RUST_LOG=debug for more details."
.into());
}
if let Some(s) = session {
s.record_phase("apt_update", None);
}
// Install essential packages
log::debug!("Installing essential packages for local build...");
@@ -272,9 +245,6 @@ pub async fn build(
if !status.success() {
return Err("Could not install essential packages for the build".into());
}
if let Some(s) = session {
s.record_phase("essentials", None);
}
// The package directory was resolved by the caller (the staged copy of
// the tree the user pointed at, or the name-pattern search fallback)
@@ -294,10 +264,11 @@ pub async fn build(
}
// Apply quilt patches if the package provides a patch series. On a
// resumed session the previous attempt's patches may still be applied:
// pop them first when the patch tree changed since.
let patch_stamp = stamps.as_ref().and_then(|s| s.patches.clone());
let pop_patches = resume && {
// build resuming over a carried upperdir the previous attempt's
// patches may still be applied: pop them first when the patch tree
// changed since.
let patch_stamp = stamps.clone();
let pop_patches = build_resume && {
let previous = session
.and_then(|s| s.manifest().phases.get("patches").map(|p| p.stamp.clone()))
.unwrap_or(None);
@@ -333,20 +304,17 @@ pub async fn build(
view,
&sink,
)?;
if let Some(s) = session {
s.record_phase(
"build_deps",
stamps.as_ref().and_then(|st| st.control.clone()),
);
}
// (The build-dependency phase reruns in every fresh chroot: nothing
// to journal.)
// A resumed build continues in the previous attempt's tree. Packages
// whose debian/rules keeps an input-untracked stamp cache (quirk:
// `resume_clear`, e.g. the kernels' `debian/stamps`, whose flavour
// config rule exports .config with no config prerequisite) would build
// from the previous attempt's configuration: drop the listed caches so
// the wrapper steps re-run over the incremental inner builds.
if resume {
// A build resuming over a carried upperdir continues in the previous
// attempt's tree. Packages whose `debian/rules` keeps an
// input-untracked stamp cache (quirk: `resume_clear`, e.g. the
// kernels' `debian/stamps`, whose flavour config rule exports .config
// with no config prerequisite) would build from the previous
// attempt's configuration: drop the listed caches so the wrapper
// steps re-run over the incremental inner builds.
if build_resume {
for rel in crate::quirks::get_resume_clear_paths(package, series) {
let path = Path::new(package_dir_str).join(&rel);
if !ctx.exists(&path).unwrap_or(false) {
@@ -383,10 +351,10 @@ pub async fn build(
s.record_phase("build", None);
}
// On a resumed session the previous attempt's debian/files would make
// On a resumed build the previous attempt's debian/files would make
// the artifact collection below surface its outputs: only the new
// attempt's registration may remain.
if resume {
if build_resume {
let _ = ctx
.command("rm")
.arg("-f")
src/deb/mod.rs
+195 -106
View File
@@ -119,36 +119,87 @@ pub struct DebBuildOptions<'a> {
/// Explicit build context; defaults to the current context.
pub ctx: Option<Arc<Context>>,
/// Incremental request (`pkh deb --incremental [<id>]`): which
/// record to fork as the base of this build, and whether this
/// build's own record is kept. Defaults to a plain throwaway run.
/// record to fork as the base of this build's staged tree, and
/// whether this build's own staged tree is retained. Defaults to a
/// plain run.
pub incremental: Incremental,
/// Where build events (phases, progress, outcome) are reported.
pub view: &'a dyn BuildView,
}
/// The `--incremental` request of a `pkh deb` build: which record the
/// build forks its environment from, and whether its own record is
/// kept afterwards.
/// build forks its staged tree from, and whether its own staged tree is
/// retained afterwards (the record's metadata and log are kept for
/// every build either way).
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub enum Incremental {
/// A plain run (`pkh deb` without `--incremental`): build from
/// scratch in a throwaway record, discarded on success, failure
/// and interrupt alike.
/// scratch in a record whose staged tree is dropped at the end —
/// metadata and log only remain.
#[default]
Off,
/// `--incremental`: fork the auto-selected base — the newest
/// successful record of this identity among the last five usable —
/// and always keep the new record. With no usable candidate the
/// build runs from scratch and its record becomes the identity's
/// first base.
/// successful record of this identity among the last five carrying
/// a staged tree — and retain the new record's staged tree. With no
/// candidate the build runs from scratch and its record becomes
/// the identity's first base.
Auto,
/// `--incremental <id>`: fork that exact record (unique id or
/// prefix, restricted to the records of this host tree; a FAILED
/// record is a valid base — its environment may be fine, and the
/// user pinned it deliberately) and always keep the new record.
/// record is a valid base — the user pinned it deliberately) and
/// retain the new record's staged tree.
Id(String),
}
/// A view delegating every event to `inner` while teeing all captured
/// subprocess lines into a build record's log through [`RecordLogSink`]:
/// everything downstream (the chroot bootstrap, the build phases) asks
/// the view for its sink, so one wrapper tees the whole run.
struct RecordLogView<'a> {
/// The caller's view.
inner: &'a dyn BuildView,
/// The record log sink, forwarding to the inner view's sink too.
log: Arc<session::RecordLogSink>,
}
impl BuildView for RecordLogView<'_> {
fn target(&self, target: crate::report::BuildTarget) {
self.inner.target(target);
}
fn phase(&self, name: &str, classifier: Box<dyn Classifier>) {
self.inner.phase(name, classifier);
}
fn message(&self, text: &str) {
self.inner.message(text);
}
fn progress(&self, label: &str, pos: usize, total: usize) {
self.inner.progress(label, pos, total);
}
fn sink(&self) -> Option<Arc<dyn crate::context::LineSink>> {
Some(self.log.clone())
}
fn finish_success(&self, artifacts: &[PathBuf]) {
self.inner.finish_success(artifacts);
}
fn finish_failure(&self) {
self.inner.finish_failure();
}
fn suspend(&self) {
self.inner.suspend();
}
fn is_enabled(&self) -> bool {
self.inner.is_enabled()
}
}
impl Default for DebBuildOptions<'_> {
fn default() -> Self {
static QUIET: crate::report::Quiet = crate::report::Quiet;
@@ -281,7 +332,7 @@ async fn build_binary_package_impl(
cross,
};
let mut persist_record = false;
let mut keep_upper = false;
let session = if session_enabled {
match resolve_record(
incremental,
@@ -291,8 +342,8 @@ async fn build_binary_package_impl(
series,
chroot_arch,
) {
Ok((record, persist)) => {
persist_record = persist;
Ok((record, keep)) => {
keep_upper = keep;
Some(record)
}
Err(e) => {
@@ -303,9 +354,26 @@ async fn build_binary_package_impl(
} else {
None
};
// Only a forked record carries a bootstrapped chroot at this point;
// fresh ones (plain runs and first `--incremental` runs) extract it.
let resume_chroot = session.as_ref().is_some_and(|s| s.chroot_is_usable());
// Tee every captured build line into the record's log: the wrapper
// view hands the combined sink to everything downstream (the chroot
// bootstrap, the build phases), so the log is a full tee whatever
// renders the output. A record-less build (non-local base context)
// keeps the plain view.
let record_log: Option<Result<Arc<session::RecordLogSink>, std::io::Error>> = session
.as_ref()
.map(|s| session::RecordLogSink::new(s, opts.view.sink()).map(Arc::new));
let view: &dyn BuildView = match record_log {
Some(Ok(log)) => &RecordLogView {
inner: opts.view,
log,
},
Some(Err(e)) => {
log::warn!("Cannot open the build record's log: {e}");
opts.view
}
None => opts.view,
};
let mut guard = if let Some(s) = session.clone() {
Some(
@@ -315,8 +383,7 @@ async fn build_binary_package_impl(
chroot_arch,
base_ctx.clone(),
view,
resume_chroot,
persist_record,
keep_upper,
)
.await?,
)
@@ -454,7 +521,7 @@ async fn build_binary_package_impl(
view,
*jobs,
session.as_ref(),
resume_chroot,
build_resume,
)
.await?
}
@@ -486,19 +553,17 @@ async fn build_binary_package_impl(
// the chroot) inherit the terminal, so they must not fight the widget.
view.suspend();
// Mark the build as successful for the guard's drop, which disposes of
// the record per the keep policy: an `--incremental` record is kept
// with its outcome recorded, a plain run's throwaway record is
// always discarded.
// Mark the build as successful for the guard's drop, which tears the
// per-run chroot down and retains the record: metadata and log for
// every build, the staged tree exactly for `--incremental` runs.
if result.is_ok()
&& let Some(g) = &mut guard
{
g.mark_build_successful();
}
// A failed `--incremental` run keeps its record for a later fork:
// record the outcome (discarded plain runs need none).
// Every record's outcome is written — plain runs leave a metadata
// record too.
if result.is_err()
&& persist_record
&& let Some(s) = &session
{
s.set_outcome(session::OUTCOME_FAILED);
@@ -518,10 +583,9 @@ fn resolve_record(
series: &str,
chroot_arch: Option<&str>,
) -> Result<(session::Session, bool), Box<dyn Error>> {
let expected = ephemeral::chroot_tarball_path(series, chroot_arch).ok();
resolve_record_in(
&session::sessions_root(),
expected.as_deref(),
&session::cache_tarball_sha256(series, chroot_arch),
incremental,
host_tree,
identity,
@@ -530,30 +594,22 @@ fn resolve_record(
}
/// The testable core of [`resolve_record`] over an explicit sessions root
/// and bootstrap tarball.
/// and cached-tarball digest (the fork gate's reference).
///
/// Every local build runs in its own record: [`Incremental::Off`] gets a
/// brand-new throwaway one, [`Incremental::Auto`] and
/// [`Incremental::Id`] fork the selected base. The returned flag says
/// whether the record is kept after the build — exactly the
/// `--incremental` runs keep theirs.
///
/// A base must be usable: its chroot passes the integrity probe and was
/// bootstrapped from the tarball this build would use (`expected_tarball`
/// is `None` when the path cannot be resolved — no record is usable
/// then, which degrades an `--incremental` run to a fresh build).
/// brand-new one, [`Incremental::Auto`] and [`Incremental::Id`] fork the
/// selected base. The returned flag says whether the record's staged
/// tree is retained after the build — exactly the `--incremental` runs
/// retain theirs. Metadata and the log are kept for every build either
/// way.
fn resolve_record_in(
records_root: &Path,
expected_tarball: Option<&Path>,
cache_tarball_sha256: &str,
incremental: &Incremental,
host_tree: &Path,
identity: &session::SessionIdentity,
version: &str,
) -> Result<(session::Session, bool), Box<dyn Error>> {
let usable = |s: &session::Session| {
s.chroot_is_usable()
&& expected_tarball.is_some_and(|t| session::tarball_matches_path(s, t))
};
match incremental {
Incremental::Off => Ok((
session::new_record_in(records_root, host_tree, identity, version)?,
@@ -561,14 +617,12 @@ fn resolve_record_in(
)),
Incremental::Auto => {
// A record locked by a live build cannot be a base: its
// chroot may be mid-mutation (apt always runs, deps land
// inside it), and the fork would copy that state.
// staged tree may be mid-mutation by the running build.
let candidates: Vec<session::Session> =
session::identity_records_in(records_root, identity)
session::fork_candidates_in(records_root, identity)
.into_iter()
.filter(|s| usable(s))
.filter(|s| !session::lock_is_held(&s.lock_path()).unwrap_or(true))
.take(session::KEEP_RECORDS)
.take(session::KEEP_UPPERS)
.collect();
for base in session::rank_base_candidates(candidates) {
let why = if base.manifest().last_outcome == session::OUTCOME_SUCCESS {
@@ -576,7 +630,14 @@ fn resolve_record_in(
} else {
"no successful record among the last usable ones; the newest record"
};
match session::fork_record_in(records_root, &base, host_tree, identity, version) {
match session::fork_record_in(
records_root,
&base,
host_tree,
identity,
version,
cache_tarball_sha256,
) {
Ok(record) => {
log::info!(
"Base of the incremental build: record {} ({why})",
@@ -591,8 +652,8 @@ fn resolve_record_in(
}
}
log::info!(
"No usable record of {identity} to fork; building from scratch — \
this run's record becomes the target's first base"
"No record of {identity} carrying a staged tree to fork; building \
from scratch — this run's record becomes the target's first base"
);
Ok((
session::new_record_in(records_root, host_tree, identity, version)?,
@@ -603,11 +664,10 @@ fn resolve_record_in(
let mut matches: Vec<session::Session> = session::by_id_prefix_in(records_root, id)
.into_iter()
.filter(|s| s.manifest().host_tree == host_tree.to_string_lossy())
.filter(|s| usable(s))
.collect();
match matches.len() {
0 => Err(format!(
"no usable build record matches id '{id}' for this tree \
"no build record matches id '{id}' for this tree \
(see `pkh deb list`)"
)
.into()),
@@ -618,7 +678,14 @@ fn resolve_record_in(
base.manifest().id
);
Ok((
session::fork_record_in(records_root, &base, host_tree, identity, version)?,
session::fork_record_in(
records_root,
&base,
host_tree,
identity,
version,
cache_tarball_sha256,
)?,
true,
))
}
@@ -1354,30 +1421,37 @@ mod tests {
// -------------------------------------------------------------------
use crate::deb::session::record_fixtures::{
record_with_environment, sessions_fixture, test_identity,
metadata_record, record_with_upper, sessions_fixture, test_identity,
};
use crate::deb::session::{self, SessionIdentity};
/// A plain run resolves to a brand-new, lineage-free record that the
/// caller will discard — including when no record exists at all yet.
/// The sha of the fixture's cached tarball (the fork gate's
/// reference).
fn cache_sha(tarball: &std::path::Path) -> String {
session::sha256_file(tarball).unwrap()
}
/// A plain run resolves to a brand-new, lineage-free record whose
/// staged tree will be dropped — including when no record exists at
/// all yet.
#[test]
fn resolve_plain_run_builds_a_fresh_discardable_record() {
let (root, tarball) = sessions_fixture();
let identity = test_identity();
let host_tree = root.path().join("tree");
let (record, persist) = resolve_record_in(
let (record, keep_upper) = resolve_record_in(
root.path(),
Some(&tarball),
&cache_sha(&tarball),
&Incremental::Off,
&host_tree,
&identity,
"1.0",
)
.expect("a plain run always resolves to a fresh record");
assert!(!persist, "a plain run discards its record");
assert!(!keep_upper, "a plain run drops its staged tree");
assert_eq!(record.manifest().base, None);
assert!(!record.chroot_is_usable());
assert!(!record.has_upper());
}
/// The user scenario, end to end over the record layout: record A
@@ -1388,7 +1462,7 @@ mod tests {
let (root, tarball) = sessions_fixture();
let identity = test_identity();
let host_tree = root.path().join("tree");
let a = record_with_environment(
let a = record_with_upper(
root.path(),
&tarball,
&host_tree,
@@ -1397,56 +1471,67 @@ mod tests {
Some(session::OUTCOME_SUCCESS),
);
let a_id = a.manifest().id;
let b = session::fork_record_in(root.path(), &a, &host_tree, &identity, "1.0").unwrap();
let b = session::fork_record_in(
root.path(),
&a,
&host_tree,
&identity,
"1.0",
&cache_sha(&tarball),
)
.unwrap();
let b_id = b.manifest().id;
b.set_outcome(session::OUTCOME_FAILED);
drop((a, b));
let (record, persist) = resolve_record_in(
let (record, keep_upper) = resolve_record_in(
root.path(),
Some(&tarball),
&cache_sha(&tarball),
&Incremental::Auto,
&host_tree,
&identity,
"1.0",
)
.expect("the auto selection must fork a usable record");
assert!(persist);
.expect("the auto selection must fork a candidate record");
assert!(keep_upper);
assert_eq!(record.manifest().base.as_deref(), Some(a_id.as_str()));
assert_ne!(record.manifest().id, b_id);
// The successful base's gate passes: its staged tree is carried.
assert!(record.has_upper());
}
/// A first `--incremental` run with no candidate builds completely
/// fresh and persists its record — it becomes the identity's first
/// base.
/// fresh and retains its staged tree — it becomes the identity's
/// first base.
#[test]
fn resolve_auto_without_candidates_builds_fresh_and_persists() {
fn resolve_auto_without_candidates_builds_fresh_and_retains() {
let (root, tarball) = sessions_fixture();
let identity = test_identity();
let host_tree = root.path().join("tree");
let (record, persist) = resolve_record_in(
let (record, keep_upper) = resolve_record_in(
root.path(),
Some(&tarball),
&cache_sha(&tarball),
&Incremental::Auto,
&host_tree,
&identity,
"1.0",
)
.expect("no candidates must degrade to a fresh build");
assert!(persist, "the first record becomes a base");
assert!(keep_upper, "the first record becomes a base");
assert_eq!(record.manifest().base, None);
}
/// Non-usable records never make it into the selection: a newer
/// successful record with a broken chroot loses to an older usable
/// failed one.
/// Records without a staged tree (plain runs, metadata-only ones)
/// never make it into the selection: a newer successful
/// metadata-only record loses to an older failed one carrying a
/// staged tree.
#[test]
fn resolve_auto_skips_non_usable_records() {
fn resolve_auto_skips_records_without_a_staged_tree() {
let (root, tarball) = sessions_fixture();
let identity = test_identity();
let host_tree = root.path().join("tree");
let usable = record_with_environment(
let carrier = record_with_upper(
root.path(),
&tarball,
&host_tree,
@@ -1454,33 +1539,37 @@ mod tests {
"1.0",
Some(session::OUTCOME_FAILED),
);
let usable_id = usable.manifest().id;
drop(usable);
// Newer, successful, but its chroot never finished bootstrapping.
let broken = session::new_record_in(root.path(), &host_tree, &identity, "1.0").unwrap();
broken.set_outcome(session::OUTCOME_SUCCESS);
drop(broken);
let carrier_id = carrier.manifest().id;
drop(carrier);
// Newer and successful, but a plain run: no staged tree.
drop(metadata_record(
root.path(),
&host_tree,
&identity,
"1.0",
session::OUTCOME_SUCCESS,
));
let (record, _) = resolve_record_in(
root.path(),
Some(&tarball),
&cache_sha(&tarball),
&Incremental::Auto,
&host_tree,
&identity,
"1.0",
)
.expect("the usable record is still a base");
assert_eq!(record.manifest().base.as_deref(), Some(usable_id.as_str()));
.expect("the record carrying a staged tree is still the base");
assert_eq!(record.manifest().base.as_deref(), Some(carrier_id.as_str()));
}
/// `--incremental <id>` may pin a FAILED record — its environment may
/// be fine and the user chose it deliberately.
/// `--incremental <id>` may pin a FAILED record — the user chose it
/// deliberately and its staged tree may be perfectly fine.
#[test]
fn resolve_pinned_id_forks_a_failed_record() {
let (root, tarball) = sessions_fixture();
let identity = test_identity();
let host_tree = root.path().join("tree");
let failed = record_with_environment(
let failed = record_with_upper(
root.path(),
&tarball,
&host_tree,
@@ -1491,17 +1580,18 @@ mod tests {
let failed_id = failed.manifest().id;
drop(failed);
let (record, persist) = resolve_record_in(
let (record, keep_upper) = resolve_record_in(
root.path(),
Some(&tarball),
&cache_sha(&tarball),
&Incremental::Id(failed_id.clone()),
&host_tree,
&identity,
"1.0",
)
.expect("a pinned failed record is a valid base");
assert!(persist);
assert!(keep_upper);
assert_eq!(record.manifest().base.as_deref(), Some(failed_id.as_str()));
assert!(record.has_upper(), "the gate passes: sha and version match");
}
/// A pinned id is restricted to the records of this host tree: an id
@@ -1512,7 +1602,7 @@ mod tests {
let identity = test_identity();
let host_tree = root.path().join("tree");
let other_tree = root.path().join("elsewhere");
let foreign = record_with_environment(
let foreign = record_with_upper(
root.path(),
&tarball,
&other_tree,
@@ -1525,7 +1615,7 @@ mod tests {
let err = resolve_record_in(
root.path(),
Some(&tarball),
&cache_sha(&tarball),
&Incremental::Id(foreign_id),
&host_tree,
&identity,
@@ -1533,8 +1623,7 @@ mod tests {
)
.expect_err("records of other trees must not be forked by id");
assert!(
err.to_string()
.contains("no usable build record matches id"),
err.to_string().contains("no build record matches id"),
"wrong error: {err}"
);
}
@@ -1545,7 +1634,7 @@ mod tests {
let (root, tarball) = sessions_fixture();
let identity = test_identity();
let host_tree = root.path().join("tree");
drop(record_with_environment(
drop(record_with_upper(
root.path(),
&tarball,
&host_tree,
@@ -1553,7 +1642,7 @@ mod tests {
"1.0",
None,
));
drop(record_with_environment(
drop(record_with_upper(
root.path(),
&tarball,
&host_tree,
@@ -1567,7 +1656,7 @@ mod tests {
let err = resolve_record_in(
root.path(),
Some(&tarball),
&cache_sha(&tarball),
&Incremental::Id(prefix),
&host_tree,
&identity,
@@ -1591,8 +1680,8 @@ mod tests {
.collect()
}
/// A pinned record of another identity is refused: its environment
/// was bootstrapped for a different target.
/// A pinned record of another identity is refused: its staged tree
/// was built for a different target.
#[test]
fn resolve_pinned_id_refuses_foreign_identities() {
let (root, tarball) = sessions_fixture();
@@ -1602,7 +1691,7 @@ mod tests {
..identity.clone()
};
let host_tree = root.path().join("tree");
let foreign = record_with_environment(
let foreign = record_with_upper(
root.path(),
&tarball,
&host_tree,
@@ -1615,7 +1704,7 @@ mod tests {
let err = resolve_record_in(
root.path(),
Some(&tarball),
&cache_sha(&tarball),
&Incremental::Id(foreign_id),
&host_tree,
&identity,
src/deb/session.rs
+808 -591
View File
File diff suppressed because it is too large Load Diff