From dd2634abe3c5023a32723f7c86311106d88f24b9 Mon Sep 17 00:00:00 2001 From: Valentin Haudiquet Date: Tue, 29 Sep 2026 14:00:34 +0200 Subject: [PATCH] deb: shrink records to upper, journal and log; keep metadata for every build MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records carried a full chroot each, and space turned out to be the hard constraint: five records per target multiplied by a multi-GB chroot does not fit. The chroot is now per-run scratch — every build extracts a fresh one from the sha-verified mmdebstrap tarball cache into its record's chroot/ dir and removes it at the end (success, failure and interrupt included); resume_chroot, the .pkh-ready probe and chroot_is_usable are gone, and crashed-run leftovers are swept when the next run of that identity starts. What a record keeps instead: manifest.json (renamed, now with the tarball sha it used, a kept_upper flag and a tree-only journal — the environment phases rerun in every fresh chroot), a log/build.log full tee of the build output (kept for EVERY build; plain runs become metadata-only records, so keep-iff-incremental now applies to the staged tree alone), and — for --incremental runs — tree-upper/ with its host-files.list. The fork gate carries a base's staged tree only while its recorded tarball sha still matches the cache and the changelog version is unchanged; a fork candidate is any record with a staged tree. The keep-last-5 cap now applies to staged trees: a sixth retention degrades the oldest record to metadata-only instead of deleting it. New `pkh deb log [-f] []` prints (or follows) a record's build log, and `pkh deb list` shows the staged-tree state and plain-run records too. README rewritten around the shrunk model. --- README.md | 58 +- src/deb/ephemeral.rs | 341 +++++----- src/deb/local.rs | 82 +-- src/deb/mod.rs | 301 +++++---- src/deb/session.rs | 1399 ++++++++++++++++++++++++------------------ src/main.rs | 41 +- src/prune.rs | 11 +- 7 files changed, 1287 insertions(+), 946 deletions(-) diff --git a/README.md b/README.md index c88c953..5e293af 100644 --- a/README.md +++ b/README.md @@ -122,44 +122,56 @@ at the oldest commit searched, which keeps the run fast on large histories. ### Incremental builds (build records) -Every `pkh deb` build runs in its own immutable **record** under -`/var/tmp/pkh/sessions///`: the bootstrapped chroot, the -installed build dependencies and the build artifacts of the staged -tree. `--incremental` forks a previous record — a full copy, the base -is never modified — so a failed run can never destroy the last-good -environment it was built from: +Every `pkh deb` build records its metadata under +`/var/tmp/pkh/sessions///`: the manifest (identity, fork +lineage, outcome, the sha of the chroot tarball it used) and a full +`log/build.log` tee of the build output. The chroot itself is per-run +scratch — extracted fresh from the sha-verified mmdebstrap tarball cache +on every run and removed at the end, plain and `--incremental` alike — +so records stay small. What `--incremental` buys is the record's +**staged tree** (`tree-upper/`): the overlay upperdir where the build's +writes land, holding the compiled objects. + +`--incremental` forks a previous record — a copy, the base is never +modified — so a failed run can never destroy the last-good state it was +built from. The staged tree is carried only when the base's recorded +tarball sha still matches the cache (the toolchain did not move) and the +changelog version is unchanged; otherwise the fork builds from a fresh +tree: ``` -pkh deb # fully ephemeral: nothing is kept, even on - # failure or interrupt +pkh deb # plain run: fresh chroot, fresh tree; the + # record keeps only metadata and the log + # (nothing is kept even on failure or + # interrupt) pkh deb --incremental # fork the newest successful record of the - # target (last five usable): the chroot, build - # deps and, at the same version, the objects - # are reused; only what changed is recompiled. - # The result is always kept (success or - # failure) for the next fork + # target (among the last five carrying a staged + # tree) and keep this run's staged tree, for + # the next fork to carry pkh deb --incremental # fork that exact record (see `pkh deb list`); - # a failed record is fine if its environment is + # a failed record is fine if its staged tree is pkh deb list # the records of this tree: id, outcome, - # version, fork lineage, age + # version, fork lineage, staged-tree state, age +pkh deb log [-f] [] # print a record's build log; -f follows a + # live build pkh prune # garbage-collect records (7-day retention) ``` -Each target (series/arch) keeps its last five records — older ones are -pruned as new ones are created, and `pkh prune` removes those untouched -for a week. `pkh deb --incremental` never forks a record built for a -different series/architecture, and a plain `pkh deb` still never -reuses anything: it builds in a throwaway record that is discarded on -success, failure and interrupt alike. +Each target keeps at most five staged trees: when a sixth is retained, +the oldest degrades to a metadata-only record (manifest and log stay), +and `pkh prune` removes records untouched for a week. `pkh deb +--incremental` never forks a record built for a different +series/architecture. Packages whose `debian/rules` keeps an input-untracked stamp cache (the kernels' `debian/stamps`) declare it in the quirks data (`resume_clear`): forked builds drop the cache so the wrapper steps — including the kernel's flavour config export — re-run over the incremental inner builds, and a config edit is never silently ignored. -If a build died hard (OOM kill), the next fork unmounts the leftover -state first (escalating through sudo when needed). +A build killed hard (OOM) can leave its chroot behind; the next run of +that target sweeps it (unmounting and escalating through sudo when +needed), and `pkh prune` collects the rest. ### Running the DEP-8 tests diff --git a/src/deb/ephemeral.rs b/src/deb/ephemeral.rs index 27e834e..ecbea69 100644 --- a/src/deb/ephemeral.rs +++ b/src/deb/ephemeral.rs @@ -13,7 +13,7 @@ use tar::Archive; use xz2::read::XzDecoder; // --------------------------------------------------------------------------- -// Interrupt-time chroot cleanup +// Interrupt-time cleanup // // On Ctrl-C, the watchdog in `crate::interrupt` runs the hook registered in // [`EphemeralContextGuard::new_with_context`] right before exiting — the @@ -22,34 +22,61 @@ use xz2::read::XzDecoder; // /proc and any overlayfs mounts. // --------------------------------------------------------------------------- -/// Interrupt-time teardown of a build tree. When `keep_tree` is set the -/// tree itself is left in place after unmounting (an `--incremental` -/// record: the interrupt is a pause, not a discard); otherwise this is -/// the historical full cleanup. For a plain `pkh deb` the `path` is the -/// whole throwaway record root, so the record is removed — the record -/// model's keep-iff-incremental policy applies to interrupts too. +/// Interrupt-time teardown of an anonymous temp chroot (no build record): +/// unmount everything under it, then remove it — a plain `rm -rf` with a +/// retry loop, best-effort. See [`sigint_cleanup_record`] for records. +fn sigint_cleanup_chroot(chroot_path: &Path) { + let is_root = unsafe { libc::geteuid() } == 0; + unmount_under_logged(chroot_path); + remove_with_retries(chroot_path, is_root); +} + +/// Interrupt-time teardown of a build record: unmount everything under +/// the record root, remove the per-run state (the chroot and the overlay +/// scratch workdir) and — unless `keep_upper` is set — the staged tree +/// and its host-tree snapshot. The manifest and the build log always +/// stay: every build leaves a metadata record, the interrupt included. /// /// Unlike [`EphemeralContextGuard::drop`], this deliberately does NOT go /// through the context manager, the ephemeral context's driver (whose -/// `cleanup()` unmounts the tracked overlays) or the base context's command -/// builder: interrupt-time hooks must be self-contained, and those -/// machineries may be mid-mutation on the interrupted thread. Instead it -/// only reads /proc/mounts and spawns umount/rm directly. -/// -/// On a full cleanup it removes the tree regardless of the build result: -/// the build was aborted, and leaving a still-mounted chroot behind is -/// exactly the leak this hook exists to prevent. +/// `cleanup()` unmounts the tracked overlays) or the base context's +/// command builder: interrupt-time hooks must be self-contained, and +/// those machineries may be mid-mutation on the interrupted thread. +/// Instead it only reads /proc/mounts and spawns umount/rm directly. /// /// Best-effort by design: if a child process still holds a mount busy or -/// privilege escalation is unavailable, individual steps fail; failures are -/// logged (pointing at `pkh prune` for the leftovers) and never panic. -fn sigint_cleanup_chroot(chroot_path: &Path, keep_tree: bool) { +/// privilege escalation is unavailable, individual steps fail; failures +/// are logged (pointing at `pkh prune` for the leftovers) and never +/// panic. +fn sigint_cleanup_record(record_root: &Path, keep_upper: bool) { let is_root = unsafe { libc::geteuid() } == 0; + unmount_under_logged(record_root); + // Per-run state first: the chroot holds the /proc bind mount and the + // device nodes, the workdir holds root-owned overlay scratch. + for dir in ["chroot", "tree-upper-work"] { + remove_with_retries(&record_root.join(dir), is_root); + } + if keep_upper { + log::debug!( + "Interrupted build record kept its staged tree at {} (fork it \ + with `pkh deb --incremental`)", + record_root.display() + ); + } else { + for path in ["tree-upper", "host-files.list"] { + remove_with_retries(&record_root.join(path), is_root); + } + } +} - // Unmount children before parents: /proc/mounts lists mounts roughly in - // creation order, so walk it in reverse - let mounts = host_mounts_under(chroot_path); - for mount_point in mounts.into_iter().rev() { +/// Unmount every mount at or under `base`, deepest first, logging the +/// failures (interrupt-time: leftover mounts are `pkh prune`'s to +/// collect). +fn unmount_under_logged(base: &Path) { + let is_root = unsafe { libc::geteuid() } == 0; + // Unmount children before parents: /proc/mounts lists mounts roughly + // in creation order, so walk it in reverse. + for mount_point in host_mounts_under(base).into_iter().rev() { if unmount_path(&mount_point, is_root) { log::debug!( "Unmounted {} during interrupt cleanup", @@ -63,19 +90,13 @@ fn sigint_cleanup_chroot(chroot_path: &Path, keep_tree: bool) { ); } } +} - if keep_tree { - log::debug!( - "Interrupted build record kept at {} (fork it with `pkh deb --incremental`)", - chroot_path.display() - ); - return; - } - - // Remove the chroot tree itself (tolerates a missing directory). A - // child the Ctrl+C interrupted may still be finishing its writeout — - // dpkg defers SIGINT until it reaches a safe state — so retry while rm - // reports the tree non-empty instead of leaving it half-removed. +/// Privileged `rm -rf` with a retry loop: a child the Ctrl+C interrupted +/// may still be finishing its writeout — dpkg defers SIGINT until it +/// reaches a safe state — so retry while rm reports the tree non-empty +/// instead of leaving it half-removed. A missing path succeeds at once. +fn remove_with_retries(path: &Path, is_root: bool) { const RETRIES: usize = 10; const RETRY_DELAY: std::time::Duration = std::time::Duration::from_millis(300); let mut last = None; @@ -86,7 +107,7 @@ fn sigint_cleanup_chroot(chroot_path: &Path, keep_tree: bool) { last = Some( privileged_command("rm", is_root) .arg("-rf") - .arg(chroot_path) + .arg(path) .status(), ); if matches!(&last, Some(Ok(status)) if status.success()) { @@ -95,22 +116,19 @@ fn sigint_cleanup_chroot(chroot_path: &Path, keep_tree: bool) { } match last { Some(Ok(status)) if status.success() => { - log::debug!( - "Removed chroot {} during interrupt cleanup", - chroot_path.display() - ); + log::debug!("Removed {} during interrupt cleanup", path.display()); } Some(Ok(status)) => { log::error!( - "Failed to remove chroot {} during interrupt cleanup \ + "Failed to remove {} during interrupt cleanup \ (rm exited with {status}); run `pkh prune`", - chroot_path.display() + path.display() ); } Some(Err(e)) => { log::error!( - "Failed to run rm for chroot {} during interrupt cleanup: {e}; run `pkh prune`", - chroot_path.display() + "Failed to run rm for {} during interrupt cleanup: {e}; run `pkh prune`", + path.display() ); } None => unreachable!("at least one rm attempt ran"), @@ -305,10 +323,11 @@ fn unescape_mount_field(field: &str) -> String { /// for building packages with unshare permissions. /// /// In session mode ([`EphemeralContextGuard::new_for_session`]) the chroot -/// lives inside a [`Session`] record: a forked record's chroot is reused -/// when the caller verified it, and the keep policy is strict — the record -/// is kept exactly for `--incremental` runs (`persist_record`), discarded -/// for plain ones whatever the outcome. +/// lives inside a [`Session`] record's `chroot/` directory — extracted +/// fresh from the tarball cache on every run (nothing chroot-shaped is +/// persisted) and removed at the end of the run. The record itself is +/// always kept as metadata and log; its staged tree is retained exactly +/// for `--incremental` runs (`keep_upper`). pub struct EphemeralContextGuard { /// The ephemeral build context this guard created (an unshare context /// bound to the chroot, parented on the base context). Held explicitly so @@ -327,8 +346,9 @@ pub struct EphemeralContextGuard { cleanup_hook: Option, /// The build record this guard works in, when any. session: Option, - /// Whether the record is kept after the build (`pkh deb --incremental`). - persist_record: bool, + /// Whether the record's staged tree is retained after the build + /// (`pkh deb --incremental`). + keep_upper: bool, } impl EphemeralContextGuard { @@ -345,7 +365,7 @@ impl EphemeralContextGuard { base_ctx: Arc, view: &dyn BuildView, ) -> Result> { - Self::new_internal(None, series, arch, base_ctx, view, false, false).await + Self::new_internal(None, series, arch, base_ctx, view, false).await } /// Create the ephemeral build context inside `session`'s chroot. @@ -357,31 +377,18 @@ impl EphemeralContextGuard { /// * `arch` - Optional target architecture, as for /// [`EphemeralContextGuard::new_with_context`] /// * `base_ctx` - The base context to use for chroot work - /// * `reuse_chroot` - Reuse the forked record's bootstrapped chroot - /// instead of re-extracting the tarball; the caller must only set - /// this after the record's chroot passed the integrity probe - /// * `persist_record` - Keep the record after the build - /// (`pkh deb --incremental`); a plain run's record is discarded - /// whatever the outcome + /// * `keep_upper` - Retain the record's staged tree after the build + /// (`pkh deb --incremental`); a plain run's record keeps metadata + /// and log only pub async fn new_for_session( session: Session, series: &str, arch: Option<&str>, base_ctx: Arc, view: &dyn BuildView, - reuse_chroot: bool, - persist_record: bool, + keep_upper: bool, ) -> Result> { - Self::new_internal( - Some(session), - series, - arch, - base_ctx, - view, - reuse_chroot, - persist_record, - ) - .await + Self::new_internal(Some(session), series, arch, base_ctx, view, keep_upper).await } async fn new_internal( @@ -390,8 +397,7 @@ impl EphemeralContextGuard { arch: Option<&str>, base_ctx: Arc, view: &dyn BuildView, - reuse_chroot: bool, - persist_record: bool, + keep_upper: bool, ) -> Result> { // Save the globally-installed context so Drop can restore exactly // this handle: concurrent builds install their own ephemeral @@ -399,8 +405,8 @@ impl EphemeralContextGuard { // before this guard swapped anything in. let previous_context = context::current(); - // In session mode the chroot lives inside the record root (and is - // reused when the caller says so); otherwise it is a fresh temp dir. + // In session mode the chroot lives inside the record root, as + // per-run scratch; otherwise it is a fresh temp dir. let chroot_path = match &session { Some(s) => { let path = s.chroot_path(); @@ -424,22 +430,23 @@ impl EphemeralContextGuard { // base context: the hook must be self-contained (stored path + // direct umount/rm subprocesses) and cannot go through `base_ctx`. // For remote or nested bases the chroot lives elsewhere, and - // leftovers stay handled by `pkh prune` as before. An - // `--incremental` record is kept on interrupt (a pause, not a - // discard); a plain run's whole record root and a plain temp chroot - // are removed. + // leftovers stay handled by `pkh prune` as before. + // + // Session mode tears the record's per-run state down (chroot and + // overlay workdir) and keeps its staged tree exactly for + // `--incremental`; a plain temp chroot is removed entirely. let cleanup_hook = if matches!(base_ctx.config, ContextConfig::Local) { - Some(crate::interrupt::register_cleanup_hook(Box::new({ - // Session mode tears the whole record root down (the - // manifest is part of the record); plain temp chroots - // only ever had the chroot dir. - let cleanup_path = match &session { - Some(s) => s.root().to_path_buf(), - None => chroot_path.clone(), - }; - let keep_tree = persist_record; - move || sigint_cleanup_chroot(&cleanup_path, keep_tree) - }))) + let hook: Box = match session.as_ref().map(|s| s.root().to_path_buf()) + { + Some(record_root) => { + Box::new(move || sigint_cleanup_record(&record_root, keep_upper)) + } + None => { + let chroot_path = chroot_path.clone(); + Box::new(move || sigint_cleanup_chroot(&chroot_path)) + } + }; + Some(crate::interrupt::register_cleanup_hook(hook)) } else { log::debug!( "Base context is not local; skipping interrupt-time cleanup registration for {}", @@ -448,18 +455,11 @@ impl EphemeralContextGuard { None }; - // Download and extract the chroot tarball (or reuse the record's - // prepared chroot) - if let Err(e) = Self::download_and_extract_chroot( - series, - arch, - &chroot_path, - base_ctx.clone(), - view, - session.as_ref(), - reuse_chroot, - ) - .await + // Download and extract the chroot tarball — always fresh; the + // record's chroot directory is per-run scratch. + if let Err(e) = + Self::download_and_extract_chroot(series, arch, &chroot_path, base_ctx.clone(), view) + .await { // On a Ctrl+C the interrupt watchdog owns the tree: keep the // hook registered (forgetting the guard) so it removes the @@ -476,6 +476,17 @@ impl EphemeralContextGuard { return Err(e); } + // Record the tarball digest this environment came from: the fork + // gate of a later `--incremental` run compares it against the + // cache to decide whether the record's staged tree is still + // reusable. + if let Some(s) = &session + && let Ok(tarball) = chroot_tarball_path(series, arch) + && let Ok(sha) = session::sha256_file(&tarball) + { + s.set_tarball_sha256(&sha); + } + // Switch to an ephemeral context to build the package in the chroot. // The parent is the base context itself (the one that bootstrapped // the chroot), wired through `with_parent` instead of a config-name @@ -502,7 +513,7 @@ impl EphemeralContextGuard { base_ctx, cleanup_hook, session, - persist_record, + keep_upper, }) } @@ -522,25 +533,10 @@ impl EphemeralContextGuard { chroot_path: &PathBuf, ctx: Arc, view: &dyn BuildView, - session: Option<&Session>, - reuse_chroot: bool, ) -> Result<(), Box> { - // Record fork: the base's chroot passed the caller's checks — - // skip the tarball extraction entirely, the environment (apt - // lists, installed build-deps) is still in there. - if let Some(s) = session { - if reuse_chroot && s.chroot_is_usable() { - log::info!( - "Reusing the prepared chroot of record {} (bootstrap skipped)", - s.manifest().id - ); - Self::finalize_chroot(chroot_path, ctx, view)?; - return Ok(()); - } - // Not reusing (fresh record or failed probe): never extract - // over stale content. - clear_dir(chroot_path)?; - } + // The chroot is per-run scratch: nothing is reused, so never + // extract over stale content. + clear_dir(chroot_path)?; // Clone ctx for use in create_device_nodes after download_chroot_tarball consumes it let ctx_for_devices = ctx.clone(); @@ -619,33 +615,13 @@ impl EphemeralContextGuard { // Create device nodes in the chroot and bind-mount /proc Self::finalize_chroot(chroot_path, ctx_for_devices, view)?; - // Record the bootstrap in the record: marker file (the integrity - // probe) and the tarball the chroot came from. - if let Some(s) = session { - std::fs::write(chroot_path.join(".pkh-ready"), b"")?; - let metadata = fs::metadata(&tarball_path)?; - let tarball_sha256 = session::sha256_file(&tarball_path) - .map_err(|e| format!("cannot hash the chroot tarball: {e}"))?; - s.set_chroot(session::ChrootInfo { - tarball: tarball_filename, - tarball_size: metadata.len(), - tarball_mtime_secs: metadata - .modified() - .ok() - .and_then(|t| t.duration_since(std::time::UNIX_EPOCH).ok()) - .map(|d| d.as_secs()) - .unwrap_or(0), - tarball_sha256, - ready: true, - }); - log::debug!("Record {} chroot is ready", s.manifest().id); - } - + // The bootstrap's tarball digest is recorded by the caller (the + // record's fork gate compares it against the cache later). Ok(()) } - /// Device nodes and the /proc bind mount for a (possibly reused) - /// chroot: idempotent finalization, safe to run on every attempt. + /// Device nodes and the /proc bind mount for a fresh chroot: + /// idempotent finalization, safe to run on every run. fn finalize_chroot( chroot_path: &Path, ctx: Arc, @@ -660,8 +636,8 @@ impl EphemeralContextGuard { // Bind mount /proc from host into chroot (before entering unshare namespace) // This allows /proc to work in containers where mounting inside unshare fails. - // A reused chroot may still carry the mount from the previous - // attempt: only mount when it is not already there. + // A chroot reused within the same run may still carry the mount: + // only mount when it is not already there. if is_mountpoint(&chroot_path.join("proc")) { log::debug!("/proc is already bind-mounted in the chroot"); } else { @@ -908,6 +884,11 @@ impl Drop for EphemeralContextGuard { // drained, forgetting is a harmless no-op). if crate::interrupt::interrupted() { context::manager().set_current_ephemeral(self.previous_context.clone()); + // The record keeps its metadata: mark the aborted build as + // failed before the hook tears the per-run state down. + if let Some(session) = &self.session { + session.set_outcome(session::OUTCOME_FAILED); + } if let Some(hook) = self.cleanup_hook.take() { std::mem::forget(hook); } @@ -942,11 +923,11 @@ impl Drop for EphemeralContextGuard { // touches it. context::manager().set_current_ephemeral(self.previous_context.clone()); - // Session mode: the tree is a build record. The /proc bind mount - // must come down in every case (a kept record has to be - // unmount-clean for the next fork); the record itself follows the - // keep policy — kept exactly for `--incremental` runs, discarded - // for plain ones whatever the outcome. + // Session mode: the record's per-run chroot comes down in every + // case (nothing chroot-shaped is persisted — success, failure and + // interrupt all end the run), then the record is retained: + // metadata and log always, the staged tree exactly for + // `--incremental` runs. if let Some(session) = &self.session { let proc_path = self.chroot_path.join("proc"); let is_root = crate::utils::root::is_root().unwrap_or(false); @@ -960,7 +941,8 @@ impl Drop for EphemeralContextGuard { .status() }; - session.dispose_after_build(self.persist_record, self.build_succeeded); + session.remove_run_state(); + session.retain_upper(self.keep_upper, self.build_succeeded); return; } @@ -1058,7 +1040,7 @@ mod chroot_cleanup_tests { fn sigint_cleanup_of_missing_chroot_is_a_noop() { let dir = tempfile::tempdir().unwrap(); let missing = dir.path().join("no-such-chroot"); - sigint_cleanup_chroot(&missing, false); + sigint_cleanup_chroot(&missing); assert!(!missing.exists()); } @@ -1081,24 +1063,61 @@ mod chroot_cleanup_tests { std::fs::create_dir_all(chroot.join("rootfs")).unwrap(); std::fs::write(chroot.join("rootfs").join("file.txt"), "data").unwrap(); - sigint_cleanup_chroot(&chroot, false); + sigint_cleanup_chroot(&chroot); assert!(!chroot.exists()); } - /// The interrupt teardown of a kept tree (an `--incremental` record) - /// unmounts and leaves the tree in place — the pause, not the discard. + /// The record interrupt teardown removes the per-run state and keeps + /// the manifest and log; an `--incremental` record keeps its staged + /// tree, a plain one loses it. #[test] - fn sigint_cleanup_keeps_the_tree_when_asked() { + fn sigint_record_teardown_keeps_metadata_and_drops_the_chroot() { + let is_root = unsafe { libc::geteuid() } == 0; + if !is_root + && !privileged_command("true", false) + .status() + .is_ok_and(|s| s.success()) + { + return; + } + let dir = tempfile::tempdir().unwrap(); - let record_root = dir.path().join("record"); - std::fs::create_dir_all(record_root.join("chroot/rootfs")).unwrap(); - std::fs::write(record_root.join("session.json"), "{}").unwrap(); + let incremental = dir.path().join("incremental"); + let plain = dir.path().join("plain"); + for root in [&incremental, &plain] { + std::fs::create_dir_all(root.join("chroot/rootfs")).unwrap(); + std::fs::create_dir_all(root.join("tree-upper/src")).unwrap(); + std::fs::create_dir_all(root.join("log")).unwrap(); + std::fs::write(root.join("manifest.json"), "{}").unwrap(); + std::fs::write(root.join("log/build.log"), "output\n").unwrap(); + std::fs::write(root.join("host-files.list"), "f\t1\t1\ta\n").unwrap(); + } - sigint_cleanup_chroot(&record_root, true); + sigint_cleanup_record(&incremental, true); + sigint_cleanup_record(&plain, false); - assert!(record_root.join("chroot/rootfs").exists()); - assert!(record_root.join("session.json").is_file()); + for root in [&incremental, &plain] { + assert!(!root.join("chroot").exists(), "the chroot is per-run"); + assert!(root.join("manifest.json").is_file(), "metadata stays"); + assert!(root.join("log/build.log").is_file(), "the log stays"); + } + assert!( + incremental.join("tree-upper/src").exists(), + "an interrupted incremental run keeps its staged tree" + ); + assert!( + incremental.join("host-files.list").is_file(), + "the snapshot stays with the staged tree" + ); + assert!( + !plain.join("tree-upper").exists(), + "an interrupted plain run drops its staged tree" + ); + assert!( + !plain.join("host-files.list").exists(), + "the snapshot goes with the staged tree" + ); } /// clear_dir removes content a plain `rm -rf` cannot: a root-owned diff --git a/src/deb/local.rs b/src/deb/local.rs index da01552..636647e 100644 --- a/src/deb/local.rs +++ b/src/deb/local.rs @@ -27,35 +27,12 @@ fn cap<'a>( cmd } -/// The session journal stamps this build works with: the input stamps of -/// the patch and build-dependency phases, computed from the host tree and -/// the resolved selectors. -struct JournalStamps { - patches: Option, - control: Option, -} - -/// Compute the journal stamps for a session build; `None` without one. -fn journal_stamps( - session: &session::Session, - arch: &str, - series: &str, - pocket: Option<&str>, - cross: bool, - ppa: &[String], - inject_packages: &[String], -) -> JournalStamps { +/// The input stamp of the patch phase, computed from the host tree: the +/// only tree phase the record journal tracks (the environment phases +/// rerun in every fresh chroot and are not recorded). +fn patch_stamp(session: &session::Session) -> Option { let host_tree = PathBuf::from(session.manifest().host_tree); - let selectors = format!( - "{arch}\n{cross}\n{series}\n{}\n{}\n{}", - pocket.unwrap_or(""), - ppa.join(","), - inject_packages.join(",") - ); - JournalStamps { - patches: session::patches_stamp(&host_tree), - control: session::control_stamp(&host_tree, &selectors), - } + session::patches_stamp(&host_tree) } #[allow(clippy::too_many_arguments)] @@ -74,13 +51,12 @@ pub async fn build( view: &dyn BuildView, jobs: Option, session: Option<&session::Session>, - resume: bool, + build_resume: bool, ) -> Result, Box> { let sink: Option> = view.sink(); - // Journal stamps for a session build; recording is a no-op without one. - let stamps = - session.map(|s| journal_stamps(s, arch, series, pocket, cross, ppa, inject_packages)); + // Journal stamp for a record build; recording is a no-op without one. + let stamps = session.and_then(patch_stamp); // Environment let mut env = HashMap::::new(); @@ -245,9 +221,6 @@ pub async fn build( or re-run with RUST_LOG=debug for more details." .into()); } - if let Some(s) = session { - s.record_phase("apt_update", None); - } // Install essential packages log::debug!("Installing essential packages for local build..."); @@ -272,9 +245,6 @@ pub async fn build( if !status.success() { return Err("Could not install essential packages for the build".into()); } - if let Some(s) = session { - s.record_phase("essentials", None); - } // The package directory was resolved by the caller (the staged copy of // the tree the user pointed at, or the name-pattern search fallback) @@ -294,10 +264,11 @@ pub async fn build( } // Apply quilt patches if the package provides a patch series. On a - // resumed session the previous attempt's patches may still be applied: - // pop them first when the patch tree changed since. - let patch_stamp = stamps.as_ref().and_then(|s| s.patches.clone()); - let pop_patches = resume && { + // build resuming over a carried upperdir the previous attempt's + // patches may still be applied: pop them first when the patch tree + // changed since. + let patch_stamp = stamps.clone(); + let pop_patches = build_resume && { let previous = session .and_then(|s| s.manifest().phases.get("patches").map(|p| p.stamp.clone())) .unwrap_or(None); @@ -333,20 +304,17 @@ pub async fn build( view, &sink, )?; - if let Some(s) = session { - s.record_phase( - "build_deps", - stamps.as_ref().and_then(|st| st.control.clone()), - ); - } + // (The build-dependency phase reruns in every fresh chroot: nothing + // to journal.) - // A resumed build continues in the previous attempt's tree. Packages - // whose debian/rules keeps an input-untracked stamp cache (quirk: - // `resume_clear`, e.g. the kernels' `debian/stamps`, whose flavour - // config rule exports .config with no config prerequisite) would build - // from the previous attempt's configuration: drop the listed caches so - // the wrapper steps re-run over the incremental inner builds. - if resume { + // A build resuming over a carried upperdir continues in the previous + // attempt's tree. Packages whose `debian/rules` keeps an + // input-untracked stamp cache (quirk: `resume_clear`, e.g. the + // kernels' `debian/stamps`, whose flavour config rule exports .config + // with no config prerequisite) would build from the previous + // attempt's configuration: drop the listed caches so the wrapper + // steps re-run over the incremental inner builds. + if build_resume { for rel in crate::quirks::get_resume_clear_paths(package, series) { let path = Path::new(package_dir_str).join(&rel); if !ctx.exists(&path).unwrap_or(false) { @@ -383,10 +351,10 @@ pub async fn build( s.record_phase("build", None); } - // On a resumed session the previous attempt's debian/files would make + // On a resumed build the previous attempt's debian/files would make // the artifact collection below surface its outputs: only the new // attempt's registration may remain. - if resume { + if build_resume { let _ = ctx .command("rm") .arg("-f") diff --git a/src/deb/mod.rs b/src/deb/mod.rs index eec6e32..db8d201 100644 --- a/src/deb/mod.rs +++ b/src/deb/mod.rs @@ -119,36 +119,87 @@ pub struct DebBuildOptions<'a> { /// Explicit build context; defaults to the current context. pub ctx: Option>, /// Incremental request (`pkh deb --incremental []`): which - /// record to fork as the base of this build, and whether this - /// build's own record is kept. Defaults to a plain throwaway run. + /// record to fork as the base of this build's staged tree, and + /// whether this build's own staged tree is retained. Defaults to a + /// plain run. pub incremental: Incremental, /// Where build events (phases, progress, outcome) are reported. pub view: &'a dyn BuildView, } /// The `--incremental` request of a `pkh deb` build: which record the -/// build forks its environment from, and whether its own record is -/// kept afterwards. +/// build forks its staged tree from, and whether its own staged tree is +/// retained afterwards (the record's metadata and log are kept for +/// every build either way). #[derive(Debug, Clone, Default, PartialEq, Eq)] pub enum Incremental { /// A plain run (`pkh deb` without `--incremental`): build from - /// scratch in a throwaway record, discarded on success, failure - /// and interrupt alike. + /// scratch in a record whose staged tree is dropped at the end — + /// metadata and log only remain. #[default] Off, /// `--incremental`: fork the auto-selected base — the newest - /// successful record of this identity among the last five usable — - /// and always keep the new record. With no usable candidate the - /// build runs from scratch and its record becomes the identity's - /// first base. + /// successful record of this identity among the last five carrying + /// a staged tree — and retain the new record's staged tree. With no + /// candidate the build runs from scratch and its record becomes + /// the identity's first base. Auto, /// `--incremental `: fork that exact record (unique id or /// prefix, restricted to the records of this host tree; a FAILED - /// record is a valid base — its environment may be fine, and the - /// user pinned it deliberately) and always keep the new record. + /// record is a valid base — the user pinned it deliberately) and + /// retain the new record's staged tree. Id(String), } +/// A view delegating every event to `inner` while teeing all captured +/// subprocess lines into a build record's log through [`RecordLogSink`]: +/// everything downstream (the chroot bootstrap, the build phases) asks +/// the view for its sink, so one wrapper tees the whole run. +struct RecordLogView<'a> { + /// The caller's view. + inner: &'a dyn BuildView, + /// The record log sink, forwarding to the inner view's sink too. + log: Arc, +} + +impl BuildView for RecordLogView<'_> { + fn target(&self, target: crate::report::BuildTarget) { + self.inner.target(target); + } + + fn phase(&self, name: &str, classifier: Box) { + self.inner.phase(name, classifier); + } + + fn message(&self, text: &str) { + self.inner.message(text); + } + + fn progress(&self, label: &str, pos: usize, total: usize) { + self.inner.progress(label, pos, total); + } + + fn sink(&self) -> Option> { + Some(self.log.clone()) + } + + fn finish_success(&self, artifacts: &[PathBuf]) { + self.inner.finish_success(artifacts); + } + + fn finish_failure(&self) { + self.inner.finish_failure(); + } + + fn suspend(&self) { + self.inner.suspend(); + } + + fn is_enabled(&self) -> bool { + self.inner.is_enabled() + } +} + impl Default for DebBuildOptions<'_> { fn default() -> Self { static QUIET: crate::report::Quiet = crate::report::Quiet; @@ -281,7 +332,7 @@ async fn build_binary_package_impl( cross, }; - let mut persist_record = false; + let mut keep_upper = false; let session = if session_enabled { match resolve_record( incremental, @@ -291,8 +342,8 @@ async fn build_binary_package_impl( series, chroot_arch, ) { - Ok((record, persist)) => { - persist_record = persist; + Ok((record, keep)) => { + keep_upper = keep; Some(record) } Err(e) => { @@ -303,9 +354,26 @@ async fn build_binary_package_impl( } else { None }; - // Only a forked record carries a bootstrapped chroot at this point; - // fresh ones (plain runs and first `--incremental` runs) extract it. - let resume_chroot = session.as_ref().is_some_and(|s| s.chroot_is_usable()); + + // Tee every captured build line into the record's log: the wrapper + // view hands the combined sink to everything downstream (the chroot + // bootstrap, the build phases), so the log is a full tee whatever + // renders the output. A record-less build (non-local base context) + // keeps the plain view. + let record_log: Option, std::io::Error>> = session + .as_ref() + .map(|s| session::RecordLogSink::new(s, opts.view.sink()).map(Arc::new)); + let view: &dyn BuildView = match record_log { + Some(Ok(log)) => &RecordLogView { + inner: opts.view, + log, + }, + Some(Err(e)) => { + log::warn!("Cannot open the build record's log: {e}"); + opts.view + } + None => opts.view, + }; let mut guard = if let Some(s) = session.clone() { Some( @@ -315,8 +383,7 @@ async fn build_binary_package_impl( chroot_arch, base_ctx.clone(), view, - resume_chroot, - persist_record, + keep_upper, ) .await?, ) @@ -454,7 +521,7 @@ async fn build_binary_package_impl( view, *jobs, session.as_ref(), - resume_chroot, + build_resume, ) .await? } @@ -486,19 +553,17 @@ async fn build_binary_package_impl( // the chroot) inherit the terminal, so they must not fight the widget. view.suspend(); - // Mark the build as successful for the guard's drop, which disposes of - // the record per the keep policy: an `--incremental` record is kept - // with its outcome recorded, a plain run's throwaway record is - // always discarded. + // Mark the build as successful for the guard's drop, which tears the + // per-run chroot down and retains the record: metadata and log for + // every build, the staged tree exactly for `--incremental` runs. if result.is_ok() && let Some(g) = &mut guard { g.mark_build_successful(); } - // A failed `--incremental` run keeps its record for a later fork: - // record the outcome (discarded plain runs need none). + // Every record's outcome is written — plain runs leave a metadata + // record too. if result.is_err() - && persist_record && let Some(s) = &session { s.set_outcome(session::OUTCOME_FAILED); @@ -518,10 +583,9 @@ fn resolve_record( series: &str, chroot_arch: Option<&str>, ) -> Result<(session::Session, bool), Box> { - let expected = ephemeral::chroot_tarball_path(series, chroot_arch).ok(); resolve_record_in( &session::sessions_root(), - expected.as_deref(), + &session::cache_tarball_sha256(series, chroot_arch), incremental, host_tree, identity, @@ -530,30 +594,22 @@ fn resolve_record( } /// The testable core of [`resolve_record`] over an explicit sessions root -/// and bootstrap tarball. +/// and cached-tarball digest (the fork gate's reference). /// /// Every local build runs in its own record: [`Incremental::Off`] gets a -/// brand-new throwaway one, [`Incremental::Auto`] and -/// [`Incremental::Id`] fork the selected base. The returned flag says -/// whether the record is kept after the build — exactly the -/// `--incremental` runs keep theirs. -/// -/// A base must be usable: its chroot passes the integrity probe and was -/// bootstrapped from the tarball this build would use (`expected_tarball` -/// is `None` when the path cannot be resolved — no record is usable -/// then, which degrades an `--incremental` run to a fresh build). +/// brand-new one, [`Incremental::Auto`] and [`Incremental::Id`] fork the +/// selected base. The returned flag says whether the record's staged +/// tree is retained after the build — exactly the `--incremental` runs +/// retain theirs. Metadata and the log are kept for every build either +/// way. fn resolve_record_in( records_root: &Path, - expected_tarball: Option<&Path>, + cache_tarball_sha256: &str, incremental: &Incremental, host_tree: &Path, identity: &session::SessionIdentity, version: &str, ) -> Result<(session::Session, bool), Box> { - let usable = |s: &session::Session| { - s.chroot_is_usable() - && expected_tarball.is_some_and(|t| session::tarball_matches_path(s, t)) - }; match incremental { Incremental::Off => Ok(( session::new_record_in(records_root, host_tree, identity, version)?, @@ -561,14 +617,12 @@ fn resolve_record_in( )), Incremental::Auto => { // A record locked by a live build cannot be a base: its - // chroot may be mid-mutation (apt always runs, deps land - // inside it), and the fork would copy that state. + // staged tree may be mid-mutation by the running build. let candidates: Vec = - session::identity_records_in(records_root, identity) + session::fork_candidates_in(records_root, identity) .into_iter() - .filter(|s| usable(s)) .filter(|s| !session::lock_is_held(&s.lock_path()).unwrap_or(true)) - .take(session::KEEP_RECORDS) + .take(session::KEEP_UPPERS) .collect(); for base in session::rank_base_candidates(candidates) { let why = if base.manifest().last_outcome == session::OUTCOME_SUCCESS { @@ -576,7 +630,14 @@ fn resolve_record_in( } else { "no successful record among the last usable ones; the newest record" }; - match session::fork_record_in(records_root, &base, host_tree, identity, version) { + match session::fork_record_in( + records_root, + &base, + host_tree, + identity, + version, + cache_tarball_sha256, + ) { Ok(record) => { log::info!( "Base of the incremental build: record {} ({why})", @@ -591,8 +652,8 @@ fn resolve_record_in( } } log::info!( - "No usable record of {identity} to fork; building from scratch — \ - this run's record becomes the target's first base" + "No record of {identity} carrying a staged tree to fork; building \ + from scratch — this run's record becomes the target's first base" ); Ok(( session::new_record_in(records_root, host_tree, identity, version)?, @@ -603,11 +664,10 @@ fn resolve_record_in( let mut matches: Vec = session::by_id_prefix_in(records_root, id) .into_iter() .filter(|s| s.manifest().host_tree == host_tree.to_string_lossy()) - .filter(|s| usable(s)) .collect(); match matches.len() { 0 => Err(format!( - "no usable build record matches id '{id}' for this tree \ + "no build record matches id '{id}' for this tree \ (see `pkh deb list`)" ) .into()), @@ -618,7 +678,14 @@ fn resolve_record_in( base.manifest().id ); Ok(( - session::fork_record_in(records_root, &base, host_tree, identity, version)?, + session::fork_record_in( + records_root, + &base, + host_tree, + identity, + version, + cache_tarball_sha256, + )?, true, )) } @@ -1354,30 +1421,37 @@ mod tests { // ------------------------------------------------------------------- use crate::deb::session::record_fixtures::{ - record_with_environment, sessions_fixture, test_identity, + metadata_record, record_with_upper, sessions_fixture, test_identity, }; use crate::deb::session::{self, SessionIdentity}; - /// A plain run resolves to a brand-new, lineage-free record that the - /// caller will discard — including when no record exists at all yet. + /// The sha of the fixture's cached tarball (the fork gate's + /// reference). + fn cache_sha(tarball: &std::path::Path) -> String { + session::sha256_file(tarball).unwrap() + } + + /// A plain run resolves to a brand-new, lineage-free record whose + /// staged tree will be dropped — including when no record exists at + /// all yet. #[test] fn resolve_plain_run_builds_a_fresh_discardable_record() { let (root, tarball) = sessions_fixture(); let identity = test_identity(); let host_tree = root.path().join("tree"); - let (record, persist) = resolve_record_in( + let (record, keep_upper) = resolve_record_in( root.path(), - Some(&tarball), + &cache_sha(&tarball), &Incremental::Off, &host_tree, &identity, "1.0", ) .expect("a plain run always resolves to a fresh record"); - assert!(!persist, "a plain run discards its record"); + assert!(!keep_upper, "a plain run drops its staged tree"); assert_eq!(record.manifest().base, None); - assert!(!record.chroot_is_usable()); + assert!(!record.has_upper()); } /// The user scenario, end to end over the record layout: record A @@ -1388,7 +1462,7 @@ mod tests { let (root, tarball) = sessions_fixture(); let identity = test_identity(); let host_tree = root.path().join("tree"); - let a = record_with_environment( + let a = record_with_upper( root.path(), &tarball, &host_tree, @@ -1397,56 +1471,67 @@ mod tests { Some(session::OUTCOME_SUCCESS), ); let a_id = a.manifest().id; - let b = session::fork_record_in(root.path(), &a, &host_tree, &identity, "1.0").unwrap(); + let b = session::fork_record_in( + root.path(), + &a, + &host_tree, + &identity, + "1.0", + &cache_sha(&tarball), + ) + .unwrap(); let b_id = b.manifest().id; b.set_outcome(session::OUTCOME_FAILED); drop((a, b)); - let (record, persist) = resolve_record_in( + let (record, keep_upper) = resolve_record_in( root.path(), - Some(&tarball), + &cache_sha(&tarball), &Incremental::Auto, &host_tree, &identity, "1.0", ) - .expect("the auto selection must fork a usable record"); - assert!(persist); + .expect("the auto selection must fork a candidate record"); + assert!(keep_upper); assert_eq!(record.manifest().base.as_deref(), Some(a_id.as_str())); assert_ne!(record.manifest().id, b_id); + // The successful base's gate passes: its staged tree is carried. + assert!(record.has_upper()); } /// A first `--incremental` run with no candidate builds completely - /// fresh and persists its record — it becomes the identity's first - /// base. + /// fresh and retains its staged tree — it becomes the identity's + /// first base. #[test] - fn resolve_auto_without_candidates_builds_fresh_and_persists() { + fn resolve_auto_without_candidates_builds_fresh_and_retains() { let (root, tarball) = sessions_fixture(); let identity = test_identity(); let host_tree = root.path().join("tree"); - let (record, persist) = resolve_record_in( + let (record, keep_upper) = resolve_record_in( root.path(), - Some(&tarball), + &cache_sha(&tarball), &Incremental::Auto, &host_tree, &identity, "1.0", ) .expect("no candidates must degrade to a fresh build"); - assert!(persist, "the first record becomes a base"); + assert!(keep_upper, "the first record becomes a base"); assert_eq!(record.manifest().base, None); } - /// Non-usable records never make it into the selection: a newer - /// successful record with a broken chroot loses to an older usable - /// failed one. + /// Records without a staged tree (plain runs, metadata-only ones) + /// never make it into the selection: a newer successful + /// metadata-only record loses to an older failed one carrying a + /// staged tree. #[test] - fn resolve_auto_skips_non_usable_records() { + fn resolve_auto_skips_records_without_a_staged_tree() { let (root, tarball) = sessions_fixture(); let identity = test_identity(); let host_tree = root.path().join("tree"); - let usable = record_with_environment( + let carrier = record_with_upper( root.path(), &tarball, &host_tree, @@ -1454,33 +1539,37 @@ mod tests { "1.0", Some(session::OUTCOME_FAILED), ); - let usable_id = usable.manifest().id; - drop(usable); - // Newer, successful, but its chroot never finished bootstrapping. - let broken = session::new_record_in(root.path(), &host_tree, &identity, "1.0").unwrap(); - broken.set_outcome(session::OUTCOME_SUCCESS); - drop(broken); + let carrier_id = carrier.manifest().id; + drop(carrier); + // Newer and successful, but a plain run: no staged tree. + drop(metadata_record( + root.path(), + &host_tree, + &identity, + "1.0", + session::OUTCOME_SUCCESS, + )); let (record, _) = resolve_record_in( root.path(), - Some(&tarball), + &cache_sha(&tarball), &Incremental::Auto, &host_tree, &identity, "1.0", ) - .expect("the usable record is still a base"); - assert_eq!(record.manifest().base.as_deref(), Some(usable_id.as_str())); + .expect("the record carrying a staged tree is still the base"); + assert_eq!(record.manifest().base.as_deref(), Some(carrier_id.as_str())); } - /// `--incremental ` may pin a FAILED record — its environment may - /// be fine and the user chose it deliberately. + /// `--incremental ` may pin a FAILED record — the user chose it + /// deliberately and its staged tree may be perfectly fine. #[test] fn resolve_pinned_id_forks_a_failed_record() { let (root, tarball) = sessions_fixture(); let identity = test_identity(); let host_tree = root.path().join("tree"); - let failed = record_with_environment( + let failed = record_with_upper( root.path(), &tarball, &host_tree, @@ -1491,17 +1580,18 @@ mod tests { let failed_id = failed.manifest().id; drop(failed); - let (record, persist) = resolve_record_in( + let (record, keep_upper) = resolve_record_in( root.path(), - Some(&tarball), + &cache_sha(&tarball), &Incremental::Id(failed_id.clone()), &host_tree, &identity, "1.0", ) .expect("a pinned failed record is a valid base"); - assert!(persist); + assert!(keep_upper); assert_eq!(record.manifest().base.as_deref(), Some(failed_id.as_str())); + assert!(record.has_upper(), "the gate passes: sha and version match"); } /// A pinned id is restricted to the records of this host tree: an id @@ -1512,7 +1602,7 @@ mod tests { let identity = test_identity(); let host_tree = root.path().join("tree"); let other_tree = root.path().join("elsewhere"); - let foreign = record_with_environment( + let foreign = record_with_upper( root.path(), &tarball, &other_tree, @@ -1525,7 +1615,7 @@ mod tests { let err = resolve_record_in( root.path(), - Some(&tarball), + &cache_sha(&tarball), &Incremental::Id(foreign_id), &host_tree, &identity, @@ -1533,8 +1623,7 @@ mod tests { ) .expect_err("records of other trees must not be forked by id"); assert!( - err.to_string() - .contains("no usable build record matches id"), + err.to_string().contains("no build record matches id"), "wrong error: {err}" ); } @@ -1545,7 +1634,7 @@ mod tests { let (root, tarball) = sessions_fixture(); let identity = test_identity(); let host_tree = root.path().join("tree"); - drop(record_with_environment( + drop(record_with_upper( root.path(), &tarball, &host_tree, @@ -1553,7 +1642,7 @@ mod tests { "1.0", None, )); - drop(record_with_environment( + drop(record_with_upper( root.path(), &tarball, &host_tree, @@ -1567,7 +1656,7 @@ mod tests { let err = resolve_record_in( root.path(), - Some(&tarball), + &cache_sha(&tarball), &Incremental::Id(prefix), &host_tree, &identity, @@ -1591,8 +1680,8 @@ mod tests { .collect() } - /// A pinned record of another identity is refused: its environment - /// was bootstrapped for a different target. + /// A pinned record of another identity is refused: its staged tree + /// was built for a different target. #[test] fn resolve_pinned_id_refuses_foreign_identities() { let (root, tarball) = sessions_fixture(); @@ -1602,7 +1691,7 @@ mod tests { ..identity.clone() }; let host_tree = root.path().join("tree"); - let foreign = record_with_environment( + let foreign = record_with_upper( root.path(), &tarball, &host_tree, @@ -1615,7 +1704,7 @@ mod tests { let err = resolve_record_in( root.path(), - Some(&tarball), + &cache_sha(&tarball), &Incremental::Id(foreign_id), &host_tree, &identity, diff --git a/src/deb/session.rs b/src/deb/session.rs index f2f74f1..2238277 100644 --- a/src/deb/session.rs +++ b/src/deb/session.rs @@ -1,39 +1,48 @@ -//! Immutable per-build records for `pkh deb`. +//! Per-build records for `pkh deb`. //! -//! A record is one (package, series, arch, cross) build environment plus -//! its phase journal, persisted under the sessions directory -//! (`/var/tmp/pkh/sessions///` by default). Records are -//! never mutated after their run ends: the outcome is written once at the -//! end, and a later `pkh deb --incremental` FORKS a record — a full copy -//! into a new record — instead of adopting it in place, so a failed run -//! can never destroy the last-good environment it was built from. +//! A record is one build's metadata: identity, lineage, outcome, the +//! SHA-256 of the chroot tarball its environment was extracted from, a +//! tree phase journal, the full build log, and — for `--incremental` +//! runs — the staged tree's overlay upperdir. Records live under +//! `/var/tmp/pkh/sessions///`. +//! +//! The chroot itself is per-run scratch: every build (plain and +//! `--incremental` alike) extracts a fresh one from the sha-verified +//! mmdebstrap tarball cache into the record's `chroot/` directory and +//! removes it at the end of the run — success, failure and interrupt +//! all included. Only the cheap, content-addressed state (upperdir, +//! log, metadata) is persisted; space is a hard constraint. //! //! The layout of one record root: //! //! ```text //! /var/tmp/pkh/sessions// -//! / # one immutable record per build -//! session.json # the manifest + phase journal -//! host-files.list # snapshot of the host tree at staging time -//! chroot/ # the chroot tree (marker: .pkh-ready), with the -//! # persistent overlay upperdir inside +//! / # one record per build +//! manifest.json # identity, lineage, outcome, tarball sha, journal +//! log/build.log # full tee of the build output (always) +//! tree-upper/ # the staged tree's overlay upperdir +//! # (--incremental runs only, post-run) +//! host-files.list # host-tree snapshot, while tree-upper is kept +//! chroot/ # DURING the run only; removed at end of run //! .lock # advisory lock (flock) marking a live build, //! # outside the record root it guards //! ``` //! -//! Recording is always on for local builds: a plain `pkh deb` builds in a -//! throwaway record that is discarded on success, failure and interrupt -//! alike; an `--incremental` run forks a base record (auto-selected or -//! pinned by id) and always keeps its own record for future forks. This +//! Metadata and the log are kept for EVERY build (plain runs become +//! metadata-only records); keeping the upperdir is what +//! `--incremental` buys. A later `--incremental` run forks a base +//! record: its upperdir is carried when the base's recorded tarball +//! sha still matches the cache (the toolchain did not move) and the +//! changelog version is unchanged, and never mutated in place. This //! module holds the manifest model, discovery, locking, forking, the -//! keep-last-5 pruning per identity, the host-tree snapshot used to -//! propagate host-side deletions into the reused overlay upperdir, and -//! the `pkh deb list` rendering. The base-selection policy lives in -//! [`crate::deb`]. +//! keep-last-5 upper cap, the host-tree snapshot used to propagate +//! host-side deletions into a carried upperdir, the `pkh deb list` +//! rendering and the `pkh deb log` streaming. The base-selection +//! policy lives in [`crate::deb`]. use std::collections::BTreeMap; use std::fs; -use std::io; +use std::io::{self, Seek, Write}; use std::os::unix::fs::{FileTypeExt, MetadataExt}; use std::os::unix::io::AsRawFd; use std::path::{Path, PathBuf}; @@ -43,6 +52,8 @@ use std::time::Duration; use serde::{Deserialize, Serialize}; use sha2::{Digest, Sha256}; +use crate::context::{LineSink, Stream}; + /// Manifest schema version: a record recorded by another schema is /// ignored (and eventually pruned) rather than forked. pub(crate) const SCHEMA_VERSION: u32 = 1; @@ -51,11 +62,11 @@ pub(crate) const SCHEMA_VERSION: u32 = 1; /// removes it. pub const SESSION_RETENTION: Duration = Duration::from_secs(7 * 24 * 60 * 60); -/// How many records of one identity are kept when a new record is created -/// for it: the oldest ones beyond this count are pruned right away (the -/// 7-day [`SESSION_RETENTION`] sweep still applies on top, through -/// `pkh prune`). -pub(crate) const KEEP_RECORDS: usize = 5; +/// How many records of one identity may retain their tree-upper at the +/// same time: when a newer upper-retaining record appears, the oldest +/// one beyond this count degrades to a metadata-only record (its +/// upperdir and host-tree snapshot are deleted; manifest and log stay). +pub(crate) const KEEP_UPPERS: usize = 5; /// The build is running (`last_outcome`). pub const OUTCOME_RUNNING: &str = "running"; @@ -64,22 +75,32 @@ pub const OUTCOME_SUCCESS: &str = "success"; /// The build recorded in this record failed (`last_outcome`). pub const OUTCOME_FAILED: &str = "failed"; -/// Marker file written inside the chroot once the bootstrap (extraction, -/// device nodes) fully succeeded: the integrity probe of a resume. -const CHROOT_READY_MARKER: &str = ".pkh-ready"; -/// Name of the manifest file inside a session root. -const MANIFEST_NAME: &str = "session.json"; -/// Name of the host-tree snapshot inside a session root. +/// Name of the manifest file inside a record root. +const MANIFEST_NAME: &str = "manifest.json"; +/// Name of the host-tree snapshot inside a record root. const SNAPSHOT_NAME: &str = "host-files.list"; -/// Advisory lock file suffix next to a session root (`.lock`). +/// Directory holding the build log inside a record root. +const LOG_DIR: &str = "log"; +/// Name of the build log inside [`LOG_DIR`]. +const LOG_NAME: &str = "build.log"; +/// The per-run chroot directory inside a record root (removed at end of +/// run). +const CHROOT_DIR: &str = "chroot"; +/// The persisted overlay upperdir inside a record root. +const UPPER_DIR: &str = "tree-upper"; +/// The per-run overlay workdir inside a record root (removed at end of +/// run; overlayfs state is meaningless between runs). +const WORK_DIR: &str = "tree-upper-work"; +/// Advisory lock file suffix next to a record root (`.lock`). const LOCK_NAME: &str = "lock"; -/// Stable build root inside the chroot for session builds: the overlay -/// upperdir mirrors paths relative to the merge point, so the staged tree -/// must live at the same in-chroot path on every attempt for the object -/// files to be reusable. +/// Stable build root inside the chroot for record builds: the overlay +/// upperdir mirrors paths relative to the merge point, so the staged +/// tree must live at the same in-chroot path on every attempt for the +/// object files to be reusable. pub(crate) const BUILD_ROOT: &str = "/tmp/pkh-build"; -/// What a session was built for: the identity keys the environment. +/// What a record was built for: the identity keys the environment and +/// the fork candidates. #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] pub struct SessionIdentity { /// Source package name. @@ -105,33 +126,18 @@ impl std::fmt::Display for SessionIdentity { } } -/// Recorded state of the chroot tarball the session was bootstrapped from. -#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] -pub struct ChrootInfo { - /// Tarball file name (e.g. `stonking-arm64-buildd.tar.xz`). - pub tarball: String, - /// Tarball size in bytes at bootstrap time. - pub tarball_size: u64, - /// Tarball modification time (Unix seconds) at bootstrap time. - pub tarball_mtime_secs: u64, - /// Tarball SHA-256 at bootstrap time. - pub tarball_sha256: String, - /// Whether the bootstrap (extract + device nodes) fully completed. - pub ready: bool, -} - /// One journal entry: when a phase last completed, and its input stamp. #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] pub struct PhaseStamp { /// RFC 3339 completion timestamp. pub at: String, - /// Hash of the phase inputs (control content, patch tree, ...), when - /// the phase has one. + /// Hash of the phase inputs (the patch tree), when the phase has + /// one. #[serde(default, skip_serializing_if = "Option::is_none")] pub stamp: Option, } -/// The record manifest: identity, journal and tree state. +/// The record manifest: identity, lineage, outcome and tree state. #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] pub struct SessionManifest { /// Manifest schema version ([`SCHEMA_VERSION`]). @@ -156,15 +162,26 @@ pub struct SessionManifest { /// (`--incremental` lineage; `None` when built from scratch). #[serde(default, skip_serializing_if = "Option::is_none")] pub base: Option, - /// Chroot bootstrap state. - pub chroot: ChrootInfo, - /// Phase journal: completed phases with their input stamps. + /// SHA-256 of the chroot tarball this record's environment was + /// extracted from: a fork carries the base's upperdir only while + /// this still equals the cached tarball's digest (the toolchain + /// moved otherwise, and carried objects would be stale). Empty + /// until the bootstrap completes. + #[serde(default)] + pub tarball_sha256: String, + /// Phase journal: completed tree phases with their input stamps. + /// Environment phases (apt, build-deps) rerun in every fresh chroot + /// and are not recorded. pub phases: BTreeMap, /// Changelog version of this record's build: a version change - /// between a base and its fork discards the build artifacts - /// (upperdir), not the environment. + /// between a base and its fork drops the carried upperdir (the + /// objects are of the old tree). pub tree_version: String, - /// Whether this record's build staged the tree over the reused + /// Whether this record retains its tree-upper after the run + /// (`--incremental` only). + #[serde(default)] + pub kept_upper: bool, + /// Whether this record's build staged the tree over a carried /// overlay upperdir (build-level resume was active). pub build_resume: bool, } @@ -200,20 +217,24 @@ impl Session { .clone() } - /// The chroot directory inside the record root. + /// The per-run chroot directory inside the record root: extracted + /// fresh from the tarball cache at the start of the run, removed at + /// its end. pub fn chroot_path(&self) -> PathBuf { - self.root.join("chroot") + self.root.join(CHROOT_DIR) } - /// The persistent overlay upperdir (host-side path): the build's - /// writes land here and are forked with the record. + /// The persisted overlay upperdir (host-side path): the build's + /// writes land here and are carried by a fork when the record + /// retains it. pub fn upper_dir(&self) -> PathBuf { - self.chroot_path().join("pkh-overlay/upper-build") + self.root.join(UPPER_DIR) } - /// The overlay workdir (host-side path). + /// The per-run overlay workdir (host-side path); scratch, removed at + /// the end of the run. pub fn work_dir(&self) -> PathBuf { - self.chroot_path().join("pkh-overlay/work-build") + self.root.join(WORK_DIR) } /// The host-tree snapshot file inside the record root. @@ -221,6 +242,17 @@ impl Session { self.root.join(SNAPSHOT_NAME) } + /// The build log file inside the record root. + pub fn log_file(&self) -> PathBuf { + self.root.join(LOG_DIR).join(LOG_NAME) + } + + /// Whether the record retains a tree-upper on disk (a fork + /// candidate). + pub fn has_upper(&self) -> bool { + self.upper_dir().is_dir() + } + /// The chroot-side path of the staged build root (stable across /// records, see [`BUILD_ROOT`]). pub fn build_root(&self) -> String { @@ -253,17 +285,11 @@ impl Session { }); } - /// Record the chroot bootstrap state and persist it. - pub fn set_chroot(&self, info: ChrootInfo) { + /// Record the SHA-256 of the chroot tarball this run's environment + /// was extracted from, and persist it. + pub fn set_tarball_sha256(&self, sha256: &str) { self.with_manifest(|manifest| { - manifest.chroot = info; - }); - } - - /// Record the changelog version of this record's build. - pub fn set_tree_version(&self, version: &str) { - self.with_manifest(|manifest| { - manifest.tree_version = version.to_string(); + manifest.tarball_sha256 = sha256.to_string(); }); } @@ -291,33 +317,104 @@ impl Session { } } - /// Dispose of the record at the end of a build, per the keep policy: - /// an `--incremental` run (`persist`) always keeps its record — the - /// outcome is recorded on success, failures already carry theirs — - /// while a plain run's throwaway record is always discarded, on - /// success and failure alike. The interrupt path is handled by the - /// SIGINT cleanup hook, which follows the same policy. - pub(crate) fn dispose_after_build(&self, persist: bool, succeeded: bool) { - if !persist { - log::debug!( - "Discarding the throwaway build record {}", - self.root.display() - ); - self.remove(); - return; - } + /// Retain the record at the end of its build: the outcome is + /// recorded on success (failures already carry theirs), and the + /// upperdir is kept exactly for `--incremental` runs — a plain + /// run's upperdir is dropped, leaving a metadata-only record + /// (manifest + log). The chroot was already removed by the caller's + /// teardown. The interrupt path is handled by the SIGINT cleanup + /// hook, which follows the same policy. + pub(crate) fn retain_upper(&self, keep_upper: bool, succeeded: bool) { if succeeded { self.set_outcome(OUTCOME_SUCCESS); + } + if keep_upper { + self.with_manifest(|manifest| { + manifest.kept_upper = true; + }); log::info!( - "Build record {} kept (the next --incremental run forks it)", + "Build record {} kept its staged tree (the next \ + --incremental run forks it)", self.manifest().id ); + self.enforce_upper_cap(); } else { - log::info!( - "Build record {} kept after the failed run (fork it with \ - `pkh deb --incremental `)", + log::debug!( + "Dropping the staged tree of build record {} (plain run)", self.manifest().id ); + self.wipe_upper(); + } + } + + /// Drop the record's persisted upperdir state: the tree-upper, its + /// host-tree snapshot and the scratch workdir. + fn wipe_upper(&self) { + for path in [self.upper_dir(), self.work_dir(), self.snapshot_file()] { + if let Err(e) = crate::deb::ephemeral::privileged_remove(&path) + && e.kind() != io::ErrorKind::NotFound + { + log::warn!("Failed to remove '{}': {}", path.display(), e); + } + } + } + + /// Enforce [`KEEP_UPPERS`] for this record's identity: degrade the + /// oldest upper-retaining records beyond the cap to metadata-only + /// (upperdir and snapshot deleted, manifest and log kept). Records + /// locked by a live build are never degraded. + fn enforce_upper_cap(&self) { + // The record root is //: the sessions + // root is two levels up, keeping the cap on the same root the + // record lives in (tests use fixture roots). + let Some(root) = self.root.parent().and_then(Path::parent) else { + return; + }; + let identity = self.manifest().identity; + let mut uppers: Vec = identity_records_in(root, &identity) + .into_iter() + .filter(|record| record.has_upper()) + .collect(); + uppers.sort_by(|a, b| record_ordering(a, b).reverse()); + for record in uppers.into_iter().skip(KEEP_UPPERS) { + if lock_is_held(&record.lock_path()).unwrap_or(true) { + log::debug!( + "Record {} is locked by a live build; skipping the \ + upper cap", + record.manifest().id + ); + continue; + } + log::debug!( + "Record {} degrades to metadata-only (beyond the newest \ + {KEEP_UPPERS} staged trees of {identity})", + record.manifest().id + ); + record.wipe_upper(); + record.with_manifest(|manifest| { + manifest.kept_upper = false; + }); + } + } + + /// Remove the per-run state of this record: the chroot directory and + /// the overlay scratch workdir, unmounting anything still mounted + /// under them first. Called at the end of every run (the chroot is + /// never persisted) and by the crashed-run sweep. + pub(crate) fn remove_run_state(&self) { + for path in [self.chroot_path(), self.work_dir()] { + unmount_under(&path); + if let Err(e) = crate::deb::ephemeral::privileged_remove(&path) + && e.kind() != io::ErrorKind::NotFound + { + log::warn!( + "Failed to remove the per-run state {} of build \ + record {}: {}", + path.display(), + self.manifest().id, + e + ); + } } } @@ -369,18 +466,9 @@ impl Session { self.root.with_file_name(format!("{name}.{LOCK_NAME}")) } - /// Whether the chroot of this record passes the integrity probe - /// (bootstrap completed marker + a runnable `/bin/sh`). - pub fn chroot_is_usable(&self) -> bool { - let manifest = self.manifest(); - manifest.chroot.ready - && self.chroot_path().join(CHROOT_READY_MARKER).exists() - && self.chroot_path().join("bin/sh").exists() - } - - /// Wipe the persistent overlay upperdir and workdir (used when the - /// changelog version changed: object files of the old version must not - /// leak into the new build). + /// Wipe the persisted overlay upperdir and workdir (used when no + /// host-tree snapshot exists: object files of an unknown tree must + /// not leak into the build). pub fn wipe_build_artifacts(&self) { for dir in [self.upper_dir(), self.work_dir()] { if let Err(e) = fs::remove_dir_all(&dir) @@ -659,9 +747,9 @@ fn unique_record_id(identity_dir: &Path) -> String { /// Reserve a new record directory of `identity` under `root` and lock it: /// the directory is created, its flock taken (so concurrent builds and -/// prune sweeps see it as live) and the manifest — built by `manifest` -/// from the reserved id — persisted. The loop absorbs the same-second -/// collisions with concurrent record creations. +/// prune sweeps see it as live), the manifest — built by `manifest` from +/// the reserved id — persisted, and the build log started. The loop +/// absorbs the same-second collisions with concurrent record creations. fn reserve_record( root: &Path, identity: &SessionIdentity, @@ -690,6 +778,20 @@ fn reserve_record( "cannot write the manifest of the new build record: {e}" )) })?; + // The log exists from creation on, so a build that + // fails before any captured line still leaves one. + fs::create_dir_all(session.root().join(LOG_DIR)) + .and_then(|()| { + fs::OpenOptions::new() + .create(true) + .append(true) + .open(session.log_file()) + }) + .map_err(|e| { + SessionOpenError::Other(format!( + "cannot start the build log of the new record: {e}" + )) + })?; return Ok(session); } // Someone stole this directory between its creation and the @@ -703,8 +805,9 @@ fn reserve_record( )) } -/// A manifest for a record built from scratch: no base, no journal, no -/// chroot yet — the bootstrap fills the chroot state as it completes. +/// A manifest for a record built from scratch: no base, no journal — +/// the run fills the tarball sha (at bootstrap) and the outcome (at its +/// end). fn fresh_manifest( id: String, host_tree: &Path, @@ -721,22 +824,19 @@ fn fresh_manifest( last_used: now_rfc3339(), last_outcome: OUTCOME_RUNNING.to_string(), base: None, - chroot: ChrootInfo { - tarball: String::new(), - tarball_size: 0, - tarball_mtime_secs: 0, - tarball_sha256: String::new(), - ready: false, - }, + tarball_sha256: String::new(), phases: BTreeMap::new(), tree_version: tree_version.to_string(), + kept_upper: false, build_resume: false, } } -/// Create the record a plain (non-incremental) `pkh deb` builds in: a -/// brand-new, empty record of `identity`, locked for this build. Whether -/// it is kept is the caller's keep policy (a plain run discards it). +/// Create the record a `pkh deb` build runs in: a brand-new, empty +/// record of `identity`, locked for this build (the caller decides +/// whether its upperdir is retained at the end). Creating one also +/// sweeps the identity's crashed-run leftovers (chroot directories no +/// live build holds). pub fn new_record( host_tree: &Path, identity: &SessionIdentity, @@ -745,69 +845,46 @@ pub fn new_record( new_record_in(&sessions_root(), host_tree, identity, tree_version) } -/// The testable core of [`new_record`] over an explicit sessions root; -/// creating a record also prunes the identity's records beyond the newest -/// [`KEEP_RECORDS`]. +/// The testable core of [`new_record`] over an explicit sessions root. pub(crate) fn new_record_in( root: &Path, host_tree: &Path, identity: &SessionIdentity, tree_version: &str, ) -> Result { - let session = reserve_record(root, identity, |id| { + sweep_identity_run_state(root, identity); + reserve_record(root, identity, |id| { fresh_manifest(id, host_tree, identity, tree_version) - })?; - prune_identity_records(root, identity); - Ok(session) -} - -/// Whether the base chroot holds device nodes (the `dev/null`/`dev/zero` -/// pair the bootstrap creates): copying them needs privileges, and a -/// plain copy of a chroot without them may still fail on root-only -/// files — the caller retries privileged in that case. -fn chroot_has_device_nodes(chroot: &Path) -> bool { - ["dev/null", "dev/zero"].iter().any(|rel| { - fs::symlink_metadata(chroot.join(rel)).is_ok_and(|meta| meta.file_type().is_char_device()) }) } -/// `cp -a --reflink=auto` of `src` into `dst` (a full, metadata-preserving -/// copy; reflinks keep it cheap on CoW filesystems), privileged when -/// `escalate` is set, with a privileged retry when the plain copy fails -/// (root-only files a device probe cannot see). -fn copy_tree(src: &Path, dst: &Path, escalate: bool) -> io::Result<()> { +/// `cp -a --reflink=auto` of `src`'s contents into `dst`: a full, +/// metadata-preserving copy (reflinks keep it cheap on CoW filesystems). +/// The overlay upperdir can hold kernel-created root-owned whiteouts +/// (char devices), so a plain-copy failure is retried privileged. +fn copy_dir(src: &Path, dst: &Path) -> io::Result<()> { let is_root = crate::utils::root::is_root().unwrap_or(false); // `src/.` into `dst`: an existing `dst` would otherwise receive a // nested `dst/` (cp's copy-into semantics for existing - // destinations), and the fork's record root already has its chroot - // directory. Plain files copy as-is (their `/.` is invalid). - let (from, into) = if src.is_dir() { - fs::create_dir_all(dst)?; - (format!("{}/.", src.display()), dst.to_path_buf()) - } else { - (src.display().to_string(), dst.to_path_buf()) - }; + // destinations), and the fork's record root already exists. + fs::create_dir_all(dst)?; + let from = format!("{}/.", src.display()); let run = |privileged: bool| { crate::deb::ephemeral::privileged_command("cp", privileged || is_root) .arg("-a") .arg("--reflink=auto") .arg(&from) - .arg(&into) + .arg(dst) .status() }; - let status = if escalate { - run(true) - } else { - match run(false) { - Ok(status) if status.success() => return Ok(()), - outcome => { - log::debug!( - "Unprivileged copy of {} failed ({:?}); retrying privileged", - src.display(), - outcome - ); - run(true) - } + let status = match run(false) { + Ok(status) if status.success() => return Ok(()), + outcome => { + log::debug!( + "Unprivileged copy of {} failed ({outcome:?}); retrying privileged", + src.display() + ); + run(true) } }; if matches!(&status, Ok(s) if s.success()) { @@ -821,32 +898,43 @@ fn copy_tree(src: &Path, dst: &Path, escalate: bool) -> io::Result<()> { } } -/// Fork `base` into a new record of `identity` under the sessions root: -/// the full chroot is copied (apt mutates it on every run, so it cannot -/// be shared between records), the tree-upper too when the changelog -/// version did not change, and the journal and chroot info are carried -/// over. The base itself is never modified. See [`fork_record_in`]. +/// Fork `base` into a new record of `identity` under the sessions root. +/// See [`fork_record_in`]. pub fn fork_record( base: &Session, host_tree: &Path, identity: &SessionIdentity, version: &str, + cache_tarball_sha256: &str, ) -> Result { - fork_record_in(&sessions_root(), base, host_tree, identity, version) + fork_record_in( + &sessions_root(), + base, + host_tree, + identity, + version, + cache_tarball_sha256, + ) } /// The testable core of [`fork_record`] over an explicit sessions root. /// -/// The copy happens while holding the base's flock: a concurrent build -/// may still be running in it (its chroot is mid-mutation), and a record -/// interrupted hard (SIGKILL) may carry leftover mounts — those are -/// unmounted first, like the crashed-session recovery of old. +/// A fork carries the base's tree-upper (and its host-tree snapshot and +/// tree journal) only through the fork gate: the base's recorded +/// tarball sha must still equal `cache_tarball_sha256` — the digest of +/// the chroot tarball this build would extract, empty when the cache +/// does not hold it yet — and the changelog version must be unchanged. +/// The toolchain moving or the tree changing invalidates the objects; +/// the fork then starts from a fresh upperdir. The base itself is never +/// modified, and the copy happens while holding the base's flock (a +/// concurrent build may still be writing into its upperdir). pub(crate) fn fork_record_in( root: &Path, base: &Session, host_tree: &Path, identity: &SessionIdentity, version: &str, + cache_tarball_sha256: &str, ) -> Result { let base_manifest = base.manifest(); if base_manifest.identity != *identity { @@ -856,27 +944,29 @@ pub(crate) fn fork_record_in( ))); } base.acquire().map_err(SessionOpenError::Busy)?; - fork_locked_record(root, base, &base_manifest, host_tree, identity, version) -} -/// Fork with the base lock held; every failure path is reported through -/// [`SessionOpenError::Unusable`] so the caller can point at `pkh prune`. -fn fork_locked_record( - root: &Path, - base: &Session, - base_manifest: &SessionManifest, - host_tree: &Path, - identity: &SessionIdentity, - version: &str, -) -> Result { - // A hard-killed base may still carry its /proc bind mount and a - // mounted overlay (the staged tree): unmount them before copying, or - // the copy would traverse the mounted filesystems instead of the - // record's own directories. - unmount_under(&base.chroot_path()); + let carry_upper = base.has_upper() + && !base_manifest.tarball_sha256.is_empty() + && base_manifest.tarball_sha256 == cache_tarball_sha256 + && base_manifest.tree_version == version; + if base.has_upper() && !carry_upper { + let why = if base_manifest.tree_version != version { + format!( + "the changelog version changed ({} -> {})", + base_manifest.tree_version, version + ) + } else if base_manifest.tarball_sha256 != cache_tarball_sha256 { + "the chroot tarball changed since the base was built".to_string() + } else { + "the base carries no staged tree".to_string() + }; + log::info!( + "Fork of record {} starts from a fresh staged tree: {why}", + base_manifest.id + ); + } - let reuse_upper = base_manifest.tree_version == version; - let escalate = chroot_has_device_nodes(&base.chroot_path()); + sweep_identity_run_state(root, identity); let record = reserve_record(root, identity, |id| SessionManifest { schema: SCHEMA_VERSION, pkh: env!("CARGO_PKG_VERSION").to_string(), @@ -887,103 +977,84 @@ fn fork_locked_record( last_used: now_rfc3339(), last_outcome: OUTCOME_RUNNING.to_string(), base: Some(base_manifest.id.clone()), - chroot: base_manifest.chroot.clone(), - phases: base_manifest.phases.clone(), + tarball_sha256: String::new(), + // The journal only makes sense with the tree state it + // describes: a fresh upperdir starts cold. + phases: if carry_upper { + base_manifest.phases.clone() + } else { + BTreeMap::new() + }, tree_version: version.to_string(), + kept_upper: false, build_resume: false, })?; - let fork_result = copy_fork_content(base, &record, reuse_upper, escalate); - if let Err(e) = fork_result { - // The partial fork is inert (its manifest is written but the - // chroot is incomplete): remove it rather than leave a trap for - // the next selection. - let _ = remove_session_dir(record.root()); - return Err(SessionOpenError::Unusable(format!( - "cannot copy record {} into {}: {e}. Free the space or \ - remove old records with `pkh prune`, then rebuild", - base.root().display(), - record.root().display() - ))); - } - if !reuse_upper { - log::info!( - "Changelog version changed since record {} ({} -> {}): the \ - environment is forked, the build artifacts are rebuilt from scratch", - base_manifest.id, - base_manifest.tree_version, - version - ); + if carry_upper { + let copied = copy_dir(&base.upper_dir(), &record.upper_dir()).and_then(|_bytes| { + if base.snapshot_file().is_file() { + fs::copy(base.snapshot_file(), record.snapshot_file()) + } else { + Ok(0) + } + }); + if let Err(e) = copied { + // The partial fork is inert (its manifest is written but + // its staged tree is incomplete): remove it rather than + // leave a trap for the next selection. + let _ = remove_session_dir(record.root()); + return Err(SessionOpenError::Unusable(format!( + "cannot copy the staged tree of record {} into {}: {e}. \ + Free the space or remove old records with `pkh prune`, \ + then rebuild", + base.root().display(), + record.root().display() + ))); + } } log::info!( - "Forked build record {} from record {}: the prepared chroot, \ - installed build dependencies{} are reused", + "Forked build record {} from record {}: {}", record.manifest().id, base_manifest.id, - if reuse_upper { - " and the staged build artifacts" + if carry_upper { + "the staged tree (objects included) is carried" } else { - "" + "a fresh staged tree; the environment comes from the apt cache" } ); - prune_identity_records(root, identity); Ok(record) } -/// Copy the forked content of `base` into `record`: the whole chroot -/// minus the per-tree staging state (`pkh-overlay`, `tmp/pkh-build`) -/// when the version changed and thus the upper must not be reused, plus -/// the host-tree snapshot (only meaningful with the upper it describes). -fn copy_fork_content( - base: &Session, - record: &Session, - reuse_upper: bool, - escalate: bool, -) -> io::Result<()> { - let src_chroot = base.chroot_path(); - let dst_chroot = record.chroot_path(); - fs::create_dir_all(&dst_chroot)?; - if reuse_upper { - copy_tree(&src_chroot, &dst_chroot, escalate)?; - } else { - // Copy the chroot entry by entry, skipping the overlay state and - // the staged tree of the previous version (a stale upper would - // shadow the new version's sources, and its objects are garbage). - for entry in fs::read_dir(&src_chroot)? { - let entry = entry?; - let name = entry.file_name(); - if name == "pkh-overlay" { - continue; - } - if name == "tmp" { - let src_tmp = entry.path(); - let dst_tmp = dst_chroot.join("tmp"); - fs::create_dir_all(&dst_tmp)?; - for tmp_entry in fs::read_dir(&src_tmp)? { - let tmp_entry = tmp_entry?; - if tmp_entry.file_name() == "pkh-build" { - continue; - } - copy_tree( - &tmp_entry.path(), - &dst_tmp.join(tmp_entry.file_name()), - escalate, - )?; - } - continue; - } - copy_tree(&entry.path(), &dst_chroot.join(&name), escalate)?; +/// Sweep the per-run leftovers of a hard-killed build (OOM, SIGKILL — +/// anything that skipped the teardown): the chroot directory and the +/// overlay scratch workdir of every record of `identity` whose lock is +/// not held by a live build. Live builds keep theirs; nothing else is +/// touched. +pub(crate) fn sweep_identity_run_state(root: &Path, identity: &SessionIdentity) { + let Ok(entries) = fs::read_dir(identity_dir(root, identity)) else { + return; + }; + for entry in entries.flatten() { + let path = entry.path(); + if !path.is_dir() || (!path.join(CHROOT_DIR).is_dir() && !path.join(WORK_DIR).is_dir()) { + continue; } + let Some(record) = open_from_dir(path) else { + continue; + }; + if lock_is_held(&record.lock_path()).unwrap_or(true) { + continue; + } + log::debug!( + "Sweeping the crashed-run leftovers of build record {}", + record.manifest().id + ); + record.remove_run_state(); } - if reuse_upper && base.snapshot_file().is_file() { - fs::copy(base.snapshot_file(), record.snapshot_file())?; - } - Ok(()) } /// Unmount every mount at or under `path` (deepest first), best-effort: -/// used before forking a possibly hard-killed base and when removing a -/// record root. +/// used before removing per-run state and record roots. pub(crate) fn unmount_under(path: &Path) { let is_root = crate::utils::root::is_root().unwrap_or(false); for mount in crate::deb::ephemeral::host_mounts_under(path) @@ -1000,42 +1071,6 @@ pub(crate) fn unmount_under(path: &Path) { } } -/// Prune the records of `identity` beyond the newest [`KEEP_RECORDS`] -/// (oldest first). Records whose lock is held by a live build are never -/// removed; directories without a readable manifest are left to -/// `pkh prune` (they may be a record mid-creation). -pub(crate) fn prune_identity_records(root: &Path, identity: &SessionIdentity) { - let mut records: Vec = Vec::new(); - let Ok(entries) = fs::read_dir(identity_dir(root, identity)) else { - return; - }; - for entry in entries.flatten() { - let path = entry.path(); - if path.is_dir() - && let Some(record) = open_from_dir(path) - { - records.push(record); - } - } - records.sort_by(|a, b| record_ordering(a, b).reverse()); - for record in records.into_iter().skip(KEEP_RECORDS) { - let held = lock_is_held(&record.lock_path()).unwrap_or(true); - if held { - log::debug!( - "Record {} is locked by a live build; skipping the pruning", - record.manifest().id - ); - continue; - } - log::debug!( - "Pruning record {} beyond the newest {KEEP_RECORDS} of {}", - record.manifest().id, - identity - ); - record.remove(); - } -} - /// Newness ordering of two records: by `created`, then id (both are /// build-start timestamps of the same format). fn record_ordering(a: &Session, b: &Session) -> std::cmp::Ordering { @@ -1068,34 +1103,15 @@ pub(crate) fn lock_is_held(path: &Path) -> io::Result { Err(err) } -/// Whether the cached chroot tarball the record was bootstrapped from is -/// still the one the current bootstrap would use: the file name recorded -/// in the manifest, then the cheap size/mtime check, falling back to the -/// recorded SHA-256 when the cheap check drifts (the tarball was -/// re-created in the same second, ...). -pub(crate) fn tarball_matches_path(session: &Session, expected: &Path) -> bool { - let Ok(metadata) = fs::metadata(expected) else { - return false; - }; - let info = &session.manifest().chroot; - if info.tarball - != expected - .file_name() - .map(|name| name.to_string_lossy().into_owned()) - .unwrap_or_default() - { - return false; - } - let mtime_secs = metadata - .modified() +/// The digest of the chroot tarball a build of `series`/`arch` would +/// extract, empty when the cache does not hold one yet (the bootstrap +/// downloads it during the run). This is the fork gate's reference +/// against the sha recorded in a base's manifest. +pub(crate) fn cache_tarball_sha256(series: &str, arch: Option<&str>) -> String { + crate::deb::ephemeral::chroot_tarball_path(series, arch) .ok() - .and_then(|t| t.duration_since(std::time::UNIX_EPOCH).ok()) - .map(|d| d.as_secs()) - .unwrap_or(0); - if info.tarball_size == metadata.len() && info.tarball_mtime_secs == mtime_secs { - return true; - } - sha256_file(expected).is_ok_and(|hash| hash == info.tarball_sha256) + .and_then(|path| sha256_file(&path).ok()) + .unwrap_or_default() } /// Load every discoverable record under `root` (any identity), newest @@ -1124,8 +1140,7 @@ pub(crate) fn load_all_in(root: &Path) -> Vec { records } -/// The records of `identity` under `root`, newest first (the -/// auto-selection candidates of `--incremental`). +/// The records of `identity` under `root`, newest first. pub(crate) fn identity_records_in(root: &Path, identity: &SessionIdentity) -> Vec { let dir = identity_dir(root, identity); let Ok(entries) = fs::read_dir(&dir) else { @@ -1142,6 +1157,16 @@ pub(crate) fn identity_records_in(root: &Path, identity: &SessionIdentity) -> Ve records } +/// The fork candidates of `identity` under `root`: its newest records +/// that still carry a tree-upper. A metadata-only record (plain run, or +/// degraded by the upper cap) holds nothing a fork could carry. +pub(crate) fn fork_candidates_in(root: &Path, identity: &SessionIdentity) -> Vec { + identity_records_in(root, identity) + .into_iter() + .filter(|record| record.has_upper()) + .collect() +} + /// Records recorded from `tree` (any identity), newest first. pub fn for_tree(tree: &Path) -> Vec { let tree = tree.to_string_lossy().into_owned(); @@ -1193,6 +1218,9 @@ pub struct SessionListRow { pub target: String, /// Id of the record this one was forked from, when any. pub base: Option, + /// Whether the record retains its staged tree (an `--incremental` + /// fork candidate). + pub upper: bool, /// Outcome of this record's build. pub outcome: String, /// Human-readable age of this record's build. @@ -1215,6 +1243,7 @@ pub fn list_for_tree(tree: &Path) -> Vec { manifest.identity.series, manifest.identity.arch, cross ), base: manifest.base, + upper: session.has_upper(), outcome: manifest.last_outcome, age: age_of(&manifest.last_used), } @@ -1255,7 +1284,7 @@ pub fn render_session_list(tree: &Path, rows: &[SessionListRow]) -> String { } out.push_str(&format!("Records for {}:\n", tree.display())); let headers = [ - "ID", "PACKAGE", "VERSION", "TARGET", "BASE", "OUTCOME", "AGE", + "ID", "PACKAGE", "VERSION", "TARGET", "BASE", "UPPER", "OUTCOME", "AGE", ]; let table: Vec> = rows .iter() @@ -1266,6 +1295,7 @@ pub fn render_session_list(tree: &Path, rows: &[SessionListRow]) -> String { row.version.clone(), row.target.clone(), row.base.clone().unwrap_or_else(|| "-".to_string()), + if row.upper { "yes" } else { "-" }.to_string(), row.outcome.clone(), row.age.clone(), ] @@ -1301,10 +1331,134 @@ pub fn render_session_list(tree: &Path, rows: &[SessionListRow]) -> String { out.push_str(&line(row)); out.push('\n'); } - out.push_str("\nFork with: pkh deb --incremental [] (* = cross build)\n"); + out.push_str( + "\nFork with: pkh deb --incremental [] (* = cross build; \ + UPPER = the record keeps its staged tree)\n", + ); out } +// --------------------------------------------------------------------------- +// Build log (`pkh deb log`) +// --------------------------------------------------------------------------- + +/// Resolve the record whose log `pkh deb log` prints: with no id, the +/// newest record of `tree`; with an id (or unique prefix), that exact +/// record of `tree`. `root` is the sessions root (`sessions_root()` in +/// production, a fixture root in tests). +pub fn record_for_log( + root: &Path, + tree: &Path, + id: Option<&str>, +) -> Result> { + let tree_str = tree.to_string_lossy().into_owned(); + match id { + None => load_all_in(root) + .into_iter() + .find(|s| s.manifest().host_tree == tree_str) + .ok_or_else(|| format!("no build record found for {}", tree.display()).into()), + Some(id) => { + let mut matches: Vec = by_id_prefix_in(root, id) + .into_iter() + .filter(|s| s.manifest().host_tree == tree_str) + .collect(); + match matches.len() { + 1 => Ok(matches.remove(0)), + 0 => Err(format!( + "no build record matches id '{id}' for this tree \ + (see `pkh deb list`)" + ) + .into()), + _ => { + let found = matches + .iter() + .map(|s| s.manifest().id.clone()) + .collect::>() + .join(", "); + Err(format!("ambiguous record id '{id}': matches {found}").into()) + } + } + } + } +} + +/// Stream a record's build log to `out`: everything written so far and, +/// with `follow`, whatever the build appends until its record reaches a +/// terminal state (outcome recorded, or the record's lock freed with the +/// outcome still `running` — a crashed build). A record whose log does +/// not exist yet streams nothing. +pub fn print_record_log(record: &Session, follow: bool, out: &mut dyn io::Write) -> io::Result<()> { + let path = record.log_file(); + let mut offset: usize = 0; + loop { + offset = stream_appended(&path, offset, out)?; + let done = open_from_dir(record.root().to_path_buf()) + .map(|live| { + live.manifest().last_outcome != OUTCOME_RUNNING + || !lock_is_held(&live.lock_path()).unwrap_or(true) + }) + .unwrap_or(true); + if !follow || done { + return out.flush(); + } + std::thread::sleep(Duration::from_millis(200)); + } +} + +/// Write the bytes of `path` beyond `offset` to `out`, returning the +/// new offset; a missing or shrunk file restarts from zero. +fn stream_appended(path: &Path, offset: usize, out: &mut dyn io::Write) -> io::Result { + let Ok(meta) = fs::metadata(path) else { + return Ok(offset); + }; + let len = meta.len() as usize; + if len < offset { + return Ok(len); // truncated/rotated: restart from the beginning + } + if len == offset { + return Ok(offset); + } + let mut file = fs::File::open(path)?; + file.seek(io::SeekFrom::Start(offset as u64))?; + io::copy(&mut file, out)?; + Ok(len) +} + +/// A sink appending every captured build line to a record's build log +/// (opened once, shared by every command of the run) and forwarding it +/// to the wrapped sink, if any: the log is a full tee of the build +/// output whatever view renders it. +pub(crate) struct RecordLogSink { + log: Mutex, + next: Option>, +} + +impl RecordLogSink { + /// Open the build log of `record` for appending, forwarding to `next` + /// as well (the view's own sink, when it has one). + pub(crate) fn new(record: &Session, next: Option>) -> io::Result { + Ok(Self { + log: Mutex::new( + fs::OpenOptions::new() + .append(true) + .open(record.log_file())?, + ), + next, + }) + } +} + +impl LineSink for RecordLogSink { + fn line(&self, stream: Stream, line: &str) { + if let Ok(mut log) = self.log.lock() { + let _ = writeln!(log, "{line}"); + } + if let Some(next) = &self.next { + next.line(stream, line); + } + } +} + // --------------------------------------------------------------------------- // Host-tree snapshot and overlay upperdir sync // @@ -1603,7 +1757,7 @@ pub(crate) fn sync_upper_for_resume(session: &Session, host_parent: &Path) -> io /// Record fixtures shared by the `deb` test modules (this one and /// `crate::deb`): a synthetic sessions root plus records carrying a -/// complete, usable environment. +/// retained staged tree. #[cfg(test)] pub(crate) mod record_fixtures { use super::*; @@ -1626,12 +1780,12 @@ pub(crate) mod record_fixtures { (root, tarball) } - /// A record with a complete, usable environment: bootstrap marker, - /// `/bin/sh`, one apt-state file, the staged-tree state of a previous - /// build (upperdir + snapshot + staged mountpoint) and a chroot - /// bootstrapped from `tarball`. The returned handle holds the record - /// lock; drop it before forking or selecting from another handle. - pub fn record_with_environment( + /// An `--incremental`-shaped record: a retained staged tree (the + /// upperdir, its host-tree snapshot and the tree journal), the sha + /// of the tarball its environment was extracted from, and an + /// outcome. The returned handle holds the record lock; drop it + /// before forking or selecting from another handle. + pub fn record_with_upper( root: &Path, tarball: &Path, host_tree: &Path, @@ -1640,31 +1794,30 @@ pub(crate) mod record_fixtures { outcome: Option<&str>, ) -> Session { let session = new_record_in(root, host_tree, identity, version).unwrap(); - let chroot = session.chroot_path(); - fs::create_dir_all(chroot.join("bin")).unwrap(); - fs::create_dir_all(chroot.join("var/lib/dpkg")).unwrap(); - fs::write(chroot.join(".pkh-ready"), b"").unwrap(); - fs::write(chroot.join("bin/sh"), b"#!/bin/sh").unwrap(); - fs::write(chroot.join("var/lib/dpkg/status"), b"installed").unwrap(); fs::create_dir_all(session.upper_dir().join("src")).unwrap(); fs::write(session.upper_dir().join("src/main.o"), b"objects").unwrap(); - fs::create_dir_all(chroot.join("tmp/pkh-build/tree")).unwrap(); fs::write(session.snapshot_file(), "f\t1\t10\ttree/src/a.c\n").unwrap(); - let metadata = fs::metadata(tarball).unwrap(); - session.set_chroot(ChrootInfo { - tarball: tarball.file_name().unwrap().to_string_lossy().into_owned(), - tarball_size: metadata.len(), - tarball_mtime_secs: metadata - .modified() - .unwrap() - .duration_since(std::time::UNIX_EPOCH) - .unwrap() - .as_secs(), - tarball_sha256: sha256_file(tarball).unwrap(), - ready: true, - }); - session.record_phase("apt_update", None); - if let Some(outcome) = outcome { + session.set_tarball_sha256(&sha256_file(tarball).unwrap()); + session.record_phase("patches", Some("stamp".to_string())); + session.retain_upper(true, outcome == Some(OUTCOME_SUCCESS)); + if let Some(outcome) = outcome.filter(|o| *o != OUTCOME_SUCCESS) { + session.set_outcome(outcome); + } + session + } + + /// A metadata-only record (a plain run's shape): manifest and log, + /// no staged tree. + pub fn metadata_record( + root: &Path, + host_tree: &Path, + identity: &SessionIdentity, + version: &str, + outcome: &str, + ) -> Session { + let session = new_record_in(root, host_tree, identity, version).unwrap(); + session.retain_upper(false, outcome == OUTCOME_SUCCESS); + if outcome != OUTCOME_SUCCESS { session.set_outcome(outcome); } session @@ -1686,7 +1839,7 @@ pub(crate) fn remove_session_dir(root: &Path) -> io::Result<()> { #[cfg(test)] mod tests { use super::*; - use record_fixtures::{record_with_environment, sessions_fixture, test_identity}; + use record_fixtures::{metadata_record, record_with_upper, sessions_fixture, test_identity}; fn meta(mtime_ns: i128, size: u64) -> EntryMeta { EntryMeta { mtime_ns, size } @@ -1875,9 +2028,9 @@ mod tests { } /// Identity slugs are sanitized and collision-free, and the manifest - /// round-trips through JSON — the optional base lineage included, - /// and its absence (records from before fork lineage existed) parses - /// as `None`. + /// round-trips through JSON — lineage, tarball sha and the kept + /// upper included, and their absence (older manifests) parsing as + /// `None`/empty/false. #[test] fn slug_and_manifest_round_trip() { let identity = SessionIdentity { @@ -1903,21 +2056,16 @@ mod tests { last_used: now_rfc3339(), last_outcome: OUTCOME_FAILED.to_string(), base: Some("20260926T140000".to_string()), - chroot: ChrootInfo { - tarball: "stonking-arm64-buildd.tar.xz".to_string(), - tarball_size: 1, - tarball_mtime_secs: 2, - tarball_sha256: "ab".to_string(), - ready: true, - }, + tarball_sha256: "ab".to_string(), phases: BTreeMap::from([( - "apt_update".to_string(), + "patches".to_string(), PhaseStamp { at: now_rfc3339(), stamp: None, }, )]), tree_version: "7.3.0-5.6~local2".to_string(), + kept_upper: true, build_resume: true, }; let base = tempfile::tempdir().unwrap(); @@ -1925,12 +2073,17 @@ mod tests { let loaded = open_from_dir(base.path().to_path_buf()).unwrap(); assert_eq!(loaded.manifest(), manifest); - // A manifest without the lineage field (the pre-fork format) - // still parses, with no base. + // A manifest without the newer fields still parses, with their + // defaults. let mut value = serde_json::to_value(&manifest).unwrap(); - value.as_object_mut().unwrap().remove("base"); - let no_base: SessionManifest = serde_json::from_value(value).unwrap(); - assert_eq!(no_base.base, None); + let object = value.as_object_mut().unwrap(); + object.remove("base"); + object.remove("tarball_sha256"); + object.remove("kept_upper"); + let older: SessionManifest = serde_json::from_value(value).unwrap(); + assert_eq!(older.base, None); + assert_eq!(older.tarball_sha256, ""); + assert!(!older.kept_upper); // A foreign schema is not discoverable. let mut foreign = manifest.clone(); @@ -1939,8 +2092,8 @@ mod tests { assert!(open_from_dir(base.path().to_path_buf()).is_none()); } - /// `pkh deb list` renders a table with the fork lineage; the empty - /// case explains itself. + /// `pkh deb list` renders a table with the fork lineage and the + /// upper-kept state; the empty case explains itself. #[test] fn session_list_rendering() { let rows = vec![ @@ -1950,6 +2103,7 @@ mod tests { version: "7.3.0-5.6~local2".to_string(), target: "stonking/arm64*".to_string(), base: Some("20260926T140000".to_string()), + upper: true, outcome: OUTCOME_FAILED.to_string(), age: "2h".to_string(), }, @@ -1959,6 +2113,7 @@ mod tests { version: "7.3.0-5.5~local1".to_string(), target: "stonking/riscv64*".to_string(), base: None, + upper: false, outcome: OUTCOME_SUCCESS.to_string(), age: "1d".to_string(), }, @@ -1970,8 +2125,10 @@ mod tests { ); assert!(rendered.contains("20260926T143505"), "{rendered}"); assert!(rendered.contains("stonking/arm64*"), "{rendered}"); - // The lineage column shows the forked base, a dash for roots. + // The lineage and upper columns: the fork's base, the metadata + // record's dash. assert!(rendered.contains("20260926T140000"), "{rendered}"); + assert!(rendered.contains("UPPER"), "{rendered}"); assert!( rendered.contains("Fork with: pkh deb --incremental []"), "{rendered}" @@ -1982,12 +2139,12 @@ mod tests { } // ------------------------------------------------------------------- - // Record layout, locking, forking, selection and pruning + // Record layout, fork gate, retention, sweep and logs // ------------------------------------------------------------------- - /// A new record lives at `//` with its manifest on disk and - /// its lock beside the root; a concurrent handle sees it busy, and - /// same-second creations get distinct ids. + /// A new record lives at `//` with its manifest and build + /// log on disk and its lock beside the root; a concurrent handle sees + /// it busy, and same-second creations get distinct ids. #[test] fn record_layout_and_locking() { let (root, _tarball) = sessions_fixture(); @@ -1998,7 +2155,8 @@ mod tests { let slug_dir = identity_dir(root.path(), &identity); let record_dir = record.root(); assert_eq!(record_dir.parent(), Some(slug_dir.as_path())); - assert!(record_dir.join("session.json").is_file()); + assert!(record_dir.join("manifest.json").is_file()); + assert!(record.log_file().is_file(), "the log exists from creation"); assert_eq!( record.lock_path(), slug_dir.join(format!("{}.lock", record.manifest().id)) @@ -2007,7 +2165,8 @@ mod tests { assert_eq!(manifest.last_outcome, OUTCOME_RUNNING); assert_eq!(manifest.tree_version, "1.0"); assert_eq!(manifest.base, None); - assert!(!manifest.chroot.ready); + assert_eq!(manifest.tarball_sha256, ""); + assert!(!manifest.kept_upper); // The handle holds the flock: another handle sees the record busy. assert!(lock_is_held(&record.lock_path()).unwrap()); @@ -2028,124 +2187,16 @@ mod tests { assert_ne!(ids[0], ids[1]); } - /// The integrity probe requires the recorded bootstrap completion AND - /// the on-disk markers. + /// The fork gate: the base's staged tree is carried only while its + /// recorded tarball sha still matches the cached tarball AND the + /// changelog version is unchanged. Everything else forks cold. #[test] - fn chroot_usable_probe() { + fn fork_gate_carries_the_upper_only_on_sha_and_version_match() { let (root, tarball) = sessions_fixture(); let identity = test_identity(); let host_tree = root.path().join("tree"); - let bare = new_record_in(root.path(), &host_tree, &identity, "1.0").unwrap(); - assert!(!bare.chroot_is_usable()); // nothing bootstrapped yet - drop(bare); - let complete = - record_with_environment(root.path(), &tarball, &host_tree, &identity, "1.0", None); - assert!(complete.chroot_is_usable()); - } - - /// A record is a usable base only while its bootstrap tarball is - /// still the recorded one: the name must match, then the cheap - /// size/mtime check, with the recorded SHA as the drift fallback. - #[test] - fn tarball_match_checks() { - let (root, tarball) = sessions_fixture(); - let identity = test_identity(); - let host_tree = root.path().join("tree"); - let record = - record_with_environment(root.path(), &tarball, &host_tree, &identity, "1.0", None); - assert!(tarball_matches_path(&record, &tarball), "recorded tarball"); - - // Same content, drifted mtime: the cheap size/mtime check fails, - // the recorded SHA-256 still certifies the tarball. - bump_mtime(&tarball, std::time::Duration::from_secs(7200)); - assert!( - tarball_matches_path(&record, &tarball), - "the recorded sha certifies a re-touched tarball" - ); - - // A re-downloaded tarball (different bytes, drifted metadata): - // no match. - fs::write(&tarball, b"re-downloaded tarball").unwrap(); - bump_mtime(&tarball, std::time::Duration::from_secs(3600)); - assert!( - !tarball_matches_path(&record, &tarball), - "different content must not match" - ); - - // A tarball under another name: never a match. - let renamed = root.path().join("other-name.tar.xz"); - fs::rename(&tarball, &renamed).unwrap(); - assert!( - !tarball_matches_path(&record, &renamed), - "a renamed tarball bootstraps a different chroot" - ); - - // Missing file: never a match. - assert!( - !tarball_matches_path(&record, &tarball), - "a missing tarball cannot certify a chroot" - ); - } - - /// Move a file's mtime `by` into the future. - fn bump_mtime(path: &Path, by: std::time::Duration) { - let file = fs::OpenOptions::new().write(true).open(path).unwrap(); - file.set_times(std::fs::FileTimes::new().set_modified(std::time::SystemTime::now() + by)) - .unwrap(); - } - - /// Forking copies the whole base into a NEW record — chroot - /// environment, staged artifacts, snapshot, journal and chroot info — - /// records the lineage, and never touches the base: writes into the - /// fork stay in the fork. - #[test] - fn fork_copies_the_base_without_touching_it() { - let (root, tarball) = sessions_fixture(); - let identity = test_identity(); - let host_tree = root.path().join("tree"); - let base = - record_with_environment(root.path(), &tarball, &host_tree, &identity, "1.0", None); - let base_id = base.manifest().id; - let base_upper = base.upper_dir().join("src/main.o"); - let base_dpkg = base.chroot_path().join("var/lib/dpkg/status"); - - let fork = fork_record_in(root.path(), &base, &host_tree, &identity, "1.0").unwrap(); - let fork_manifest = fork.manifest(); - assert_ne!(fork_manifest.id, base_id); - assert_eq!(fork_manifest.base.as_deref(), Some(base_id.as_str())); - assert_eq!(fork_manifest.tree_version, "1.0"); - assert_eq!( - fork_manifest.chroot.tarball_sha256, - base.manifest().chroot.tarball_sha256 - ); - assert!(fork_manifest.phases.contains_key("apt_update")); - assert_eq!( - fs::read_to_string(fork.chroot_path().join("var/lib/dpkg/status")).unwrap(), - "installed" - ); - assert_eq!( - fs::read_to_string(fork.upper_dir().join("src/main.o")).unwrap(), - "objects" - ); - assert!(fork.snapshot_file().is_file()); - assert!(fork.chroot_is_usable()); - - // Mutating the fork leaves the base byte-identical. - fs::write(fork.upper_dir().join("src/main.o"), b"changed").unwrap(); - fs::write(fork.chroot_path().join("var/lib/dpkg/status"), b"mutated").unwrap(); - assert_eq!(fs::read_to_string(&base_upper).unwrap(), "objects"); - assert_eq!(fs::read_to_string(&base_dpkg).unwrap(), "installed"); - } - - /// A version change at the fork drops the tree state — upperdir, - /// staged tree, host-tree snapshot — while the environment (chroot, - /// journal, chroot info) is carried and the new version is recorded. - #[test] - fn fork_on_version_change_rebuilds_the_tree_state() { - let (root, tarball) = sessions_fixture(); - let identity = test_identity(); - let host_tree = root.path().join("tree"); - let base = record_with_environment( + let sha = sha256_file(&tarball).unwrap(); + let base = record_with_upper( root.path(), &tarball, &host_tree, @@ -2153,32 +2204,70 @@ mod tests { "1.0", Some(OUTCOME_SUCCESS), ); + let base_id = base.manifest().id; - let fork = fork_record_in(root.path(), &base, &host_tree, &identity, "1.1").unwrap(); + // Everything matches: the staged tree, its snapshot and the tree + // journal are carried. + let fork = fork_record_in(root.path(), &base, &host_tree, &identity, "1.0", &sha).unwrap(); + assert_eq!(fork.manifest().base.as_deref(), Some(base_id.as_str())); + assert_eq!( + fs::read_to_string(fork.upper_dir().join("src/main.o")).unwrap(), + "objects" + ); + assert!(fork.snapshot_file().is_file()); + assert!(fork.manifest().phases.contains_key("patches")); + drop(fork); + + // The changelog version changed: cold fork. + let fork = fork_record_in(root.path(), &base, &host_tree, &identity, "1.1", &sha).unwrap(); assert_eq!(fork.manifest().tree_version, "1.1"); - assert!(fork.chroot_path().join("var/lib/dpkg/status").is_file()); - assert!(fork.chroot_path().join(".pkh-ready").is_file()); - assert!(fork.chroot_is_usable()); - assert!(fork_manifest_has_phase(&fork, "apt_update")); - // The stale objects of version 1.0 must not survive into 1.1. assert!(!fork.upper_dir().join("src/main.o").exists()); - assert!(!fork.chroot_path().join("tmp/pkh-build/tree").exists()); assert!(!fork.snapshot_file().exists()); + assert!(fork.manifest().phases.is_empty()); + drop(fork); + + // The toolchain moved (the cached tarball changed): cold fork. + let fork = + fork_record_in(root.path(), &base, &host_tree, &identity, "1.0", "deadbeef").unwrap(); + assert!(!fork.upper_dir().join("src/main.o").exists()); + drop(fork); + + // No cached tarball yet (the sha is unknown): cold fork. + let fork = fork_record_in(root.path(), &base, &host_tree, &identity, "1.0", "").unwrap(); + assert!(!fork.upper_dir().join("src/main.o").exists()); } - fn fork_manifest_has_phase(record: &Session, phase: &str) -> bool { - record.manifest().phases.contains_key(phase) + /// Forking never touches the base: writes into the fork's staged + /// tree stay in the fork. + #[test] + fn fork_isolation() { + let (root, tarball) = sessions_fixture(); + let identity = test_identity(); + let host_tree = root.path().join("tree"); + let sha = sha256_file(&tarball).unwrap(); + let base = record_with_upper(root.path(), &tarball, &host_tree, &identity, "1.0", None); + let base_upper = base.upper_dir().join("src/main.o"); + + let fork = fork_record_in(root.path(), &base, &host_tree, &identity, "1.0", &sha).unwrap(); + fs::write(fork.upper_dir().join("src/main.o"), b"changed").unwrap(); + fs::write(fork.snapshot_file(), "mutated\n").unwrap(); + assert_eq!(fs::read_to_string(&base_upper).unwrap(), "objects"); + assert_eq!( + fs::read_to_string(base.snapshot_file()).unwrap(), + "f\t1\t10\ttree/src/a.c\n" + ); } - /// The user scenario: a successful record A, a failed fork B of it — - /// the default selection must pick A (the newest success), not B - /// (the newest record); with no success at all it takes the newest. + /// The user scenario: record A succeeds, a fork B of it fails — the + /// default selection must pick A (the newest success), not B (the + /// newest record); with no success at all it takes the newest. #[test] fn selection_prefers_the_newest_success_over_the_newest_failure() { let (root, tarball) = sessions_fixture(); let identity = test_identity(); let host_tree = root.path().join("tree"); - let a = record_with_environment( + let sha = sha256_file(&tarball).unwrap(); + let a = record_with_upper( root.path(), &tarball, &host_tree, @@ -2187,29 +2276,33 @@ mod tests { Some(OUTCOME_SUCCESS), ); let a_id = a.manifest().id; - let b = fork_record_in(root.path(), &a, &host_tree, &identity, "1.0").unwrap(); + let b = fork_record_in(root.path(), &a, &host_tree, &identity, "1.0", &sha).unwrap(); let b_id = b.manifest().id; b.set_outcome(OUTCOME_FAILED); drop((a, b)); - let candidates = identity_records_in(root.path(), &identity); + let candidates = fork_candidates_in(root.path(), &identity); assert_eq!(candidates.len(), 2); assert_eq!(candidates[0].manifest().id, b_id, "newest first"); let ranked = rank_base_candidates(candidates); assert_eq!(ranked[0].manifest().id, a_id, "the success wins"); assert_eq!(ranked[1].manifest().id, b_id); - // No success anywhere: the newest record is the fallback. - ranked[0].set_outcome(OUTCOME_FAILED); - drop(ranked); - let ranked = rank_base_candidates(identity_records_in(root.path(), &identity)); + // No success anywhere (A fails too): the newest record is the + // fallback. + identity_records_in(root.path(), &identity) + .into_iter() + .find(|r| r.manifest().id == a_id) + .expect("record A is still there") + .set_outcome(OUTCOME_FAILED); + let ranked = rank_base_candidates(fork_candidates_in(root.path(), &identity)); assert_eq!(ranked[0].manifest().id, b_id); } - /// Records are scoped per identity: another package or target never - /// shows up in the candidates. + /// Metadata-only records (plain runs, degraded ones) are not fork + /// candidates; records are scoped per identity. #[test] - fn identity_records_are_scoped_to_the_identity() { + fn metadata_only_records_are_not_fork_candidates() { let (root, tarball) = sessions_fixture(); let identity = test_identity(); let other = SessionIdentity { @@ -2217,15 +2310,14 @@ mod tests { ..identity.clone() }; let host_tree = root.path().join("tree"); - drop(record_with_environment( + drop(metadata_record( root.path(), - &tarball, &host_tree, &identity, "1.0", - None, + OUTCOME_SUCCESS, )); - drop(record_with_environment( + drop(record_with_upper( root.path(), &tarball, &host_tree, @@ -2233,22 +2325,27 @@ mod tests { "1.0", None, )); + assert!( + fork_candidates_in(root.path(), &identity).is_empty(), + "a metadata-only record carries nothing to fork" + ); + assert_eq!(fork_candidates_in(root.path(), &other).len(), 1); assert_eq!(identity_records_in(root.path(), &identity).len(), 1); - assert_eq!(identity_records_in(root.path(), &other).len(), 1); assert_eq!(load_all_in(root.path()).len(), 2); } - /// Creating a record prunes the identity's records beyond the newest - /// five, oldest first. + /// Retaining a sixth staged tree degrades the oldest one to a + /// metadata-only record (upperdir and snapshot deleted, manifest and + /// log kept, `kept_upper` flipped): at most five staged trees per + /// target at any time. #[test] - fn creating_records_keeps_only_the_last_five() { + fn retaining_a_sixth_upper_degrades_the_oldest() { let (root, tarball) = sessions_fixture(); let identity = test_identity(); let host_tree = root.path().join("tree"); - let mut first_id = None; - let mut last_id = None; - for i in 0..7 { - let record = record_with_environment( + let mut oldest = None; + for i in 0..6 { + let record = record_with_upper( root.path(), &tarball, &host_tree, @@ -2256,64 +2353,184 @@ mod tests { &format!("1.{i}"), None, ); - let id = record.manifest().id; + oldest.get_or_insert_with(|| record.manifest().id.clone()); drop(record); - first_id.get_or_insert(id.clone()); - last_id = Some(id); - // Older records vanish as soon as the count passes five. - let kept: Vec = identity_records_in(root.path(), &identity) - .into_iter() - .map(|r| r.manifest().id) - .collect(); - assert!(kept.len() <= KEEP_RECORDS, "left {kept:?} behind"); - if i >= KEEP_RECORDS { - assert_eq!(kept.len(), KEEP_RECORDS); - } } - let kept: Vec = identity_records_in(root.path(), &identity) + let oldest = oldest.unwrap(); + let degraded = identity_records_in(root.path(), &identity) .into_iter() - .map(|r| r.manifest().id) - .collect(); - assert_eq!(kept.len(), KEEP_RECORDS); - // The five kept are the newest five: the first record is gone, - // the last one stays. - assert!(!kept.contains(first_id.as_ref().unwrap())); - assert!(kept.contains(last_id.as_ref().unwrap())); + .find(|r| r.manifest().id == oldest) + .expect("the oldest record is still there, metadata-only"); + assert!(!degraded.has_upper(), "the oldest staged tree is gone"); + assert!(!degraded.snapshot_file().exists()); + assert!(!degraded.manifest().kept_upper); + assert!(degraded.root().join("manifest.json").is_file()); + assert!(degraded.log_file().is_file()); + assert_eq!( + fork_candidates_in(root.path(), &identity).len(), + KEEP_UPPERS, + "exactly five staged trees remain" + ); } - /// The end-of-build disposition follows keep-iff-incremental: a - /// plain run's record is always discarded (success and failure), an - /// incremental run's record always kept, with the outcome recorded on - /// success. + /// The end-of-run retention: metadata and the log are kept for every + /// build; the staged tree exactly for `--incremental` ones. #[test] - fn dispose_keeps_iff_incremental() { + fn retain_upper_keeps_metadata_for_every_build() { let (root, tarball) = sessions_fixture(); let identity = test_identity(); let host_tree = root.path().join("tree"); - let plain_success = - record_with_environment(root.path(), &tarball, &host_tree, &identity, "1.0", None); - plain_success.dispose_after_build(false, true); - assert!(!plain_success.root().exists(), "plain success discards"); + // A plain run, success or failure: metadata-only. + for succeeded in [true, false] { + let record = + record_with_upper(root.path(), &tarball, &host_tree, &identity, "1.0", None); + record.retain_upper(false, succeeded); + assert!(record.root().join("manifest.json").is_file(), "metadata"); + assert!(record.log_file().is_file(), "the log is always kept"); + assert!(!record.has_upper(), "a plain run drops its staged tree"); + assert!(!record.snapshot_file().exists()); + let expected = if succeeded { + OUTCOME_SUCCESS + } else { + OUTCOME_RUNNING // the failure outcome is the flow's to set + }; + assert_eq!(record.manifest().last_outcome, expected); + drop(record); + } - let plain_failure = - record_with_environment(root.path(), &tarball, &host_tree, &identity, "1.0", None); - plain_failure.dispose_after_build(false, false); - assert!(!plain_failure.root().exists(), "plain failure discards"); + // An incremental run: the staged tree is kept, success or + // failure. + for outcome in [OUTCOME_SUCCESS, OUTCOME_FAILED] { + let record = record_with_upper( + root.path(), + &tarball, + &host_tree, + &identity, + "1.0", + Some(outcome), + ); + assert!(record.has_upper(), "an incremental run keeps its tree"); + assert!(record.manifest().kept_upper); + assert!(record.root().join("manifest.json").is_file()); + assert!(record.log_file().is_file()); + drop(record); + } + } - let incremental_success = - record_with_environment(root.path(), &tarball, &host_tree, &identity, "1.0", None); - incremental_success.dispose_after_build(true, true); - assert!(incremental_success.root().exists(), "incremental keeps"); - assert_eq!(incremental_success.manifest().last_outcome, OUTCOME_SUCCESS); - drop(incremental_success); + /// The per-run chroot and overlay workdir are removed at the end of + /// the run — nothing in a record root survives except the manifest, + /// the log and the retained staged tree. + #[test] + fn run_state_is_removed_at_end_of_run() { + let (root, tarball) = sessions_fixture(); + let identity = test_identity(); + let host_tree = root.path().join("tree"); + let record = record_with_upper(root.path(), &tarball, &host_tree, &identity, "1.0", None); + fs::create_dir_all(record.chroot_path().join("bin")).unwrap(); + fs::write(record.chroot_path().join("bin/sh"), b"").unwrap(); + fs::create_dir_all(record.work_dir().join("scratch")).unwrap(); - let incremental_failure = - record_with_environment(root.path(), &tarball, &host_tree, &identity, "1.0", None); - incremental_failure.set_outcome(OUTCOME_FAILED); - incremental_failure.dispose_after_build(true, false); - assert!(incremental_failure.root().exists(), "incremental keeps"); - assert_eq!(incremental_failure.manifest().last_outcome, OUTCOME_FAILED); + record.remove_run_state(); + + assert!(!record.chroot_path().exists(), "the chroot is gone"); + assert!(!record.work_dir().exists(), "the overlay workdir is gone"); + assert!(record.has_upper(), "the staged tree survives"); + assert!(record.root().join("manifest.json").is_file()); + } + + /// A crashed run's leftover chroot is swept when the next run of the + /// identity starts; a live build's chroot (lock held) is left alone. + #[test] + fn crashed_chroots_are_swept_on_the_next_run() { + let (root, tarball) = sessions_fixture(); + let identity = test_identity(); + let host_tree = root.path().join("tree"); + let crashed = record_with_upper( + root.path(), + &tarball, + &host_tree, + &identity, + "1.0", + Some(OUTCOME_FAILED), + ); + fs::create_dir_all(crashed.chroot_path().join("bin")).unwrap(); + let live = record_with_upper(root.path(), &tarball, &host_tree, &identity, "1.0", None); + fs::create_dir_all(live.chroot_path().join("bin")).unwrap(); + drop(crashed); + + // The next run of the identity sweeps the unlocked leftovers. + drop(new_record_in(root.path(), &host_tree, &identity, "1.0").unwrap()); + + let swept = identity_records_in(root.path(), &identity) + .into_iter() + .find(|r| !lock_is_held(&r.lock_path()).unwrap()) + .expect("the crashed record is unlocked"); + assert!(!swept.chroot_path().exists(), "the leftover is swept"); + assert!( + live.chroot_path().exists(), + "the live build keeps its chroot" + ); + drop(live); + } + + /// `pkh deb log` resolves the newest record of a tree by default, an + /// exact id or unique prefix when given, and streams its log; a + /// terminal record's --follow just prints. + #[test] + fn record_log_resolution_and_streaming() { + let (root, tarball) = sessions_fixture(); + let identity = test_identity(); + let host_tree = root.path().join("tree"); + let first = metadata_record(root.path(), &host_tree, &identity, "1.0", OUTCOME_FAILED); + fs::write(first.log_file(), "first build log\n").unwrap(); + let first_id = first.manifest().id; + drop(first); + // The next record must land in another second: same-second ids + // are prefix-related (`` / `-02`), and the pinned + // lookup below must resolve the first record's id unambiguously. + std::thread::sleep(Duration::from_millis(1100)); + let newest = record_with_upper( + root.path(), + &tarball, + &host_tree, + &identity, + "1.1", + Some(OUTCOME_SUCCESS), + ); + fs::write(newest.log_file(), "newest build log\n").unwrap(); + drop(newest); + + // No id: the newest record of the tree. + let record = record_for_log(root.path(), &host_tree, None).unwrap(); + let mut out = Vec::new(); + print_record_log(&record, false, &mut out).unwrap(); + assert_eq!(String::from_utf8(out).unwrap(), "newest build log\n"); + + // The exact id of the older record; the record is terminal, so + // --follow just prints. + let record = record_for_log(root.path(), &host_tree, Some(&first_id)).unwrap(); + let mut out = Vec::new(); + print_record_log(&record, true, &mut out).unwrap(); + assert_eq!(String::from_utf8(out).unwrap(), "first build log\n"); + + // A wrong id is an error. + assert!(record_for_log(root.path(), &host_tree, Some("nonesuch")).is_err()); + } + + /// A record whose build captured no line yet streams an empty log. + #[test] + fn record_log_of_a_silent_build_is_empty() { + let (root, _tarball) = sessions_fixture(); + let identity = test_identity(); + let host_tree = root.path().join("tree"); + let record = new_record_in(root.path(), &host_tree, &identity, "1.0").unwrap(); + record.set_outcome(OUTCOME_SUCCESS); + drop(record); + let record = record_for_log(root.path(), &host_tree, None).unwrap(); + let mut out = Vec::new(); + print_record_log(&record, true, &mut out).unwrap(); + assert_eq!(out.len(), 0); } /// Ages render in the coarse units the list promises. diff --git a/src/main.rs b/src/main.rs index f5e4c0d..cc2a962 100644 --- a/src/main.rs +++ b/src/main.rs @@ -371,15 +371,22 @@ fn main() { .num_args(0..=1) .require_equals(false) .default_missing_value("") - .help("Incremental build: fork a previous build record of this target and keep the result") - .long_help("Incremental build: fork a previous build record of this target — its prepared chroot and installed build dependencies are copied into a new record, and at the same changelog version its build artifacts too (make recompiles only what changed).\nWithout a value, the newest successful record of the target (among the last five usable) is the base; with an ID (as shown by 'pkh deb list'), that exact record — a failed one is fine if its environment is.\nEvery --incremental run keeps its record, success or failure, so the next one can fork it; a plain 'pkh deb' is fully ephemeral (nothing is kept, even on failure or interrupt)."), + .help("Incremental build: fork a previous build record of this target and keep its staged tree") + .long_help("Incremental build: fork a previous build record of this target — at an unchanged changelog version and while the chroot tarball is unchanged, the base's staged tree is copied into a new record, so make recompiles only what changed.\nWithout a value, the newest successful record of the target (among the last five carrying a staged tree) is the base; with an ID (as shown by 'pkh deb list'), that exact record — a failed one is fine if its staged tree is.\nEvery --incremental run keeps its staged tree, success or failure, so the next one can fork it; a plain 'pkh deb' keeps only metadata and its build log (the chroot is per-run and always discarded)."), ) .arg(arg!(--verbose "Show raw tool output instead of the live build view").required(false) .long_help("Show raw tool output instead of the live build view.\nAlso implied by RUST_LOG=debug for pkh's own logs.")) .subcommand( Command::new("list") .about("List the build records recorded from this tree") - .long_about("List the build records recorded from this tree, with their outcome, version, fork lineage and the id to pass to 'pkh deb --incremental '.\nRead-only: never builds or removes anything (only the last five records per target are kept, 'pkh prune' removes the ones older than a week)."), + .long_about("List the build records recorded from this tree (plain runs too, as metadata records), with their outcome, version, fork lineage, whether their staged tree is kept, and the id to pass to 'pkh deb --incremental '.\nRead-only: never builds or removes anything (at most five staged trees per target are kept, 'pkh prune' removes records older than a week)."), + ) + .subcommand( + Command::new("log") + .about("Print a recorded build's log") + .long_about("Print the build log of a recorded build: the full tee of its output.\nWithout an ID, the newest record of this tree; with an ID (as shown by 'pkh deb list'), that exact record.\n--follow keeps reading a live build's log as it is written and exits once the build's record reaches a terminal state.") + .arg(arg!([id] "Record id (default: the newest record of this tree)").required(false)) + .arg(arg!(-f --follow "Keep reading the log as the build writes it")), ) ) .subcommand( @@ -896,6 +903,34 @@ fn main() { return; } + // `pkh deb log`: print a recorded build's log (optionally + // following a live build); read-only too. + if let Some(("log", log_matches)) = sub_matches.subcommand() { + let tree = cwd.canonicalize().unwrap_or(cwd.clone()); + let id = log_matches.get_one::("id").map(String::as_str); + let follow = log_matches.get_flag("follow"); + let record = match pkh::deb::session::record_for_log( + &pkh::deb::session::sessions_root(), + &tree, + id, + ) { + Ok(record) => record, + Err(e) => { + error!("{e}"); + std::process::exit(1); + } + }; + let mut out = std::io::stdout(); + if let Err(e) = pkh::deb::session::print_record_log(&record, follow, &mut out) { + error!( + "cannot print the build log of {}: {e}", + record.root().display() + ); + std::process::exit(1); + } + return; + } + // Ctrl+C during the build must say what happened and release the // ephemeral chroot instead of dying on the default disposition. // The live view (when enabled) registers its own reporter on top diff --git a/src/prune.rs b/src/prune.rs index 4bf4de3..58b542a 100644 --- a/src/prune.rs +++ b/src/prune.rs @@ -98,7 +98,8 @@ enum Artifact { Tarball(PathBuf), /// A build log file under `/logs`. LogFile(PathBuf), - /// A build record root (chroot + journal), one per build. + /// A build record root (manifest + log, and the staged tree of + /// `--incremental` runs), one per build. Session(PathBuf), /// A `pkh test` run directory (report artifacts). TestRun(PathBuf), @@ -184,7 +185,7 @@ fn discover_sessions(sessions_dir: &Path, all: bool) -> Vec { } else if is_stale_path(&path, all) { // A record root only ever holds directories; a plain // file inside a slug directory is a leftover of the old - // flat layout (its session.json / host-tree snapshot). + // flat layout (its manifest / host-tree snapshot). artifacts.push(Artifact::Session(path)); } } @@ -240,7 +241,7 @@ fn is_prunable_session(path: &Path, all: bool) -> bool { if all { return true; } - let last_used = fs::read_to_string(path.join("session.json")) + let last_used = fs::read_to_string(path.join("manifest.json")) .ok() .and_then(|content| serde_json::from_str::(&content).ok()) .and_then(|value| { @@ -1277,7 +1278,7 @@ none /tmp/other proc rw 0 0 fn write_session_manifest(session_dir: &Path, last_used: &str) { fs::create_dir_all(session_dir).unwrap(); fs::write( - session_dir.join("session.json"), + session_dir.join("manifest.json"), format!(r#"{{ "last_used": "{last_used}" }}"#), ) .unwrap(); @@ -1299,7 +1300,7 @@ none /tmp/other proc rw 0 0 &(chrono::Utc::now() - chrono::Duration::hours(1)).to_rfc3339(), ); fs::create_dir_all(&corrupt).unwrap(); - fs::write(corrupt.join("session.json"), "not json").unwrap(); + fs::write(corrupt.join("manifest.json"), "not json").unwrap(); let report = prune_in_roots( temp.path(),