vhaudiquet/pkh
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

deb: shrink records to upper, journal and log; keep metadata for every build

Browse Source 
Records carried a full chroot each, and space turned out to be the hard
constraint: five records per target multiplied by a multi-GB chroot does
not fit. The chroot is now per-run scratch — every build extracts a
fresh one from the sha-verified mmdebstrap tarball cache into its
record's chroot/ dir and removes it at the end (success, failure and
interrupt included); resume_chroot, the .pkh-ready probe and
chroot_is_usable are gone, and crashed-run leftovers are swept when the
next run of that identity starts.

What a record keeps instead: manifest.json (renamed, now with the
tarball sha it used, a kept_upper flag and a tree-only journal — the
environment phases rerun in every fresh chroot), a log/build.log full
tee of the build output (kept for EVERY build; plain runs become
metadata-only records, so keep-iff-incremental now applies to the
staged tree alone), and — for --incremental runs — tree-upper/ with its
host-files.list. The fork gate carries a base's staged tree only while
its recorded tarball sha still matches the cache and the changelog
version is unchanged; a fork candidate is any record with a staged tree.

The keep-last-5 cap now applies to staged trees: a sixth retention
degrades the oldest record to metadata-only instead of deleting it.
New `pkh deb log [-f] [<id>]` prints (or follows) a record's build log,
and `pkh deb list` shows the staged-tree state and plain-run records
too. README rewritten around the shrunk model.
This commit is contained in:
vhaudiquet
2026-09-29 14:00:34 +02:00
parent 81f0d52f85
commit dd2634abe3
7 changed files with 1287 additions and 946 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+35 -23
View File
@@ -122,44 +122,56 @@ at the oldest commit searched, which keeps the run fast on large histories.
### Incremental builds (build records) ### Incremental builds (build records)
Every `pkh deb` build runs in its own immutable **record** under Every `pkh deb` build records its metadata under
`/var/tmp/pkh/sessions/<target>/<id>/`: the bootstrapped chroot, the `/var/tmp/pkh/sessions/<target>/<id>/`: the manifest (identity, fork
installed build dependencies and the build artifacts of the staged lineage, outcome, the sha of the chroot tarball it used) and a full
tree. `--incremental` forks a previous record — a full copy, the base `log/build.log` tee of the build output. The chroot itself is per-run
is never modified — so a failed run can never destroy the last-good scratch — extracted fresh from the sha-verified mmdebstrap tarball cache
environment it was built from: on every run and removed at the end, plain and `--incremental` alike —
so records stay small. What `--incremental` buys is the record's
**staged tree** (`tree-upper/`): the overlay upperdir where the build's
writes land, holding the compiled objects.
`--incremental` forks a previous record — a copy, the base is never
modified — so a failed run can never destroy the last-good state it was
built from. The staged tree is carried only when the base's recorded
tarball sha still matches the cache (the toolchain did not move) and the
changelog version is unchanged; otherwise the fork builds from a fresh
tree:
``` ```
pkh deb # fully ephemeral: nothing is kept, even on pkh deb # plain run: fresh chroot, fresh tree; the
# failure or interrupt # record keeps only metadata and the log
# (nothing is kept even on failure or
# interrupt)
pkh deb --incremental # fork the newest successful record of the pkh deb --incremental # fork the newest successful record of the
# target (last five usable): the chroot, build # target (among the last five carrying a staged
# deps and, at the same version, the objects # tree) and keep this run's staged tree, for
# are reused; only what changed is recompiled. # the next fork to carry
# The result is always kept (success or
# failure) for the next fork
pkh deb --incremental <id> pkh deb --incremental <id>
# fork that exact record (see `pkh deb list`); # fork that exact record (see `pkh deb list`);
# a failed record is fine if its environment is # a failed record is fine if its staged tree is
pkh deb list # the records of this tree: id, outcome, pkh deb list # the records of this tree: id, outcome,
# version, fork lineage, age # version, fork lineage, staged-tree state, age
pkh deb log [-f] [<id>] # print a record's build log; -f follows a
# live build
pkh prune # garbage-collect records (7-day retention) pkh prune # garbage-collect records (7-day retention)
``` ```
Each target (series/arch) keeps its last five records — older ones are Each target keeps at most five staged trees: when a sixth is retained,
pruned as new ones are created, and `pkh prune` removes those untouched the oldest degrades to a metadata-only record (manifest and log stay),
for a week. `pkh deb --incremental` never forks a record built for a and `pkh prune` removes records untouched for a week. `pkh deb
different series/architecture, and a plain `pkh deb` still never --incremental` never forks a record built for a different
reuses anything: it builds in a throwaway record that is discarded on series/architecture.
success, failure and interrupt alike.
Packages whose `debian/rules` keeps an input-untracked stamp cache (the Packages whose `debian/rules` keeps an input-untracked stamp cache (the
kernels' `debian/stamps`) declare it in the quirks data kernels' `debian/stamps`) declare it in the quirks data
(`resume_clear`): forked builds drop the cache so the wrapper steps — (`resume_clear`): forked builds drop the cache so the wrapper steps —
including the kernel's flavour config export — re-run over the including the kernel's flavour config export — re-run over the
incremental inner builds, and a config edit is never silently ignored. incremental inner builds, and a config edit is never silently ignored.
If a build died hard (OOM kill), the next fork unmounts the leftover A build killed hard (OOM) can leave its chroot behind; the next run of
state first (escalating through sudo when needed). that target sweeps it (unmounting and escalating through sudo when
needed), and `pkh prune` collects the rest.
### Running the DEP-8 tests ### Running the DEP-8 tests
src/deb/ephemeral.rs
+180 -161
View File
@@ -13,7 +13,7 @@ use tar::Archive;
use xz2::read::XzDecoder; use xz2::read::XzDecoder;
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Interrupt-time chroot cleanup // Interrupt-time cleanup
// //
// On Ctrl-C, the watchdog in `crate::interrupt` runs the hook registered in // On Ctrl-C, the watchdog in `crate::interrupt` runs the hook registered in
// [`EphemeralContextGuard::new_with_context`] right before exiting — the // [`EphemeralContextGuard::new_with_context`] right before exiting — the
@@ -22,34 +22,61 @@ use xz2::read::XzDecoder;
// /proc and any overlayfs mounts. // /proc and any overlayfs mounts.
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
/// Interrupt-time teardown of a build tree. When `keep_tree` is set the /// Interrupt-time teardown of an anonymous temp chroot (no build record):
/// tree itself is left in place after unmounting (an `--incremental` /// unmount everything under it, then remove it — a plain `rm -rf` with a
/// record: the interrupt is a pause, not a discard); otherwise this is /// retry loop, best-effort. See [`sigint_cleanup_record`] for records.
/// the historical full cleanup. For a plain `pkh deb` the `path` is the fn sigint_cleanup_chroot(chroot_path: &Path) {
/// whole throwaway record root, so the record is removed — the record let is_root = unsafe { libc::geteuid() } == 0;
/// model's keep-iff-incremental policy applies to interrupts too. unmount_under_logged(chroot_path);
remove_with_retries(chroot_path, is_root);
}
/// Interrupt-time teardown of a build record: unmount everything under
/// the record root, remove the per-run state (the chroot and the overlay
/// scratch workdir) and — unless `keep_upper` is set — the staged tree
/// and its host-tree snapshot. The manifest and the build log always
/// stay: every build leaves a metadata record, the interrupt included.
/// ///
/// Unlike [`EphemeralContextGuard::drop`], this deliberately does NOT go /// Unlike [`EphemeralContextGuard::drop`], this deliberately does NOT go
/// through the context manager, the ephemeral context's driver (whose /// through the context manager, the ephemeral context's driver (whose
/// `cleanup()` unmounts the tracked overlays) or the base context's command /// `cleanup()` unmounts the tracked overlays) or the base context's
/// builder: interrupt-time hooks must be self-contained, and those /// command builder: interrupt-time hooks must be self-contained, and
/// machineries may be mid-mutation on the interrupted thread. Instead it /// those machineries may be mid-mutation on the interrupted thread.
/// only reads /proc/mounts and spawns umount/rm directly. /// Instead it only reads /proc/mounts and spawns umount/rm directly.
///
/// On a full cleanup it removes the tree regardless of the build result:
/// the build was aborted, and leaving a still-mounted chroot behind is
/// exactly the leak this hook exists to prevent.
/// ///
/// Best-effort by design: if a child process still holds a mount busy or /// Best-effort by design: if a child process still holds a mount busy or
/// privilege escalation is unavailable, individual steps fail; failures are /// privilege escalation is unavailable, individual steps fail; failures
/// logged (pointing at `pkh prune` for the leftovers) and never panic. /// are logged (pointing at `pkh prune` for the leftovers) and never
fn sigint_cleanup_chroot(chroot_path: &Path, keep_tree: bool) { /// panic.
fn sigint_cleanup_record(record_root: &Path, keep_upper: bool) {
let is_root = unsafe { libc::geteuid() } == 0; let is_root = unsafe { libc::geteuid() } == 0;
unmount_under_logged(record_root);
// Per-run state first: the chroot holds the /proc bind mount and the
// device nodes, the workdir holds root-owned overlay scratch.
for dir in ["chroot", "tree-upper-work"] {
remove_with_retries(&record_root.join(dir), is_root);
}
if keep_upper {
log::debug!(
"Interrupted build record kept its staged tree at {} (fork it \
with `pkh deb --incremental`)",
record_root.display()
);
} else {
for path in ["tree-upper", "host-files.list"] {
remove_with_retries(&record_root.join(path), is_root);
}
}
}
// Unmount children before parents: /proc/mounts lists mounts roughly in /// Unmount every mount at or under `base`, deepest first, logging the
// creation order, so walk it in reverse /// failures (interrupt-time: leftover mounts are `pkh prune`'s to
let mounts = host_mounts_under(chroot_path); /// collect).
for mount_point in mounts.into_iter().rev() { fn unmount_under_logged(base: &Path) {
let is_root = unsafe { libc::geteuid() } == 0;
// Unmount children before parents: /proc/mounts lists mounts roughly
// in creation order, so walk it in reverse.
for mount_point in host_mounts_under(base).into_iter().rev() {
if unmount_path(&mount_point, is_root) { if unmount_path(&mount_point, is_root) {
log::debug!( log::debug!(
"Unmounted {} during interrupt cleanup", "Unmounted {} during interrupt cleanup",
@@ -63,19 +90,13 @@ fn sigint_cleanup_chroot(chroot_path: &Path, keep_tree: bool) {
); );
} }
} }
}
if keep_tree { /// Privileged `rm -rf` with a retry loop: a child the Ctrl+C interrupted
log::debug!( /// may still be finishing its writeout — dpkg defers SIGINT until it
"Interrupted build record kept at {} (fork it with `pkh deb --incremental`)", /// reaches a safe state — so retry while rm reports the tree non-empty
chroot_path.display() /// instead of leaving it half-removed. A missing path succeeds at once.
); fn remove_with_retries(path: &Path, is_root: bool) {
return;
}
// Remove the chroot tree itself (tolerates a missing directory). A
// child the Ctrl+C interrupted may still be finishing its writeout —
// dpkg defers SIGINT until it reaches a safe state — so retry while rm
// reports the tree non-empty instead of leaving it half-removed.
const RETRIES: usize = 10; const RETRIES: usize = 10;
const RETRY_DELAY: std::time::Duration = std::time::Duration::from_millis(300); const RETRY_DELAY: std::time::Duration = std::time::Duration::from_millis(300);
let mut last = None; let mut last = None;
@@ -86,7 +107,7 @@ fn sigint_cleanup_chroot(chroot_path: &Path, keep_tree: bool) {
last = Some( last = Some(
privileged_command("rm", is_root) privileged_command("rm", is_root)
.arg("-rf") .arg("-rf")
.arg(chroot_path) .arg(path)
.status(), .status(),
); );
if matches!(&last, Some(Ok(status)) if status.success()) { if matches!(&last, Some(Ok(status)) if status.success()) {
@@ -95,22 +116,19 @@ fn sigint_cleanup_chroot(chroot_path: &Path, keep_tree: bool) {
} }
match last { match last {
Some(Ok(status)) if status.success() => { Some(Ok(status)) if status.success() => {
log::debug!( log::debug!("Removed {} during interrupt cleanup", path.display());
"Removed chroot {} during interrupt cleanup",
chroot_path.display()
);
} }
Some(Ok(status)) => { Some(Ok(status)) => {
log::error!( log::error!(
"Failed to remove chroot {} during interrupt cleanup \ "Failed to remove {} during interrupt cleanup \
(rm exited with {status}); run `pkh prune`", (rm exited with {status}); run `pkh prune`",
chroot_path.display() path.display()
); );
} }
Some(Err(e)) => { Some(Err(e)) => {
log::error!( log::error!(
"Failed to run rm for chroot {} during interrupt cleanup: {e}; run `pkh prune`", "Failed to run rm for {} during interrupt cleanup: {e}; run `pkh prune`",
chroot_path.display() path.display()
); );
} }
None => unreachable!("at least one rm attempt ran"), None => unreachable!("at least one rm attempt ran"),
@@ -305,10 +323,11 @@ fn unescape_mount_field(field: &str) -> String {
/// for building packages with unshare permissions. /// for building packages with unshare permissions.
/// ///
/// In session mode ([`EphemeralContextGuard::new_for_session`]) the chroot /// In session mode ([`EphemeralContextGuard::new_for_session`]) the chroot
/// lives inside a [`Session`] record: a forked record's chroot is reused /// lives inside a [`Session`] record's `chroot/` directory — extracted
/// when the caller verified it, and the keep policy is strict — the record /// fresh from the tarball cache on every run (nothing chroot-shaped is
/// is kept exactly for `--incremental` runs (`persist_record`), discarded /// persisted) and removed at the end of the run. The record itself is
/// for plain ones whatever the outcome. /// always kept as metadata and log; its staged tree is retained exactly
/// for `--incremental` runs (`keep_upper`).
pub struct EphemeralContextGuard { pub struct EphemeralContextGuard {
/// The ephemeral build context this guard created (an unshare context /// The ephemeral build context this guard created (an unshare context
/// bound to the chroot, parented on the base context). Held explicitly so /// bound to the chroot, parented on the base context). Held explicitly so
@@ -327,8 +346,9 @@ pub struct EphemeralContextGuard {
cleanup_hook: Option<CleanupHookGuard>, cleanup_hook: Option<CleanupHookGuard>,
/// The build record this guard works in, when any. /// The build record this guard works in, when any.
session: Option<Session>, session: Option<Session>,
/// Whether the record is kept after the build (`pkh deb --incremental`). /// Whether the record's staged tree is retained after the build
persist_record: bool, /// (`pkh deb --incremental`).
keep_upper: bool,
} }
impl EphemeralContextGuard { impl EphemeralContextGuard {
@@ -345,7 +365,7 @@ impl EphemeralContextGuard {
base_ctx: Arc<Context>, base_ctx: Arc<Context>,
view: &dyn BuildView, view: &dyn BuildView,
) -> Result<Self, Box<dyn Error>> { ) -> Result<Self, Box<dyn Error>> {
Self::new_internal(None, series, arch, base_ctx, view, false, false).await Self::new_internal(None, series, arch, base_ctx, view, false).await
} }
/// Create the ephemeral build context inside `session`'s chroot. /// Create the ephemeral build context inside `session`'s chroot.
@@ -357,31 +377,18 @@ impl EphemeralContextGuard {
/// * `arch` - Optional target architecture, as for /// * `arch` - Optional target architecture, as for
/// [`EphemeralContextGuard::new_with_context`] /// [`EphemeralContextGuard::new_with_context`]
/// * `base_ctx` - The base context to use for chroot work /// * `base_ctx` - The base context to use for chroot work
/// * `reuse_chroot` - Reuse the forked record's bootstrapped chroot /// * `keep_upper` - Retain the record's staged tree after the build
/// instead of re-extracting the tarball; the caller must only set /// (`pkh deb --incremental`); a plain run's record keeps metadata
/// this after the record's chroot passed the integrity probe /// and log only
/// * `persist_record` - Keep the record after the build
/// (`pkh deb --incremental`); a plain run's record is discarded
/// whatever the outcome
pub async fn new_for_session( pub async fn new_for_session(
session: Session, session: Session,
series: &str, series: &str,
arch: Option<&str>, arch: Option<&str>,
base_ctx: Arc<Context>, base_ctx: Arc<Context>,
view: &dyn BuildView, view: &dyn BuildView,
reuse_chroot: bool, keep_upper: bool,
persist_record: bool,
) -> Result<Self, Box<dyn Error>> { ) -> Result<Self, Box<dyn Error>> {
Self::new_internal( Self::new_internal(Some(session), series, arch, base_ctx, view, keep_upper).await
Some(session),
series,
arch,
base_ctx,
view,
reuse_chroot,
persist_record,
)
.await
} }
async fn new_internal( async fn new_internal(
@@ -390,8 +397,7 @@ impl EphemeralContextGuard {
arch: Option<&str>, arch: Option<&str>,
base_ctx: Arc<Context>, base_ctx: Arc<Context>,
view: &dyn BuildView, view: &dyn BuildView,
reuse_chroot: bool, keep_upper: bool,
persist_record: bool,
) -> Result<Self, Box<dyn Error>> { ) -> Result<Self, Box<dyn Error>> {
// Save the globally-installed context so Drop can restore exactly // Save the globally-installed context so Drop can restore exactly
// this handle: concurrent builds install their own ephemeral // this handle: concurrent builds install their own ephemeral
@@ -399,8 +405,8 @@ impl EphemeralContextGuard {
// before this guard swapped anything in. // before this guard swapped anything in.
let previous_context = context::current(); let previous_context = context::current();
// In session mode the chroot lives inside the record root (and is // In session mode the chroot lives inside the record root, as
// reused when the caller says so); otherwise it is a fresh temp dir. // per-run scratch; otherwise it is a fresh temp dir.
let chroot_path = match &session { let chroot_path = match &session {
Some(s) => { Some(s) => {
let path = s.chroot_path(); let path = s.chroot_path();
@@ -424,22 +430,23 @@ impl EphemeralContextGuard {
// base context: the hook must be self-contained (stored path + // base context: the hook must be self-contained (stored path +
// direct umount/rm subprocesses) and cannot go through `base_ctx`. // direct umount/rm subprocesses) and cannot go through `base_ctx`.
// For remote or nested bases the chroot lives elsewhere, and // For remote or nested bases the chroot lives elsewhere, and
// leftovers stay handled by `pkh prune` as before. An // leftovers stay handled by `pkh prune` as before.
// `--incremental` record is kept on interrupt (a pause, not a //
// discard); a plain run's whole record root and a plain temp chroot // Session mode tears the record's per-run state down (chroot and
// are removed. // overlay workdir) and keeps its staged tree exactly for
// `--incremental`; a plain temp chroot is removed entirely.
let cleanup_hook = if matches!(base_ctx.config, ContextConfig::Local) { let cleanup_hook = if matches!(base_ctx.config, ContextConfig::Local) {
Some(crate::interrupt::register_cleanup_hook(Box::new({ let hook: Box<dyn Fn() + Send> = match session.as_ref().map(|s| s.root().to_path_buf())
// Session mode tears the whole record root down (the {
// manifest is part of the record); plain temp chroots Some(record_root) => {
// only ever had the chroot dir. Box::new(move || sigint_cleanup_record(&record_root, keep_upper))
let cleanup_path = match &session { }
Some(s) => s.root().to_path_buf(), None => {
None => chroot_path.clone(), let chroot_path = chroot_path.clone();
}; Box::new(move || sigint_cleanup_chroot(&chroot_path))
let keep_tree = persist_record; }
move || sigint_cleanup_chroot(&cleanup_path, keep_tree) };
}))) Some(crate::interrupt::register_cleanup_hook(hook))
} else { } else {
log::debug!( log::debug!(
"Base context is not local; skipping interrupt-time cleanup registration for {}", "Base context is not local; skipping interrupt-time cleanup registration for {}",
@@ -448,18 +455,11 @@ impl EphemeralContextGuard {
None None
}; };
// Download and extract the chroot tarball (or reuse the record's // Download and extract the chroot tarball — always fresh; the
// prepared chroot) // record's chroot directory is per-run scratch.
if let Err(e) = Self::download_and_extract_chroot( if let Err(e) =
series, Self::download_and_extract_chroot(series, arch, &chroot_path, base_ctx.clone(), view)
arch, .await
&chroot_path,
base_ctx.clone(),
view,
session.as_ref(),
reuse_chroot,
)
.await
{ {
// On a Ctrl+C the interrupt watchdog owns the tree: keep the // On a Ctrl+C the interrupt watchdog owns the tree: keep the
// hook registered (forgetting the guard) so it removes the // hook registered (forgetting the guard) so it removes the
@@ -476,6 +476,17 @@ impl EphemeralContextGuard {
return Err(e); return Err(e);
} }
// Record the tarball digest this environment came from: the fork
// gate of a later `--incremental` run compares it against the
// cache to decide whether the record's staged tree is still
// reusable.
if let Some(s) = &session
&& let Ok(tarball) = chroot_tarball_path(series, arch)
&& let Ok(sha) = session::sha256_file(&tarball)
{
s.set_tarball_sha256(&sha);
}
// Switch to an ephemeral context to build the package in the chroot. // Switch to an ephemeral context to build the package in the chroot.
// The parent is the base context itself (the one that bootstrapped // The parent is the base context itself (the one that bootstrapped
// the chroot), wired through `with_parent` instead of a config-name // the chroot), wired through `with_parent` instead of a config-name
@@ -502,7 +513,7 @@ impl EphemeralContextGuard {
base_ctx, base_ctx,
cleanup_hook, cleanup_hook,
session, session,
persist_record, keep_upper,
}) })
} }
@@ -522,25 +533,10 @@ impl EphemeralContextGuard {
chroot_path: &PathBuf, chroot_path: &PathBuf,
ctx: Arc<context::Context>, ctx: Arc<context::Context>,
view: &dyn BuildView, view: &dyn BuildView,
session: Option<&Session>,
reuse_chroot: bool,
) -> Result<(), Box<dyn Error>> { ) -> Result<(), Box<dyn Error>> {
// Record fork: the base's chroot passed the caller's checks — // The chroot is per-run scratch: nothing is reused, so never
// skip the tarball extraction entirely, the environment (apt // extract over stale content.
// lists, installed build-deps) is still in there. clear_dir(chroot_path)?;
if let Some(s) = session {
if reuse_chroot && s.chroot_is_usable() {
log::info!(
"Reusing the prepared chroot of record {} (bootstrap skipped)",
s.manifest().id
);
Self::finalize_chroot(chroot_path, ctx, view)?;
return Ok(());
}
// Not reusing (fresh record or failed probe): never extract
// over stale content.
clear_dir(chroot_path)?;
}
// Clone ctx for use in create_device_nodes after download_chroot_tarball consumes it // Clone ctx for use in create_device_nodes after download_chroot_tarball consumes it
let ctx_for_devices = ctx.clone(); let ctx_for_devices = ctx.clone();
@@ -619,33 +615,13 @@ impl EphemeralContextGuard {
// Create device nodes in the chroot and bind-mount /proc // Create device nodes in the chroot and bind-mount /proc
Self::finalize_chroot(chroot_path, ctx_for_devices, view)?; Self::finalize_chroot(chroot_path, ctx_for_devices, view)?;
// Record the bootstrap in the record: marker file (the integrity // The bootstrap's tarball digest is recorded by the caller (the
// probe) and the tarball the chroot came from. // record's fork gate compares it against the cache later).
if let Some(s) = session {
std::fs::write(chroot_path.join(".pkh-ready"), b"")?;
let metadata = fs::metadata(&tarball_path)?;
let tarball_sha256 = session::sha256_file(&tarball_path)
.map_err(|e| format!("cannot hash the chroot tarball: {e}"))?;
s.set_chroot(session::ChrootInfo {
tarball: tarball_filename,
tarball_size: metadata.len(),
tarball_mtime_secs: metadata
.modified()
.ok()
.and_then(|t| t.duration_since(std::time::UNIX_EPOCH).ok())
.map(|d| d.as_secs())
.unwrap_or(0),
tarball_sha256,
ready: true,
});
log::debug!("Record {} chroot is ready", s.manifest().id);
}
Ok(()) Ok(())
} }
/// Device nodes and the /proc bind mount for a (possibly reused) /// Device nodes and the /proc bind mount for a fresh chroot:
/// chroot: idempotent finalization, safe to run on every attempt. /// idempotent finalization, safe to run on every run.
fn finalize_chroot( fn finalize_chroot(
chroot_path: &Path, chroot_path: &Path,
ctx: Arc<context::Context>, ctx: Arc<context::Context>,
@@ -660,8 +636,8 @@ impl EphemeralContextGuard {
// Bind mount /proc from host into chroot (before entering unshare namespace) // Bind mount /proc from host into chroot (before entering unshare namespace)
// This allows /proc to work in containers where mounting inside unshare fails. // This allows /proc to work in containers where mounting inside unshare fails.
// A reused chroot may still carry the mount from the previous // A chroot reused within the same run may still carry the mount:
// attempt: only mount when it is not already there. // only mount when it is not already there.
if is_mountpoint(&chroot_path.join("proc")) { if is_mountpoint(&chroot_path.join("proc")) {
log::debug!("/proc is already bind-mounted in the chroot"); log::debug!("/proc is already bind-mounted in the chroot");
} else { } else {
@@ -908,6 +884,11 @@ impl Drop for EphemeralContextGuard {
// drained, forgetting is a harmless no-op). // drained, forgetting is a harmless no-op).
if crate::interrupt::interrupted() { if crate::interrupt::interrupted() {
context::manager().set_current_ephemeral(self.previous_context.clone()); context::manager().set_current_ephemeral(self.previous_context.clone());
// The record keeps its metadata: mark the aborted build as
// failed before the hook tears the per-run state down.
if let Some(session) = &self.session {
session.set_outcome(session::OUTCOME_FAILED);
}
if let Some(hook) = self.cleanup_hook.take() { if let Some(hook) = self.cleanup_hook.take() {
std::mem::forget(hook); std::mem::forget(hook);
} }
@@ -942,11 +923,11 @@ impl Drop for EphemeralContextGuard {
// touches it. // touches it.
context::manager().set_current_ephemeral(self.previous_context.clone()); context::manager().set_current_ephemeral(self.previous_context.clone());
// Session mode: the tree is a build record. The /proc bind mount // Session mode: the record's per-run chroot comes down in every
// must come down in every case (a kept record has to be // case (nothing chroot-shaped is persisted — success, failure and
// unmount-clean for the next fork); the record itself follows the // interrupt all end the run), then the record is retained:
// keep policy — kept exactly for `--incremental` runs, discarded // metadata and log always, the staged tree exactly for
// for plain ones whatever the outcome. // `--incremental` runs.
if let Some(session) = &self.session { if let Some(session) = &self.session {
let proc_path = self.chroot_path.join("proc"); let proc_path = self.chroot_path.join("proc");
let is_root = crate::utils::root::is_root().unwrap_or(false); let is_root = crate::utils::root::is_root().unwrap_or(false);
@@ -960,7 +941,8 @@ impl Drop for EphemeralContextGuard {
.status() .status()
}; };
session.dispose_after_build(self.persist_record, self.build_succeeded); session.remove_run_state();
session.retain_upper(self.keep_upper, self.build_succeeded);
return; return;
} }
@@ -1058,7 +1040,7 @@ mod chroot_cleanup_tests {
fn sigint_cleanup_of_missing_chroot_is_a_noop() { fn sigint_cleanup_of_missing_chroot_is_a_noop() {
let dir = tempfile::tempdir().unwrap(); let dir = tempfile::tempdir().unwrap();
let missing = dir.path().join("no-such-chroot"); let missing = dir.path().join("no-such-chroot");
sigint_cleanup_chroot(&missing, false); sigint_cleanup_chroot(&missing);
assert!(!missing.exists()); assert!(!missing.exists());
} }
@@ -1081,24 +1063,61 @@ mod chroot_cleanup_tests {
std::fs::create_dir_all(chroot.join("rootfs")).unwrap(); std::fs::create_dir_all(chroot.join("rootfs")).unwrap();
std::fs::write(chroot.join("rootfs").join("file.txt"), "data").unwrap(); std::fs::write(chroot.join("rootfs").join("file.txt"), "data").unwrap();
sigint_cleanup_chroot(&chroot, false); sigint_cleanup_chroot(&chroot);
assert!(!chroot.exists()); assert!(!chroot.exists());
} }
/// The interrupt teardown of a kept tree (an `--incremental` record) /// The record interrupt teardown removes the per-run state and keeps
/// unmounts and leaves the tree in place — the pause, not the discard. /// the manifest and log; an `--incremental` record keeps its staged
/// tree, a plain one loses it.
#[test] #[test]
fn sigint_cleanup_keeps_the_tree_when_asked() { fn sigint_record_teardown_keeps_metadata_and_drops_the_chroot() {
let is_root = unsafe { libc::geteuid() } == 0;
if !is_root
&& !privileged_command("true", false)
.status()
.is_ok_and(|s| s.success())
{
return;
}
let dir = tempfile::tempdir().unwrap(); let dir = tempfile::tempdir().unwrap();
let record_root = dir.path().join("record"); let incremental = dir.path().join("incremental");
std::fs::create_dir_all(record_root.join("chroot/rootfs")).unwrap(); let plain = dir.path().join("plain");
std::fs::write(record_root.join("session.json"), "{}").unwrap(); for root in [&incremental, &plain] {
std::fs::create_dir_all(root.join("chroot/rootfs")).unwrap();
std::fs::create_dir_all(root.join("tree-upper/src")).unwrap();
std::fs::create_dir_all(root.join("log")).unwrap();
std::fs::write(root.join("manifest.json"), "{}").unwrap();
std::fs::write(root.join("log/build.log"), "output\n").unwrap();
std::fs::write(root.join("host-files.list"), "f\t1\t1\ta\n").unwrap();
}
sigint_cleanup_chroot(&record_root, true); sigint_cleanup_record(&incremental, true);
sigint_cleanup_record(&plain, false);
assert!(record_root.join("chroot/rootfs").exists()); for root in [&incremental, &plain] {
assert!(record_root.join("session.json").is_file()); assert!(!root.join("chroot").exists(), "the chroot is per-run");
assert!(root.join("manifest.json").is_file(), "metadata stays");
assert!(root.join("log/build.log").is_file(), "the log stays");
}
assert!(
incremental.join("tree-upper/src").exists(),
"an interrupted incremental run keeps its staged tree"
);
assert!(
incremental.join("host-files.list").is_file(),
"the snapshot stays with the staged tree"
);
assert!(
!plain.join("tree-upper").exists(),
"an interrupted plain run drops its staged tree"
);
assert!(
!plain.join("host-files.list").exists(),
"the snapshot goes with the staged tree"
);
} }
/// clear_dir removes content a plain `rm -rf` cannot: a root-owned /// clear_dir removes content a plain `rm -rf` cannot: a root-owned
src/deb/local.rs
+25 -57
View File
@@ -27,35 +27,12 @@ fn cap<'a>(
cmd cmd
} }
/// The session journal stamps this build works with: the input stamps of /// The input stamp of the patch phase, computed from the host tree: the
/// the patch and build-dependency phases, computed from the host tree and /// only tree phase the record journal tracks (the environment phases
/// the resolved selectors. /// rerun in every fresh chroot and are not recorded).
struct JournalStamps { fn patch_stamp(session: &session::Session) -> Option<String> {
patches: Option<String>,
control: Option<String>,
}
/// Compute the journal stamps for a session build; `None` without one.
fn journal_stamps(
session: &session::Session,
arch: &str,
series: &str,
pocket: Option<&str>,
cross: bool,
ppa: &[String],
inject_packages: &[String],
) -> JournalStamps {
let host_tree = PathBuf::from(session.manifest().host_tree); let host_tree = PathBuf::from(session.manifest().host_tree);
let selectors = format!( session::patches_stamp(&host_tree)
"{arch}\n{cross}\n{series}\n{}\n{}\n{}",
pocket.unwrap_or(""),
ppa.join(","),
inject_packages.join(",")
);
JournalStamps {
patches: session::patches_stamp(&host_tree),
control: session::control_stamp(&host_tree, &selectors),
}
} }
#[allow(clippy::too_many_arguments)] #[allow(clippy::too_many_arguments)]
@@ -74,13 +51,12 @@ pub async fn build(
view: &dyn BuildView, view: &dyn BuildView,
jobs: Option<usize>, jobs: Option<usize>,
session: Option<&session::Session>, session: Option<&session::Session>,
resume: bool, build_resume: bool,
) -> Result<Vec<PathBuf>, Box<dyn Error>> { ) -> Result<Vec<PathBuf>, Box<dyn Error>> {
let sink: Option<Arc<dyn LineSink>> = view.sink(); let sink: Option<Arc<dyn LineSink>> = view.sink();
// Journal stamps for a session build; recording is a no-op without one. // Journal stamp for a record build; recording is a no-op without one.
let stamps = let stamps = session.and_then(patch_stamp);
session.map(|s| journal_stamps(s, arch, series, pocket, cross, ppa, inject_packages));
// Environment // Environment
let mut env = HashMap::<String, String>::new(); let mut env = HashMap::<String, String>::new();
@@ -245,9 +221,6 @@ pub async fn build(
or re-run with RUST_LOG=debug for more details." or re-run with RUST_LOG=debug for more details."
.into()); .into());
} }
if let Some(s) = session {
s.record_phase("apt_update", None);
}
// Install essential packages // Install essential packages
log::debug!("Installing essential packages for local build..."); log::debug!("Installing essential packages for local build...");
@@ -272,9 +245,6 @@ pub async fn build(
if !status.success() { if !status.success() {
return Err("Could not install essential packages for the build".into()); return Err("Could not install essential packages for the build".into());
} }
if let Some(s) = session {
s.record_phase("essentials", None);
}
// The package directory was resolved by the caller (the staged copy of // The package directory was resolved by the caller (the staged copy of
// the tree the user pointed at, or the name-pattern search fallback) // the tree the user pointed at, or the name-pattern search fallback)
@@ -294,10 +264,11 @@ pub async fn build(
} }
// Apply quilt patches if the package provides a patch series. On a // Apply quilt patches if the package provides a patch series. On a
// resumed session the previous attempt's patches may still be applied: // build resuming over a carried upperdir the previous attempt's
// pop them first when the patch tree changed since. // patches may still be applied: pop them first when the patch tree
let patch_stamp = stamps.as_ref().and_then(|s| s.patches.clone()); // changed since.
let pop_patches = resume && { let patch_stamp = stamps.clone();
let pop_patches = build_resume && {
let previous = session let previous = session
.and_then(|s| s.manifest().phases.get("patches").map(|p| p.stamp.clone())) .and_then(|s| s.manifest().phases.get("patches").map(|p| p.stamp.clone()))
.unwrap_or(None); .unwrap_or(None);
@@ -333,20 +304,17 @@ pub async fn build(
view, view,
&sink, &sink,
)?; )?;
if let Some(s) = session { // (The build-dependency phase reruns in every fresh chroot: nothing
s.record_phase( // to journal.)
"build_deps",
stamps.as_ref().and_then(|st| st.control.clone()),
);
}
// A resumed build continues in the previous attempt's tree. Packages // A build resuming over a carried upperdir continues in the previous
// whose debian/rules keeps an input-untracked stamp cache (quirk: // attempt's tree. Packages whose `debian/rules` keeps an
// `resume_clear`, e.g. the kernels' `debian/stamps`, whose flavour // input-untracked stamp cache (quirk: `resume_clear`, e.g. the
// config rule exports .config with no config prerequisite) would build // kernels' `debian/stamps`, whose flavour config rule exports .config
// from the previous attempt's configuration: drop the listed caches so // with no config prerequisite) would build from the previous
// the wrapper steps re-run over the incremental inner builds. // attempt's configuration: drop the listed caches so the wrapper
if resume { // steps re-run over the incremental inner builds.
if build_resume {
for rel in crate::quirks::get_resume_clear_paths(package, series) { for rel in crate::quirks::get_resume_clear_paths(package, series) {
let path = Path::new(package_dir_str).join(&rel); let path = Path::new(package_dir_str).join(&rel);
if !ctx.exists(&path).unwrap_or(false) { if !ctx.exists(&path).unwrap_or(false) {
@@ -383,10 +351,10 @@ pub async fn build(
s.record_phase("build", None); s.record_phase("build", None);
} }
// On a resumed session the previous attempt's debian/files would make // On a resumed build the previous attempt's debian/files would make
// the artifact collection below surface its outputs: only the new // the artifact collection below surface its outputs: only the new
// attempt's registration may remain. // attempt's registration may remain.
if resume { if build_resume {
let _ = ctx let _ = ctx
.command("rm") .command("rm")
.arg("-f") .arg("-f")
src/deb/mod.rs
+195 -106
View File
@@ -119,36 +119,87 @@ pub struct DebBuildOptions<'a> {
/// Explicit build context; defaults to the current context. /// Explicit build context; defaults to the current context.
pub ctx: Option<Arc<Context>>, pub ctx: Option<Arc<Context>>,
/// Incremental request (`pkh deb --incremental [<id>]`): which /// Incremental request (`pkh deb --incremental [<id>]`): which
/// record to fork as the base of this build, and whether this /// record to fork as the base of this build's staged tree, and
/// build's own record is kept. Defaults to a plain throwaway run. /// whether this build's own staged tree is retained. Defaults to a
/// plain run.
pub incremental: Incremental, pub incremental: Incremental,
/// Where build events (phases, progress, outcome) are reported. /// Where build events (phases, progress, outcome) are reported.
pub view: &'a dyn BuildView, pub view: &'a dyn BuildView,
} }
/// The `--incremental` request of a `pkh deb` build: which record the /// The `--incremental` request of a `pkh deb` build: which record the
/// build forks its environment from, and whether its own record is /// build forks its staged tree from, and whether its own staged tree is
/// kept afterwards. /// retained afterwards (the record's metadata and log are kept for
/// every build either way).
#[derive(Debug, Clone, Default, PartialEq, Eq)] #[derive(Debug, Clone, Default, PartialEq, Eq)]
pub enum Incremental { pub enum Incremental {
/// A plain run (`pkh deb` without `--incremental`): build from /// A plain run (`pkh deb` without `--incremental`): build from
/// scratch in a throwaway record, discarded on success, failure /// scratch in a record whose staged tree is dropped at the end —
/// and interrupt alike. /// metadata and log only remain.
#[default] #[default]
Off, Off,
/// `--incremental`: fork the auto-selected base — the newest /// `--incremental`: fork the auto-selected base — the newest
/// successful record of this identity among the last five usable — /// successful record of this identity among the last five carrying
/// and always keep the new record. With no usable candidate the /// a staged tree — and retain the new record's staged tree. With no
/// build runs from scratch and its record becomes the identity's /// candidate the build runs from scratch and its record becomes
/// first base. /// the identity's first base.
Auto, Auto,
/// `--incremental <id>`: fork that exact record (unique id or /// `--incremental <id>`: fork that exact record (unique id or
/// prefix, restricted to the records of this host tree; a FAILED /// prefix, restricted to the records of this host tree; a FAILED
/// record is a valid base — its environment may be fine, and the /// record is a valid base — the user pinned it deliberately) and
/// user pinned it deliberately) and always keep the new record. /// retain the new record's staged tree.
Id(String), Id(String),
} }
/// A view delegating every event to `inner` while teeing all captured
/// subprocess lines into a build record's log through [`RecordLogSink`]:
/// everything downstream (the chroot bootstrap, the build phases) asks
/// the view for its sink, so one wrapper tees the whole run.
struct RecordLogView<'a> {
/// The caller's view.
inner: &'a dyn BuildView,
/// The record log sink, forwarding to the inner view's sink too.
log: Arc<session::RecordLogSink>,
}
impl BuildView for RecordLogView<'_> {
fn target(&self, target: crate::report::BuildTarget) {
self.inner.target(target);
}
fn phase(&self, name: &str, classifier: Box<dyn Classifier>) {
self.inner.phase(name, classifier);
}
fn message(&self, text: &str) {
self.inner.message(text);
}
fn progress(&self, label: &str, pos: usize, total: usize) {
self.inner.progress(label, pos, total);
}
fn sink(&self) -> Option<Arc<dyn crate::context::LineSink>> {
Some(self.log.clone())
}
fn finish_success(&self, artifacts: &[PathBuf]) {
self.inner.finish_success(artifacts);
}
fn finish_failure(&self) {
self.inner.finish_failure();
}
fn suspend(&self) {
self.inner.suspend();
}
fn is_enabled(&self) -> bool {
self.inner.is_enabled()
}
}
impl Default for DebBuildOptions<'_> { impl Default for DebBuildOptions<'_> {
fn default() -> Self { fn default() -> Self {
static QUIET: crate::report::Quiet = crate::report::Quiet; static QUIET: crate::report::Quiet = crate::report::Quiet;
@@ -281,7 +332,7 @@ async fn build_binary_package_impl(
cross, cross,
}; };
let mut persist_record = false; let mut keep_upper = false;
let session = if session_enabled { let session = if session_enabled {
match resolve_record( match resolve_record(
incremental, incremental,
@@ -291,8 +342,8 @@ async fn build_binary_package_impl(
series, series,
chroot_arch, chroot_arch,
) { ) {
Ok((record, persist)) => { Ok((record, keep)) => {
persist_record = persist; keep_upper = keep;
Some(record) Some(record)
} }
Err(e) => { Err(e) => {
@@ -303,9 +354,26 @@ async fn build_binary_package_impl(
} else { } else {
None None
}; };
// Only a forked record carries a bootstrapped chroot at this point;
// fresh ones (plain runs and first `--incremental` runs) extract it. // Tee every captured build line into the record's log: the wrapper
let resume_chroot = session.as_ref().is_some_and(|s| s.chroot_is_usable()); // view hands the combined sink to everything downstream (the chroot
// bootstrap, the build phases), so the log is a full tee whatever
// renders the output. A record-less build (non-local base context)
// keeps the plain view.
let record_log: Option<Result<Arc<session::RecordLogSink>, std::io::Error>> = session
.as_ref()
.map(|s| session::RecordLogSink::new(s, opts.view.sink()).map(Arc::new));
let view: &dyn BuildView = match record_log {
Some(Ok(log)) => &RecordLogView {
inner: opts.view,
log,
},
Some(Err(e)) => {
log::warn!("Cannot open the build record's log: {e}");
opts.view
}
None => opts.view,
};
let mut guard = if let Some(s) = session.clone() { let mut guard = if let Some(s) = session.clone() {
Some( Some(
@@ -315,8 +383,7 @@ async fn build_binary_package_impl(
chroot_arch, chroot_arch,
base_ctx.clone(), base_ctx.clone(),
view, view,
resume_chroot, keep_upper,
persist_record,
) )
.await?, .await?,
) )
@@ -454,7 +521,7 @@ async fn build_binary_package_impl(
view, view,
*jobs, *jobs,
session.as_ref(), session.as_ref(),
resume_chroot, build_resume,
) )
.await? .await?
} }
@@ -486,19 +553,17 @@ async fn build_binary_package_impl(
// the chroot) inherit the terminal, so they must not fight the widget. // the chroot) inherit the terminal, so they must not fight the widget.
view.suspend(); view.suspend();
// Mark the build as successful for the guard's drop, which disposes of // Mark the build as successful for the guard's drop, which tears the
// the record per the keep policy: an `--incremental` record is kept // per-run chroot down and retains the record: metadata and log for
// with its outcome recorded, a plain run's throwaway record is // every build, the staged tree exactly for `--incremental` runs.
// always discarded.
if result.is_ok() if result.is_ok()
&& let Some(g) = &mut guard && let Some(g) = &mut guard
{ {
g.mark_build_successful(); g.mark_build_successful();
} }
// A failed `--incremental` run keeps its record for a later fork: // Every record's outcome is written — plain runs leave a metadata
// record the outcome (discarded plain runs need none). // record too.
if result.is_err() if result.is_err()
&& persist_record
&& let Some(s) = &session && let Some(s) = &session
{ {
s.set_outcome(session::OUTCOME_FAILED); s.set_outcome(session::OUTCOME_FAILED);
@@ -518,10 +583,9 @@ fn resolve_record(
series: &str, series: &str,
chroot_arch: Option<&str>, chroot_arch: Option<&str>,
) -> Result<(session::Session, bool), Box<dyn Error>> { ) -> Result<(session::Session, bool), Box<dyn Error>> {
let expected = ephemeral::chroot_tarball_path(series, chroot_arch).ok();
resolve_record_in( resolve_record_in(
&session::sessions_root(), &session::sessions_root(),
expected.as_deref(), &session::cache_tarball_sha256(series, chroot_arch),
incremental, incremental,
host_tree, host_tree,
identity, identity,
@@ -530,30 +594,22 @@ fn resolve_record(
} }
/// The testable core of [`resolve_record`] over an explicit sessions root /// The testable core of [`resolve_record`] over an explicit sessions root
/// and bootstrap tarball. /// and cached-tarball digest (the fork gate's reference).
/// ///
/// Every local build runs in its own record: [`Incremental::Off`] gets a /// Every local build runs in its own record: [`Incremental::Off`] gets a
/// brand-new throwaway one, [`Incremental::Auto`] and /// brand-new one, [`Incremental::Auto`] and [`Incremental::Id`] fork the
/// [`Incremental::Id`] fork the selected base. The returned flag says /// selected base. The returned flag says whether the record's staged
/// whether the record is kept after the build — exactly the /// tree is retained after the build — exactly the `--incremental` runs
/// `--incremental` runs keep theirs. /// retain theirs. Metadata and the log are kept for every build either
/// /// way.
/// A base must be usable: its chroot passes the integrity probe and was
/// bootstrapped from the tarball this build would use (`expected_tarball`
/// is `None` when the path cannot be resolved — no record is usable
/// then, which degrades an `--incremental` run to a fresh build).
fn resolve_record_in( fn resolve_record_in(
records_root: &Path, records_root: &Path,
expected_tarball: Option<&Path>, cache_tarball_sha256: &str,
incremental: &Incremental, incremental: &Incremental,
host_tree: &Path, host_tree: &Path,
identity: &session::SessionIdentity, identity: &session::SessionIdentity,
version: &str, version: &str,
) -> Result<(session::Session, bool), Box<dyn Error>> { ) -> Result<(session::Session, bool), Box<dyn Error>> {
let usable = |s: &session::Session| {
s.chroot_is_usable()
&& expected_tarball.is_some_and(|t| session::tarball_matches_path(s, t))
};
match incremental { match incremental {
Incremental::Off => Ok(( Incremental::Off => Ok((
session::new_record_in(records_root, host_tree, identity, version)?, session::new_record_in(records_root, host_tree, identity, version)?,
@@ -561,14 +617,12 @@ fn resolve_record_in(
)), )),
Incremental::Auto => { Incremental::Auto => {
// A record locked by a live build cannot be a base: its // A record locked by a live build cannot be a base: its
// chroot may be mid-mutation (apt always runs, deps land // staged tree may be mid-mutation by the running build.
// inside it), and the fork would copy that state.
let candidates: Vec<session::Session> = let candidates: Vec<session::Session> =
session::identity_records_in(records_root, identity) session::fork_candidates_in(records_root, identity)
.into_iter() .into_iter()
.filter(|s| usable(s))
.filter(|s| !session::lock_is_held(&s.lock_path()).unwrap_or(true)) .filter(|s| !session::lock_is_held(&s.lock_path()).unwrap_or(true))
.take(session::KEEP_RECORDS) .take(session::KEEP_UPPERS)
.collect(); .collect();
for base in session::rank_base_candidates(candidates) { for base in session::rank_base_candidates(candidates) {
let why = if base.manifest().last_outcome == session::OUTCOME_SUCCESS { let why = if base.manifest().last_outcome == session::OUTCOME_SUCCESS {
@@ -576,7 +630,14 @@ fn resolve_record_in(
} else { } else {
"no successful record among the last usable ones; the newest record" "no successful record among the last usable ones; the newest record"
}; };
match session::fork_record_in(records_root, &base, host_tree, identity, version) { match session::fork_record_in(
records_root,
&base,
host_tree,
identity,
version,
cache_tarball_sha256,
) {
Ok(record) => { Ok(record) => {
log::info!( log::info!(
"Base of the incremental build: record {} ({why})", "Base of the incremental build: record {} ({why})",
@@ -591,8 +652,8 @@ fn resolve_record_in(
} }
} }
log::info!( log::info!(
"No usable record of {identity} to fork; building from scratch — \ "No record of {identity} carrying a staged tree to fork; building \
this run's record becomes the target's first base" from scratch — this run's record becomes the target's first base"
); );
Ok(( Ok((
session::new_record_in(records_root, host_tree, identity, version)?, session::new_record_in(records_root, host_tree, identity, version)?,
@@ -603,11 +664,10 @@ fn resolve_record_in(
let mut matches: Vec<session::Session> = session::by_id_prefix_in(records_root, id) let mut matches: Vec<session::Session> = session::by_id_prefix_in(records_root, id)
.into_iter() .into_iter()
.filter(|s| s.manifest().host_tree == host_tree.to_string_lossy()) .filter(|s| s.manifest().host_tree == host_tree.to_string_lossy())
.filter(|s| usable(s))
.collect(); .collect();
match matches.len() { match matches.len() {
0 => Err(format!( 0 => Err(format!(
"no usable build record matches id '{id}' for this tree \ "no build record matches id '{id}' for this tree \
(see `pkh deb list`)" (see `pkh deb list`)"
) )
.into()), .into()),
@@ -618,7 +678,14 @@ fn resolve_record_in(
base.manifest().id base.manifest().id
); );
Ok(( Ok((
session::fork_record_in(records_root, &base, host_tree, identity, version)?, session::fork_record_in(
records_root,
&base,
host_tree,
identity,
version,
cache_tarball_sha256,
)?,
true, true,
)) ))
} }
@@ -1354,30 +1421,37 @@ mod tests {
// ------------------------------------------------------------------- // -------------------------------------------------------------------
use crate::deb::session::record_fixtures::{ use crate::deb::session::record_fixtures::{
record_with_environment, sessions_fixture, test_identity, metadata_record, record_with_upper, sessions_fixture, test_identity,
}; };
use crate::deb::session::{self, SessionIdentity}; use crate::deb::session::{self, SessionIdentity};
/// A plain run resolves to a brand-new, lineage-free record that the /// The sha of the fixture's cached tarball (the fork gate's
/// caller will discard — including when no record exists at all yet. /// reference).
fn cache_sha(tarball: &std::path::Path) -> String {
session::sha256_file(tarball).unwrap()
}
/// A plain run resolves to a brand-new, lineage-free record whose
/// staged tree will be dropped — including when no record exists at
/// all yet.
#[test] #[test]
fn resolve_plain_run_builds_a_fresh_discardable_record() { fn resolve_plain_run_builds_a_fresh_discardable_record() {
let (root, tarball) = sessions_fixture(); let (root, tarball) = sessions_fixture();
let identity = test_identity(); let identity = test_identity();
let host_tree = root.path().join("tree"); let host_tree = root.path().join("tree");
let (record, persist) = resolve_record_in( let (record, keep_upper) = resolve_record_in(
root.path(), root.path(),
Some(&tarball), &cache_sha(&tarball),
&Incremental::Off, &Incremental::Off,
&host_tree, &host_tree,
&identity, &identity,
"1.0", "1.0",
) )
.expect("a plain run always resolves to a fresh record"); .expect("a plain run always resolves to a fresh record");
assert!(!persist, "a plain run discards its record"); assert!(!keep_upper, "a plain run drops its staged tree");
assert_eq!(record.manifest().base, None); assert_eq!(record.manifest().base, None);
assert!(!record.chroot_is_usable()); assert!(!record.has_upper());
} }
/// The user scenario, end to end over the record layout: record A /// The user scenario, end to end over the record layout: record A
@@ -1388,7 +1462,7 @@ mod tests {
let (root, tarball) = sessions_fixture(); let (root, tarball) = sessions_fixture();
let identity = test_identity(); let identity = test_identity();
let host_tree = root.path().join("tree"); let host_tree = root.path().join("tree");
let a = record_with_environment( let a = record_with_upper(
root.path(), root.path(),
&tarball, &tarball,
&host_tree, &host_tree,
@@ -1397,56 +1471,67 @@ mod tests {
Some(session::OUTCOME_SUCCESS), Some(session::OUTCOME_SUCCESS),
); );
let a_id = a.manifest().id; let a_id = a.manifest().id;
let b = session::fork_record_in(root.path(), &a, &host_tree, &identity, "1.0").unwrap(); let b = session::fork_record_in(
root.path(),
&a,
&host_tree,
&identity,
"1.0",
&cache_sha(&tarball),
)
.unwrap();
let b_id = b.manifest().id; let b_id = b.manifest().id;
b.set_outcome(session::OUTCOME_FAILED); b.set_outcome(session::OUTCOME_FAILED);
drop((a, b)); drop((a, b));
let (record, persist) = resolve_record_in( let (record, keep_upper) = resolve_record_in(
root.path(), root.path(),
Some(&tarball), &cache_sha(&tarball),
&Incremental::Auto, &Incremental::Auto,
&host_tree, &host_tree,
&identity, &identity,
"1.0", "1.0",
) )
.expect("the auto selection must fork a usable record"); .expect("the auto selection must fork a candidate record");
assert!(persist); assert!(keep_upper);
assert_eq!(record.manifest().base.as_deref(), Some(a_id.as_str())); assert_eq!(record.manifest().base.as_deref(), Some(a_id.as_str()));
assert_ne!(record.manifest().id, b_id); assert_ne!(record.manifest().id, b_id);
// The successful base's gate passes: its staged tree is carried.
assert!(record.has_upper());
} }
/// A first `--incremental` run with no candidate builds completely /// A first `--incremental` run with no candidate builds completely
/// fresh and persists its record — it becomes the identity's first /// fresh and retains its staged tree — it becomes the identity's
/// base. /// first base.
#[test] #[test]
fn resolve_auto_without_candidates_builds_fresh_and_persists() { fn resolve_auto_without_candidates_builds_fresh_and_retains() {
let (root, tarball) = sessions_fixture(); let (root, tarball) = sessions_fixture();
let identity = test_identity(); let identity = test_identity();
let host_tree = root.path().join("tree"); let host_tree = root.path().join("tree");
let (record, persist) = resolve_record_in( let (record, keep_upper) = resolve_record_in(
root.path(), root.path(),
Some(&tarball), &cache_sha(&tarball),
&Incremental::Auto, &Incremental::Auto,
&host_tree, &host_tree,
&identity, &identity,
"1.0", "1.0",
) )
.expect("no candidates must degrade to a fresh build"); .expect("no candidates must degrade to a fresh build");
assert!(persist, "the first record becomes a base"); assert!(keep_upper, "the first record becomes a base");
assert_eq!(record.manifest().base, None); assert_eq!(record.manifest().base, None);
} }
/// Non-usable records never make it into the selection: a newer /// Records without a staged tree (plain runs, metadata-only ones)
/// successful record with a broken chroot loses to an older usable /// never make it into the selection: a newer successful
/// failed one. /// metadata-only record loses to an older failed one carrying a
/// staged tree.
#[test] #[test]
fn resolve_auto_skips_non_usable_records() { fn resolve_auto_skips_records_without_a_staged_tree() {
let (root, tarball) = sessions_fixture(); let (root, tarball) = sessions_fixture();
let identity = test_identity(); let identity = test_identity();
let host_tree = root.path().join("tree"); let host_tree = root.path().join("tree");
let usable = record_with_environment( let carrier = record_with_upper(
root.path(), root.path(),
&tarball, &tarball,
&host_tree, &host_tree,
@@ -1454,33 +1539,37 @@ mod tests {
"1.0", "1.0",
Some(session::OUTCOME_FAILED), Some(session::OUTCOME_FAILED),
); );
let usable_id = usable.manifest().id; let carrier_id = carrier.manifest().id;
drop(usable); drop(carrier);
// Newer, successful, but its chroot never finished bootstrapping. // Newer and successful, but a plain run: no staged tree.
let broken = session::new_record_in(root.path(), &host_tree, &identity, "1.0").unwrap(); drop(metadata_record(
broken.set_outcome(session::OUTCOME_SUCCESS); root.path(),
drop(broken); &host_tree,
&identity,
"1.0",
session::OUTCOME_SUCCESS,
));
let (record, _) = resolve_record_in( let (record, _) = resolve_record_in(
root.path(), root.path(),
Some(&tarball), &cache_sha(&tarball),
&Incremental::Auto, &Incremental::Auto,
&host_tree, &host_tree,
&identity, &identity,
"1.0", "1.0",
) )
.expect("the usable record is still a base"); .expect("the record carrying a staged tree is still the base");
assert_eq!(record.manifest().base.as_deref(), Some(usable_id.as_str())); assert_eq!(record.manifest().base.as_deref(), Some(carrier_id.as_str()));
} }
/// `--incremental <id>` may pin a FAILED record — its environment may /// `--incremental <id>` may pin a FAILED record — the user chose it
/// be fine and the user chose it deliberately. /// deliberately and its staged tree may be perfectly fine.
#[test] #[test]
fn resolve_pinned_id_forks_a_failed_record() { fn resolve_pinned_id_forks_a_failed_record() {
let (root, tarball) = sessions_fixture(); let (root, tarball) = sessions_fixture();
let identity = test_identity(); let identity = test_identity();
let host_tree = root.path().join("tree"); let host_tree = root.path().join("tree");
let failed = record_with_environment( let failed = record_with_upper(
root.path(), root.path(),
&tarball, &tarball,
&host_tree, &host_tree,
@@ -1491,17 +1580,18 @@ mod tests {
let failed_id = failed.manifest().id; let failed_id = failed.manifest().id;
drop(failed); drop(failed);
let (record, persist) = resolve_record_in( let (record, keep_upper) = resolve_record_in(
root.path(), root.path(),
Some(&tarball), &cache_sha(&tarball),
&Incremental::Id(failed_id.clone()), &Incremental::Id(failed_id.clone()),
&host_tree, &host_tree,
&identity, &identity,
"1.0", "1.0",
) )
.expect("a pinned failed record is a valid base"); .expect("a pinned failed record is a valid base");
assert!(persist); assert!(keep_upper);
assert_eq!(record.manifest().base.as_deref(), Some(failed_id.as_str())); assert_eq!(record.manifest().base.as_deref(), Some(failed_id.as_str()));
assert!(record.has_upper(), "the gate passes: sha and version match");
} }
/// A pinned id is restricted to the records of this host tree: an id /// A pinned id is restricted to the records of this host tree: an id
@@ -1512,7 +1602,7 @@ mod tests {
let identity = test_identity(); let identity = test_identity();
let host_tree = root.path().join("tree"); let host_tree = root.path().join("tree");
let other_tree = root.path().join("elsewhere"); let other_tree = root.path().join("elsewhere");
let foreign = record_with_environment( let foreign = record_with_upper(
root.path(), root.path(),
&tarball, &tarball,
&other_tree, &other_tree,
@@ -1525,7 +1615,7 @@ mod tests {
let err = resolve_record_in( let err = resolve_record_in(
root.path(), root.path(),
Some(&tarball), &cache_sha(&tarball),
&Incremental::Id(foreign_id), &Incremental::Id(foreign_id),
&host_tree, &host_tree,
&identity, &identity,
@@ -1533,8 +1623,7 @@ mod tests {
) )
.expect_err("records of other trees must not be forked by id"); .expect_err("records of other trees must not be forked by id");
assert!( assert!(
err.to_string() err.to_string().contains("no build record matches id"),
.contains("no usable build record matches id"),
"wrong error: {err}" "wrong error: {err}"
); );
} }
@@ -1545,7 +1634,7 @@ mod tests {
let (root, tarball) = sessions_fixture(); let (root, tarball) = sessions_fixture();
let identity = test_identity(); let identity = test_identity();
let host_tree = root.path().join("tree"); let host_tree = root.path().join("tree");
drop(record_with_environment( drop(record_with_upper(
root.path(), root.path(),
&tarball, &tarball,
&host_tree, &host_tree,
@@ -1553,7 +1642,7 @@ mod tests {
"1.0", "1.0",
None, None,
)); ));
drop(record_with_environment( drop(record_with_upper(
root.path(), root.path(),
&tarball, &tarball,
&host_tree, &host_tree,
@@ -1567,7 +1656,7 @@ mod tests {
let err = resolve_record_in( let err = resolve_record_in(
root.path(), root.path(),
Some(&tarball), &cache_sha(&tarball),
&Incremental::Id(prefix), &Incremental::Id(prefix),
&host_tree, &host_tree,
&identity, &identity,
@@ -1591,8 +1680,8 @@ mod tests {
.collect() .collect()
} }
/// A pinned record of another identity is refused: its environment /// A pinned record of another identity is refused: its staged tree
/// was bootstrapped for a different target. /// was built for a different target.
#[test] #[test]
fn resolve_pinned_id_refuses_foreign_identities() { fn resolve_pinned_id_refuses_foreign_identities() {
let (root, tarball) = sessions_fixture(); let (root, tarball) = sessions_fixture();
@@ -1602,7 +1691,7 @@ mod tests {
..identity.clone() ..identity.clone()
}; };
let host_tree = root.path().join("tree"); let host_tree = root.path().join("tree");
let foreign = record_with_environment( let foreign = record_with_upper(
root.path(), root.path(),
&tarball, &tarball,
&host_tree, &host_tree,
@@ -1615,7 +1704,7 @@ mod tests {
let err = resolve_record_in( let err = resolve_record_in(
root.path(), root.path(),
Some(&tarball), &cache_sha(&tarball),
&Incremental::Id(foreign_id), &Incremental::Id(foreign_id),
&host_tree, &host_tree,
&identity, &identity,
src/deb/session.rs
+808 -591
View File
File diff suppressed because it is too large Load Diff
src/main.rs
+38 -3
View File
@@ -371,15 +371,22 @@ fn main() {
.num_args(0..=1) .num_args(0..=1)
.require_equals(false) .require_equals(false)
.default_missing_value("") .default_missing_value("")
.help("Incremental build: fork a previous build record of this target and keep the result") .help("Incremental build: fork a previous build record of this target and keep its staged tree")
.long_help("Incremental build: fork a previous build record of this target — its prepared chroot and installed build dependencies are copied into a new record, and at the same changelog version its build artifacts too (make recompiles only what changed).\nWithout a value, the newest successful record of the target (among the last five usable) is the base; with an ID (as shown by 'pkh deb list'), that exact record — a failed one is fine if its environment is.\nEvery --incremental run keeps its record, success or failure, so the next one can fork it; a plain 'pkh deb' is fully ephemeral (nothing is kept, even on failure or interrupt)."), .long_help("Incremental build: fork a previous build record of this target — at an unchanged changelog version and while the chroot tarball is unchanged, the base's staged tree is copied into a new record, so make recompiles only what changed.\nWithout a value, the newest successful record of the target (among the last five carrying a staged tree) is the base; with an ID (as shown by 'pkh deb list'), that exact record — a failed one is fine if its staged tree is.\nEvery --incremental run keeps its staged tree, success or failure, so the next one can fork it; a plain 'pkh deb' keeps only metadata and its build log (the chroot is per-run and always discarded)."),
) )
.arg(arg!(--verbose "Show raw tool output instead of the live build view").required(false) .arg(arg!(--verbose "Show raw tool output instead of the live build view").required(false)
.long_help("Show raw tool output instead of the live build view.\nAlso implied by RUST_LOG=debug for pkh's own logs.")) .long_help("Show raw tool output instead of the live build view.\nAlso implied by RUST_LOG=debug for pkh's own logs."))
.subcommand( .subcommand(
Command::new("list") Command::new("list")
.about("List the build records recorded from this tree") .about("List the build records recorded from this tree")
.long_about("List the build records recorded from this tree, with their outcome, version, fork lineage and the id to pass to 'pkh deb --incremental <id>'.\nRead-only: never builds or removes anything (only the last five records per target are kept, 'pkh prune' removes the ones older than a week)."), .long_about("List the build records recorded from this tree (plain runs too, as metadata records), with their outcome, version, fork lineage, whether their staged tree is kept, and the id to pass to 'pkh deb --incremental <id>'.\nRead-only: never builds or removes anything (at most five staged trees per target are kept, 'pkh prune' removes records older than a week)."),
)
.subcommand(
Command::new("log")
.about("Print a recorded build's log")
.long_about("Print the build log of a recorded build: the full tee of its output.\nWithout an ID, the newest record of this tree; with an ID (as shown by 'pkh deb list'), that exact record.\n--follow keeps reading a live build's log as it is written and exits once the build's record reaches a terminal state.")
.arg(arg!([id] "Record id (default: the newest record of this tree)").required(false))
.arg(arg!(-f --follow "Keep reading the log as the build writes it")),
) )
) )
.subcommand( .subcommand(
@@ -896,6 +903,34 @@ fn main() {
return; return;
} }
// `pkh deb log`: print a recorded build's log (optionally
// following a live build); read-only too.
if let Some(("log", log_matches)) = sub_matches.subcommand() {
let tree = cwd.canonicalize().unwrap_or(cwd.clone());
let id = log_matches.get_one::<String>("id").map(String::as_str);
let follow = log_matches.get_flag("follow");
let record = match pkh::deb::session::record_for_log(
&pkh::deb::session::sessions_root(),
&tree,
id,
) {
Ok(record) => record,
Err(e) => {
error!("{e}");
std::process::exit(1);
}
};
let mut out = std::io::stdout();
if let Err(e) = pkh::deb::session::print_record_log(&record, follow, &mut out) {
error!(
"cannot print the build log of {}: {e}",
record.root().display()
);
std::process::exit(1);
}
return;
}
// Ctrl+C during the build must say what happened and release the // Ctrl+C during the build must say what happened and release the
// ephemeral chroot instead of dying on the default disposition. // ephemeral chroot instead of dying on the default disposition.
// The live view (when enabled) registers its own reporter on top // The live view (when enabled) registers its own reporter on top
src/prune.rs
+6 -5
View File
@@ -98,7 +98,8 @@ enum Artifact {
Tarball(PathBuf), Tarball(PathBuf),
/// A build log file under `<cache>/logs`. /// A build log file under `<cache>/logs`.
LogFile(PathBuf), LogFile(PathBuf),
/// A build record root (chroot + journal), one per build. /// A build record root (manifest + log, and the staged tree of
/// `--incremental` runs), one per build.
Session(PathBuf), Session(PathBuf),
/// A `pkh test` run directory (report artifacts). /// A `pkh test` run directory (report artifacts).
TestRun(PathBuf), TestRun(PathBuf),
@@ -184,7 +185,7 @@ fn discover_sessions(sessions_dir: &Path, all: bool) -> Vec<Artifact> {
} else if is_stale_path(&path, all) { } else if is_stale_path(&path, all) {
// A record root only ever holds directories; a plain // A record root only ever holds directories; a plain
// file inside a slug directory is a leftover of the old // file inside a slug directory is a leftover of the old
// flat layout (its session.json / host-tree snapshot). // flat layout (its manifest / host-tree snapshot).
artifacts.push(Artifact::Session(path)); artifacts.push(Artifact::Session(path));
} }
} }
@@ -240,7 +241,7 @@ fn is_prunable_session(path: &Path, all: bool) -> bool {
if all { if all {
return true; return true;
} }
let last_used = fs::read_to_string(path.join("session.json")) let last_used = fs::read_to_string(path.join("manifest.json"))
.ok() .ok()
.and_then(|content| serde_json::from_str::<serde_json::Value>(&content).ok()) .and_then(|content| serde_json::from_str::<serde_json::Value>(&content).ok())
.and_then(|value| { .and_then(|value| {
@@ -1277,7 +1278,7 @@ none /tmp/other proc rw 0 0
fn write_session_manifest(session_dir: &Path, last_used: &str) { fn write_session_manifest(session_dir: &Path, last_used: &str) {
fs::create_dir_all(session_dir).unwrap(); fs::create_dir_all(session_dir).unwrap();
fs::write( fs::write(
session_dir.join("session.json"), session_dir.join("manifest.json"),
format!(r#"{{ "last_used": "{last_used}" }}"#), format!(r#"{{ "last_used": "{last_used}" }}"#),
) )
.unwrap(); .unwrap();
@@ -1299,7 +1300,7 @@ none /tmp/other proc rw 0 0
&(chrono::Utc::now() - chrono::Duration::hours(1)).to_rfc3339(), &(chrono::Utc::now() - chrono::Duration::hours(1)).to_rfc3339(),
); );
fs::create_dir_all(&corrupt).unwrap(); fs::create_dir_all(&corrupt).unwrap();
fs::write(corrupt.join("session.json"), "not json").unwrap(); fs::write(corrupt.join("manifest.json"), "not json").unwrap();
let report = prune_in_roots( let report = prune_in_roots(
temp.path(), temp.path(),