vhaudiquet/pkh
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

deb: add resumable build sessions

Browse Source 
Every local build records a session under /var/tmp/pkh/sessions:
the bootstrapped chroot, the installed build dependencies, the
phase journal and the staged tree with a persistent overlay
upperdir. Rebuilding after a failure currently redoes the tarball
extraction, apt update, build-dep resolution and the whole compile;
a kernel-sized package loses half an hour per iteration.

Recording is always on, reuse is opt-in (--resume [<id>]): the
newest session of the tree, or the one matching an id as shown by
'pkh deb list' (one session per series/arch/cross identity, ids
are build-start timestamps). Resume skips the chroot bootstrap
(integrity marker + tarball check), re-mounts the previous
upperdir so make recompiles only what changed, pops the quilt
series first when debian/patches changed, and clears debian/files
before re-packaging so artifact collection stays exact. A version
bump keeps the environment and discards the build artifacts.

Explicit selectors disagreeing with the adopted session are an
error, never a silent environment switch. Failures and Ctrl+C keep
the session (the interrupt hook unmounts but preserves the tree); a
success consumes it unless --keep. Concurrent same-identity builds
serialize on a lock file kept OUTSIDE the session root: teardown
removes the root while holding the lock, and a lock inside it would
be deleted under the holder, letting the next opener lock a fresh
inode. The apt phases rerun on resume (idempotent, seconds-cheap)
rather than being stamp-gated.
This commit is contained in:
vhaudiquet
2026-09-26 11:51:06 +02:00
parent 0f1446ad4d
commit 8ff423ffb8
4 changed files with 2136 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/deb/ephemeral.rs
+298 -31
View File
@@ -1,4 +1,5 @@
use crate::context::{self, Context, ContextConfig};
use crate::deb::session::{self, Session};
use crate::deb::{Phase, enter_phase};
use crate::interrupt::CleanupHookGuard;
use crate::report::BuildView;
@@ -21,9 +22,10 @@ use xz2::read::XzDecoder;
// /proc and any overlayfs mounts.
// ---------------------------------------------------------------------------
/// Interrupt-time cleanup of an ephemeral chroot: unmount every host-side
/// mount at or below `chroot_path` (the /proc bind mount, any overlay mounts)
/// and then remove the directory tree.
/// Interrupt-time teardown of a chroot tree. When `keep_tree` is set the
/// tree itself is left in place after unmounting (a resumable build
/// session: the interrupt is a pause, not a discard); otherwise this is
/// the historical full cleanup.
///
/// Unlike [`EphemeralContextGuard::drop`], this deliberately does NOT go
/// through the context manager, the ephemeral context's driver (whose
@@ -32,14 +34,14 @@ use xz2::read::XzDecoder;
/// machineries may be mid-mutation on the interrupted thread. Instead it
/// only reads /proc/mounts and spawns umount/rm directly.
///
/// It also differs from `drop` in that it removes the chroot regardless of
/// the build result: the build was aborted, and leaving a still-mounted
/// chroot behind is exactly the leak this hook exists to prevent.
/// On a full cleanup it removes the tree regardless of the build result:
/// the build was aborted, and leaving a still-mounted chroot behind is
/// exactly the leak this hook exists to prevent.
///
/// Best-effort by design: if a child process still holds a mount busy or
/// privilege escalation is unavailable, individual steps fail; failures are
/// logged (pointing at `pkh prune` for the leftovers) and never panic.
fn sigint_cleanup_chroot(chroot_path: &Path) {
fn sigint_cleanup_chroot(chroot_path: &Path, keep_tree: bool) {
let is_root = unsafe { libc::geteuid() } == 0;
// Unmount children before parents: /proc/mounts lists mounts roughly in
@@ -60,6 +62,14 @@ fn sigint_cleanup_chroot(chroot_path: &Path) {
}
}
if keep_tree {
log::debug!(
"Interrupted build session kept at {} (resume with `pkh deb --resume`)",
chroot_path.display()
);
return;
}
// Remove the chroot tree itself (tolerates a missing directory). A
// child the Ctrl+C interrupted may still be finishing its writeout —
// dpkg defers SIGINT until it reaches a safe state — so retry while rm
@@ -108,7 +118,7 @@ fn sigint_cleanup_chroot(chroot_path: &Path) {
/// Build a `Command` for `program`, wrapped in non-interactive sudo when not
/// running as root: interrupt cleanup must never block on a password prompt,
/// so without cached credentials the command fails fast and is logged instead
fn privileged_command(program: &str, is_root: bool) -> Command {
pub(crate) fn privileged_command(program: &str, is_root: bool) -> Command {
if is_root {
Command::new(program)
} else {
@@ -118,10 +128,74 @@ fn privileged_command(program: &str, is_root: bool) -> Command {
}
}
/// Privileged `rm -rf` of a tree (residual chroots hold root-owned device
/// nodes); shared with the session removal in [`crate::deb::session`].
pub(crate) fn privileged_remove(path: &Path) -> std::io::Result<()> {
let is_root = crate::utils::root::is_root().unwrap_or(false);
let status = privileged_command("rm", is_root)
.arg("-rf")
.arg(path)
.status()
.map_err(|e| std::io::Error::other(format!("failed to run rm: {e}")))?;
if status.success() {
Ok(())
} else {
Err(std::io::Error::other(format!(
"rm -rf {} exited with {status}",
path.display()
)))
}
}
/// Remove the contents of `dir` but keep the directory itself.
fn wipe_dir_contents(dir: &Path) -> std::io::Result<()> {
fs::create_dir_all(dir)?;
for entry in fs::read_dir(dir)?.flatten() {
let path = entry.path();
if path.is_dir() {
fs::remove_dir_all(&path)?;
} else {
fs::remove_file(&path)?;
}
}
Ok(())
}
/// Whether `path` is currently a mount point (according to /proc/mounts).
fn is_mountpoint(path: &Path) -> bool {
let path = path.canonicalize().unwrap_or_else(|_| path.to_path_buf());
let Ok(mounts_text) = fs::read_to_string("/proc/mounts") else {
return false;
};
mounts_text.lines().any(|line| {
line.split_whitespace()
.nth(1)
.map(|mp| unescape_mount_field(mp) == path.to_string_lossy())
.unwrap_or(false)
})
}
/// The cached chroot tarball path for a series/arch, as used by the
/// bootstrap: the resume path compares it against the tarball recorded in
/// the session manifest.
pub(crate) fn chroot_tarball_path(
series: &str,
arch: Option<&str>,
) -> Result<PathBuf, Box<dyn Error>> {
let proj_dirs =
ProjectDirs::from("com", "pkh", "pkh").ok_or("Could not determine project directories")?;
let tarball_filename = if let Some(a) = arch {
format!("{}-{}-buildd.tar.xz", series, a)
} else {
format!("{}-buildd.tar.xz", series)
};
Ok(proj_dirs.cache_dir().join(tarball_filename))
}
/// Unmount `path`, falling back to a lazy unmount if the first attempt fails
/// because something still holds the mount busy (e.g. an interrupted child
/// that has not exited yet); returns whether the mount is gone
fn unmount_path(path: &Path, is_root: bool) -> bool {
pub(crate) fn unmount_path(path: &Path, is_root: bool) -> bool {
if privileged_command("umount", is_root)
.arg(path)
.status()
@@ -138,7 +212,7 @@ fn unmount_path(path: &Path, is_root: bool) -> bool {
/// Collect the host-side mount points at or below `base`, in /proc/mounts
/// order (empty if /proc/mounts cannot be read)
fn host_mounts_under(base: &Path) -> Vec<PathBuf> {
pub(crate) fn host_mounts_under(base: &Path) -> Vec<PathBuf> {
let mut mounts = Vec::new();
let Ok(mounts_text) = fs::read_to_string("/proc/mounts") else {
return mounts;
@@ -185,6 +259,12 @@ fn unescape_mount_field(field: &str) -> String {
/// An ephemeral unshare context guard that creates and manages a temporary chroot environment
/// for building packages with unshare permissions.
///
/// In session mode ([`EphemeralContextGuard::new_for_session`]) the chroot
/// lives inside a [`Session`]: a previous attempt's chroot is reused when
/// the caller verified it, teardown keeps the tree for a later `--resume`,
/// and a successful build removes the session unless `keep_on_success` is
/// set.
pub struct EphemeralContextGuard {
/// The ephemeral build context this guard created (an unshare context
/// bound to the chroot, parented on the base context). Held explicitly so
@@ -201,6 +281,10 @@ pub struct EphemeralContextGuard {
/// Registration of the interrupt-time cleanup hook; deregistered when
/// this guard drops, so the hook can never fire after the normal cleanup
cleanup_hook: Option<CleanupHookGuard>,
/// The resumable build session this guard works in, when any.
session: Option<Session>,
/// Whether a successful build keeps the session (`pkh deb --keep`).
keep_on_success: bool,
}
impl EphemeralContextGuard {
@@ -216,6 +300,53 @@ impl EphemeralContextGuard {
arch: Option<&str>,
base_ctx: Arc<Context>,
view: &dyn BuildView,
) -> Result<Self, Box<dyn Error>> {
Self::new_internal(None, series, arch, base_ctx, view, false, false).await
}
/// Create the ephemeral build context inside `session`'s chroot.
///
/// # Arguments
/// * `session` - The resumable session owning the chroot (locked by the
/// caller)
/// * `series` - The distribution series (e.g., "noble", "sid")
/// * `arch` - Optional target architecture, as for
/// [`EphemeralContextGuard::new_with_context`]
/// * `base_ctx` - The base context to use for chroot work
/// * `reuse_chroot` - Adopt the previous attempt's bootstrapped chroot
/// instead of re-extracting the tarball; the caller must only set
/// this after the session's chroot passed the integrity probe
/// * `keep_on_success` - Keep the session after a successful build
/// (`pkh deb --keep`)
pub async fn new_for_session(
session: Session,
series: &str,
arch: Option<&str>,
base_ctx: Arc<Context>,
view: &dyn BuildView,
reuse_chroot: bool,
keep_on_success: bool,
) -> Result<Self, Box<dyn Error>> {
Self::new_internal(
Some(session),
series,
arch,
base_ctx,
view,
reuse_chroot,
keep_on_success,
)
.await
}
async fn new_internal(
session: Option<Session>,
series: &str,
arch: Option<&str>,
base_ctx: Arc<Context>,
view: &dyn BuildView,
reuse_chroot: bool,
keep_on_success: bool,
) -> Result<Self, Box<dyn Error>> {
// Save the globally-installed context so Drop can restore exactly
// this handle: concurrent builds install their own ephemeral
@@ -223,9 +354,16 @@ impl EphemeralContextGuard {
// before this guard swapped anything in.
let previous_context = context::current();
// Create a temporary directory for the chroot
let chroot_path_str = base_ctx.create_temp_dir()?;
let chroot_path = PathBuf::from(chroot_path_str);
// In session mode the chroot lives inside the session root (and is
// reused when the caller says so); otherwise it is a fresh temp dir.
let chroot_path = match &session {
Some(s) => {
let path = s.chroot_path();
fs::create_dir_all(&path)?;
path
}
None => PathBuf::from(base_ctx.create_temp_dir()?),
};
log::debug!(
"Creating new chroot for {} (arch: {:?}) at {}...",
@@ -236,16 +374,19 @@ impl EphemeralContextGuard {
// Register the interrupt-time cleanup hook before any heavy work: if
// the user hits Ctrl-C during bootstrap or the build itself, the
// interrupt watchdog unmounts and removes the chroot through this
// hook (see `sigint_cleanup_chroot`). This only works for a local
// interrupt watchdog unmounts the chroot's mounts through this hook
// (see `sigint_cleanup_chroot`). This only works for a local
// base context: the hook must be self-contained (stored path +
// direct umount/rm subprocesses) and cannot go through `base_ctx`.
// For remote or nested bases the chroot lives elsewhere, and
// leftovers stay handled by `pkh prune` as before.
// leftovers stay handled by `pkh prune` as before. A session tree is
// kept on interrupt (a pause, not a discard); a plain temp chroot is
// still removed.
let cleanup_hook = if matches!(base_ctx.config, ContextConfig::Local) {
Some(crate::interrupt::register_cleanup_hook(Box::new({
let chroot_path = chroot_path.clone();
move || sigint_cleanup_chroot(&chroot_path)
let keep_tree = session.is_some();
move || sigint_cleanup_chroot(&chroot_path, keep_tree)
})))
} else {
log::debug!(
@@ -255,10 +396,18 @@ impl EphemeralContextGuard {
None
};
// Download and extract the chroot tarball
if let Err(e) =
Self::download_and_extract_chroot(series, arch, &chroot_path, base_ctx.clone(), view)
.await
// Download and extract the chroot tarball (or reuse the session's
// prepared chroot)
if let Err(e) = Self::download_and_extract_chroot(
series,
arch,
&chroot_path,
base_ctx.clone(),
view,
session.as_ref(),
reuse_chroot,
)
.await
{
// On a Ctrl+C the interrupt watchdog owns the tree: keep the
// hook registered (forgetting the guard) so it removes the
@@ -300,6 +449,8 @@ impl EphemeralContextGuard {
build_succeeded: false,
base_ctx,
cleanup_hook,
session,
keep_on_success,
})
}
@@ -319,7 +470,26 @@ impl EphemeralContextGuard {
chroot_path: &PathBuf,
ctx: Arc<context::Context>,
view: &dyn BuildView,
session: Option<&Session>,
reuse_chroot: bool,
) -> Result<(), Box<dyn Error>> {
// Session resume: the previous attempt's chroot passed the caller's
// checks — skip the tarball extraction entirely, the environment
// (apt lists, installed build-deps) is still in there.
if let Some(s) = session {
if reuse_chroot && s.chroot_is_usable() {
log::info!(
"Reusing the prepared chroot of session {} (bootstrap skipped)",
s.manifest().id
);
Self::finalize_chroot(chroot_path, ctx, view)?;
return Ok(());
}
// Not reusing (fresh session or failed probe): never extract
// over stale content.
wipe_dir_contents(chroot_path)?;
}
// Clone ctx for use in create_device_nodes after download_chroot_tarball consumes it
let ctx_for_devices = ctx.clone();
// Get project directories for caching
@@ -394,16 +564,58 @@ impl EphemeralContextGuard {
enter_phase(view, Phase::ExtractingChroot);
Self::extract_tarball(&tarball_path, chroot_path, view)?;
// Create device nodes in the chroot
log::debug!("Creating device nodes in chroot...");
enter_phase(view, Phase::FinalizingChroot);
Self::create_device_nodes(chroot_path, ctx_for_devices.clone())?;
// Create device nodes in the chroot and bind-mount /proc
Self::finalize_chroot(chroot_path, ctx_for_devices, view)?;
// Record the bootstrap in the session: marker file (the resume
// integrity probe) and the tarball the chroot came from.
if let Some(s) = session {
std::fs::write(chroot_path.join(".pkh-ready"), b"")?;
let metadata = fs::metadata(&tarball_path)?;
let tarball_sha256 = session::sha256_file(&tarball_path)
.map_err(|e| format!("cannot hash the chroot tarball: {e}"))?;
s.set_chroot(session::ChrootInfo {
tarball: tarball_filename,
tarball_size: metadata.len(),
tarball_mtime_secs: metadata
.modified()
.ok()
.and_then(|t| t.duration_since(std::time::UNIX_EPOCH).ok())
.map(|d| d.as_secs())
.unwrap_or(0),
tarball_sha256,
ready: true,
});
log::debug!("Session {} chroot is ready", s.manifest().id);
}
Ok(())
}
/// Device nodes and the /proc bind mount for a (possibly reused)
/// chroot: idempotent finalization, safe to run on every attempt.
fn finalize_chroot(
chroot_path: &Path,
ctx: Arc<context::Context>,
view: &dyn BuildView,
) -> Result<(), Box<dyn Error>> {
if !chroot_path.join("dev/null").exists() {
// Create device nodes in the chroot
log::debug!("Creating device nodes in chroot...");
enter_phase(view, Phase::FinalizingChroot);
Self::create_device_nodes(chroot_path, ctx.clone())?;
}
// Bind mount /proc from host into chroot (before entering unshare namespace)
// This allows /proc to work in containers where mounting inside unshare fails
log::debug!("Bind-mounting /proc into chroot...");
Self::bind_mount_proc(chroot_path, ctx_for_devices)?;
// This allows /proc to work in containers where mounting inside unshare fails.
// A reused chroot may still carry the mount from the previous
// attempt: only mount when it is not already there.
if is_mountpoint(&chroot_path.join("proc")) {
log::debug!("/proc is already bind-mounted in the chroot");
} else {
log::debug!("Bind-mounting /proc into chroot...");
Self::bind_mount_proc(chroot_path, ctx)?;
}
Ok(())
}
@@ -617,6 +829,21 @@ impl EphemeralContextGuard {
pub fn mark_build_successful(&mut self) {
self.build_succeeded = true;
}
/// The staged build root inside the chroot: the stable
/// `/tmp/pkh-build` for session builds (the overlay upperdir mirrors
/// paths relative to the merge point, so the staged tree must live at
/// the same in-chroot path on every attempt for the build artifacts to
/// be reusable), a fresh timestamped temp dir otherwise.
pub fn build_root(&self) -> Result<String, Box<dyn Error>> {
match &self.session {
Some(s) => {
std::fs::create_dir_all(s.chroot_path().join("tmp/pkh-build"))?;
Ok(s.build_root())
}
None => Ok(self.ephemeral_ctx.create_temp_dir()?),
}
}
}
impl Drop for EphemeralContextGuard {
@@ -663,6 +890,46 @@ impl Drop for EphemeralContextGuard {
// touches it.
context::manager().set_current_ephemeral(self.previous_context.clone());
// Session mode: the tree is a resumable build session. The /proc
// bind mount must come down in every case (a kept session has to be
// unmount-clean for the next resume); the tree itself stays unless
// a successful build without `--keep` consumes it.
if let Some(session) = &self.session {
let proc_path = self.chroot_path.join("proc");
let is_root = crate::utils::root::is_root().unwrap_or(false);
let _ = if is_root {
self.base_ctx.command("umount").arg(&proc_path).status()
} else {
self.base_ctx
.command("sudo")
.arg("umount")
.arg(&proc_path)
.status()
};
if self.build_succeeded {
if self.keep_on_success {
session.set_outcome(session::OUTCOME_SUCCESS);
log::info!(
"Build session {} kept (resume with `pkh deb --resume`)",
session.manifest().id
);
} else {
log::debug!(
"Build succeeded, removing build session {}",
session.root().display()
);
session.remove();
}
} else {
log::debug!(
"Build did not succeed: session {} kept (resume with `pkh deb --resume`)",
session.manifest().id
);
}
return;
}
// Remove chroot directory only if build succeeded
if self.build_succeeded {
log::debug!(
@@ -757,7 +1024,7 @@ mod chroot_cleanup_tests {
fn sigint_cleanup_of_missing_chroot_is_a_noop() {
let dir = tempfile::tempdir().unwrap();
let missing = dir.path().join("no-such-chroot");
sigint_cleanup_chroot(&missing);
sigint_cleanup_chroot(&missing, false);
assert!(!missing.exists());
}
@@ -780,7 +1047,7 @@ mod chroot_cleanup_tests {
std::fs::create_dir_all(chroot.join("rootfs")).unwrap();
std::fs::write(chroot.join("rootfs").join("file.txt"), "data").unwrap();
sigint_cleanup_chroot(&chroot);
sigint_cleanup_chroot(&chroot, false);
assert!(!chroot.exists());
}