vhaudiquet/pkh
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

deb: add resumable build sessions

Browse Source 
Every local build records a session under /var/tmp/pkh/sessions:
the bootstrapped chroot, the installed build dependencies, the
phase journal and the staged tree with a persistent overlay
upperdir. Rebuilding after a failure currently redoes the tarball
extraction, apt update, build-dep resolution and the whole compile;
a kernel-sized package loses half an hour per iteration.

Recording is always on, reuse is opt-in (--resume [<id>]): the
newest session of the tree, or the one matching an id as shown by
'pkh deb list' (one session per series/arch/cross identity, ids
are build-start timestamps). Resume skips the chroot bootstrap
(integrity marker + tarball check), re-mounts the previous
upperdir so make recompiles only what changed, pops the quilt
series first when debian/patches changed, and clears debian/files
before re-packaging so artifact collection stays exact. A version
bump keeps the environment and discards the build artifacts.

Explicit selectors disagreeing with the adopted session are an
error, never a silent environment switch. Failures and Ctrl+C keep
the session (the interrupt hook unmounts but preserves the tree); a
success consumes it unless --keep. Concurrent same-identity builds
serialize on a lock file kept OUTSIDE the session root: teardown
removes the root while holding the lock, and a lock inside it would
be deleted under the holder, letting the next opener lock a fresh
inode. The apt phases rerun on resume (idempotent, seconds-cheap)
rather than being stamp-gated.
This commit is contained in:
vhaudiquet
2026-09-26 11:51:06 +02:00
parent 0f1446ad4d
commit 8ff423ffb8
4 changed files with 2136 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/deb/ephemeral.rs
+293 -26
View File
@@ -1,4 +1,5 @@
use crate::context::{self, Context, ContextConfig};
use crate::deb::session::{self, Session};
use crate::deb::{Phase, enter_phase};
use crate::interrupt::CleanupHookGuard;
use crate::report::BuildView;
@@ -21,9 +22,10 @@ use xz2::read::XzDecoder;
// /proc and any overlayfs mounts.
// ---------------------------------------------------------------------------
/// Interrupt-time cleanup of an ephemeral chroot: unmount every host-side
/// mount at or below `chroot_path` (the /proc bind mount, any overlay mounts)
/// and then remove the directory tree.
/// Interrupt-time teardown of a chroot tree. When `keep_tree` is set the
/// tree itself is left in place after unmounting (a resumable build
/// session: the interrupt is a pause, not a discard); otherwise this is
/// the historical full cleanup.
///
/// Unlike [`EphemeralContextGuard::drop`], this deliberately does NOT go
/// through the context manager, the ephemeral context's driver (whose
@@ -32,14 +34,14 @@ use xz2::read::XzDecoder;
/// machineries may be mid-mutation on the interrupted thread. Instead it
/// only reads /proc/mounts and spawns umount/rm directly.
///
/// It also differs from `drop` in that it removes the chroot regardless of
/// the build result: the build was aborted, and leaving a still-mounted
/// chroot behind is exactly the leak this hook exists to prevent.
/// On a full cleanup it removes the tree regardless of the build result:
/// the build was aborted, and leaving a still-mounted chroot behind is
/// exactly the leak this hook exists to prevent.
///
/// Best-effort by design: if a child process still holds a mount busy or
/// privilege escalation is unavailable, individual steps fail; failures are
/// logged (pointing at `pkh prune` for the leftovers) and never panic.
fn sigint_cleanup_chroot(chroot_path: &Path) {
fn sigint_cleanup_chroot(chroot_path: &Path, keep_tree: bool) {
let is_root = unsafe { libc::geteuid() } == 0;
// Unmount children before parents: /proc/mounts lists mounts roughly in
@@ -60,6 +62,14 @@ fn sigint_cleanup_chroot(chroot_path: &Path) {
}
}
if keep_tree {
log::debug!(
"Interrupted build session kept at {} (resume with `pkh deb --resume`)",
chroot_path.display()
);
return;
}
// Remove the chroot tree itself (tolerates a missing directory). A
// child the Ctrl+C interrupted may still be finishing its writeout —
// dpkg defers SIGINT until it reaches a safe state — so retry while rm
@@ -108,7 +118,7 @@ fn sigint_cleanup_chroot(chroot_path: &Path) {
/// Build a `Command` for `program`, wrapped in non-interactive sudo when not
/// running as root: interrupt cleanup must never block on a password prompt,
/// so without cached credentials the command fails fast and is logged instead
fn privileged_command(program: &str, is_root: bool) -> Command {
pub(crate) fn privileged_command(program: &str, is_root: bool) -> Command {
if is_root {
Command::new(program)
} else {
@@ -118,10 +128,74 @@ fn privileged_command(program: &str, is_root: bool) -> Command {
}
}
/// Privileged `rm -rf` of a tree (residual chroots hold root-owned device
/// nodes); shared with the session removal in [`crate::deb::session`].
pub(crate) fn privileged_remove(path: &Path) -> std::io::Result<()> {
let is_root = crate::utils::root::is_root().unwrap_or(false);
let status = privileged_command("rm", is_root)
.arg("-rf")
.arg(path)
.status()
.map_err(|e| std::io::Error::other(format!("failed to run rm: {e}")))?;
if status.success() {
Ok(())
} else {
Err(std::io::Error::other(format!(
"rm -rf {} exited with {status}",
path.display()
)))
}
}
/// Remove the contents of `dir` but keep the directory itself.
fn wipe_dir_contents(dir: &Path) -> std::io::Result<()> {
fs::create_dir_all(dir)?;
for entry in fs::read_dir(dir)?.flatten() {
let path = entry.path();
if path.is_dir() {
fs::remove_dir_all(&path)?;
} else {
fs::remove_file(&path)?;
}
}
Ok(())
}
/// Whether `path` is currently a mount point (according to /proc/mounts).
fn is_mountpoint(path: &Path) -> bool {
let path = path.canonicalize().unwrap_or_else(|_| path.to_path_buf());
let Ok(mounts_text) = fs::read_to_string("/proc/mounts") else {
return false;
};
mounts_text.lines().any(|line| {
line.split_whitespace()
.nth(1)
.map(|mp| unescape_mount_field(mp) == path.to_string_lossy())
.unwrap_or(false)
})
}
/// The cached chroot tarball path for a series/arch, as used by the
/// bootstrap: the resume path compares it against the tarball recorded in
/// the session manifest.
pub(crate) fn chroot_tarball_path(
series: &str,
arch: Option<&str>,
) -> Result<PathBuf, Box<dyn Error>> {
let proj_dirs =
ProjectDirs::from("com", "pkh", "pkh").ok_or("Could not determine project directories")?;
let tarball_filename = if let Some(a) = arch {
format!("{}-{}-buildd.tar.xz", series, a)
} else {
format!("{}-buildd.tar.xz", series)
};
Ok(proj_dirs.cache_dir().join(tarball_filename))
}
/// Unmount `path`, falling back to a lazy unmount if the first attempt fails
/// because something still holds the mount busy (e.g. an interrupted child
/// that has not exited yet); returns whether the mount is gone
fn unmount_path(path: &Path, is_root: bool) -> bool {
pub(crate) fn unmount_path(path: &Path, is_root: bool) -> bool {
if privileged_command("umount", is_root)
.arg(path)
.status()
@@ -138,7 +212,7 @@ fn unmount_path(path: &Path, is_root: bool) -> bool {
/// Collect the host-side mount points at or below `base`, in /proc/mounts
/// order (empty if /proc/mounts cannot be read)
fn host_mounts_under(base: &Path) -> Vec<PathBuf> {
pub(crate) fn host_mounts_under(base: &Path) -> Vec<PathBuf> {
let mut mounts = Vec::new();
let Ok(mounts_text) = fs::read_to_string("/proc/mounts") else {
return mounts;
@@ -185,6 +259,12 @@ fn unescape_mount_field(field: &str) -> String {
/// An ephemeral unshare context guard that creates and manages a temporary chroot environment
/// for building packages with unshare permissions.
///
/// In session mode ([`EphemeralContextGuard::new_for_session`]) the chroot
/// lives inside a [`Session`]: a previous attempt's chroot is reused when
/// the caller verified it, teardown keeps the tree for a later `--resume`,
/// and a successful build removes the session unless `keep_on_success` is
/// set.
pub struct EphemeralContextGuard {
/// The ephemeral build context this guard created (an unshare context
/// bound to the chroot, parented on the base context). Held explicitly so
@@ -201,6 +281,10 @@ pub struct EphemeralContextGuard {
/// Registration of the interrupt-time cleanup hook; deregistered when
/// this guard drops, so the hook can never fire after the normal cleanup
cleanup_hook: Option<CleanupHookGuard>,
/// The resumable build session this guard works in, when any.
session: Option<Session>,
/// Whether a successful build keeps the session (`pkh deb --keep`).
keep_on_success: bool,
}
impl EphemeralContextGuard {
@@ -216,6 +300,53 @@ impl EphemeralContextGuard {
arch: Option<&str>,
base_ctx: Arc<Context>,
view: &dyn BuildView,
) -> Result<Self, Box<dyn Error>> {
Self::new_internal(None, series, arch, base_ctx, view, false, false).await
}
/// Create the ephemeral build context inside `session`'s chroot.
///
/// # Arguments
/// * `session` - The resumable session owning the chroot (locked by the
/// caller)
/// * `series` - The distribution series (e.g., "noble", "sid")
/// * `arch` - Optional target architecture, as for
/// [`EphemeralContextGuard::new_with_context`]
/// * `base_ctx` - The base context to use for chroot work
/// * `reuse_chroot` - Adopt the previous attempt's bootstrapped chroot
/// instead of re-extracting the tarball; the caller must only set
/// this after the session's chroot passed the integrity probe
/// * `keep_on_success` - Keep the session after a successful build
/// (`pkh deb --keep`)
pub async fn new_for_session(
session: Session,
series: &str,
arch: Option<&str>,
base_ctx: Arc<Context>,
view: &dyn BuildView,
reuse_chroot: bool,
keep_on_success: bool,
) -> Result<Self, Box<dyn Error>> {
Self::new_internal(
Some(session),
series,
arch,
base_ctx,
view,
reuse_chroot,
keep_on_success,
)
.await
}
async fn new_internal(
session: Option<Session>,
series: &str,
arch: Option<&str>,
base_ctx: Arc<Context>,
view: &dyn BuildView,
reuse_chroot: bool,
keep_on_success: bool,
) -> Result<Self, Box<dyn Error>> {
// Save the globally-installed context so Drop can restore exactly
// this handle: concurrent builds install their own ephemeral
@@ -223,9 +354,16 @@ impl EphemeralContextGuard {
// before this guard swapped anything in.
let previous_context = context::current();
// Create a temporary directory for the chroot
let chroot_path_str = base_ctx.create_temp_dir()?;
let chroot_path = PathBuf::from(chroot_path_str);
// In session mode the chroot lives inside the session root (and is
// reused when the caller says so); otherwise it is a fresh temp dir.
let chroot_path = match &session {
Some(s) => {
let path = s.chroot_path();
fs::create_dir_all(&path)?;
path
}
None => PathBuf::from(base_ctx.create_temp_dir()?),
};
log::debug!(
"Creating new chroot for {} (arch: {:?}) at {}...",
@@ -236,16 +374,19 @@ impl EphemeralContextGuard {
// Register the interrupt-time cleanup hook before any heavy work: if
// the user hits Ctrl-C during bootstrap or the build itself, the
// interrupt watchdog unmounts and removes the chroot through this
// hook (see `sigint_cleanup_chroot`). This only works for a local
// interrupt watchdog unmounts the chroot's mounts through this hook
// (see `sigint_cleanup_chroot`). This only works for a local
// base context: the hook must be self-contained (stored path +
// direct umount/rm subprocesses) and cannot go through `base_ctx`.
// For remote or nested bases the chroot lives elsewhere, and
// leftovers stay handled by `pkh prune` as before.
// leftovers stay handled by `pkh prune` as before. A session tree is
// kept on interrupt (a pause, not a discard); a plain temp chroot is
// still removed.
let cleanup_hook = if matches!(base_ctx.config, ContextConfig::Local) {
Some(crate::interrupt::register_cleanup_hook(Box::new({
let chroot_path = chroot_path.clone();
move || sigint_cleanup_chroot(&chroot_path)
let keep_tree = session.is_some();
move || sigint_cleanup_chroot(&chroot_path, keep_tree)
})))
} else {
log::debug!(
@@ -255,9 +396,17 @@ impl EphemeralContextGuard {
None
};
// Download and extract the chroot tarball
if let Err(e) =
Self::download_and_extract_chroot(series, arch, &chroot_path, base_ctx.clone(), view)
// Download and extract the chroot tarball (or reuse the session's
// prepared chroot)
if let Err(e) = Self::download_and_extract_chroot(
series,
arch,
&chroot_path,
base_ctx.clone(),
view,
session.as_ref(),
reuse_chroot,
)
.await
{
// On a Ctrl+C the interrupt watchdog owns the tree: keep the
@@ -300,6 +449,8 @@ impl EphemeralContextGuard {
build_succeeded: false,
base_ctx,
cleanup_hook,
session,
keep_on_success,
})
}
@@ -319,7 +470,26 @@ impl EphemeralContextGuard {
chroot_path: &PathBuf,
ctx: Arc<context::Context>,
view: &dyn BuildView,
session: Option<&Session>,
reuse_chroot: bool,
) -> Result<(), Box<dyn Error>> {
// Session resume: the previous attempt's chroot passed the caller's
// checks — skip the tarball extraction entirely, the environment
// (apt lists, installed build-deps) is still in there.
if let Some(s) = session {
if reuse_chroot && s.chroot_is_usable() {
log::info!(
"Reusing the prepared chroot of session {} (bootstrap skipped)",
s.manifest().id
);
Self::finalize_chroot(chroot_path, ctx, view)?;
return Ok(());
}
// Not reusing (fresh session or failed probe): never extract
// over stale content.
wipe_dir_contents(chroot_path)?;
}
// Clone ctx for use in create_device_nodes after download_chroot_tarball consumes it
let ctx_for_devices = ctx.clone();
// Get project directories for caching
@@ -394,16 +564,58 @@ impl EphemeralContextGuard {
enter_phase(view, Phase::ExtractingChroot);
Self::extract_tarball(&tarball_path, chroot_path, view)?;
// Create device nodes in the chroot and bind-mount /proc
Self::finalize_chroot(chroot_path, ctx_for_devices, view)?;
// Record the bootstrap in the session: marker file (the resume
// integrity probe) and the tarball the chroot came from.
if let Some(s) = session {
std::fs::write(chroot_path.join(".pkh-ready"), b"")?;
let metadata = fs::metadata(&tarball_path)?;
let tarball_sha256 = session::sha256_file(&tarball_path)
.map_err(|e| format!("cannot hash the chroot tarball: {e}"))?;
s.set_chroot(session::ChrootInfo {
tarball: tarball_filename,
tarball_size: metadata.len(),
tarball_mtime_secs: metadata
.modified()
.ok()
.and_then(|t| t.duration_since(std::time::UNIX_EPOCH).ok())
.map(|d| d.as_secs())
.unwrap_or(0),
tarball_sha256,
ready: true,
});
log::debug!("Session {} chroot is ready", s.manifest().id);
}
Ok(())
}
/// Device nodes and the /proc bind mount for a (possibly reused)
/// chroot: idempotent finalization, safe to run on every attempt.
fn finalize_chroot(
chroot_path: &Path,
ctx: Arc<context::Context>,
view: &dyn BuildView,
) -> Result<(), Box<dyn Error>> {
if !chroot_path.join("dev/null").exists() {
// Create device nodes in the chroot
log::debug!("Creating device nodes in chroot...");
enter_phase(view, Phase::FinalizingChroot);
Self::create_device_nodes(chroot_path, ctx_for_devices.clone())?;
Self::create_device_nodes(chroot_path, ctx.clone())?;
}
// Bind mount /proc from host into chroot (before entering unshare namespace)
// This allows /proc to work in containers where mounting inside unshare fails
// This allows /proc to work in containers where mounting inside unshare fails.
// A reused chroot may still carry the mount from the previous
// attempt: only mount when it is not already there.
if is_mountpoint(&chroot_path.join("proc")) {
log::debug!("/proc is already bind-mounted in the chroot");
} else {
log::debug!("Bind-mounting /proc into chroot...");
Self::bind_mount_proc(chroot_path, ctx_for_devices)?;
Self::bind_mount_proc(chroot_path, ctx)?;
}
Ok(())
}
@@ -617,6 +829,21 @@ impl EphemeralContextGuard {
pub fn mark_build_successful(&mut self) {
self.build_succeeded = true;
}
/// The staged build root inside the chroot: the stable
/// `/tmp/pkh-build` for session builds (the overlay upperdir mirrors
/// paths relative to the merge point, so the staged tree must live at
/// the same in-chroot path on every attempt for the build artifacts to
/// be reusable), a fresh timestamped temp dir otherwise.
pub fn build_root(&self) -> Result<String, Box<dyn Error>> {
match &self.session {
Some(s) => {
std::fs::create_dir_all(s.chroot_path().join("tmp/pkh-build"))?;
Ok(s.build_root())
}
None => Ok(self.ephemeral_ctx.create_temp_dir()?),
}
}
}
impl Drop for EphemeralContextGuard {
@@ -663,6 +890,46 @@ impl Drop for EphemeralContextGuard {
// touches it.
context::manager().set_current_ephemeral(self.previous_context.clone());
// Session mode: the tree is a resumable build session. The /proc
// bind mount must come down in every case (a kept session has to be
// unmount-clean for the next resume); the tree itself stays unless
// a successful build without `--keep` consumes it.
if let Some(session) = &self.session {
let proc_path = self.chroot_path.join("proc");
let is_root = crate::utils::root::is_root().unwrap_or(false);
let _ = if is_root {
self.base_ctx.command("umount").arg(&proc_path).status()
} else {
self.base_ctx
.command("sudo")
.arg("umount")
.arg(&proc_path)
.status()
};
if self.build_succeeded {
if self.keep_on_success {
session.set_outcome(session::OUTCOME_SUCCESS);
log::info!(
"Build session {} kept (resume with `pkh deb --resume`)",
session.manifest().id
);
} else {
log::debug!(
"Build succeeded, removing build session {}",
session.root().display()
);
session.remove();
}
} else {
log::debug!(
"Build did not succeed: session {} kept (resume with `pkh deb --resume`)",
session.manifest().id
);
}
return;
}
// Remove chroot directory only if build succeeded
if self.build_succeeded {
log::debug!(
@@ -757,7 +1024,7 @@ mod chroot_cleanup_tests {
fn sigint_cleanup_of_missing_chroot_is_a_noop() {
let dir = tempfile::tempdir().unwrap();
let missing = dir.path().join("no-such-chroot");
sigint_cleanup_chroot(&missing);
sigint_cleanup_chroot(&missing, false);
assert!(!missing.exists());
}
@@ -780,7 +1047,7 @@ mod chroot_cleanup_tests {
std::fs::create_dir_all(chroot.join("rootfs")).unwrap();
std::fs::write(chroot.join("rootfs").join("file.txt"), "data").unwrap();
sigint_cleanup_chroot(&chroot);
sigint_cleanup_chroot(&chroot, false);
assert!(!chroot.exists());
}
src/deb/local.rs
+111 -5
View File
@@ -12,6 +12,7 @@ use std::sync::Arc;
use crate::apt;
use crate::deb::cross;
use crate::deb::session;
use crate::debian::control::ControlInfo;
use crate::debian::deps::{Deps, Facts, ParseOpts, PkgRelation};
@@ -26,6 +27,37 @@ fn cap<'a>(
cmd
}
/// The session journal stamps this build works with: the input stamps of
/// the patch and build-dependency phases, computed from the host tree and
/// the resolved selectors.
struct JournalStamps {
patches: Option<String>,
control: Option<String>,
}
/// Compute the journal stamps for a session build; `None` without one.
fn journal_stamps(
session: &session::Session,
arch: &str,
series: &str,
pocket: Option<&str>,
cross: bool,
ppa: &[String],
inject_packages: &[String],
) -> JournalStamps {
let host_tree = PathBuf::from(session.manifest().host_tree);
let selectors = format!(
"{arch}\n{cross}\n{series}\n{}\n{}\n{}",
pocket.unwrap_or(""),
ppa.join(","),
inject_packages.join(",")
);
JournalStamps {
patches: session::patches_stamp(&host_tree),
control: session::control_stamp(&host_tree, &selectors),
}
}
#[allow(clippy::too_many_arguments)]
pub async fn build(
package: &str,
@@ -41,9 +73,15 @@ pub async fn build(
ctx: Arc<Context>,
view: &dyn BuildView,
jobs: Option<usize>,
session: Option<&session::Session>,
resume: bool,
) -> Result<Vec<PathBuf>, Box<dyn Error>> {
let sink: Option<Arc<dyn LineSink>> = view.sink();
// Journal stamps for a session build; recording is a no-op without one.
let stamps =
session.map(|s| journal_stamps(s, arch, series, pocket, cross, ppa, inject_packages));
// Environment
let mut env = HashMap::<String, String>::new();
env.insert("LANG".to_string(), "C".to_string());
@@ -207,6 +245,9 @@ pub async fn build(
or re-run with RUST_LOG=debug for more details."
.into());
}
if let Some(s) = session {
s.record_phase("apt_update", None);
}
// Install essential packages
log::debug!("Installing essential packages for local build...");
@@ -231,6 +272,9 @@ pub async fn build(
if !status.success() {
return Err("Could not install essential packages for the build".into());
}
if let Some(s) = session {
s.record_phase("essentials", None);
}
// The package directory was resolved by the caller (the staged copy of
// the tree the user pointed at, or the name-pattern search fallback)
@@ -249,8 +293,23 @@ pub async fn build(
Err(e) => log::debug!("cannot read changelog for SOURCE_DATE_EPOCH: {}", e),
}
// Apply quilt patches if the package provides a patch series
apply_quilt_patches(package_dir_str, &env, ctx.clone(), view, &sink)?;
// Apply quilt patches if the package provides a patch series. On a
// resumed session the previous attempt's patches may still be applied:
// pop them first when the patch tree changed since.
let patch_stamp = stamps.as_ref().and_then(|s| s.patches.clone());
let pop_patches = resume && {
let previous = session
.and_then(|s| s.manifest().phases.get("patches").map(|p| p.stamp.clone()))
.unwrap_or(None);
let applied = ctx
.exists(&Path::new(package_dir_str).join(".pc/applied-patches"))
.unwrap_or(false);
applied && previous != patch_stamp
};
apply_quilt_patches(package_dir_str, &env, ctx.clone(), view, &sink, pop_patches)?;
if let Some(s) = session {
s.record_phase("patches", patch_stamp);
}
// Install injected packages if specified
if !inject_packages.is_empty() {
@@ -274,6 +333,12 @@ pub async fn build(
view,
&sink,
)?;
if let Some(s) = session {
s.record_phase(
"build_deps",
stamps.as_ref().and_then(|st| st.control.clone()),
);
}
// Run the build step
log::debug!("Building (debian/rules build) package...");
@@ -289,6 +354,20 @@ pub async fn build(
if !status.success() {
return Err("Error while building the package".into());
}
if let Some(s) = session {
s.record_phase("build", None);
}
// On a resumed session the previous attempt's debian/files would make
// the artifact collection below surface its outputs: only the new
// attempt's registration may remain.
if resume {
let _ = ctx
.command("rm")
.arg("-f")
.arg(Path::new(package_dir_str).join("debian/files"))
.status();
}
// Run the 'binary' step to produce deb
enter_phase(view, Phase::ProducingBinaries);
@@ -334,6 +413,10 @@ pub async fn build(
}
}
if let Some(s) = session {
s.record_phase("binary", None);
}
Ok(artifacts)
}
@@ -843,13 +926,15 @@ fn generate_upload_metadata(
}
/// Apply quilt patches before building, if the package provides a
/// 'debian/patches/series' file
/// 'debian/patches/series' file. `pop_first` un-applies the previous
/// attempt's patches first (resumed session whose patch tree changed).
fn apply_quilt_patches(
package_dir: &str,
env: &HashMap<String, String>,
ctx: Arc<Context>,
view: &dyn BuildView,
sink: &Option<Arc<dyn LineSink>>,
pop_first: bool,
) -> Result<(), Box<dyn Error>> {
let series_path = Path::new(package_dir).join("debian/patches/series");
if !ctx.exists(&series_path)? {
@@ -907,13 +992,33 @@ fn apply_quilt_patches(
return Err("Could not install 'quilt', required to apply patches".into());
}
let mut patch_env = env.clone();
patch_env.insert("QUILT_PATCHES".to_string(), "debian/patches".to_string());
// Un-apply the previous attempt's patches when the patch tree changed:
// best-effort (a failing pop leaves the state quilt reports; the fresh
// `push -a` below then fails loudly on the actual problem).
if pop_first {
log::info!("Patch series changed since the previous attempt, un-applying patches");
let status = cap(
ctx.command("quilt")
.current_dir(package_dir)
.envs(patch_env.clone())
.arg("pop")
.arg("-a"),
sink,
)
.status()?;
if !status.success() {
warn!("'quilt pop -a' failed; continuing with 'quilt push -a'");
}
}
// Apply all patches listed in the series
view.phase(
Phase::ApplyingPatches.label(),
Box::new(QuiltClassifier::new(total_patches)),
);
let mut patch_env = env.clone();
patch_env.insert("QUILT_PATCHES".to_string(), "debian/patches".to_string());
let status = cap(
ctx.command("quilt")
.current_dir(package_dir)
@@ -1395,6 +1500,7 @@ mod tests {
ctx,
&crate::report::Quiet,
&None,
false,
)
.unwrap();
}
src/deb/mod.rs
+272 -11
View File
@@ -3,6 +3,9 @@ mod cross;
/// cleanup-hook registry drained by the SIGINT handler
pub(crate) mod ephemeral;
mod local;
/// Resumable build sessions: manifest, discovery, `pkh deb list` and the
/// host-tree snapshot/sync that keeps build artifacts alive across attempts
pub mod session;
use crate::context::{self, Context};
use crate::logfmt::{
@@ -115,10 +118,33 @@ pub struct DebBuildOptions<'a> {
pub jobs: Option<usize>,
/// Explicit build context; defaults to the current context.
pub ctx: Option<Arc<Context>>,
/// Resumable-session request: adopt a previous attempt's session
/// (`pkh deb --resume [<id>]`). Defaults to a fresh build, which
/// replaces the session of this identity without reading it.
pub resume: SessionResume,
/// Keep the session after a successful build (`pkh deb --keep`).
/// Without it a success consumes the session, as today's successful
/// build removes its chroot.
pub keep_session: bool,
/// Where build events (phases, progress, outcome) are reported.
pub view: &'a dyn BuildView,
}
/// Which session a `pkh deb` build should adopt, if any.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub enum SessionResume {
/// No adoption: build from scratch, replacing the session of this
/// identity (recording stays on).
#[default]
Fresh,
/// Adopt the newest usable session recorded from this tree
/// (`pkh deb --resume`).
Latest,
/// Adopt the session with this id or unique id prefix
/// (`pkh deb --resume <id>`).
Id(String),
}
impl Default for DebBuildOptions<'_> {
fn default() -> Self {
static QUIET: crate::report::Quiet = crate::report::Quiet;
@@ -133,6 +159,8 @@ impl Default for DebBuildOptions<'_> {
inject: Vec::new(),
jobs: None,
ctx: None,
resume: SessionResume::Fresh,
keep_session: false,
view: &QUIET,
}
}
@@ -175,6 +203,8 @@ async fn build_binary_package_impl(
ref inject,
ref jobs,
ref ctx,
ref resume,
keep_session,
view,
} = opts;
let cwd = cwd.as_deref().unwrap_or_else(|| Path::new("."));
@@ -233,7 +263,50 @@ async fn build_binary_package_impl(
// Create an ephemeral unshare context for all Local builds. It is kept in
// this scope so it outlives the guarded section below and is only dropped
// once the live view has been cleared.
let mut guard = if *mode == BuildMode::Local {
//
// Resumable build sessions are a local-only feature: with a local base
// context the chroot lives in a session root (recording always on,
// adoption only when requested); every other base context keeps the
// historical anonymous temp chroot.
let session_enabled = *mode == BuildMode::Local
&& matches!(base_ctx.config, crate::context::ContextConfig::Local);
let host_tree = cwd.canonicalize().unwrap_or_else(|_| cwd.to_path_buf());
let identity = session::SessionIdentity {
package: package.clone(),
series: series.to_string(),
arch: arch.to_string(),
cross,
};
let session = if session_enabled {
match resolve_session(resume, &host_tree, &identity, &version, series, chroot_arch) {
Ok(session) => session,
Err(e) => {
view.finish_failure();
return Err(e);
}
}
} else {
None
};
let resume_chroot = session
.as_ref()
.is_some_and(|s| s.chroot_is_usable() && resume != &SessionResume::Fresh);
let mut guard = if let Some(s) = session.clone() {
Some(
ephemeral::EphemeralContextGuard::new_for_session(
s,
series,
chroot_arch,
base_ctx.clone(),
view,
resume_chroot,
keep_session,
)
.await?,
)
} else if *mode == BuildMode::Local {
Some(
ephemeral::EphemeralContextGuard::new_with_context(
series,
@@ -258,16 +331,71 @@ async fn build_binary_package_impl(
};
let result = async {
// Prepare build directory
let build_root = build_ctx.create_temp_dir()?;
// Prepare build directory: the stable /tmp/pkh-build for session
// builds (see EphemeralContextGuard::build_root — the overlay
// upperdir mirrors paths relative to the merge point), a fresh
// timestamped temp dir otherwise.
let build_root = match guard.as_ref() {
Some(g) => g.build_root()?,
None => build_ctx.create_temp_dir()?,
};
// Ensure availability of all needed files for the build
let parent_dir = cwd.parent().ok_or("Cannot find parent directory")?;
build_ctx.ensure_available(parent_dir, &build_root)?;
let parent_dir_name = parent_dir
.file_name()
.ok_or("Cannot find parent directory name")?;
let build_root = format!("{}/{}", build_root, parent_dir_name.to_str().unwrap());
// Ensure availability of all needed files for the build. Session
// builds sync the persistent overlay upperdir with the host tree
// first (propagating host-side deletions and modifications into
// the upperdir) and re-mount it, so the previous attempt's build
// artifacts (object files) are visible to make; a cold staging
// (no snapshot, or no overlay support) stages a fresh copy.
let parent_dir = host_tree
.parent()
.ok_or("Cannot find parent directory")?
.to_path_buf();
let mut build_resume = false;
if let Some(s) = &session {
build_resume = session::sync_upper_for_resume(s, &parent_dir)?;
if build_resume {
log::info!(
"Reusing the build artifacts of the previous attempt (incremental build)"
);
}
}
let staged_path = if let Some(s) = &session {
// Re-mount the session's persistent upperdir over the host
// tree; a driver without overlay support (or a failed mount)
// falls back to a fresh copy — still environment-resumable,
// just not artifact-resumable.
let staging = match build_ctx.ensure_available_with_overlay(
&parent_dir,
&build_root,
&s.upper_dir(),
&s.work_dir(),
) {
Ok(staging) => staging,
Err(e) if e.kind() == std::io::ErrorKind::Unsupported => {
let path = build_ctx.ensure_available(&parent_dir, &build_root)?;
crate::context::OverlayStaging {
path,
overlay: false,
}
}
Err(e) => return Err(e.into()),
};
if !staging.overlay {
// Copy fallback: the staged tree is fresh, there is nothing
// incremental to build on.
build_resume = false;
}
staging.path
} else {
build_ctx.ensure_available(&parent_dir, &build_root)?
};
// The staged parent (build root + workspace name) is where the
// build steps and the artifact collection work.
let build_root = staged_path.to_string_lossy().into_owned();
if let Some(s) = &session {
s.set_build_resume(build_resume);
}
// Resolve the package directory inside the staging area. The tree
// the caller pointed at is authoritative (its changelog defined the
@@ -302,6 +430,8 @@ async fn build_binary_package_impl(
build_ctx.clone(),
view,
*jobs,
session.as_ref(),
resume_chroot,
)
.await?
}
@@ -333,16 +463,147 @@ async fn build_binary_package_impl(
// the chroot) inherit the terminal, so they must not fight the widget.
view.suspend();
// Mark build as successful to trigger chroot cleanup
// Mark build as successful to trigger chroot cleanup. The guard's drop
// disposes of the session: removed on a plain success, kept with
// `--keep` (outcome recorded there).
if result.is_ok()
&& let Some(ref mut g) = guard
{
g.mark_build_successful();
}
// Failures keep the session for a later `--resume`: record the outcome.
if result.is_err()
&& let Some(s) = &session
{
s.set_outcome(session::OUTCOME_FAILED);
}
result
}
/// Resolve the session a build runs in.
///
/// - [`SessionResume::Fresh`]: open (or create) the session of this
/// identity and reset it for a fresh attempt — the previous attempt's
/// content is replaced without being read. When a concurrent build holds
/// the session, build without one (the historical anonymous temp chroot).
/// - [`SessionResume::Latest`]: adopt the newest usable session recorded
/// from this tree whose identity matches this build; sessions for other
/// identities are a conflict (refused, never silently switched).
/// - [`SessionResume::Id`]: adopt the session matching the id (or unique
/// prefix) recorded from this tree.
///
/// Returns the session handle, if any; `Ok(None, ..)` means "build without
/// a session". The bool says whether the previous attempt's chroot may be
/// reused (it passed the integrity probe and the bootstrap tarball still
/// matches).
fn resolve_session(
resume: &SessionResume,
host_tree: &Path,
identity: &session::SessionIdentity,
version: &str,
series: &str,
chroot_arch: Option<&str>,
) -> Result<Option<session::Session>, Box<dyn Error>> {
let usable = |s: &session::Session| {
s.chroot_is_usable() && session::tarball_matches(s, series, chroot_arch)
};
match resume {
SessionResume::Fresh => match session::fresh_session(host_tree, identity, version) {
Ok(s) => Ok(Some(s)),
Err(session::SessionOpenError::Busy(busy)) => {
log::info!("{busy}; building without a session");
Ok(None)
}
Err(session::SessionOpenError::Other(e)) => Err(e.into()),
},
SessionResume::Latest => {
let candidates: Vec<session::Session> = session::for_tree(host_tree)
.into_iter()
.filter(|s| usable(s))
.collect();
match candidates
.iter()
.find(|s| s.manifest().identity == *identity)
{
Some(s) => Ok(Some(adopt_session(s, identity, version)?)),
None if candidates.is_empty() => {
log::info!(
"No resumable build session found for this tree; building from scratch"
);
Ok(None)
}
None => {
let found = candidates
.iter()
.map(|s| s.manifest().identity.to_string())
.collect::<Vec<_>>()
.join(", ");
Err(format!(
"cannot resume: the sessions of this tree were built for {found}, \
but this build targets {identity}. Use `pkh deb list` and \
`pkh deb --resume <id>`, or adjust the selectors"
)
.into())
}
}
}
SessionResume::Id(id) => {
let mut matches: Vec<session::Session> = session::by_id_prefix(id)
.into_iter()
.filter(|s| s.manifest().host_tree == host_tree.to_string_lossy())
.filter(|s| usable(s))
.collect();
match matches.len() {
0 => Err(format!(
"no usable build session matches id '{id}' for this tree \
(see `pkh deb list`)"
)
.into()),
1 => Ok(Some(adopt_session(&matches.remove(0), identity, version)?)),
_ => {
let found = matches
.iter()
.map(|s| s.manifest().id.clone())
.collect::<Vec<_>>()
.join(", ");
Err(format!("ambiguous session id '{id}': matches {found}").into())
}
}
}
}
}
/// Lock a session for adoption and update its journal: outcome resets to
/// running, a changed changelog version drops the build artifacts (the
/// environment is kept), and the reuse decision for the chroot is made.
fn adopt_session(
session: &session::Session,
identity: &session::SessionIdentity,
version: &str,
) -> Result<session::Session, Box<dyn Error>> {
session
.acquire()
.map_err(|busy| -> Box<dyn Error> { format!("cannot resume: {busy}").into() })?;
let manifest = session.manifest();
if manifest.tree_version != version {
log::info!(
"Version changed since the previous attempt ({}): build artifacts \
are discarded, the environment is reused",
manifest.tree_version
);
session.wipe_build_artifacts();
session.set_tree_version(version);
}
session.set_outcome(session::OUTCOME_RUNNING);
log::info!(
"Resuming session {} for {identity}: the prepared chroot and \
installed build dependencies are reused",
manifest.id
);
Ok(session.clone())
}
/// Resolve the package directory for a build inside the staged build root.
///
/// The tree the caller pointed at is authoritative: `cwd`'s changelog
src/deb/session.rs
+1455
View File
File diff suppressed because it is too large Load Diff