deb: survive crashed sessions and config-blind resumes
CI / build (push) Successful in 3m9s
CI / test (push) Skipped
CI / publish (push) Skipped
CI / snap (push) Successful in 6m46s

Two failures from resumed kernel sessions:

A resumed build re-runs debian/rules build, but the kernels keep
their wrapper-step stamps in debian/stamps and the flavour config
rule there exports .config from the annotations with no config
prerequisite: the resumed build silently kept the previous
attempt's configuration and a config edit never reached the .deb.
Dropping the stamp cache unconditionally would impose the kernel
packaging's shape on every package, so it is data-driven instead:
packages declare the caches under a resume_clear quirk, and
resumed builds remove those tree-relative paths before the build.
The inner kbuild keeps its own incremental state, so only the
cheap wrapper passes re-run and config-affected objects recompile.

A SIGKILLed build (OOM) leaves its overlay mounts and /proc bind
mount behind, and overlayfs creates root-owned work state inside
the workdir: the next plain build (recording is always on, it
replaces the session of its identity) failed to clear the
leftovers with a permission error. Session clearing now unmounts
everything under each entry at depth (re-reading /proc/mounts,
tolerating mount stacks from consecutive crashes) and escalates
through sudo; when the tree still cannot be cleared, the build
reports it and continues without a session instead of layering
over a half-cleared one.
This commit is contained in:
2026-09-28 00:05:14 +02:00
parent 6bb8ab5016
commit d927dc93ab
8 changed files with 230 additions and 42 deletions
+76 -6
View File
@@ -147,15 +147,48 @@ pub(crate) fn privileged_remove(path: &Path) -> std::io::Result<()> {
}
}
/// Remove the contents of `dir` but keep the directory itself.
fn wipe_dir_contents(dir: &Path) -> std::io::Result<()> {
/// Clear a directory's contents, keeping the directory itself.
///
/// Built for re-cleaning a crashed build's leftovers: a SIGKILLed build
/// (OOM, power loss) leaves its overlay mounts and /proc bind mount behind,
/// and the kernel creates root-owned `work` state inside an overlay
/// workdir that a plain `rm -rf` cannot remove. So every child is first
/// unmounted at depth (re-reading /proc/mounts until nothing is left under
/// it), then removed with a privileged `rm -rf` fallback.
///
/// Returns an error when some content could not be removed: callers should
/// refuse to layer a fresh build over a half-cleared tree.
pub(crate) fn clear_dir(dir: &Path) -> std::io::Result<()> {
fs::create_dir_all(dir)?;
const MAX_UNMOUNT_PASSES: usize = 10;
for entry in fs::read_dir(dir)?.flatten() {
let path = entry.path();
if path.is_dir() {
fs::remove_dir_all(&path)?;
// Unmount everything under the child, deepest first; the loop
// handles stacks (a re-mount over an earlier, never-unmounted one —
// exactly what consecutive crashed builds leave behind).
for _ in 0..MAX_UNMOUNT_PASSES {
let mounts = host_mounts_under(&path);
if mounts.is_empty() {
break;
}
let is_root = crate::utils::root::is_root().unwrap_or(false);
for mount in mounts.into_iter().rev() {
if !unmount_path(&mount, is_root) {
log::warn!("Failed to unmount {} (retrying)", mount.display());
}
}
}
let result = if path.is_dir() {
fs::remove_dir_all(&path)
} else {
fs::remove_file(&path)?;
fs::remove_file(&path)
}
.or_else(|_| privileged_remove(&path));
if let Err(e) = result {
return Err(std::io::Error::other(format!(
"cannot remove {}: {e}",
path.display()
)));
}
}
Ok(())
@@ -487,7 +520,7 @@ impl EphemeralContextGuard {
}
// Not reusing (fresh session or failed probe): never extract
// over stale content.
wipe_dir_contents(chroot_path)?;
clear_dir(chroot_path)?;
}
// Clone ctx for use in create_device_nodes after download_chroot_tarball consumes it
@@ -1051,4 +1084,41 @@ mod chroot_cleanup_tests {
assert!(!chroot.exists());
}
/// clear_dir removes content a plain `rm -rf` cannot: a root-owned
/// 0700 directory (the shape of an overlay workdir left by a crashed
/// build). Skipped when neither root nor working non-interactive sudo
/// is available, since escalation is then legitimately impossible.
#[test]
fn clear_dir_removes_root_owned_subdirectories() {
let is_root = unsafe { libc::geteuid() } == 0;
if !is_root
&& !privileged_command("true", false)
.status()
.is_ok_and(|s| s.success())
{
return;
}
let dir = tempfile::tempdir().unwrap();
let cleared = dir.path().join("chroot");
std::fs::create_dir_all(cleared.join("stale")).unwrap();
std::fs::write(cleared.join("stale").join("f"), "x").unwrap();
// Root-owned, non-traversable: the crashed-workdir shape.
let _ = privileged_command("chmod", is_root)
.arg("-R")
.arg("0700")
.arg(cleared.join("stale"))
.status();
let _ = privileged_command("chown", is_root)
.arg("-R")
.arg("root:root")
.arg(cleared.join("stale"))
.status();
clear_dir(&cleared).expect("clear_dir removes root-owned leftovers");
let read = std::fs::read_dir(&cleared).unwrap();
assert_eq!(read.count(), 0, "the directory must be empty");
}
}