vhaudiquet/pkh
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

deb,config: the staged-tree budget replaces the size cap

Browse Source 
The size cap was the wrong shape: an --incremental user asked for a
carried tree, and the invariant that matters is which trees survive —
the newest successful one (the next run's automatic fork base) and the
newest one (what a failed run is re-forked from explicitly). The cap
is gone, with the tree measurement and manifest field it needed; the
retention planner now works on the records' outcomes alone and fills a
budget: staged_trees staged trees per target, the newer ones first,
the pair always among them. When the budget cannot hold the pair (one
staged tree configured, the newest build failed and an older
successful tree present) only one of the two is successful and that
one is kept; with no successful tree at all, the most recent one is.

The budget is a setting of the nested config file:

    deb:
      incremental:
        staged_trees: 2

(default 2; 0 clamps to 1 with a warning — the newest successful tree
always survives). The fork-base selection no longer truncates the
candidate list to the budget before ranking it: with a budget of one
and a failed newest tree, that truncation hid the successful record
and forked the failed one instead.
This commit is contained in:
vhaudiquet
2026-10-03 23:43:10 +02:00
parent a9a6d2001c
commit 9acf650fc9
8 changed files with 284 additions and 414 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+11 -14
View File
@@ -183,26 +183,23 @@ pkh deb profile [<id>] # a record's per-phase timing profile: how
pkh deb --no-ccache # build without the shared compile cache
```
Each target keeps two staged trees: the newest one and the newest
successful one — the tree the next `--incremental` run forks
Each target keeps two staged trees by default: the newest one and the
newest successful one — the tree the next `--incremental` run forks
automatically, and the tree a failed run can be re-forked from
explicitly. An `--incremental` run always keeps its own tree, so at
least one staged tree exists even when the newest build failed. Every
other staged tree degrades to a metadata-only record (manifest and log
stay).
A staged tree larger than 20 GiB comes down as well, as soon as the
pair has another member covering the resume — a tree that size costs
more disk than the resume it buys. The cap is a setting:
explicitly — with any newer trees filling the budget. An
`--incremental` run always keeps its own tree, so at least one staged
tree exists even when the newest build failed; every other staged tree
degrades to a metadata-only record (manifest and log stay). The
budget is a setting:
```yaml
# ~/.config/pkh/config.yml (missing file or keys = defaults)
staged_maxsize: 20G
deb:
incremental:
staged_trees: 2
```
`0` disables the size rule entirely, and `PKH_STAGED_MAXSIZE`
overrides the file when set. `pkh prune` removes records older than a
week. `pkh prune --dry-run` previews everything it would remove,
`pkh prune` removes records older than a week. `pkh prune --dry-run` previews everything it would remove,
grouped by kind with sizes; `--all` ignores the age and removes
everything a live build does not hold, recent records and staged trees
included; `-f/--force` skips the size pass and the confirmation
src/config.rs
+75 -33
View File
@@ -1,20 +1,26 @@
//! User settings: the optional `config.yml` in the pkh configuration
//! directory (the same `~/.config/pkh/` that holds `contexts.json`),
//! its defaults, and the environment overrides.
//! and their defaults.
//!
//! A missing file carries the defaults — the file is written only when
//! a user creates it, never generated. A file that cannot be read or
//! parsed must not take the whole program down: like the context
//! configuration, it is backed up to `<path>.bak` (best effort) so a
//! later save cannot silently destroy it, and the defaults apply. Every
//! field is individually defaulted (`#[serde(default)]`), so any
//! hand-written file may carry a subset of the settings.
//! struct level is individually defaulted (`#[serde(default)]`), so any
//! hand-written file may carry a subset of the settings:
//!
//! ```yaml
//! deb:
//! incremental:
//! staged_trees: 2
//! ```
use serde::Deserialize;
use std::path::PathBuf;
/// The pkh configuration directory (the one holding `contexts.json`
/// and `config.json`).
/// and `config.yml`).
pub fn config_dir() -> Option<PathBuf> {
directories::ProjectDirs::from("com", "pkh", "pkh").map(|dirs| dirs.config_dir().to_path_buf())
}
@@ -29,32 +35,35 @@ fn config_path() -> Option<PathBuf> {
#[derive(Debug, Clone, Deserialize, Default)]
#[serde(default)]
pub struct Settings {
/// The retention size cap of a staged build tree: trees larger than
/// this are not retained after a build (the record degrades to
/// metadata-only), and existing oversized trees degrade at the next
/// build's retention. A size string (`20G`, `512M`, bytes); `0`
/// disables the cap.
pub staged_maxsize: String,
/// `pkh deb` settings.
pub deb: DebSettings,
}
/// The staged-tree retention cap in bytes: `PKH_STAGED_MAXSIZE` overrides
/// the settings file, which overrides the default (20 GiB — the scale
/// where a carried kernel tree stops earning its disk).
pub fn staged_maxsize() -> u64 {
if let Ok(text) = std::env::var("PKH_STAGED_MAXSIZE")
&& let Some(bytes) = crate::utils::size::parse_size(&text)
{
return bytes;
}
file_staged_maxsize()
/// The `pkh deb` settings.
#[derive(Debug, Clone, Deserialize, Default)]
#[serde(default)]
pub struct DebSettings {
/// `pkh deb --incremental` settings.
pub incremental: IncrementalSettings,
}
/// The settings-file layer of [`staged_maxsize`]: the stored value
/// parsed, the default when unset, unparseable, or the file is missing
/// or corrupt (a corrupt file is backed up, as with the context
/// configuration).
fn file_staged_maxsize() -> u64 {
const DEFAULT: u64 = 20 * 1024 * 1024 * 1024;
/// The `pkh deb --incremental` settings.
#[derive(Debug, Clone, Deserialize, Default)]
#[serde(default)]
pub struct IncrementalSettings {
/// How many staged trees `pkh deb` retains per target: the newer
/// ones first, with the newest successful one and the newest one
/// always among them.
pub staged_trees: Option<usize>,
}
/// How many staged trees `pkh deb` retains per target: the setting's
/// value, or the default (2 — the newest tree and the newest successful
/// one). A configured 0 cannot honor the retention invariant (the
/// newest successful tree always survives), so it clamps to 1 with a
/// warning.
pub fn staged_trees() -> usize {
const DEFAULT: usize = 2;
let Some(path) = config_path() else {
return DEFAULT;
};
@@ -72,15 +81,18 @@ fn file_staged_maxsize() -> u64 {
return DEFAULT;
}
};
crate::utils::size::parse_size(&settings.staged_maxsize).unwrap_or_else(|| {
match settings.deb.incremental.staged_trees {
Some(0) => {
log::warn!(
"Settings file {} carries an unparseable staged_maxsize \
'{}'; using the default",
path.display(),
settings.staged_maxsize
"Settings file {} carries staged_trees: 0; the newest \
successful staged tree is always retained, so 1 applies",
path.display()
);
DEFAULT
})
1
}
Some(count) => count,
None => DEFAULT,
}
}
/// Back up a corrupt settings file next to itself (best effort), so a
@@ -100,3 +112,33 @@ fn backup(path: &std::path::Path) {
),
}
}
#[cfg(test)]
mod tests {
use super::*;
/// The nested setting parses; every level may be missing and every
/// missing level defaults.
#[test]
fn settings_parse_the_nested_shape() {
let settings: Settings =
serde_yaml::from_str("deb:\n incremental:\n staged_trees: 3").unwrap();
assert_eq!(settings.deb.incremental.staged_trees, Some(3));
let settings: Settings = serde_yaml::from_str("deb: {}").unwrap();
assert_eq!(settings.deb.incremental.staged_trees, None);
let settings: Settings = serde_yaml::from_str("{}").unwrap();
assert_eq!(settings.deb.incremental.staged_trees, None);
}
/// A value of the wrong type is a file error, not a default for the
/// field: the whole file is corrupt (the caller backs it up and
/// applies the defaults).
#[test]
fn a_mistyped_value_is_a_corrupt_file() {
let result =
serde_yaml::from_str::<Settings>("deb:\n incremental:\n staged_trees: two\n");
assert!(result.is_err());
}
}
src/deb/local.rs
+7 -1
View File
@@ -291,7 +291,7 @@ fn resumed_upper_dir(session: Option<&session::Session>, build_resume: bool) ->
/// chroot-local uid, host-side a subuid). The metadata is hot when the
/// resume sync just walked the same tree.
fn has_foreign_owner(root: &Path) -> bool {
use std::os::unix::fs::MetadataExt;
use std::os::unix::fs::{FileTypeExt, MetadataExt};
let euid = unsafe { libc::geteuid() };
let mut stack = vec![root.to_path_buf()];
while let Some(dir) = stack.pop() {
@@ -302,6 +302,12 @@ fn has_foreign_owner(root: &Path) -> bool {
let Ok(meta) = fs::symlink_metadata(entry.path()) else {
continue;
};
// Overlayfs whiteouts (char device 0:0) are the overlay's
// own bookkeeping, root-owned in every build — not the mark
// of a setpriv-era tree.
if meta.file_type().is_char_device() && meta.rdev() == 0 {
continue;
}
if meta.uid() != euid {
return true;
}
src/deb/mod.rs
-1
View File
@@ -891,7 +891,6 @@ fn resolve_record_in(
session::fork_candidates_in(records_root, identity)
.into_iter()
.filter(|s| !session::lock_is_held(&s.lock_path()).unwrap_or(true))
.take(session::KEEP_UPPERS)
.collect();
for base in session::rank_base_candidates(candidates) {
let why = if base.manifest().last_outcome == session::OUTCOME_SUCCESS {
src/deb/session.rs
+181 -299
View File
@@ -64,15 +64,6 @@ pub(crate) const SCHEMA_VERSION: u32 = 1;
/// removes it.
pub const SESSION_RETENTION: Duration = Duration::from_secs(7 * 24 * 60 * 60);
/// How many staged trees one identity retains: the newest one and the
/// newest successful one — a single record when they coincide. Every
/// other upper-retaining record degrades to metadata-only (its upperdir
/// and host-tree snapshot are deleted; manifest and log stay): the
/// newest successful tree is what the next `--incremental` run forks
/// automatically, the newest tree is what a failed run can be re-forked
/// from explicitly, and anything older earns no disk.
pub(crate) const KEEP_UPPERS: usize = 2;
/// The build is running (`last_outcome`).
pub const OUTCOME_RUNNING: &str = "running";
/// The build recorded in this record succeeded (`last_outcome`).
@@ -268,11 +259,6 @@ pub struct SessionManifest {
/// before any phase produced a measurement).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub profile: Option<BuildProfile>,
/// Size of the retained staged tree in bytes, measured at retention
/// (`None`: never measured — the record predates the field, its tree
/// was kept by an interrupted build, or it retains nothing).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub staged_bytes: Option<u64>,
}
/// State shared through the [`Session`] handle: the manifest plus the
@@ -325,61 +311,43 @@ pub(crate) fn upper_retention(requested: bool, upper_exists: bool) -> UpperReten
/// Why the retention plan keeps or degrades a staged tree.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum UpperDecision {
/// The newest staged tree, or the newest successful one: kept.
/// Within the retained set: the newest staged tree, the newest
/// successful one, or one of the newer trees filling the budget.
Keep,
/// Neither the newest staged tree nor the newest successful one.
DropCount,
/// Part of the retained pair but over the size cap, with the
/// pair's other member still covering the incremental resume.
DropSize,
}
/// One upper-retaining record as the retention planner sees it,
/// newest first.
pub(crate) struct UpperPlanRecord {
/// Whether the record's build succeeded — the newest successful
/// tree is what the next `--incremental` run forks automatically.
pub success: bool,
/// The tree's measured size in bytes.
pub bytes: u64,
/// Beyond the retained set.
Drop,
}
/// The retention plan for upper-retaining records (newest-first
/// input): keep the newest staged tree and the newest successful one,
/// degrade everything else, and apply the size cap to the pair without
/// ever emptying it — an `--incremental` user always has at least one
/// staged tree. When the cap forces a choice inside the pair, the
/// failed newest tree yields first: the successful tree is the one the
/// next run forks automatically.
pub(crate) fn upper_retention_plan(
records: &[UpperPlanRecord],
max_bytes: u64,
) -> Vec<UpperDecision> {
let mut decisions = vec![UpperDecision::DropCount; records.len()];
if records.is_empty() {
/// `success` flags, one per record): keep `staged_trees` trees — the
/// newer ones first — and always the newest successful one (the tree
/// the next `--incremental` run forks automatically) and the newest one
/// (what a failed run is re-forked from explicitly). When the budget
/// cannot hold the pair — one staged tree configured with the newest
/// build failed and an older successful tree present — only one of the
/// two is successful, and that one is kept; with no successful tree at
/// all, the most recent one is.
pub(crate) fn upper_retention_plan(success: &[bool], staged_trees: usize) -> Vec<UpperDecision> {
let mut decisions = vec![UpperDecision::Drop; success.len()];
if success.is_empty() {
return decisions;
}
let mut keep: Vec<usize> = vec![0];
if let Some(success_idx) = records.iter().position(|record| record.success)
&& success_idx != 0
let budget = staged_trees.max(1);
let mut keep: Vec<usize> = (0..budget.min(success.len())).collect();
if let Some(success_idx) = success.iter().position(|succeeded| *succeeded)
&& !keep.contains(&success_idx)
{
if budget == 1 {
// The newest and the newest successful are two records and
// only one tree is kept: exactly one of them is successful —
// that one — or neither is, and the most recent stays.
keep = vec![success_idx];
} else {
// Make room for the mandatory member: the oldest filled
// slot yields.
keep.pop();
keep.push(success_idx);
}
if max_bytes != 0 {
// Try the cap on the pair, least valuable first: the failed
// newest tree, then the successful one — and never the last
// tree standing.
let mut drop_order: Vec<usize> = keep.clone();
drop_order.sort_by_key(|&idx| records[idx].success as usize);
for idx in drop_order {
if keep.len() <= 1 {
break;
}
if records[idx].bytes > max_bytes {
keep.retain(|&kept| kept != idx);
decisions[idx] = UpperDecision::DropSize;
}
}
}
for &idx in &keep {
decisions[idx] = UpperDecision::Keep;
@@ -387,25 +355,6 @@ pub(crate) fn upper_retention_plan(
decisions
}
/// The staged-tree retention cap in bytes (`0` = unlimited), from the
/// user settings (see [`crate::config`]).
pub(crate) fn staged_maxsize() -> u64 {
crate::config::staged_maxsize()
}
/// The staged tree's size in bytes of a record: the manifest's stored
/// measurement when it has one, a fresh one (recorded for the next
/// sweeps) otherwise — a tree only changes while its own build runs,
/// so a stored value stays accurate between builds.
fn upper_size(record: &Session) -> u64 {
if let Some(bytes) = record.manifest().staged_bytes {
return bytes;
}
let bytes = crate::utils::size::tree_sizes(&[record.upper_dir().as_path()])[0];
record.set_staged_bytes(bytes);
bytes
}
impl Session {
/// The record root directory.
pub fn root(&self) -> &Path {
@@ -520,14 +469,6 @@ impl Session {
});
}
/// Record the measured size of the retained staged tree and persist
/// it (the upper-cap sweeps reuse it instead of re-walking the tree).
pub(crate) fn set_staged_bytes(&self, bytes: u64) {
self.with_manifest(|manifest| {
manifest.staged_bytes = Some(bytes);
});
}
/// Run `f` over the manifest and persist the result; failures are
/// logged and never abort the build (the journal is a cache).
fn with_manifest(&self, f: impl FnOnce(&mut SessionManifest)) {
@@ -548,39 +489,39 @@ impl Session {
/// Retain the record at the end of its build: the outcome is
/// recorded on success (failures already carry theirs), and the staged
/// tree follows [`upper_retention`] — kept exactly for `--incremental`
/// runs that actually staged one, whatever its size (an `--incremental`
/// user always has at least one staged tree); a plain run's tree is
/// dropped and a run that ended before staging keeps nothing, both
/// leaving a metadata-only record (manifest + log). The retention
/// sweep afterwards applies the count and size policies to the
/// identity's older trees. The chroot was already removed by the
/// caller's teardown. The interrupt path is handled by the SIGINT
/// cleanup hook, which follows the same policy.
/// runs that actually staged one (the user asked for a carried tree);
/// a plain run's tree is dropped and a run that ended before staging
/// keeps nothing, both leaving a metadata-only record (manifest +
/// log). The retention sweep afterwards applies the staged-tree
/// budget to the identity's older trees. The chroot was already
/// removed by the caller's teardown. The interrupt path is handled
/// by the SIGINT cleanup hook, which follows the same policy.
pub(crate) fn retain_upper(&self, keep_upper: bool, succeeded: bool) {
self.retain_upper_with_cap(keep_upper, succeeded, staged_maxsize());
self.retain_upper_keeping(keep_upper, succeeded, crate::config::staged_trees());
}
/// [`Self::retain_upper`] with the retention cap explicit (the test
/// seam; the cap usually comes from [`staged_maxsize`]).
pub(crate) fn retain_upper_with_cap(&self, keep_upper: bool, succeeded: bool, max_bytes: u64) {
/// [`Self::retain_upper`] with the staged-tree budget explicit (the
/// test seam; the budget usually comes from [`crate::config`]).
pub(crate) fn retain_upper_keeping(
&self,
keep_upper: bool,
succeeded: bool,
staged_trees: usize,
) {
if succeeded {
self.set_outcome(OUTCOME_SUCCESS);
}
let id = self.manifest().id;
let upper_bytes = (keep_upper && self.has_upper())
.then(|| crate::utils::size::tree_sizes(&[self.upper_dir().as_path()])[0]);
match upper_retention(keep_upper, self.has_upper()) {
UpperRetention::Keep => {
self.with_manifest(|manifest| {
manifest.kept_upper = true;
manifest.staged_bytes = upper_bytes;
});
log::info!(
"Build record {id} kept its staged tree ({}, the next \
--incremental run forks it)",
crate::utils::size::format_bytes(upper_bytes.unwrap_or(0))
"Build record {id} kept its staged tree (the next \
--incremental run forks it)"
);
self.enforce_upper_cap(max_bytes);
self.enforce_upper_cap(staged_trees);
}
UpperRetention::KeepNothing => {
log::info!(
@@ -597,7 +538,7 @@ impl Session {
/// Drop the record's persisted upperdir state: the tree-upper, its
/// host-tree snapshot and the scratch workdir, and the manifest's
/// claims about them (`kept_upper`, the measured size).
/// claim about the tree (`kept_upper`).
fn wipe_upper(&self) {
for path in [self.upper_dir(), self.work_dir(), self.snapshot_file()] {
if let Err(e) = crate::deb::ephemeral::privileged_remove(&path)
@@ -608,22 +549,20 @@ impl Session {
}
self.with_manifest(|manifest| {
manifest.kept_upper = false;
manifest.staged_bytes = None;
});
}
/// Enforce the retention policy for this record's identity: keep the
/// newest staged tree and the newest successful one
/// ([`KEEP_UPPERS`]), degrade every other upper-retaining record to
/// metadata-only (upperdir and snapshot deleted, manifest and log
/// kept), and apply the size cap to the kept pair without ever
/// emptying it — covering trees kept oversized by older pkh versions
/// or interrupted builds too. The record whose build just finished
/// (and any record locked by a live build) is never degraded: an
/// `--incremental` run's own tree is the at-least-one tree the user
/// asked for, and the sweep of a later build applies the cap to it
/// Enforce the retention policy for this record's identity: keep
/// `staged_trees` trees — the newer ones first, the newest
/// successful one always among them (see
/// [`upper_retention_plan`]) — and degrade every other
/// upper-retaining record to metadata-only (upperdir and snapshot
/// deleted, manifest and log kept). The record whose build just
/// finished (and any record locked by a live build) is never
/// degraded: an `--incremental` run's own tree is what the user
/// asked to carry, and the sweep of a later build decides on it
/// once its lock is free.
fn enforce_upper_cap(&self, max_bytes: u64) {
fn enforce_upper_cap(&self, staged_trees: usize) {
// The record root is <sessions root>/<slug>/<id>: the sessions
// root is two levels up, keeping the cap on the same root the
// record lives in (tests use fixture roots).
@@ -636,20 +575,17 @@ impl Session {
.filter(|record| record.has_upper())
.collect();
uppers.sort_by(|a, b| record_ordering(a, b).reverse());
let plan_records: Vec<UpperPlanRecord> = uppers
let success: Vec<bool> = uppers
.iter()
.map(|record| UpperPlanRecord {
success: record.manifest().last_outcome == OUTCOME_SUCCESS,
bytes: upper_size(record),
})
.map(|record| record.manifest().last_outcome == OUTCOME_SUCCESS)
.collect();
for (record, decision) in uppers
.into_iter()
.zip(upper_retention_plan(&plan_records, max_bytes))
.zip(upper_retention_plan(&success, staged_trees))
{
match decision {
UpperDecision::Keep => {}
UpperDecision::DropCount | UpperDecision::DropSize => {
UpperDecision::Drop => {
if record.root == self.root || lock_is_held(&record.lock_path()).unwrap_or(true)
{
log::debug!(
@@ -659,28 +595,11 @@ impl Session {
);
continue;
}
match decision {
UpperDecision::DropSize => {
log::info!(
"Record {} degrades to metadata-only (its \
staged tree is {}, over the retention cap \
of {})",
record.manifest().id,
crate::utils::size::format_bytes(
record.manifest().staged_bytes.unwrap_or(0)
),
crate::utils::size::format_bytes(max_bytes)
);
}
_ => {
log::debug!(
"Record {} degrades to metadata-only (neither \
the newest staged tree of {identity} nor the \
newest successful one)",
"Record {} degrades to metadata-only (beyond the \
retained staged trees of {identity})",
record.manifest().id
);
}
}
record.wipe_upper();
}
}
@@ -1151,7 +1070,6 @@ fn fresh_manifest(
build_resume: false,
timings: None,
profile: None,
staged_bytes: None,
}
}
@@ -1373,7 +1291,6 @@ pub(crate) fn fork_record_in(
build_resume: false,
timings: None,
profile: None,
staged_bytes: None,
})?;
if carry_upper {
@@ -2430,7 +2347,7 @@ pub(crate) mod record_fixtures {
fs::write(session.snapshot_file(), "f\t1\t10\ttree/src/a.c\n").unwrap();
session.set_tarball_sha256(&sha256_file(tarball).unwrap());
session.record_phase("patches", Some("stamp".to_string()));
session.retain_upper(true, outcome == Some(OUTCOME_SUCCESS));
session.retain_upper_keeping(true, outcome == Some(OUTCOME_SUCCESS), 2);
if let Some(outcome) = outcome.filter(|o| *o != OUTCOME_SUCCESS) {
session.set_outcome(outcome);
}
@@ -2761,7 +2678,6 @@ mod tests {
},
],
}),
staged_bytes: None,
};
let base = tempfile::tempdir().unwrap();
save_manifest(base.path(), &manifest).unwrap();
@@ -3273,100 +3189,22 @@ mod tests {
assert!(degraded.log_file().is_file());
assert_eq!(
fork_candidates_in(root.path(), &identity).len(),
KEEP_UPPERS,
2,
"the newest and the newest successful staged trees remain"
);
}
/// The build's own tree is kept by its own run whatever its size —
/// an `--incremental` user always has at least one staged tree — and
/// its measured size is recorded for the sweeps. Sizes are apparent
/// sizes, so the 21 GiB probe file is sparse and costs nothing on
/// disk.
/// A budget of one keeps the newest successful tree: the sweep of
/// a later build takes both the failed newest tree and any other
/// surplus down, while the just-built tree of the running sweep
/// survives its own lock (it comes down at the next retention).
#[test]
fn retain_upper_keeps_the_built_tree_over_the_size_cap() {
fn a_budget_of_one_keeps_the_newest_successful_tree() {
let (root, tarball) = sessions_fixture();
let identity = test_identity();
let host_tree = root.path().join("tree");
let over_cap = 21 * 1024 * 1024 * 1024;
let cap = 20 * 1024 * 1024 * 1024;
let oversized =
record_with_upper(root.path(), &tarball, &host_tree, &identity, "1.0", None);
let file = fs::File::create(oversized.upper_dir().join("big.o")).unwrap();
file.set_len(over_cap).unwrap();
drop(file);
oversized.retain_upper_with_cap(true, true, cap);
assert!(
oversized.has_upper(),
"an --incremental run's own tree is always kept"
);
assert!(oversized.manifest().kept_upper);
assert_eq!(
oversized.manifest().staged_bytes,
Some(crate::utils::size::tree_sizes(&[oversized.upper_dir().as_path()])[0]),
"the recorded size is the tree's measured size"
);
assert!(oversized.root().join("manifest.json").is_file());
assert!(oversized.log_file().is_file());
}
/// The retention sweep applies the size cap to the kept pair: the
/// newest successful tree over the cap comes down once the failed
/// newest tree covers the resume — oversized trees left by older pkh
/// versions or interrupted builds come down at the next build's
/// retention too.
#[test]
fn the_cap_sweep_drops_an_oversized_tree_of_the_pair() {
let (root, tarball) = sessions_fixture();
let identity = test_identity();
let host_tree = root.path().join("tree");
let cap = 20 * 1024 * 1024 * 1024;
// The newest successful record, staged tree oversized (the
// stored measurement stands in for a tree the sweep must not
// have to walk).
let old = record_with_upper(root.path(), &tarball, &host_tree, &identity, "1.0", None);
old.set_staged_bytes(30 * 1024 * 1024 * 1024);
let old_root = old.root().to_path_buf();
drop(old); // release the record lock: a held lock is never degraded
// The newest record failed: the pair is it (tiny) and the
// oversized successful one — the sweep takes the oversized one
// down while the failed tree covers the resume.
let new = record_with_upper(
root.path(),
&tarball,
&host_tree,
&identity,
"1.1",
Some(OUTCOME_FAILED),
);
new.retain_upper_with_cap(true, false, cap);
assert!(new.has_upper(), "the newest tree covers the resume");
let degraded = identity_records_in(root.path(), &identity)
.into_iter()
.find(|r| r.root() == old_root)
.expect("the old record is still there");
assert!(!degraded.has_upper(), "the oversized tree is gone");
assert!(!degraded.manifest().kept_upper);
assert_eq!(degraded.manifest().staged_bytes, None);
assert!(degraded.root().join("manifest.json").is_file());
assert!(degraded.log_file().is_file());
}
/// The size cap never empties the pair: when both kept trees are
/// over the cap, the newest successful one (the auto-fork base) is
/// the last one standing and stays.
#[test]
fn the_cap_sweep_never_drops_the_last_tree_standing() {
let (root, tarball) = sessions_fixture();
let identity = test_identity();
let host_tree = root.path().join("tree");
let cap = 20 * 1024 * 1024 * 1024;
// The newest successful record: the auto-fork base.
let success = record_with_upper(
root.path(),
&tarball,
@@ -3375,33 +3213,81 @@ mod tests {
"1.0",
Some(OUTCOME_SUCCESS),
);
success.set_staged_bytes(30 * 1024 * 1024 * 1024);
let success_root = success.root().to_path_buf();
drop(success);
// The newest record failed, its own tree over the cap as well:
// staged manually (the fixture helper retains internally, which
// would run the sweep before the tree is oversized).
// An older failed record with a staged tree.
let failed = new_record_in(root.path(), &host_tree, &identity, "1.1").unwrap();
fs::create_dir_all(failed.upper_dir().join("src")).unwrap();
fs::write(failed.upper_dir().join("src/main.o"), b"objects").unwrap();
let file = fs::File::create(failed.upper_dir().join("big.o")).unwrap();
file.set_len(30 * 1024 * 1024 * 1024).unwrap();
drop(file);
failed.set_outcome(OUTCOME_FAILED);
failed.retain_upper_with_cap(true, false, cap);
let failed_root = failed.root().to_path_buf();
drop(failed);
// The newest build failed as well, its tree staged: under a
// budget of one the successful tree is the one kept.
let newest = new_record_in(root.path(), &host_tree, &identity, "1.2").unwrap();
fs::create_dir_all(newest.upper_dir().join("src")).unwrap();
fs::write(newest.upper_dir().join("src/main.o"), b"objects").unwrap();
newest.set_outcome(OUTCOME_FAILED);
newest.retain_upper_keeping(true, false, 1);
assert!(
failed.has_upper(),
"an --incremental run's own tree is always kept"
newest.has_upper(),
"the just-built tree survives its own run's lock"
);
let success = identity_records_in(root.path(), &identity)
.into_iter()
.find(|r| r.root() == success_root)
.expect("the successful record is still there");
assert!(success.has_upper(), "the newest successful tree is kept");
let failed = identity_records_in(root.path(), &identity)
.into_iter()
.find(|r| r.root() == failed_root)
.expect("the failed record is still there");
assert!(!failed.has_upper(), "the surplus failed tree is gone");
}
/// A budget beyond two fills with the newer trees, the newest
/// successful one always among them.
#[test]
fn a_budget_of_three_fills_with_the_newer_trees() {
let (root, tarball) = sessions_fixture();
let identity = test_identity();
let host_tree = root.path().join("tree");
// Newest-first after the newest build: [newest, 1.2, 1.1, 1.0]
// with 1.1 the newest successful one.
for (version, outcome) in [("1.0", None), ("1.1", Some(OUTCOME_SUCCESS)), ("1.2", None)] {
let record = record_with_upper(
root.path(),
&tarball,
&host_tree,
&identity,
version,
outcome,
);
drop(record);
}
let surplus = new_record_in(root.path(), &host_tree, &identity, "1.3").unwrap();
fs::create_dir_all(surplus.upper_dir().join("src")).unwrap();
fs::write(surplus.upper_dir().join("src/main.o"), b"objects").unwrap();
surplus.set_outcome(OUTCOME_FAILED);
surplus.retain_upper_keeping(true, false, 3);
let kept = identity_records_in(root.path(), &identity)
.into_iter()
.filter(|record| record.has_upper())
.map(|record| record.manifest().tree_version)
.collect::<Vec<_>>();
assert_eq!(
kept.len(),
3,
"the budget bounds the retained trees: {kept:?}"
);
assert!(
success.has_upper(),
"the last tree standing is never dropped"
kept.contains(&"1.1".to_string()),
"the newest successful tree is always kept: {kept:?}"
);
}
@@ -3465,71 +3351,67 @@ mod tests {
}
/// The retention plan: the newest staged tree and the newest
/// successful one are kept, everything else degrades by count.
/// successful one are kept, everything else degrades; the budget
/// fills with the newer trees.
#[test]
fn the_retention_plan_keeps_the_pair() {
use UpperDecision::{DropCount, Keep};
let plan = |records: &[(bool, u64)], max: u64| {
upper_retention_plan(
&records
.iter()
.map(|(success, bytes)| UpperPlanRecord {
success: *success,
bytes: *bytes,
})
.collect::<Vec<_>>(),
max,
)
};
use UpperDecision::{Drop, Keep};
// Newest-first: [failed, success, failed]. The pair is the two
// first distinct ones — the newest failed tree and the newest
// successful one.
assert_eq!(
plan(&[(false, 1), (true, 1), (false, 1)], 0),
vec![Keep, Keep, DropCount]
upper_retention_plan(&[false, true, false], 2),
vec![Keep, Keep, Drop]
);
// A successful newest record is both members at once: only it
// survives.
assert_eq!(plan(&[(true, 1), (true, 1)], 0), vec![Keep, DropCount]);
// No successful record at all: the newest tree is the one kept
// (a failed record is a legitimate explicit-fork base).
assert_eq!(plan(&[(false, 1), (false, 1)], 0), vec![Keep, DropCount]);
assert_eq!(plan(&[], 0), Vec::<UpperDecision>::new());
// A successful newest record is both members at once; the
// budget still fills with the next-newer tree.
assert_eq!(
upper_retention_plan(&[true, true, false], 2),
vec![Keep, Keep, Drop]
);
// No successful record at all: the newest tree is the one the
// pair keeps (a failed record is a legitimate explicit-fork
// base), and the budget fills with the next ones.
assert_eq!(
upper_retention_plan(&[false, false, false], 2),
vec![Keep, Keep, Drop]
);
assert_eq!(upper_retention_plan(&[], 2), Vec::<UpperDecision>::new());
}
/// The size cap thins the pair without ever emptying it: the failed
/// newest tree yields first, the successful one only while another
/// tree remains, and a cap of 0 never applies.
/// A budget of one cannot hold the pair: when only one of the two
/// is successful, that one is kept; otherwise the most recent.
#[test]
fn the_retention_plan_applies_the_size_cap_to_the_pair() {
use UpperDecision::{DropSize, Keep};
let plan = |records: &[(bool, u64)], max: u64| {
upper_retention_plan(
&records
.iter()
.map(|(success, bytes)| UpperPlanRecord {
success: *success,
bytes: *bytes,
})
.collect::<Vec<_>>(),
max,
)
};
fn the_retention_plan_reduces_the_pair_to_one() {
use UpperDecision::{Drop, Keep};
// The oversized failed newest tree yields to the successful one.
assert_eq!(plan(&[(false, 30), (true, 1)], 20), vec![DropSize, Keep]);
// The oversized failed tree yields, the next-successful is over
// the cap too but is the last tree standing: kept.
assert_eq!(plan(&[(false, 30), (true, 30)], 20), vec![DropSize, Keep]);
// A failed newest tree under the cap keeps the pair; the
// oversized successful tree is dropped while the failed one
// still covers the resume.
assert_eq!(plan(&[(false, 1), (true, 30)], 20), vec![Keep, DropSize]);
// Both members under the cap: untouched.
assert_eq!(plan(&[(false, 1), (true, 20)], 20), vec![Keep, Keep]);
// The cap of 0 (unlimited) keeps anything.
assert_eq!(plan(&[(false, 99), (true, 99)], 0), vec![Keep, Keep]);
// Newest failed, older successful: the successful tree wins —
// it is what the next run forks automatically.
assert_eq!(upper_retention_plan(&[false, true], 1), vec![Drop, Keep]);
// No successful tree: the most recent stays.
assert_eq!(upper_retention_plan(&[false, false], 1), vec![Keep, Drop]);
// The newest is successful: it is both members at once.
assert_eq!(upper_retention_plan(&[true, false], 1), vec![Keep, Drop]);
}
/// A budget beyond two fills with the newer trees, the newest
/// successful one always among them.
#[test]
fn the_retention_plan_fills_the_budget() {
use UpperDecision::{Drop, Keep};
// Newest-first: the success sits within the fill.
assert_eq!(
upper_retention_plan(&[false, false, true, false], 3),
vec![Keep, Keep, Keep, Drop]
);
// The success sits beyond the fill: it displaces the oldest
// filled slot.
assert_eq!(
upper_retention_plan(&[false, false, false, true], 3),
vec![Keep, Keep, Drop, Keep]
);
}
/// Driving the keep path on an incremental record whose staged tree
src/main.rs
+2 -2
View File
@@ -393,7 +393,7 @@ fn main() {
.require_equals(false)
.default_missing_value("")
.help("Incremental build: fork a previous build record of this target and keep its staged tree")
.long_help("Incremental build: fork a previous build record of this target and keep its staged tree, so make recompiles only what changed. The base's staged tree carries over only while the changelog version and the chroot tarball are unchanged.\nWithout a value, the base is the newest successful record of the target — pkh keeps the newest and the newest successful staged trees per target. With an ID as shown by 'pkh deb list', that exact record. A failed record is fine as a base if its staged tree is still there.\nEvery --incremental run keeps its staged tree, success or failure, so the next run can fork it. A plain 'pkh deb' keeps only metadata and the build log."),
.long_help("Incremental build: fork a previous build record of this target and keep its staged tree, so make recompiles only what changed. The base's staged tree carries over only while the changelog version and the chroot tarball are unchanged.\nWithout a value, the base is the newest successful record of the target — pkh keeps the newest and the newest successful staged trees per target (two by default, 'deb.incremental.staged_trees' in ~/.config/pkh/config.yml). With an ID as shown by 'pkh deb list', that exact record. A failed record is fine as a base if its staged tree is still there.\nEvery --incremental run keeps its staged tree, success or failure, so the next run can fork it. A plain 'pkh deb' keeps only metadata and the build log."),
)
.arg(arg!(--"no-ccache" "Build without the shared ccache").required(false))
.arg(arg!(--verbose "Show raw tool output instead of the live build view").required(false)
@@ -401,7 +401,7 @@ fn main() {
.subcommand(
Command::new("list")
.about("List the build records recorded from this tree")
.long_about("List the build records recorded from this tree (plain runs too, as metadata records), with their outcome, version, fork lineage, whether their staged tree is kept, how long the dependency installation and the package build took, and the id to pass to 'pkh deb --incremental <id>'.\nRead-only: never builds or removes anything (the newest and the newest successful staged trees are kept per target, 'pkh prune' removes records older than a week)."),
.long_about("List the build records recorded from this tree (plain runs too, as metadata records), with their outcome, version, fork lineage, whether their staged tree is kept, how long the dependency installation and the package build took, and the id to pass to 'pkh deb --incremental <id>'.\nRead-only: never builds or removes anything (the newest and the newest successful staged trees are kept per target by default, 'pkh prune' removes records older than a week)."),
)
.subcommand(
Command::new("log")
src/prune.rs
+2 -1
View File
@@ -17,7 +17,8 @@
//! removes every record — recent builds and staged trees included,
//! which is where the space is. A record a live build holds the lock
//! of is never prunable, whatever the policy (the newest and the
//! newest successful staged trees per target are kept by `pkh deb`).
//! newest successful staged trees per target are kept by `pkh deb`,
//! two by default).
//! - **Shared apt download caches** under `/var/tmp/pkh/apt-cache/`
//! (see [`crate::deb::ephemeral::apt_cache_dir`]): the default policy
//! removes the ones untouched for longer than the retention window,
src/utils/size.rs
-57
View File
@@ -111,35 +111,6 @@ pub(crate) fn format_bytes(bytes: u64) -> String {
}
}
/// Parse a human size into bytes: an integer with an optional unit
/// suffix — `B`, `K`, `M`, `G`, `T` (optionally `iB`, case-insensitive),
/// the binary multiples ccache's configuration uses (`20G` = 20 GiB). A
/// bare number is bytes. `None` on anything else (empty, negative,
/// overflow, unknown unit).
pub(crate) fn parse_size(text: &str) -> Option<u64> {
let text = text.trim();
let split = text
.rfind(|c: char| c.is_ascii_digit())
.map(|idx| idx + 1)
.unwrap_or(0);
let (digits, unit) = text.split_at(split);
let power = match unit.trim().to_ascii_lowercase().as_str() {
"" | "b" => 0,
"k" | "kib" => 1,
"m" | "mib" => 2,
"g" | "gib" => 3,
"t" | "tib" => 4,
_ => return None,
};
if digits.is_empty() || !digits.bytes().all(|b| b.is_ascii_digit()) {
return None;
}
digits
.parse::<u64>()
.ok()
.and_then(|value| value.checked_mul(1u64.checked_shl(10 * power)?))
}
#[cfg(test)]
mod tests {
use super::*;
@@ -188,32 +159,4 @@ mod tests {
// this size; the mount-crossing walk keeps it out of the totals.
assert_eq!(format_bytes(140_737_477_959_680), "128.0 TiB");
}
/// Unit spellings and their byte values: ccache-style binary
/// multiples, optional B/iB decorations, case-insensitive.
#[test]
fn parse_size_reads_the_unit_suffixes() {
assert_eq!(parse_size("0"), Some(0));
assert_eq!(parse_size("512"), Some(512));
assert_eq!(parse_size("512B"), Some(512));
assert_eq!(parse_size("512 b"), Some(512));
assert_eq!(parse_size("10K"), Some(10 * 1024));
assert_eq!(parse_size("512m"), Some(512 * 1024 * 1024));
assert_eq!(parse_size("20G"), Some(20 * 1024 * 1024 * 1024));
assert_eq!(parse_size("2TiB"), Some(2 * 1024 * 1024 * 1024 * 1024));
assert_eq!(parse_size(" 20G "), Some(20 * 1024 * 1024 * 1024));
}
/// Garbage is an error, not a guess: empty, negative, junk units and
/// overflow all parse to None.
#[test]
fn parse_size_rejects_garbage() {
assert_eq!(parse_size(""), None);
assert_eq!(parse_size(" "), None);
assert_eq!(parse_size("G"), None);
assert_eq!(parse_size("-1"), None);
assert_eq!(parse_size("20GB"), None);
assert_eq!(parse_size("20 kg"), None);
assert_eq!(parse_size("999999999999999999999999T"), None);
}
}