Compare commits

...
17 Commits
Author SHA1 Message Date
vhaudiquet 4af8dbddb0 fmt: fix clippy warnings on all targets
CI / build (push) Successful in 2m48s
CI / test (push) Skipped
CI / snap (push) Failing after 11s
Drop an unused test fixture constant, move download_trust_ppa_key above
the test module that precedes it, and simplify two test borrows.
2026-09-16 09:15:52 +02:00
vhaudiquet 3a454b0811 build: record the actual build environment in .buildinfo
The binary build exported DEB_BUILD_OPTIONS='parallel=<context nproc>
nocheck' (or the -j override) but the generated .buildinfo recomputed
the environment from host state: host core count, no nocheck, and
vendor profiles that ignored DEB_BUILD_PROFILES (a cross build recorded
no 'cross' profile). generate_binary_metadata now records the exact env
map that was exported to the build steps, and the recorded profiles
come from the exported DEB_BUILD_PROFILES when set.

Also unifies vendor parsing on one helper (the context-side copy lacked
the Origin: fallback of the source-build path).
2026-09-16 04:22:43 +02:00
vhaudiquet 50ae12cafe prune: match real log names, order retention by time, spare fresh locks
Log retention only matched 'deb-*' logs, so source-build ('build-*')
and placeholder ('pkh-*') logs accumulated forever, and the 'keep the
newest' sort was lexicographic on names that sort by package/version
first, so arbitrary logs were kept. All three log shapes are matched
now and retention orders by the timestamp embedded in the name (mtime
fallback). Stale-lockfile pruning no longer deletes lockfiles younger
than 24h: a fresh <tarball>.lock is the mutual-exclusion signal of a
concurrent download and deleting it could corrupt the shared tarball
cache.
2026-09-16 04:04:50 +02:00
vhaudiquet 38562abe2c debian: fix deb822 writer/parser asymmetries
write_paragraph emitted a bare-space continuation line for empty lines
inside a value, which parse_paragraphs treated as a paragraph separator
and silently dropped the rest of the value; blank lines are now encoded
as ' .' like dpkg does and decoded back on read. Tab-indented
continuation lines now strip exactly one tab instead of keeping it.

Clearsigned .dsc content no longer leaks armor metadata into parsed
fields: the Hash:/Comment: header and the signature trailer are
stripped before parse_paragraphs at both .dsc parse sites.
2026-09-16 03:42:30 +02:00
vhaudiquet dc6a019a13 build: parse .dsc checksum fields through one shared, validating parser
The source-build pipeline and the binNMU metadata path had drifted into
two inline parsers with different acceptance rules: binary.rs filled
names from any line with a third column but partials only from
exactly-three-column lines, so a 4+ column Checksums line made
&partials[name] panic by map index. Both paths now share one parser
that accepts the modern 3-column and the legacy 5-column Files layout,
rejects anything else with an error naming the field and line, and all
remaining lookups go through .get() with a clear error instead of
indexing. Legacy 5-column Files md5s were previously attributed to the
section token instead of the file name.
2026-09-16 03:20:01 +02:00
vhaudiquet f72b35acfa Handle malformed remote and edge-case data instead of panicking
- distro_info: malformed CSV rows are skipped with a warning, dates
  that fail to parse become None, and all plain HTTP requests go
  through a shared reqwest client with connect/total timeouts
- package_info: the Sources stanza iterator is iterative (a crafted
  index with many blank stanzas overflowed the stack), stanzas missing
  a Version are skipped, and failed series/pocket probes are summarized
  in the final 'not found' error instead of being silently dropped
- pull: no double unwrap on the remote-derived artifact filename, an
  empty series list is an error, and streaming downloads get a
  per-request timeout
- deb/cross: dpkg-architecture output parsing skips unexpected lines
  and its exit status is checked, as is dpkg --add-architecture
- changelog: version increments parse as u64 with checked arithmetic
  (1.0-20250123123456 used to panic on the u32 parse)
2026-09-16 02:44:45 +02:00
vhaudiquet 6a5c5a7106 context: fix deadlock and panics when resolving parented contexts
set_current held the config RwLock for writing across make_context,
which for a context with a parent re-entered the same lock through
Context::new's global-manager lookup, deadlocking 'pkh context use'.
Context building is now lock-free by construction: make_context
resolves parent chains against a snapshot map (also rejecting parent
cycles), and neither set_current nor remove_context holds a guard
while building a Context.

A corrupt contexts.json no longer aborts every command at manager
init: load falls back to the default local-only config, backs the
corrupt file up to contexts.json.bak so a later save cannot silently
destroy it, and a dangling current/parent context falls back to local
with an error log instead of panicking.
2026-09-16 02:27:11 +02:00
vhaudiquet 592e98c1e9 deb: pass the build context explicitly instead of swapping the global
build_binary_package installed its ephemeral chroot context into the
process-global manager and read it back with context::current(),
ignoring its ctx parameter: two concurrent builds would re-point each
other's global and each drop would clean up whichever chroot was
current at the time. The guard now keeps the Arc of the context it
created (parented directly on the base context, not on a config-name
lookup), exposes it via context(), and Drop cleans up exactly that
context and restores the exact handle that was current at creation,
so overlapping builds no longer cross-destroy each other.
2026-09-16 02:05:40 +02:00
vhaudiquet 512a1cb778 deb: unmount and remove the ephemeral chroot on Ctrl-C
The SIGINT handler libc::_exit(130)s, skipping EphemeralContextGuard's
drop and leaking the freshly bootstrapped chroot with its bind-mounted
/proc and overlay mounts. Resources now register a self-contained
cleanup hook in a process-global registry that the handler drains right
before exiting: the hook unmounts everything under the chroot path
(children first, lazy fallback) and removes the tree, using only stored
paths and direct umount/rm subprocesses so it cannot deadlock on a lock
the interrupted thread may hold; sudo -n keeps it from ever hanging on
a password prompt. Drop deregisters the hook first, so the normal
cleanup path is unchanged.

Also fixes the hex grouping of the CRC-24 polynomial in apt::release.
2026-09-16 01:55:35 +02:00
vhaudiquet 93176aa479 deps: give legacy < and > relations their documented dpkg semantics
Debian Policy 7 defines the deprecated single-character spellings as
'earlier/later or equal' (i.e. <= and >=), and dpkg still accepts them
that way; the parser mapped them to the strict << and >> instead, so
'foo (< 1.0)' was wrongly reported unmet against installed 1.0.
2026-09-16 01:30:25 +02:00
vhaudiquet 60ee99adc8 pull: search pockets in release order by default
Without an explicit --pocket, find_package stops at the first pocket
containing the package, but the search order listed '-proposed' first
and never included '-security': unreleased proposed packages won by
default and security-only updates were unreachable. Search the main
archive first, then updates, security, and proposed last.
2026-09-16 01:27:25 +02:00
vhaudiquet f6fed7328b pull: reject malformed --ppa values instead of silently using the archive
A --ppa value that was not exactly 'user/name' (full URL, extra
segment, empty halves) made base_url None and pulled the package from
the main archive without any warning. Error out naming the expected
format instead, and document the format in --help.
2026-09-16 01:24:02 +02:00
vhaudiquet 213668fa82 pull: authenticate archive indexes against signed Release files
The Sources index was downloaded with no authentication: per-artifact
checksums were verified, but against hashes taken from an index a MITM
could substitute along with the artifacts. Fetch each suite's InRelease
(or Release + Release.gpg), verify the signature with gpgv against the
archive keyring (or the PPA signing key) the same way apt does, and
checksum-check every Sources index against it before parsing.

Distro archives and PPAs verify strictly: an invalid or unverifiable
signature, or a missing gpgv binary, is a hard error. Flat repositories
keep working without a Release file or without a verifiable one (warned
as unauthenticated), but tampering evidence is a hard error there too.

Also switches all archive, PPA and keyring base URLs to https, and
reads suite components from the verified Release instead of fetching
them separately over an unauthenticated channel.
2026-09-16 01:22:22 +02:00
vhaudiquet f508f20846 context: quote program, args, cwd and env in ssh, schroot and unshare drivers
All three non-local drivers assembled remote/chroot command strings by
raw concatenation: ssh pushed args verbatim (TODO: escape), schroot
interpolated env values raw so DEB_BUILD_OPTIONS='parallel=4 nocheck'
made sh treat 'nocheck' as the command, and unshare wrapped args in
unescaped double quotes letting quotes break out and $/backticks
expand. Add a shared POSIX shell_quote helper and use it for every
component interpolated into a shell string, including ssh copy_path
(which used Rust's {:?}, not shell quoting) and schroot write_file
(which also switches echo -ne to printf %s so backslash sequences in
content are no longer interpreted).
2026-09-16 00:36:02 +02:00
vhaudiquet 685538e637 apt: round-trip sources in place instead of consolidating them into sources.list
Saving the modified source entries with save_legacy rewrote every entry
into /etc/apt/sources.list in legacy format, destroying deb-src entries
and signed-by/trusted options, duplicating every distro entry that came
from a deb822 file (which stayed in place), and hardcoding the Ubuntu
keyring on cross builds. Entries now remember the file and format they
were loaded from and are written back there; new entries (PPAs, ports)
go to a pkh-owned /etc/apt/sources.list.d/pkh-added.list, and a one-time
<path>.pkh-backup copy is made before overwriting an existing file.

Also fixes: 'Types: deb deb-src' stanzas are split instead of being
treated as binary-only, commented-out legacy entries are kept disabled
instead of deleted, debian.sources is actually read on Debian (the old
else-if never fired), and the double blank lines save_legacy emitted.
2026-09-16 00:10:45 +02:00
vhaudiquet ea70ddc10d apt: keep the keyring cache private to the invoking user
The shared world-writable /tmp/pkh-keyrings directory, combined with the
skip-if-exists logic, let any local user pre-plant keyrings that pkh
then trusts into the chroot's trusted.gpg.d. Use a per-uid 0700
directory instead, refuse to reuse a pre-existing directory that is not
owned by the current user or is group/other-writable, and drop the now
unnecessary world-accessibility chmods (mmdebstrap in unshare mode runs
with the same real uid).
2026-09-15 23:47:21 +02:00
vhaudiquet 48f6e6ce4e pull: confine tar extraction to the destination directory
Entry::unpack performs no path sanitization, so a malicious or malformed
tarball (PPA, flat repository) could write files outside the package
directory via '..' components or absolute entry paths. Refuse such
entries with an error naming the offending path.
2026-09-15 23:30:28 +02:00
29 changed files with 5088 additions and 667 deletions
+7 -5
View File
@@ -9,20 +9,22 @@ dist_info:
network: https://salsa.debian.org/debian/distro-info-data/-/raw/main/
dist:
debian:
base_url: http://deb.debian.org/debian
base_url: https://deb.debian.org/debian
archive_keyring: https://ftp-master.debian.org/keys/archive-key-{series_num}.asc
pockets:
- proposed-updates
- updates
- security
- proposed-updates
series:
local: /usr/share/distro-info/debian.csv
network: https://salsa.debian.org/debian/distro-info-data/-/raw/main/debian.csv
ubuntu:
base_url: http://archive.ubuntu.com/ubuntu
archive_keyring: http://archive.ubuntu.com/ubuntu/project/ubuntu-archive-keyring.gpg
base_url: https://archive.ubuntu.com/ubuntu
archive_keyring: https://archive.ubuntu.com/ubuntu/project/ubuntu-archive-keyring.gpg
pockets:
- proposed
- updates
- security
- proposed
series:
local: /usr/share/distro-info/ubuntu.csv
network: https://salsa.debian.org/debian/distro-info-data/-/raw/main/ubuntu.csv
+101 -15
View File
@@ -7,6 +7,7 @@ use crate::context;
use crate::distro_info;
use serde::Deserialize;
use std::error::Error;
use std::os::unix::fs::MetadataExt;
use std::path::{Path, PathBuf};
use std::sync::Arc;
@@ -47,19 +48,48 @@ pub async fn download_cache_keyrings(
// Use system temp directory for keyrings since it's accessible from unshare mode
// The home directory may not be accessible from mmdebstrap's unshare namespace
let temp_dir = std::env::temp_dir();
let keyring_dir = temp_dir.join("pkh-keyrings");
// Name the cache directory per-uid: a single shared /tmp directory would
// be writable by any local user, and the skip-if-exists logic below
// trusts pre-existing keyrings, so it must never be shared.
let euid = current_euid();
let keyring_dir = temp_dir.join(format!("pkh-keyrings-{euid}"));
// Create keyring directory if it doesn't exist
if !ctx.exists(&keyring_dir)? {
ctx.command("mkdir").arg("-p").arg(&keyring_dir).status()?;
if ctx.exists(&keyring_dir)? {
if let context::ContextConfig::Local = ctx.config {
// Cached keyrings are trusted as-is whenever they already exist,
// so refuse to reuse a directory that is not owned by the current
// user or is writable by group/others (it could have been planted
// by another local user).
let metadata = std::fs::symlink_metadata(&keyring_dir)?;
validate_keyring_dir(metadata.uid(), metadata.mode(), euid).map_err(|reason| {
format!(
"Refusing to use keyring cache directory {}: {reason}; \
remove the directory and re-run pkh",
keyring_dir.display()
)
})?;
} else {
// Remote contexts (e.g. ssh) have no stat/metadata access through
// the context API, so the ownership guard cannot be performed;
// keep the previous best-effort behavior of tightening the
// directory permissions instead (0700 instead of the former
// world-writable a+rwx).
ctx.command("chmod").arg("700").arg(&keyring_dir).status()?;
}
} else {
// Create the directory private to the invoking user (0700). This is
// sufficient for mmdebstrap in unshare mode: it runs with the same
// real uid (the user namespace only maps that uid to root, file
// access still happens as the real uid), so no world-accessible
// permissions are needed.
ctx.command("mkdir")
.arg("-p")
.arg("-m")
.arg("700")
.arg(&keyring_dir)
.status()?;
}
// Make keyring directory world-accessible so mmdebstrap in unshare mode can access it
ctx.command("chmod")
.arg("a+rwx")
.arg(&keyring_dir)
.status()?;
for keyring_url in keyring_urls {
// Extract the original filename from the keyring URL
let filename = keyring_url
@@ -116,9 +146,6 @@ pub async fn download_cache_keyrings(
let _ = ctx.command("rm").arg("-f").arg(&download_path).status();
}
// Make the keyring file world-readable so mmdebstrap in unshare mode can access it
ctx.command("chmod").arg("a+r").arg(&binary_path).status()?;
log::info!(
"Successfully downloaded keyring for {} to {}",
series,
@@ -129,8 +156,6 @@ pub async fn download_cache_keyrings(
"Keyring already exists at {}, skipping download",
binary_path.display()
);
// Ensure existing keyring is world-readable
ctx.command("chmod").arg("a+r").arg(&binary_path).status()?;
}
}
@@ -143,6 +168,36 @@ pub async fn download_cache_keyrings(
Ok(keyring_dir)
}
/// Effective uid of the current process
fn current_euid() -> u32 {
unsafe { libc::geteuid() }
}
/// Check that an existing keyring cache directory is safe to reuse
///
/// Cached keyrings are trusted whenever the files already exist (see the
/// skip-if-exists logic in [`download_cache_keyrings`]), so the directory
/// must be owned by the current user and must not be writable by group or
/// others, otherwise another local user could plant a malicious keyring.
///
/// Takes the directory's owner uid and permission mode (e.g. from
/// `std::fs::symlink_metadata`) so it can be unit tested without touching
/// the filesystem.
fn validate_keyring_dir(dir_uid: u32, mode: u32, euid: u32) -> Result<(), String> {
if dir_uid != euid {
return Err(format!(
"owned by uid {dir_uid}, not by the current user (uid {euid})"
));
}
if mode & 0o022 != 0 {
return Err(format!(
"writable by group or others (permissions {:04o})",
mode & 0o7777
));
}
Ok(())
}
/// Download and import a PPA key using Launchpad API
///
/// # Arguments
@@ -246,3 +301,34 @@ pub async fn download_trust_ppa_key(
);
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_validate_keyring_dir_accepts_private_dir_owned_by_current_user() {
assert!(validate_keyring_dir(1000, 0o700, 1000).is_ok());
assert!(validate_keyring_dir(1000, 0o750, 1000).is_ok());
assert!(validate_keyring_dir(1000, 0o1744, 1000).is_ok());
assert!(validate_keyring_dir(0, 0o700, 0).is_ok());
}
#[test]
fn test_validate_keyring_dir_rejects_foreign_owner() {
let err = validate_keyring_dir(1000, 0o700, 1001).unwrap_err();
assert!(err.contains("owned by uid 1000"));
let err = validate_keyring_dir(1001, 0o700, 1000).unwrap_err();
assert!(err.contains("owned by uid 1001"));
}
#[test]
fn test_validate_keyring_dir_rejects_group_or_other_writable() {
assert!(validate_keyring_dir(1000, 0o770, 1000).is_err());
assert!(validate_keyring_dir(1000, 0o706, 1000).is_err());
assert!(validate_keyring_dir(1000, 0o707, 1000).is_err());
assert!(validate_keyring_dir(1000, 0o777, 1000).is_err());
// Sticky bit does not neutralize the group/other write bits.
assert!(validate_keyring_dir(1000, 0o1777, 1000).is_err());
}
}
+2
View File
@@ -1,2 +1,4 @@
pub mod keyring;
/// Release-file signature and checksum verification for repositories
pub mod release;
pub mod sources;
+1303
View File
File diff suppressed because it is too large Load Diff
+547 -118
View File
@@ -1,97 +1,188 @@
//! APT sources.list management
//! Provides a simple structure for managing APT repository sources
use crate::context;
//!
//! Entries carry enough information (kind, signed-by, trusted, enabled) to
//! be written back without loss, and remember the file they were loaded
//! from ([`SourceEntry::origin`]) so that saving writes each entry back to
//! its own file, in that file's own format.
use crate::context::{self, Context};
use crate::debian::control::{Paragraph, parse_paragraphs, write_paragraph};
use std::error::Error;
use std::path::Path;
use std::path::{Path, PathBuf};
use std::sync::Arc;
/// Sources file owned by pkh, holding entries added by pkh (e.g. PPAs).
///
/// New entries never end up in distro-managed files.
const PKH_ADDED_PATH: &str = "/etc/apt/sources.list.d/pkh-added.list";
/// Suffix appended to an origin file path to build its backup path
const BACKUP_SUFFIX: &str = ".pkh-backup";
/// Kind of packages provided by a source entry
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum SourceKind {
/// Binary packages ('deb')
Deb,
/// Source packages ('deb-src')
DebSrc,
}
impl SourceKind {
/// Token used in legacy lines and deb822 'Types' fields
pub fn as_str(self) -> &'static str {
match self {
SourceKind::Deb => "deb",
SourceKind::DebSrc => "deb-src",
}
}
/// Parse a type token ('deb' or 'deb-src')
fn parse(token: &str) -> Option<Self> {
match token {
"deb" => Some(SourceKind::Deb),
"deb-src" => Some(SourceKind::DebSrc),
_ => None,
}
}
}
/// On-disk format of a sources file
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum SourceFormat {
/// Legacy one-line-per-entry format (sources.list, *.list)
Legacy,
/// deb822 format (*.sources)
Deb822,
}
/// File a source entry was loaded from
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct SourceOrigin {
/// Path of the origin file, inside the context
pub path: PathBuf,
/// Format of the origin file
pub format: SourceFormat,
}
/// Represents a single source entry in sources.list
#[derive(Debug, Clone)]
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct SourceEntry {
/// Is the source enabled?
pub enabled: bool,
/// Kind of packages provided by the source (binary or source)
pub kind: SourceKind,
/// Source components (universe, main, contrib)
pub components: Vec<String>,
/// Source architectures (amd64, riscv64, arm64)
pub architectures: Vec<String>,
/// Keyring the repository is signed with ('signed-by' option)
pub signed_by: Option<String>,
/// Explicit trust flag ('trusted' option), when set
pub trusted: Option<bool>,
/// Source URI
pub uri: String,
/// Source suites (series-pocket)
pub suite: Vec<String>,
/// File and format the entry was loaded from
///
/// Entries without an origin are new (e.g. repositories added by pkh);
/// they are saved to the pkh-owned added-sources file.
pub origin: Option<SourceOrigin>,
}
impl SourceEntry {
/// Parse a string describing a source entry in deb822 format
pub fn from_deb822(data: &str) -> Option<Self> {
let mut current_entry = SourceEntry {
enabled: true,
components: Vec::new(),
architectures: Vec::new(),
uri: String::new(),
suite: Vec::new(),
};
for line in data.lines() {
let line = line.trim();
if line.starts_with('#') {
continue;
}
// Empty line: end of an entry, or beginning
if line.is_empty() {
if !current_entry.uri.is_empty() {
return Some(current_entry);
} else {
continue;
}
}
if let Some((key, value)) = line.split_once(':') {
let key = key.trim();
let value = value.trim();
match key {
"Types" => {
// We only care about deb types
}
"URIs" => current_entry.uri = value.to_string(),
"Suites" => {
current_entry.suite =
value.split_whitespace().map(|s| s.to_string()).collect();
}
"Components" => {
current_entry.components =
value.split_whitespace().map(|s| s.to_string()).collect();
}
"Architectures" => {
current_entry.architectures =
value.split_whitespace().map(|s| s.to_string()).collect();
}
_ => {}
}
}
/// Build entries from a single deb822 stanza
///
/// A stanza declaring several types ('Types: deb deb-src') yields one
/// entry per type.
fn from_deb822_stanza(p: &Paragraph) -> Vec<Self> {
// apt defaults 'Types' to 'deb' when the field is absent
let mut kinds: Vec<SourceKind> = p
.get("Types")
.unwrap_or("deb")
.split_whitespace()
.filter_map(SourceKind::parse)
.collect();
if kinds.is_empty() {
kinds.push(SourceKind::Deb);
}
// End of entry, or empty file?
if !current_entry.uri.is_empty() {
Some(current_entry)
} else {
None
let enabled = p
.get("Enabled")
.map(|v| {
let v = v.trim();
!v.eq_ignore_ascii_case("no") && !v.eq_ignore_ascii_case("false")
})
.unwrap_or(true);
let signed_by = p
.get("Signed-By")
.map(str::trim)
.filter(|v| !v.is_empty())
.map(str::to_string);
let trusted = p
.get("Trusted")
.map(|v| v.trim().eq_ignore_ascii_case("yes"));
let uri = p.get("URIs").unwrap_or("").trim().to_string();
if uri.is_empty() {
return Vec::new();
}
let suite: Vec<String> = p
.get("Suites")
.unwrap_or("")
.split_whitespace()
.map(|s| s.to_string())
.collect();
let components: Vec<String> = p
.get("Components")
.unwrap_or("")
.split_whitespace()
.map(|s| s.to_string())
.collect();
let architectures: Vec<String> = p
.get("Architectures")
.unwrap_or("")
.split_whitespace()
.map(|s| s.to_string())
.collect();
kinds
.into_iter()
.map(|kind| SourceEntry {
enabled,
kind,
components: components.clone(),
architectures: architectures.clone(),
signed_by: signed_by.clone(),
trusted,
uri: uri.clone(),
suite: suite.clone(),
origin: None,
})
.collect()
}
/// Parse a line describing a legacy source entry
pub fn from_legacy(data: &str) -> Option<Self> {
let line = data.lines().next()?.trim();
let raw = data.lines().next()?.trim();
if line.is_empty() || line.starts_with("#") {
if raw.is_empty() {
return None;
}
// Parse legacy deb line format: deb [arch=... / signed_by=] uri suite [components...]
// Entries commented out with '#' are disabled, not deleted
let (enabled, line) = match raw.strip_prefix('#') {
Some(rest) => (false, rest.trim_start()),
None => (true, raw),
};
// Parse legacy deb line format:
// deb [arch=... signed-by=... trusted=...] uri suite [components...]
// Extract bracket parameters first
let mut architectures = Vec::new();
let mut signed_by = None;
let mut trusted = None;
let mut line_without_brackets = line.to_string();
// Find and process bracket parameters
@@ -102,14 +193,13 @@ impl SourceEntry {
// Parse parameters inside brackets
for param in bracket_content.split_whitespace() {
if param.starts_with("arch=") {
let arch_values = param.split('=').nth(1).unwrap_or("");
architectures = arch_values
.split(',')
.map(|s| s.trim().to_string())
.collect();
if let Some(values) = param.strip_prefix("arch=") {
architectures = values.split(',').map(|s| s.trim().to_string()).collect();
} else if let Some(keyring) = param.strip_prefix("signed-by=") {
signed_by = Some(keyring.trim_matches('"').to_string());
} else if let Some(flag) = param.strip_prefix("trusted=") {
trusted = Some(flag.eq_ignore_ascii_case("yes") || flag == "1");
}
// signed-by parameter is parsed but not stored
}
// Remove the bracket section from the line
@@ -120,37 +210,61 @@ impl SourceEntry {
let line_without_brackets = line_without_brackets.trim();
let parts: Vec<&str> = line_without_brackets.split_whitespace().collect();
// We need at least: deb, uri, suite
if parts.len() < 3 || parts[0] != "deb" {
// We need at least: type, uri, suite
if parts.len() < 3 {
return None;
}
let kind = SourceKind::parse(parts[0])?;
let uri = parts[1].to_string();
let suite = vec![parts[2].to_string()];
let components: Vec<String> = parts[3..].iter().map(|&s| s.to_string()).collect();
Some(SourceEntry {
enabled: true,
enabled,
kind,
components,
architectures,
signed_by,
trusted,
uri,
suite,
origin: None,
})
}
/// Convert this source entry to legacy format
///
/// Entries holding several suites are rendered as one line per suite.
/// Disabled entries are commented out.
pub fn to_legacy(&self) -> String {
let mut result = String::new();
// Legacy entries contain one suite per line
for suite in &self.suite {
// Start with "deb" type
result.push_str("deb");
if !self.enabled {
result.push_str("# ");
}
result.push_str(self.kind.as_str());
// Add architectures if present
// Bracket options: architectures, signing keyring and trust
let mut options = Vec::new();
if !self.architectures.is_empty() {
result.push_str(" [arch=");
result.push_str(&self.architectures.join(","));
options.push(format!("arch={}", self.architectures.join(",")));
}
if let Some(keyring) = &self.signed_by {
if keyring.contains(char::is_whitespace) {
options.push(format!("signed-by=\"{keyring}\""));
} else {
options.push(format!("signed-by={keyring}"));
}
}
if let Some(trusted) = self.trusted {
options.push(format!("trusted={}", if trusted { "yes" } else { "no" }));
}
if !options.is_empty() {
result.push_str(" [");
result.push_str(&options.join(" "));
result.push(']');
}
@@ -171,88 +285,199 @@ impl SourceEntry {
result
}
/// Convert this source entry to a deb822 stanza (with a trailing newline)
pub fn to_deb822(&self) -> String {
let mut stanza = Paragraph::new();
stanza.set("Types", self.kind.as_str());
stanza.set("URIs", &self.uri);
stanza.set("Suites", &self.suite.join(" "));
stanza.set("Components", &self.components.join(" "));
if let Some(keyring) = &self.signed_by {
stanza.set("Signed-By", keyring);
}
if !self.architectures.is_empty() {
stanza.set("Architectures", &self.architectures.join(" "));
}
if let Some(trusted) = self.trusted {
stanza.set("Trusted", if trusted { "yes" } else { "no" });
}
if !self.enabled {
stanza.set("Enabled", "no");
}
write_paragraph(&stanza)
}
}
/// Parse a 'source list' string in deb822 format into a SourceEntry vector
///
/// A stanza declaring several types ('Types: deb deb-src') yields one entry
/// per type.
pub fn parse_deb822(data: &str) -> Vec<SourceEntry> {
data.split("\n\n")
.flat_map(SourceEntry::from_deb822)
parse_paragraphs(data)
.iter()
.flat_map(SourceEntry::from_deb822_stanza)
.collect()
}
/// Parse a 'source list' string in legacy format into a SourceEntry vector
pub fn parse_legacy(data: &str) -> Vec<SourceEntry> {
data.split("\n")
data.split('\n')
.flat_map(SourceEntry::from_legacy)
.collect()
}
/// Load sources from context (or current context by default)
pub fn load(ctx: Option<Arc<crate::context::Context>>) -> Result<Vec<SourceEntry>, Box<dyn Error>> {
///
/// Reads the deb822 distro sources (ubuntu.sources or debian.sources), the
/// legacy '/etc/apt/sources.list' and the pkh-owned added-sources file when
/// they exist. Every entry remembers the file and format it came from.
pub fn load(ctx: Option<Arc<Context>>) -> Result<Vec<SourceEntry>, Box<dyn Error>> {
let mut sources = Vec::new();
let ctx = ctx.unwrap_or_else(context::current);
// Try DEB822 format first (Ubuntu 24.04+ and Debian Trixie+)
if let Ok(entries) = load_deb822(&ctx, "/etc/apt/sources.list.d/ubuntu.sources") {
sources.extend(entries);
} else if let Ok(entries) = load_deb822(&ctx, "/etc/apt/sources.list.d/debian.sources") {
sources.extend(entries);
}
load_file(
&ctx,
"/etc/apt/sources.list.d/ubuntu.sources",
SourceFormat::Deb822,
&mut sources,
)?;
load_file(
&ctx,
"/etc/apt/sources.list.d/debian.sources",
SourceFormat::Deb822,
&mut sources,
)?;
// Fall back to legacy format
if let Ok(entries) = load_legacy(&ctx, "/etc/apt/sources.list") {
sources.extend(entries);
}
load_file(
&ctx,
"/etc/apt/sources.list",
SourceFormat::Legacy,
&mut sources,
)?;
// Entries added by a previous pkh run
load_file(&ctx, PKH_ADDED_PATH, SourceFormat::Legacy, &mut sources)?;
Ok(sources)
}
/// Save sources back to context
pub fn save_legacy(
ctx: Option<Arc<crate::context::Context>>,
sources: Vec<SourceEntry>,
path: &str,
) -> Result<(), Box<dyn Error>> {
let ctx = if let Some(c) = ctx {
c
} else {
context::current()
};
/// Save sources back to the context
///
/// Each entry is written back to the file it was loaded from
/// ([`SourceEntry::origin`]), in that file's format. Entries without an
/// origin (e.g. repositories added by pkh) go to the pkh-owned
/// added-sources file in legacy format, never to distro-managed files.
///
/// Files whose rendered content is byte-identical to their current content
/// are left untouched; otherwise a '<path>.pkh-backup' copy is created once
/// before the first overwrite.
pub fn save(ctx: Option<Arc<Context>>, sources: Vec<SourceEntry>) -> Result<(), Box<dyn Error>> {
let ctx = ctx.unwrap_or_else(context::current);
for (path, _format, content) in plan_writes(&sources) {
let original = if ctx.exists(&path)? {
Some(ctx.read_file(&path)?)
} else {
None
};
if original.as_deref() == Some(content.as_str()) {
// Nothing changed: leave the file untouched
continue;
}
// One-time backup before overwriting an existing file
if original.is_some() {
let backup = backup_path(&path);
if !ctx.exists(&backup)? {
ctx.copy_path(&path, &backup)?;
}
}
ctx.write_file(&path, &content)?;
}
let content = sources
.into_iter()
.map(|s| s.to_legacy())
.collect::<Vec<_>>()
.join("\n");
ctx.write_file(Path::new(path), &content)?;
Ok(())
}
/// Load sources from DEB822 format
fn load_deb822(ctx: &context::Context, path: &str) -> Result<Vec<SourceEntry>, Box<dyn Error>> {
let path = Path::new(path);
if path.exists() {
let content = ctx.read_file(path)?;
return Ok(parse_deb822(&content));
/// Load entries from one sources file, if it exists, tagging them with
/// their origin
fn load_file(
ctx: &Context,
path: &str,
format: SourceFormat,
out: &mut Vec<SourceEntry>,
) -> Result<(), Box<dyn Error>> {
let path = PathBuf::from(path);
if !ctx.exists(&path)? {
return Ok(());
}
Ok(Vec::new())
let content = ctx.read_file(&path)?;
let mut entries = match format {
SourceFormat::Deb822 => parse_deb822(&content),
SourceFormat::Legacy => parse_legacy(&content),
};
for entry in &mut entries {
entry.origin = Some(SourceOrigin {
path: path.clone(),
format,
});
}
out.append(&mut entries);
Ok(())
}
/// Load sources from legacy format
fn load_legacy(ctx: &context::Context, path: &str) -> Result<Vec<SourceEntry>, Box<dyn Error>> {
let path = Path::new(path);
if path.exists() {
let content = ctx.read_file(path)?;
return Ok(content.lines().flat_map(SourceEntry::from_legacy).collect());
/// Compute the writes needed to persist entries: one
/// (path, format, content) triple per destination file, entries kept in order
///
/// Entries without an origin are routed to the pkh-owned added-sources file.
fn plan_writes(sources: &[SourceEntry]) -> Vec<(PathBuf, SourceFormat, String)> {
let mut plan: Vec<(PathBuf, SourceFormat, Vec<&SourceEntry>)> = Vec::new();
for entry in sources {
let (path, format) = match &entry.origin {
Some(origin) => (origin.path.clone(), origin.format),
None => (PathBuf::from(PKH_ADDED_PATH), SourceFormat::Legacy),
};
if let Some((_, _, group)) = plan.iter_mut().find(|(p, _, _)| *p == path) {
group.push(entry);
} else {
plan.push((path, format, vec![entry]));
}
}
Ok(Vec::new())
plan.into_iter()
.map(|(path, format, entries)| {
let content = match format {
// Legacy entries end with '\n': plain concatenation, no
// blank lines in between
SourceFormat::Legacy => entries.iter().map(|e| e.to_legacy()).collect(),
// deb822 stanzas end with '\n': a '\n' join gives one blank
// line between stanzas
SourceFormat::Deb822 => entries
.iter()
.map(|e| e.to_deb822())
.collect::<Vec<_>>()
.join("\n"),
};
(path, format, content)
})
.collect()
}
/// Backup path for a sources file ('<path>.pkh-backup')
fn backup_path(path: &Path) -> PathBuf {
let mut with_suffix = path.as_os_str().to_os_string();
with_suffix.push(BACKUP_SUFFIX);
PathBuf::from(with_suffix)
}
#[cfg(test)]
mod tests {
use super::*;
use crate::context::ContextConfig;
#[tokio::test]
async fn test_parse_deb822() {
@@ -333,4 +558,208 @@ mod tests {
assert_eq!(sources[2].suite, vec!["resolute-security"]);
assert_eq!(sources[2].components, vec!["main"]);
}
/// Legacy round-trip: kind, signed-by, trusted and arch are preserved,
/// and rendering introduces no blank lines
#[test]
fn legacy_roundtrip_preserves_options() {
let input = "\
deb [arch=amd64 signed-by=/k.gpg] http://x noble main\n\
deb-src http://x noble main\n\
deb [trusted=yes] http://x noble universe\n\
# deb [arch=i386] http://x noble main\n";
let sources = parse_legacy(input);
assert_eq!(sources.len(), 4);
assert_eq!(sources[0].kind, SourceKind::Deb);
assert_eq!(sources[0].signed_by.as_deref(), Some("/k.gpg"));
assert_eq!(sources[0].architectures, vec!["amd64"]);
assert_eq!(sources[1].kind, SourceKind::DebSrc);
assert_eq!(sources[2].trusted, Some(true));
assert!(!sources[3].enabled);
// Render as a legacy file through the save planning path
let origin = SourceOrigin {
path: PathBuf::from("/etc/apt/sources.list"),
format: SourceFormat::Legacy,
};
let mut sources = sources;
for entry in &mut sources {
entry.origin = Some(origin.clone());
}
let plan = plan_writes(&sources);
assert_eq!(plan.len(), 1);
let rendered = &plan[0].2;
// Rendering is faithful: byte-identical and without blank lines
assert_eq!(rendered, input);
assert!(!rendered.contains("\n\n"));
let reparsed = parse_legacy(rendered);
assert_eq!(reparsed, parse_legacy(input));
}
/// deb822 round-trip: multiple types are split into one entry per type,
/// Signed-By and Enabled are preserved
#[test]
fn deb822_roundtrip_preserves_types_and_options() {
let input = "\
Types: deb deb-src\n\
URIs: http://archive.ubuntu.com/ubuntu\n\
Suites: noble\n\
Components: main\n\
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg\n\
Enabled: false\n\
\n\
Types: deb\n\
URIs: http://archive.ubuntu.com/ubuntu\n\
Suites: noble-updates\n\
Components: main universe\n";
let sources = parse_deb822(input);
// The first stanza declares two types: one entry per type
assert_eq!(sources.len(), 3);
assert_eq!(sources[0].kind, SourceKind::Deb);
assert_eq!(sources[1].kind, SourceKind::DebSrc);
assert_eq!(sources[2].kind, SourceKind::Deb);
assert!(!sources[0].enabled);
assert!(!sources[1].enabled);
assert!(sources[2].enabled);
assert_eq!(
sources[0].signed_by.as_deref(),
Some("/usr/share/keyrings/ubuntu-archive-keyring.gpg")
);
assert_eq!(sources[1].signed_by, sources[0].signed_by);
assert_eq!(sources[2].signed_by, None);
// Render as a deb822 file through the save planning path
let mut sources = sources;
for entry in &mut sources {
entry.origin = Some(SourceOrigin {
path: PathBuf::from("/etc/apt/sources.list.d/ubuntu.sources"),
format: SourceFormat::Deb822,
});
}
let plan = plan_writes(&sources);
assert_eq!(plan.len(), 1);
let rendered = &plan[0].2;
let reparsed = parse_deb822(rendered);
// Parse/render round-trip preserves the model (origin excepted)
assert_eq!(reparsed, parse_deb822(input));
assert_eq!(reparsed[0].kind, SourceKind::Deb);
assert_eq!(reparsed[1].kind, SourceKind::DebSrc);
assert_eq!(reparsed[2].kind, SourceKind::Deb);
assert!(!reparsed[0].enabled);
assert!(!reparsed[1].enabled);
assert!(reparsed[2].enabled);
assert_eq!(
reparsed[0].signed_by.as_deref(),
Some("/usr/share/keyrings/ubuntu-archive-keyring.gpg")
);
// 'Enabled' is only emitted for disabled entries
assert_eq!(rendered.matches("Enabled: no").count(), 2);
}
/// Entries are routed to their origin file in its own format, and new
/// entries (no origin) go to the pkh-owned added-sources file
#[test]
fn plan_writes_routes_by_origin() {
let origin_a = SourceOrigin {
path: PathBuf::from("/etc/apt/sources.list.d/ubuntu.sources"),
format: SourceFormat::Deb822,
};
let mut sources = parse_deb822(
"Types: deb\nURIs: http://archive.ubuntu.com/ubuntu\nSuites: noble\nComponents: main\n",
);
sources[0].origin = Some(origin_a.clone());
// Modify the origin-A entry and add a brand new (PPA) entry
sources[0].components.push("universe".to_string());
sources.push(SourceEntry {
enabled: true,
kind: SourceKind::Deb,
components: vec!["main".to_string()],
architectures: vec![],
signed_by: None,
trusted: None,
uri: "http://ppa.example.org/user/ppa/ubuntu".to_string(),
suite: vec!["noble".to_string()],
origin: None,
});
let plan = plan_writes(&sources);
assert_eq!(plan.len(), 2);
assert_eq!(plan[0].0, origin_a.path);
assert_eq!(plan[0].1, SourceFormat::Deb822);
assert!(plan[0].2.starts_with("Types: deb\n"));
assert!(plan[0].2.contains("main universe"));
assert_eq!(
plan[1].0,
PathBuf::from("/etc/apt/sources.list.d/pkh-added.list")
);
assert_eq!(plan[1].1, SourceFormat::Legacy);
assert!(plan[1].2.starts_with("deb http://ppa.example.org/"));
}
/// save() leaves unchanged files untouched, and backs up existing files
/// once before overwriting them; the backup also works for the
/// pkh-owned added-sources file
#[test]
fn save_skips_unchanged_and_backs_up() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("ubuntu.sources");
std::fs::write(
&path,
"Types: deb\nURIs: http://a\nSuites: noble\nComponents: main\n",
)
.unwrap();
let ctx = Arc::new(Context::new(ContextConfig::Local).unwrap());
let origin = SourceOrigin {
path: path.clone(),
format: SourceFormat::Deb822,
};
let mut entries =
parse_deb822("Types: deb\nURIs: http://a\nSuites: noble\nComponents: main\n");
entries[0].origin = Some(origin.clone());
// Unchanged content: no write, no backup
save(Some(ctx.clone()), entries.clone()).unwrap();
assert_eq!(
std::fs::read_to_string(&path).unwrap(),
"Types: deb\nURIs: http://a\nSuites: noble\nComponents: main\n"
);
assert!(!backup_path(&path).exists());
// Modified content: backup created, file rewritten in its own format
entries[0].components.push("universe".to_string());
save(Some(ctx.clone()), entries).unwrap();
assert_eq!(
std::fs::read_to_string(backup_path(&path)).unwrap(),
"Types: deb\nURIs: http://a\nSuites: noble\nComponents: main\n"
);
assert_eq!(
std::fs::read_to_string(&path).unwrap(),
"Types: deb\nURIs: http://a\nSuites: noble\nComponents: main universe\n"
);
// A second save does not overwrite the first backup
let mut entries =
parse_deb822("Types: deb\nURIs: http://a\nSuites: noble\nComponents: main universe\n");
entries[0].origin = Some(origin);
entries[0].components.push("restricted".to_string());
save(Some(ctx.clone()), entries).unwrap();
assert_eq!(
std::fs::read_to_string(backup_path(&path)).unwrap(),
"Types: deb\nURIs: http://a\nSuites: noble\nComponents: main\n"
);
assert_eq!(
std::fs::read_to_string(&path).unwrap(),
"Types: deb\nURIs: http://a\nSuites: noble\nComponents: main universe restricted\n"
);
}
}
+177 -69
View File
@@ -18,6 +18,8 @@ use crate::debian::{
ChecksumEntry, ControlInfo, FileChecksums, FilesList, parse_changelog_entry_from_str,
};
use super::parse_checksum_field;
/// Digests of one artifact.
#[derive(Debug, Clone, Default)]
struct ArtifactHashes {
@@ -34,10 +36,13 @@ pub struct BinaryMetadataOptions {
pub profiles: Vec<String>,
/// Vendor name (`Build-Origin`).
pub vendor: String,
/// Parallel job count advertised in `DEB_BUILD_OPTIONS`.
pub parallel: usize,
/// Reproducible-builds epoch exported to the build.
pub source_date_epoch: i64,
/// Environment variables pkh exported to the build steps (e.g. `LANG`,
/// `DEB_BUILD_OPTIONS` with the real parallel count and `nocheck`,
/// `SOURCE_DATE_EPOCH`, cross `DEB_*` variables). Recorded — filtered to
/// dpkg's allow-list — in the `.buildinfo` `Environment` field, taking
/// precedence over whatever the host process inherited, so the metadata
/// describes the environment the build actually ran in.
pub exported_env: BTreeMap<String, String>,
/// Build architecture (the machine inside the build context).
pub build_arch: String,
/// Host architecture (the packages' target); equals the build
@@ -208,8 +213,10 @@ pub fn generate_binary_metadata(
// ------------------------------------------------------------------
// .buildinfo generation, then registration in debian/files
// ------------------------------------------------------------------
let pipeline_env = pipeline_environment(opts);
let environment = crate::build::env::buildinfo_environment(&pipeline_env);
// Record exactly the environment that was exported to the build steps,
// overriding any host-inherited value (dpkg-style allowed-variable
// filtering, export precedence).
let environment = crate::build::env::buildinfo_environment(&opts.exported_env);
// dpkg-genbuildinfo sorts the accumulated architecture values, while
// dpkg-genchanges keeps encounter order.
@@ -292,24 +299,6 @@ pub fn generate_binary_metadata(
Ok((buildinfo_path, changes_path))
}
/// Environment exported to the build steps; recorded (filtered) in the
/// `.buildinfo` `Environment` field.
fn pipeline_environment(opts: &BinaryMetadataOptions) -> BTreeMap<String, String> {
let mut env = BTreeMap::new();
env.insert(
"SOURCE_DATE_EPOCH".to_string(),
opts.source_date_epoch.to_string(),
);
env.insert(
"DEB_BUILD_OPTIONS".to_string(),
format!("parallel={}", opts.parallel),
);
if !opts.profiles.is_empty() {
env.insert("DEB_BUILD_PROFILES".to_string(), opts.profiles.join(","));
}
env
}
/// Compute md5/sha1/sha256 digests and sizes for the named files inside the
/// context directory `dir`, using coreutils.
fn hashes_in_context(
@@ -388,46 +377,36 @@ fn include_dsc_artifacts(
checksums: &mut FileChecksums,
) -> Result<(), Box<dyn Error>> {
let dsc_content = ctx.read_file(&upload_dir.join(dsc_name))?;
let para = crate::debian::control::parse_paragraphs(&dsc_content)
.into_iter()
.next()
.ok_or_else(|| format!("'{dsc_name}' is empty"))?;
let para = crate::debian::control::parse_paragraphs(
crate::debian::control::strip_clearsigned_armour(&dsc_content),
)
.into_iter()
.next()
.ok_or_else(|| format!("'{dsc_name}' is empty"))?;
// Names and partial checksums are filled from the very same validated
// lines, so a listed name can never miss its checksum entry.
// Distribution order follows the Checksums fields (Checksums-Sha1 then
// Checksums-Sha256), like dpkg-genchanges; the `Files` field only
// supplements the md5 digests.
let mut names: Vec<String> = Vec::new();
let mut partials: BTreeMap<String, PartialDscChecksums> = BTreeMap::new();
for field in ["Checksums-Sha1", "Checksums-Sha256"] {
if let Some(value) = para.get(field) {
for line in value.lines() {
let tokens: Vec<&str> = line.split_whitespace().collect();
if tokens.len() != 3 {
continue;
}
let slot = partials.entry(tokens[2].to_string()).or_default();
if field == "Checksums-Sha1" {
slot.sha1 = Some(tokens[0].to_string());
} else {
slot.sha256 = Some(tokens[0].to_string());
}
slot.size = tokens[1].parse().ok().or(slot.size);
let mut partials: BTreeMap<String, super::PartialChecksum> = BTreeMap::new();
for field in ["Checksums-Sha1", "Checksums-Sha256", "Files"] {
let Some(value) = para.get(field) else {
continue;
};
for cl in parse_checksum_field(field, value)
.map_err(|e| format!("cannot parse '{dsc_name}': {e}"))?
{
let slot = partials.entry(cl.name.clone()).or_default();
match field {
"Checksums-Sha1" => slot.sha1 = Some(cl.digest),
"Checksums-Sha256" => slot.sha256 = Some(cl.digest),
_ => slot.md5 = Some(cl.digest),
}
}
}
if let Some(files_value) = para.get("Files") {
for line in files_value.lines() {
let tokens: Vec<&str> = line.split_whitespace().collect();
if tokens.len() >= 3 {
let slot = partials.entry(tokens[2].to_string()).or_default();
slot.md5 = Some(tokens[0].to_string());
slot.size = tokens[1].parse().ok().or(slot.size);
}
}
}
for field in ["Checksums-Sha1", "Checksums-Sha256"] {
if let Some(value) = para.get(field) {
for line in value.lines() {
if let Some(name) = line.split_whitespace().nth(2) {
names.push(name.to_string());
}
slot.size = Some(cl.size);
if field != "Files" && !names.contains(&cl.name) {
names.push(cl.name);
}
}
}
@@ -451,7 +430,9 @@ fn include_dsc_artifacts(
if name == dsc_name {
continue;
}
let p = &partials[name];
let p = partials
.get(name)
.ok_or_else(|| format!("file '{name}' listed in '{dsc_name}' has no checksum entry"))?;
checksums.insert_entry(
name,
ChecksumEntry {
@@ -465,11 +446,138 @@ fn include_dsc_artifacts(
Ok(())
}
/// Partially-known checksums taken from a `.dsc` checksum field.
#[derive(Debug, Default)]
struct PartialDscChecksums {
size: Option<u64>,
md5: Option<String>,
sha1: Option<String>,
sha256: Option<String>,
#[cfg(test)]
mod tests {
use super::*;
/// The recorded `.buildinfo` `Environment` must carry the environment
/// actually exported to the build steps (`parallel=N nocheck`, `LANG=C`,
/// ...), taking precedence over any host-inherited value, instead of
/// values recomputed from host state at generation time.
#[test]
fn environment_records_exported_env_not_host_defaults() {
let mut exported_env = BTreeMap::new();
exported_env.insert("LANG".to_string(), "C".to_string());
exported_env.insert(
"DEB_BUILD_OPTIONS".to_string(),
"parallel=7 nocheck".to_string(),
);
let opts = BinaryMetadataOptions {
profiles: Vec::new(),
vendor: "debian".to_string(),
exported_env,
build_arch: "amd64".to_string(),
host_arch: "amd64".to_string(),
};
let environment = crate::build::env::buildinfo_environment(&opts.exported_env);
assert!(
environment.contains("DEB_BUILD_OPTIONS=\"parallel=7 nocheck\""),
"recorded Environment must carry the exported DEB_BUILD_OPTIONS: {environment}"
);
assert!(
environment.contains("LANG=\"C\""),
"recorded Environment must carry the exported LANG: {environment}"
);
// Not in dpkg's allowed-variable list: never recorded.
assert!(!environment.contains("DEBIAN_FRONTEND"), "{environment}");
}
/// A minimal previous-version `.dsc` with a 3-column Checksums-Sha1
/// field, a 4-column Checksums-Sha256 line and a 3-column `Files`.
/// Regression: the old code filled `names` from any line with a third
/// column but `partials` only from exactly-3-column lines, so the
/// "bogus" name landed in `names` alone and `&partials["bogus"]`
/// panicked. It must produce a build error instead.
#[test]
fn dsc_four_column_checksum_line_errors_instead_of_panicking() {
let dsc_name = "hello_1.0-1.dsc";
let dsc = "\
Format: 3.0 (native)
Source: hello
Binary: hello
Architecture: any
Version: 1.0-1
Maintainer: A B <a@b.c>
Checksums-Sha1:
aaa111 12 hello_1.0.orig.tar.xz
Checksums-Sha256:
bbb222 12 bogus hello_1.0-1.debian.tar.xz
Files:
ddd333 12 hello_1.0.orig.tar.xz
";
let base = tempfile::tempdir().expect("tempdir");
std::fs::write(base.path().join(dsc_name), dsc).expect("write dsc");
let ctx = Arc::new(
crate::context::Context::new(crate::context::ContextConfig::Local).expect("context"),
);
let mut checksums = FileChecksums::new();
let err = include_dsc_artifacts(&ctx, base.path(), dsc_name, &mut checksums)
.expect_err("malformed Checksums-Sha256 line must fail the build");
let err = err.to_string();
assert!(err.contains("Checksums-Sha256"), "{err}");
assert!(err.contains("hello_1.0-1.debian.tar.xz"), "{err}");
}
/// Happy path: tarball entries are assembled from the Checksums fields
/// (sha1/sha256) and merged with the legacy 5-column `Files` md5, in
/// Checksums-Sha1 order, with the `.dsc` itself hashed fresh first.
#[test]
fn include_dsc_artifacts_merges_legacy_files_layout() {
let dsc_name = "hello_1.0-1.dsc";
let tarball = "hello_1.0.orig.tar.xz";
let dsc = "\
Format: 3.0 (quilt)
Source: hello
Binary: hello
Architecture: any
Version: 1.0-1
Maintainer: A B <a@b.c>
Checksums-Sha1:
aaa111 12 hello_1.0.orig.tar.xz
Checksums-Sha256:
bbb222 12 hello_1.0.orig.tar.xz
Files:
ddd333 12 devel optional hello_1.0.orig.tar.xz
";
let base = tempfile::tempdir().expect("tempdir");
std::fs::write(base.path().join(dsc_name), dsc).expect("write dsc");
std::fs::write(base.path().join(tarball), "tarball bytes").expect("write tarball");
let ctx = Arc::new(
crate::context::Context::new(crate::context::ContextConfig::Local).expect("context"),
);
let mut checksums = FileChecksums::new();
include_dsc_artifacts(&ctx, base.path(), dsc_name, &mut checksums)
.expect("valid dsc must parse");
let collected: Vec<(String, crate::debian::ChecksumEntry)> = checksums
.iter()
.map(|(k, e)| (k.clone(), e.clone()))
.collect();
assert_eq!(
collected
.iter()
.map(|(k, _)| k.as_str())
.collect::<Vec<_>>(),
vec![dsc_name, tarball],
".dsc first, then Checksums-Sha1 order"
);
// The .dsc is hashed fresh from disk.
let dsc_entry = &collected[0].1;
assert_eq!(dsc_entry.size, dsc.len() as u64);
assert_eq!(dsc_entry.md5.len(), 32);
assert_eq!(dsc_entry.sha1.len(), 40);
assert_eq!(dsc_entry.sha256.len(), 64);
// The tarball reuses the .dsc-recorded digests, including the
// legacy 5-column `Files` md5 (section/priority skipped).
let tar_entry = &collected[1].1;
assert_eq!(tar_entry.size, 12);
assert_eq!(tar_entry.md5, "ddd333");
assert_eq!(tar_entry.sha1, "aaa111");
assert_eq!(tar_entry.sha256, "bbb222");
}
}
+27 -3
View File
@@ -60,11 +60,16 @@ pub fn arch_env(host_arch: Option<&str>) -> Result<BTreeMap<String, String>, Str
/// Read the current vendor name from `/etc/dpkg/origins/default`
/// (its `Vendor:` or `Origin:` field), defaulting to `"debian"`.
pub fn current_vendor() -> String {
read_vendor_from(Path::new("/etc/dpkg/origins/default")).unwrap_or_else(|| "debian".to_string())
std::fs::read_to_string(Path::new("/etc/dpkg/origins/default"))
.ok()
.and_then(|content| vendor_from_origins_content(&content))
.unwrap_or_else(|| "debian".to_string())
}
fn read_vendor_from(path: &Path) -> Option<String> {
let content = std::fs::read_to_string(path).ok()?;
/// Extract the vendor name from the content of a dpkg origins file: its
/// `Vendor:` field, falling back to `Origin:` when absent. `None` when
/// neither field carries a non-empty value.
pub fn vendor_from_origins_content(content: &str) -> Option<String> {
for line in content.lines() {
if let Some(value) = line.strip_prefix("Vendor:") {
let v = value.trim();
@@ -282,6 +287,25 @@ mod tests {
);
}
#[test]
fn vendor_from_origins_content_prefers_vendor_then_origin() {
assert_eq!(
vendor_from_origins_content("Vendor: Ubuntu\nSuite: noble\n"),
Some("Ubuntu".to_string())
);
// Origin fallback when no Vendor field is present.
assert_eq!(
vendor_from_origins_content("Origin: Debian\nSuite: stable\n"),
Some("Debian".to_string())
);
// Empty Vendor falls through to Origin.
assert_eq!(
vendor_from_origins_content("Vendor: \nOrigin: Debian\n"),
Some("Debian".to_string())
);
assert_eq!(vendor_from_origins_content("Suite: stable\n"), None);
}
#[test]
fn environment_escaping() {
// The function reads the process env; just verify formatting helpers
+161 -36
View File
@@ -376,41 +376,36 @@ pub fn run_source_build(
// order the .dsc itself lists them.
let dsc_content = std::fs::read_to_string(&ref_dsc_path)
.map_err(|e| format!("cannot read '{}': {}", ref_dsc_path.display(), e))?;
let dsc_para = parse_paragraphs(&dsc_content)
.into_iter()
.next()
.ok_or_else(|| format!("'{}' is empty", ref_dsc_path.display()))?;
let dsc_para = parse_paragraphs(crate::debian::control::strip_clearsigned_armour(
&dsc_content,
))
.into_iter()
.next()
.ok_or_else(|| format!("'{}' is empty", ref_dsc_path.display()))?;
let mut tarball_paths = Vec::new();
let mut dsc_file_names: Vec<String> = Vec::new();
let mut dsc_files: HashMap<String, PartialChecksum> = HashMap::new();
for field in ["Checksums-Sha1", "Checksums-Sha256"] {
if let Some(value) = dsc_para.get(field) {
for line in value.lines() {
let tokens: Vec<&str> = line.split_whitespace().collect();
if tokens.len() != 3 {
continue;
}
if !dsc_files.contains_key(tokens[2]) {
dsc_file_names.push(tokens[2].to_string());
}
let slot = dsc_files.entry(tokens[2].to_string()).or_default();
match field {
"Checksums-Sha1" => slot.sha1 = Some(tokens[0].to_string()),
_ => slot.sha256 = Some(tokens[0].to_string()),
}
slot.size = tokens[1].parse().ok().or(slot.size);
// Distribution order follows the Checksums fields (Checksums-Sha1 then
// Checksums-Sha256), like dpkg-genchanges; the `Files` field only
// supplements the md5 digests.
for field in ["Checksums-Sha1", "Checksums-Sha256", "Files"] {
let Some(value) = dsc_para.get(field) else {
continue;
};
for cl in parse_checksum_field(field, value)
.map_err(|e| format!("cannot parse '{}': {e}", ref_dsc_path.display()))?
{
if !dsc_files.contains_key(&cl.name) {
dsc_file_names.push(cl.name.clone());
}
}
}
if let Some(files_value) = dsc_para.get("Files") {
for line in files_value.lines() {
let tokens: Vec<&str> = line.split_whitespace().collect();
if tokens.len() >= 3 {
let slot = dsc_files.entry(tokens[2].to_string()).or_default();
slot.md5 = Some(tokens[0].to_string());
slot.size = tokens[1].parse().ok().or(slot.size);
let slot = dsc_files.entry(cl.name.clone()).or_default();
match field {
"Checksums-Sha1" => slot.sha1 = Some(cl.digest),
"Checksums-Sha256" => slot.sha256 = Some(cl.digest),
_ => slot.md5 = Some(cl.digest),
}
slot.size = Some(cl.size);
}
}
for name in &dsc_file_names {
@@ -426,7 +421,12 @@ pub fn run_source_build(
)
.into());
}
let partial = &dsc_files[name];
let partial = dsc_files.get(name).ok_or_else(|| {
format!(
"file '{name}' listed in '{}' has no checksum entry",
ref_dsc_path.display()
)
})?;
checksums.insert_entry(
name,
ChecksumEntry {
@@ -546,6 +546,63 @@ struct PartialChecksum {
sha256: Option<String>,
}
/// One validated line of a `Checksums-Sha1` / `Checksums-Sha256` / `Files`
/// field body.
#[derive(Debug, Clone, PartialEq, Eq)]
struct ChecksumLine {
/// Digest as written in the first column.
digest: String,
/// File size in bytes.
size: u64,
/// File name (last column of the line).
name: String,
}
/// Parse the body of a `Checksums-Sha1` / `Checksums-Sha256` / `Files` field
/// (one file per line) into validated entries. Shared by the source-build
/// pipeline and the binary-only metadata generation so both accept exactly
/// the same lines.
///
/// Both layouts are accepted, detected by column count:
/// - 3 columns: `<digest> <size> <name>` (modern `Checksums-*` fields and
/// the `Files` field of freshly built `.dsc`/`.changes`),
/// - 5 columns: `<digest> <size> <section> <priority> <name>` (legacy
/// `Files` fields, where the name is the last token).
///
/// Any other line (notably 4 columns) or a non-numeric size is a malformed
/// line and yields an error naming `field` and the offending line.
fn parse_checksum_field(field: &str, value: &str) -> Result<Vec<ChecksumLine>, String> {
let mut entries = Vec::new();
for line in value.lines() {
if line.trim().is_empty() {
continue;
}
let tokens: Vec<&str> = line.split_whitespace().collect();
let (digest, size, name) = match tokens.as_slice() {
[digest, size, name] => (*digest, *size, *name),
// Legacy 5-column `Files` layout: digest size section priority name.
[digest, size, _section, _priority, name] => (*digest, *size, *name),
_ => {
return Err(format!(
"malformed '{field}' line (expected 'checksum size name' \
or legacy 'checksum size section priority name', got {} \
columns): '{line}'",
tokens.len()
));
}
};
let size: u64 = size.parse().map_err(|_| {
format!("malformed '{field}' line (size '{size}' is not a number): '{line}'")
})?;
entries.push(ChecksumLine {
digest: digest.to_string(),
size,
name: name.to_string(),
});
}
Ok(entries)
}
/// Run a build command in `cwd` with extra environment variables layered on
/// top of the inherited environment.
///
@@ -633,6 +690,73 @@ mod tests {
let v = DebianVersion::parse("1.0-2").unwrap();
assert_eq!(v.no_epoch(), "1.0-2");
}
#[test]
fn checksum_field_parses_three_column_lines() {
let entries = parse_checksum_field(
"Checksums-Sha256",
" aaa111 12 hello_1.0.orig.tar.xz\n bbb222 3 hello_1.0-1.debian.tar.xz",
)
.expect("valid 3-column field");
assert_eq!(
entries,
vec![
ChecksumLine {
digest: "aaa111".into(),
size: 12,
name: "hello_1.0.orig.tar.xz".into(),
},
ChecksumLine {
digest: "bbb222".into(),
size: 3,
name: "hello_1.0-1.debian.tar.xz".into(),
},
]
);
}
#[test]
fn checksum_field_parses_legacy_five_column_files() {
// Old archive .dsc/.changes carry `Files` as
// md5 size section priority name.
let entries = parse_checksum_field(
"Files",
" d111 100 editors optional hello_1.0.orig.tar.gz\n \
d222 55 web optional hello_1.0-1.diff.gz",
)
.expect("valid legacy 5-column field");
assert_eq!(entries.len(), 2);
assert_eq!(entries[0].digest, "d111");
assert_eq!(entries[0].size, 100);
assert_eq!(entries[0].name, "hello_1.0.orig.tar.gz");
assert_eq!(entries[1].name, "hello_1.0-1.diff.gz");
}
#[test]
fn checksum_field_rejects_four_column_line() {
let err = parse_checksum_field(
"Checksums-Sha256",
" aaa111 12 hello_1.0.orig.tar.xz\n ccc333 12 bogus hello_1.0-1.debian.tar.xz",
)
.expect_err("4-column line must be rejected");
assert!(err.contains("Checksums-Sha256"), "{err}");
assert!(err.contains("hello_1.0-1.debian.tar.xz"), "{err}");
}
#[test]
fn checksum_field_rejects_non_numeric_size() {
let err = parse_checksum_field("Files", " d111 twelve hello.tar.xz")
.expect_err("non-numeric size must be rejected");
assert!(err.contains("twelve"), "{err}");
}
#[test]
fn checksum_field_skips_blank_lines() {
let entries = parse_checksum_field("Files", "\n d111 100 hello.tar.xz\n\n")
.expect("blank lines are ignored");
assert_eq!(entries.len(), 1);
assert_eq!(entries[0].name, "hello.tar.xz");
}
}
/// Differential tests: build synthetic (or real archive) source packages
@@ -1315,7 +1439,7 @@ Provides: virtual-thing (= 2.0), plain-virtual
let vendor = env::current_vendor();
let profiles = env::resolve_build_profiles(&[], &vendor);
let parallel = env::num_parallel();
let build_env_vars: Vec<(String, String)> = [
let build_env_vars: BTreeMap<String, String> = [
("LANG".to_string(), "C".to_string()),
(
"DEB_BUILD_OPTIONS".to_string(),
@@ -1336,15 +1460,16 @@ Provides: virtual-thing (= 2.0), plain-virtual
assert!(status.success(), "debian/rules {target} failed");
}
let ctx = std::sync::Arc::new(crate::context::Context::new(
crate::context::ContextConfig::Local,
));
let ctx = std::sync::Arc::new(
crate::context::Context::new(crate::context::ContextConfig::Local).unwrap(),
);
let native_arch = crate::debian::arch::native().unwrap_or_else(|_| "amd64".into());
let opts = crate::build::binary::BinaryMetadataOptions {
profiles,
vendor,
parallel,
source_date_epoch: entry.timestamp,
// The metadata records exactly the environment exported to the
// build steps above.
exported_env: build_env_vars,
build_arch: native_arch.clone(),
host_arch: native_arch,
};
+68 -23
View File
@@ -40,7 +40,7 @@ pub fn generate_entry(
version.to_string()
} else {
// TODO: Pass these flags from CLI
compute_new_version(&old_version, false, false, false)
compute_new_version(&old_version, false, false, false)?
};
let (maintainer_name, maintainer_email) = get_maintainer_info()?;
@@ -68,7 +68,7 @@ fn compute_new_version(
is_ubuntu: bool,
is_rebuild: bool,
is_nmu: bool,
) -> String {
) -> Result<String, Box<dyn std::error::Error>> {
if is_ubuntu {
return increment_suffix(old_version, "ubuntu");
}
@@ -86,7 +86,7 @@ fn compute_new_version(
}
/// Increment a version number by 1, for a given suffix
fn increment_suffix(version: &str, suffix: &str) -> String {
fn increment_suffix(version: &str, suffix: &str) -> Result<String, Box<dyn std::error::Error>> {
// If suffix is empty, we just look for trailing digits
// If suffix is not empty, we look for suffix followed by digits
@@ -100,19 +100,33 @@ fn increment_suffix(version: &str, suffix: &str) -> String {
if let Some(caps) = re.captures(version) {
let num_str = caps.get(1).unwrap().as_str();
let num: u32 = num_str.parse().unwrap();
// Parse as u64 so that large trailing numbers (e.g. date-based
// versions like '1.0-20250123123456', which do not fit in a u32)
// still increment normally
let num: u64 = num_str.parse().map_err(|_| {
format!(
"Cannot increment version '{version}': trailing number '{num_str}' \
is too large to be incremented. Specify a version explicitly instead."
)
})?;
let range = caps.get(1).unwrap().range();
let new_num = num.checked_add(1).ok_or_else(|| {
format!(
"Cannot increment version '{version}': trailing number {num} \
is too large to be incremented. Specify a version explicitly instead."
)
})?;
let mut new_ver = version.to_string();
new_ver.replace_range(range, &(num + 1).to_string());
return new_ver;
new_ver.replace_range(range, &new_num.to_string());
return Ok(new_ver);
}
// If pattern not found, append suffix + "1"
// But if suffix is empty, we default to appending "-1" (standard Debian revision start)
if suffix.is_empty() {
format!("{}-1", version)
Ok(format!("{}-1", version))
} else {
format!("{}{}{}", version, suffix, 1)
Ok(format!("{}{}{}", version, suffix, 1))
}
}
@@ -381,70 +395,101 @@ mod tests {
fn test_compute_new_version() {
// Debian upload
assert_eq!(
compute_new_version("15.2.0-8", false, false, false),
compute_new_version("15.2.0-8", false, false, false).unwrap(),
"15.2.0-9"
);
assert_eq!(
compute_new_version("15.2.0-9", false, false, false),
compute_new_version("15.2.0-9", false, false, false).unwrap(),
"15.2.0-10"
);
// Ubuntu upload
assert_eq!(
compute_new_version("15.2.0-9", true, false, false),
compute_new_version("15.2.0-9", true, false, false).unwrap(),
"15.2.0-9ubuntu1"
);
assert_eq!(
compute_new_version("15.2.0-9ubuntu1", true, false, false),
compute_new_version("15.2.0-9ubuntu1", true, false, false).unwrap(),
"15.2.0-9ubuntu2"
);
// No change rebuild
assert_eq!(
compute_new_version("15.2.0-9", false, true, false),
compute_new_version("15.2.0-9", false, true, false).unwrap(),
"15.2.0-9build1"
);
assert_eq!(
compute_new_version("15.2.0-9build1", false, true, false),
compute_new_version("15.2.0-9build1", false, true, false).unwrap(),
"15.2.0-9build2"
);
// Rebuild of Ubuntu version
assert_eq!(
compute_new_version("15.2.0-9ubuntu1", false, true, false),
compute_new_version("15.2.0-9ubuntu1", false, true, false).unwrap(),
"15.2.0-9ubuntu1build1"
);
// NMU
// Native
assert_eq!(compute_new_version("1.0", false, false, true), "1.0+nmu1");
assert_eq!(
compute_new_version("1.0+nmu1", false, false, true),
compute_new_version("1.0", false, false, true).unwrap(),
"1.0+nmu1"
);
assert_eq!(
compute_new_version("1.0+nmu1", false, false, true).unwrap(),
"1.0+nmu2"
);
// Non-native
assert_eq!(compute_new_version("1.0-1", false, false, true), "1.0-1.1");
assert_eq!(
compute_new_version("1.0-1.1", false, false, true),
compute_new_version("1.0-1", false, false, true).unwrap(),
"1.0-1.1"
);
assert_eq!(
compute_new_version("1.0-1.1", false, false, true).unwrap(),
"1.0-1.2"
);
// NMU of NMU?
assert_eq!(
compute_new_version("1.0-1.2", false, false, true),
compute_new_version("1.0-1.2", false, false, true).unwrap(),
"1.0-1.3"
);
// Native package uploads
assert_eq!(compute_new_version("1.0", false, false, false), "1.1");
assert_eq!(compute_new_version("1.0.5", false, false, false), "1.0.6");
assert_eq!(
compute_new_version("20241126", false, false, false),
compute_new_version("1.0", false, false, false).unwrap(),
"1.1"
);
assert_eq!(
compute_new_version("1.0.5", false, false, false).unwrap(),
"1.0.6"
);
assert_eq!(
compute_new_version("20241126", false, false, false).unwrap(),
"20241127"
);
}
#[test]
fn test_compute_new_version_large_trailing_number() {
// Date-based versions with a trailing number larger than u32::MAX
// must increment normally (they fit in a u64)
assert_eq!(
compute_new_version("1.0-20250123123456", false, false, false).unwrap(),
"1.0-20250123123457"
);
// A number that does not even fit in a u64 yields a clear error
// instead of panicking
let err = compute_new_version("1.0-99999999999999999999999999", false, false, false);
assert!(err.is_err());
// u64::MAX itself cannot be incremented
let err = compute_new_version("1.0-18446744073709551615", false, false, false);
assert!(err.is_err());
}
#[test]
fn test_get_maintainer_info() {
// Test with env vars
+67 -15
View File
@@ -131,8 +131,46 @@ pub struct Context {
impl Context {
/// Create a context from configuration
pub fn new(config: ContextConfig) -> Self {
let parent = match &config {
///
/// Parent contexts named in the configuration are resolved through the
/// global context manager; a dangling parent name is reported as an
/// error instead of panicking.
///
/// Note that this takes a read lock on the global manager's
/// configuration: never call it while holding that lock for writing.
/// [`crate::context::ContextManager`] itself goes through
/// [`Context::with_lookup`] instead, which takes no locks.
pub fn new(config: ContextConfig) -> io::Result<Self> {
Self::with_lookup(config, &|name| {
crate::context::manager::MANAGER
.get_config()
.contexts
.get(name)
.cloned()
})
}
/// Create a context from configuration, resolving `parent` context names
/// through `lookup` instead of the global context manager.
///
/// `lookup` must be lock-free: this is what allows
/// [`crate::context::ContextManager`] to build contexts while holding
/// (or before the very existence of) its configuration lock. Returns an
/// error when a referenced parent does not exist or when the parent
/// chain contains a cycle.
pub(crate) fn with_lookup(
config: ContextConfig,
lookup: &dyn Fn(&str) -> Option<ContextConfig>,
) -> io::Result<Self> {
Self::with_lookup_inner(config, lookup, &mut Vec::new())
}
fn with_lookup_inner(
config: ContextConfig,
lookup: &dyn Fn(&str) -> Option<ContextConfig>,
chain: &mut Vec<String>,
) -> io::Result<Self> {
let parent_name = match &config {
ContextConfig::Schroot {
parent: Some(parent_name),
..
@@ -140,23 +178,37 @@ impl Context {
| ContextConfig::Unshare {
parent: Some(parent_name),
..
} => {
let config_lock = crate::context::manager::MANAGER.get_config();
let parent_config = config_lock
.contexts
.get(parent_name)
.cloned()
.expect("Parent context not found");
Some(Arc::new(Context::new(parent_config)))
} => parent_name.clone(),
_ => {
return Ok(Self {
config,
parent: None,
driver: Mutex::new(None),
});
}
_ => None,
};
Self {
config,
parent,
driver: Mutex::new(None),
if chain.iter().any(|name| name == &parent_name) {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
format!("Parent context cycle: '{parent_name}' appears in its own parent chain"),
));
}
let parent_config = lookup(&parent_name).ok_or_else(|| {
io::Error::new(
io::ErrorKind::NotFound,
format!("Parent context '{parent_name}' not found"),
)
})?;
chain.push(parent_name);
let parent = Self::with_lookup_inner(parent_config, lookup, chain);
chain.pop();
Ok(Self {
config,
parent: Some(Arc::new(parent?)),
driver: Mutex::new(None),
})
}
/// Create a context with an explicit parent context
+149 -41
View File
@@ -3,7 +3,7 @@ use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::fs;
use std::io;
use std::path::PathBuf;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::sync::RwLock;
@@ -48,26 +48,88 @@ impl ContextManager {
fs::create_dir_all(config_dir)?;
let config_path = config_dir.join("contexts.json");
let config = if config_path.exists() {
// Load existing configuration file
let content = fs::read_to_string(&config_path)?;
serde_json::from_str(&content)
.map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))?
} else {
// Create a new configuration file
Config::default()
let mut config = Self::load_config(&config_path);
// Build the initial Context against the freshly loaded map, before
// the manager itself exists: resolution must not go through the
// global MANAGER here, since a parented current context would
// re-enter its own LazyLock initialization.
let initial = match Self::make_context(&config.context, &config.contexts) {
Ok(context) => context,
Err(e) => {
log::error!(
"Cannot build current context '{}' from {}: {e}; falling back to 'local'",
config.context,
config_path.display()
);
config.context = "local".to_string();
Self::make_context("local", &config.contexts).unwrap_or_else(|e| {
// Only possible in a hand-edited configuration without
// any 'local' entry; a plain Local context has no parent
// and cannot fail to build.
log::error!(
"'local' context missing from {}: {e}",
config_path.display()
);
Context::new(ContextConfig::Local).expect("Local context cannot fail")
})
}
};
Ok(Self {
context: RwLock::new(Arc::new(Self::make_context(
config.context.as_str(),
&config,
))),
context: RwLock::new(Arc::new(initial)),
config_path,
config: RwLock::new(config),
})
}
/// Load the configuration stored at `path`.
///
/// A missing file yields [`Config::default`]. A file that cannot be read
/// or parsed must not take the whole program down: this falls back to
/// the default (local-only) configuration and logs an error. Because a
/// later [`ContextManager::save`] would otherwise silently overwrite the
/// corrupt file and destroy its content, the corrupt file is first
/// backed up to `<path>.bak` (best effort).
pub(crate) fn load_config(path: &Path) -> Config {
if !path.exists() {
return Config::default();
}
let loaded = fs::read_to_string(path).and_then(|content| {
serde_json::from_str(&content)
.map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))
});
match loaded {
Ok(config) => config,
Err(e) => {
log::error!(
"Context configuration {} is corrupt ({e}); using the default (local-only) configuration",
path.display()
);
Self::backup_corrupt_file(path);
Config::default()
}
}
}
/// Back up a corrupt configuration file (best effort) so a later save
/// cannot silently destroy its content.
fn backup_corrupt_file(path: &Path) {
let mut os = path.as_os_str().to_os_string();
os.push(".bak");
let backup_path = PathBuf::from(os);
match fs::copy(path, &backup_path) {
Ok(_) => log::warn!(
"Corrupt context configuration backed up to {}",
backup_path.display()
),
Err(e) => log::warn!(
"Could not back up corrupt context configuration to {}: {e}",
backup_path.display()
),
}
}
/// Obtain current ContextManager configuration
pub fn get_config(&self) -> std::sync::RwLockReadGuard<'_, Config> {
self.config.read().unwrap()
@@ -77,7 +139,12 @@ impl ContextManager {
pub fn with_path(path: PathBuf) -> Self {
let config = Config::default();
Self {
context: RwLock::new(Arc::new(Self::make_context("local", &config))),
// 'local' is always present in Config::default and has no
// parent, so this cannot fail.
context: RwLock::new(Arc::new(
Self::make_context("local", &config.contexts)
.expect("default 'local' context cannot fail"),
)),
config_path: path,
config: RwLock::new(config),
}
@@ -92,13 +159,22 @@ impl ContextManager {
Ok(())
}
fn make_context(name: &str, config: &Config) -> Context {
let context_config = config
.contexts
.get(name)
.cloned()
.expect("Context not found in config");
Context::new(context_config)
/// Build a [`Context`] for `name` from `contexts`.
///
/// Lock-free by construction: parent references are resolved against
/// `contexts` itself (see [`Context::with_lookup`]), never against the
/// manager's configuration lock. This is what keeps [`ContextManager::new`]
/// working before the global [`MANAGER`] exists, and what allows callers
/// to build contexts without risking a re-entrant read on a lock they
/// already hold for writing.
fn make_context(name: &str, contexts: &HashMap<String, ContextConfig>) -> io::Result<Context> {
let context_config = contexts.get(name).cloned().ok_or_else(|| {
io::Error::new(
io::ErrorKind::NotFound,
format!("Context '{name}' not found in configuration"),
)
})?;
Context::with_lookup(context_config, &|parent| contexts.get(parent).cloned())
}
/// List contexts from configuration
@@ -124,45 +200,77 @@ impl ContextManager {
/// Remove context from configuration
pub fn remove_context(&self, name: &str) -> io::Result<()> {
let mut config = self.config.write().unwrap();
if name == "local" {
return Err(io::Error::new(
io::ErrorKind::InvalidInput,
"Cannot remove local context",
));
}
if config.contexts.remove(name).is_some() {
// If we are removing the current context, fallback to local
if name == config.context {
config.context = "local".to_string();
self.set_current_ephemeral(Self::make_context("local", &config));
// Mutate under the write lock, snapshotting the remaining map when
// the removed context was current; the fallback Context is built
// after the lock is released (same discipline as `set_current`).
let fallback_contexts = {
let mut config = self.config.write().unwrap();
if config.contexts.remove(name).is_none() {
return Ok(());
}
if name == config.context {
// If we are removing the current context, fallback to local
config.context = "local".to_string();
Some(config.contexts.clone())
} else {
None
}
};
drop(config); // Drop write lock before saving
self.save()?;
if let Some(contexts) = fallback_contexts {
self.set_current_ephemeral(Self::make_context("local", &contexts)?);
}
self.save()?;
Ok(())
}
/// Set current context from name (modifying configuration)
pub fn set_current(&self, name: &str) -> io::Result<()> {
let mut config = self.config.write().unwrap();
if config.contexts.contains_key(name) {
config.context = name.to_string();
self.set_current_ephemeral(Self::make_context(name, &config));
drop(config); // Drop write lock before saving
self.save()?;
Ok(())
} else {
Err(io::Error::new(
// Snapshot what `make_context` needs and release the lock before
// building the Context. Building resolves parent contexts, and this
// code path used to hold the config write guard while re-entering
// the same lock for a read — a guaranteed deadlock on a
// std::sync::RwLock (same-thread write-then-read).
let contexts = self.config.read().unwrap().contexts.clone();
if !contexts.contains_key(name) {
return Err(io::Error::new(
io::ErrorKind::NotFound,
format!("Context '{}' not found", name),
))
format!("Context '{name}' not found"),
));
}
let context = Self::make_context(name, &contexts)?;
// Re-take the write lock briefly to commit. The name may have been
// removed between snapshot and commit; report the same NotFound
// error instead of persisting a dangling current-context reference.
let mut config = self.config.write().unwrap();
if !config.contexts.contains_key(name) {
return Err(io::Error::new(
io::ErrorKind::NotFound,
format!("Context '{name}' not found"),
));
}
config.context = name.to_string();
drop(config); // Drop write lock before saving
self.set_current_ephemeral(context);
self.save()?;
Ok(())
}
/// Set current context, without modifying configuration
pub fn set_current_ephemeral(&self, context: Context) {
///
/// Accepts either an owned [`Context`] or an already-shared
/// `Arc<Context>`: callers that keep their own handle to the context
/// they install (e.g. [`crate::deb::ephemeral::EphemeralContextGuard`])
/// pass the Arc so they can restore exactly this context afterwards
/// instead of relying on whatever happens to be current at that time.
pub fn set_current_ephemeral(&self, context: impl Into<Arc<Context>>) {
*self.context.write().unwrap() = context.into();
}
+142 -4
View File
@@ -3,6 +3,7 @@ pub(crate) mod capture;
mod local;
mod manager;
mod schroot;
pub(crate) mod shell;
mod ssh;
mod unshare;
@@ -75,7 +76,7 @@ mod tests {
let src_file = temp_dir.path().join("src.txt");
fs::write(&src_file, "local").unwrap();
let ctx = Context::new(ContextConfig::Local);
let ctx = Context::new(ContextConfig::Local).unwrap();
let dest = ctx.ensure_available(&src_file, "/tmp").unwrap();
// Should return a path that exists and has the same content
@@ -154,10 +155,147 @@ mod tests {
assert!(mgr.list_contexts().contains(&"local".to_string()));
}
/// `set_current` on a context whose configuration carries a `parent`
/// must complete without deadlocking: building the Context resolves the
/// parent chain, which used to re-enter the config lock while
/// `set_current` still held it for writing (a guaranteed deadlock on a
/// std::sync::RwLock, same-thread write-then-read).
#[test]
fn test_set_current_parented_context_no_deadlock() {
let temp_file = NamedTempFile::new().unwrap();
let mgr = Arc::new(ContextManager::with_path(temp_file.path().to_path_buf()));
mgr.add_context("base", ContextConfig::Local).unwrap();
mgr.add_context(
"child",
ContextConfig::Schroot {
name: "testchroot".to_string(),
parent: Some("base".to_string()),
},
)
.unwrap();
// Run with a timeout so a regression fails fast instead of hanging
// the test binary forever.
let (tx, rx) = std::sync::mpsc::channel();
let worker = {
let mgr = mgr.clone();
std::thread::spawn(move || {
let result = mgr.set_current("child");
tx.send(()).expect("receiver still waiting");
result
})
};
match rx.recv_timeout(std::time::Duration::from_secs(30)) {
Ok(()) => {}
Err(std::sync::mpsc::RecvTimeoutError::Timeout) => {
panic!("set_current() deadlocked building a parented context");
}
Err(std::sync::mpsc::RecvTimeoutError::Disconnected) => {
panic!("set_current() thread panicked before completing");
}
}
worker.join().unwrap().unwrap();
assert_eq!(mgr.current_name(), "child");
}
/// A context referencing a missing parent must produce an error, not a
/// panic (the parent lookup used to `.expect()`).
#[test]
fn test_set_current_dangling_parent_errors() {
let temp_file = NamedTempFile::new().unwrap();
let mgr = ContextManager::with_path(temp_file.path().to_path_buf());
mgr.add_context(
"orphan",
ContextConfig::Unshare {
path: "/some/chroot".to_string(),
parent: Some("missing".to_string()),
},
)
.unwrap();
let err = mgr.set_current("orphan").unwrap_err();
assert_eq!(err.kind(), std::io::ErrorKind::NotFound);
// Nothing was committed: the current context is unchanged.
assert_eq!(mgr.current_name(), "local");
}
/// A parent cycle in a hand-edited configuration must be rejected with
/// an error instead of recursing until the stack overflows.
#[test]
fn test_set_current_parent_cycle_errors() {
let temp_file = NamedTempFile::new().unwrap();
let mgr = ContextManager::with_path(temp_file.path().to_path_buf());
mgr.add_context(
"a",
ContextConfig::Schroot {
name: "schroot-a".to_string(),
parent: Some("b".to_string()),
},
)
.unwrap();
mgr.add_context(
"b",
ContextConfig::Unshare {
path: "/chroot-b".to_string(),
parent: Some("a".to_string()),
},
)
.unwrap();
let err = mgr.set_current("a").unwrap_err();
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
}
/// A corrupt contexts.json must not take the manager down:
/// `load_config` falls back to the default (local-only) configuration,
/// keeps the corrupt file in place and backs it up to contexts.json.bak
/// so a later save cannot silently destroy its content.
#[test]
fn test_load_config_corrupt_file_falls_back_and_backs_up() {
let temp_dir = tempfile::tempdir().unwrap();
let config_path = temp_dir.path().join("contexts.json");
let garbage = "{ this is definitely not valid json";
fs::write(&config_path, garbage).unwrap();
let config = ContextManager::load_config(&config_path);
// Falls back to the default (local-only) configuration...
assert_eq!(config.context, "local");
assert!(config.contexts.contains_key("local"));
// ...preserving the corrupt file via the backup, original untouched.
let backup_path = temp_dir.path().join("contexts.json.bak");
assert_eq!(fs::read_to_string(&backup_path).unwrap(), garbage);
assert_eq!(fs::read_to_string(&config_path).unwrap(), garbage);
// A subsequent save replaces only the original, never the backup.
let mgr = ContextManager::with_path(config_path.clone());
mgr.add_context("newctx", ContextConfig::Local).unwrap();
let rewritten = fs::read_to_string(&config_path).unwrap();
serde_json::from_str::<super::manager::Config>(&rewritten).unwrap();
assert_eq!(fs::read_to_string(&backup_path).unwrap(), garbage);
}
/// A missing contexts.json yields the default configuration and writes
/// nothing (no file, no backup) until an explicit save.
#[test]
fn test_load_config_missing_file_defaults() {
let temp_dir = tempfile::tempdir().unwrap();
let config_path = temp_dir.path().join("contexts.json");
let config = ContextManager::load_config(&config_path);
assert_eq!(config.context, "local");
assert!(config.contexts.contains_key("local"));
assert!(!config_path.exists());
assert!(!temp_dir.path().join("contexts.json.bak").exists());
}
#[test]
fn test_context_file_ops() {
let temp_dir = tempfile::tempdir().unwrap();
let ctx = Context::new(ContextConfig::Local);
let ctx = Context::new(ContextConfig::Local).unwrap();
let file_path = temp_dir.path().join("test.txt");
let content = "hello world";
@@ -199,7 +337,7 @@ mod tests {
fn test_context_copy_preserves_dangling_symlink() {
use std::os::unix::fs::symlink;
let temp_dir = tempfile::tempdir().unwrap();
let ctx = Context::new(ContextConfig::Local);
let ctx = Context::new(ContextConfig::Local).unwrap();
let src_dir = temp_dir.path().join("src");
std::fs::create_dir_all(&src_dir).unwrap();
@@ -235,7 +373,7 @@ mod tests {
fs::create_dir_all(src_root.join("src/.svn")).unwrap();
fs::write(src_root.join("src/hello.c"), "int main() {}").unwrap();
let ctx = Context::new(ContextConfig::Local);
let ctx = Context::new(ContextConfig::Local).unwrap();
let dest = ctx.ensure_available(&src_root, "/tmp").unwrap();
assert!(dest.join("src/hello.c").exists());
+96 -12
View File
@@ -1,6 +1,7 @@
/// Schroot context: execute commands in a schroot session
/// Not tested, will need more work!
use super::api::{ContextDriver, LineSink};
use super::api::{Context, ContextConfig, ContextDriver, LineSink};
use super::shell::shell_quote;
use std::io;
use std::path::{Path, PathBuf};
use std::sync::Arc;
@@ -11,13 +12,12 @@ pub struct SchrootDriver {
pub parent: Option<Arc<super::api::Context>>,
}
use super::api::{Context, ContextConfig};
impl SchrootDriver {
fn parent(&self) -> Arc<Context> {
self.parent
.clone()
.unwrap_or_else(|| Arc::new(Context::new(ContextConfig::Local)))
self.parent.clone().unwrap_or_else(|| {
// ContextConfig::Local has no parent, so this cannot fail.
Arc::new(Context::new(ContextConfig::Local).expect("Local context cannot fail"))
})
}
fn ensure_session(&self) -> io::Result<String> {
@@ -106,6 +106,11 @@ impl SchrootDriver {
/// Wrap `(program, args)` in `sh -c` when a working directory or
/// environment variables are needed.
///
/// Everything interpolated into the resulting shell string — the `cd`
/// target, env keys and values, the program and each argument — is
/// POSIX-shell-quoted (see [`shell_quote`]), so metacharacters (spaces,
/// quotes, `$`, ...) can neither split words nor trigger expansion.
fn wrap_command(
program: &str,
args: &[String],
@@ -119,17 +124,21 @@ impl SchrootDriver {
let mut shell_cmd = String::new();
if let Some(dir) = cwd {
shell_cmd.push_str(&format!("cd {} && ", dir));
shell_cmd.push_str(&format!("cd {} && ", shell_quote(dir)));
}
if !env.is_empty() {
shell_cmd.push_str("env ");
for (k, v) in env {
shell_cmd.push_str(&format!("{}={} ", k, v));
shell_cmd.push_str(&format!("{}={} ", shell_quote(k), shell_quote(v)));
}
}
shell_cmd.push_str(&format!("{} {}", program, args.join(" ")));
shell_cmd.push_str(&shell_quote(program));
for arg in args {
shell_cmd.push(' ');
shell_cmd.push_str(&shell_quote(arg));
}
actual_program = "sh".to_string();
actual_args = vec!["-c".to_string(), shell_cmd];
@@ -260,9 +269,13 @@ impl ContextDriver for SchrootDriver {
&[
"-c".to_string(),
format!(
"echo -ne '{}' > '{}'",
content.replace("'", "'\\''"),
path.to_string_lossy()
// `printf '%s'` writes the content verbatim (the previous
// `echo -ne` mangled backslashes, and dash's echo prints
// "-ne" literally). Content and path are shell-quoted so
// metacharacters in either cannot break out.
"printf '%s' {} > {}",
shell_quote(content),
shell_quote(&path.to_string_lossy())
),
],
&[],
@@ -284,3 +297,74 @@ impl ContextDriver for SchrootDriver {
Ok(status.success())
}
}
#[cfg(test)]
mod tests {
use super::SchrootDriver;
/// Without cwd/env the program and args go to schroot as direct argv
/// (no shell involved), so they must pass through untouched.
#[test]
fn wrap_command_passthrough_without_env_or_cwd() {
let (prog, args) = SchrootDriver::wrap_command(
"make",
&["install".to_string(), "DEST=x y".to_string()],
&[],
None,
);
assert_eq!(prog, "make");
assert_eq!(args, vec!["install".to_string(), "DEST=x y".to_string()]);
}
/// A value with a space must stay a single env assignment: previously
/// DEB_BUILD_OPTIONS="parallel=4 nocheck" made sh treat `nocheck` as
/// the command to run.
#[test]
fn wrap_command_quotes_env_values_cwd_and_args() {
let (prog, args) = SchrootDriver::wrap_command(
"dpkg-buildpackage",
&["-us".to_string(), "-uc".to_string()],
&[(
"DEB_BUILD_OPTIONS".to_string(),
"parallel=4 nocheck".to_string(),
)],
Some("/build/pkg 1.0"),
);
assert_eq!(prog, "sh");
assert_eq!(args[0], "-c");
assert_eq!(
args[1],
"cd '/build/pkg 1.0' && env 'DEB_BUILD_OPTIONS'='parallel=4 nocheck' \
'dpkg-buildpackage' '-us' '-uc'"
);
}
/// The definitive check: a real shell must execute the wrapped command
/// exactly as intended — cwd applied, env set verbatim, the inner
/// program invoked with its argument — despite quotes in the values.
#[cfg(unix)]
#[test]
fn wrapped_command_survives_shell_parsing() {
let dir = tempfile::tempdir().unwrap();
let (prog, args) = SchrootDriver::wrap_command(
"printenv",
&["SOME_OPT".to_string()],
&[(
"SOME_OPT".to_string(),
"parallel=4 noch'eck \"x\"".to_string(),
)],
Some(dir.path().to_str().unwrap()),
);
let output = std::process::Command::new(&prog)
.arg(&args[0])
.arg(&args[1])
.output()
.unwrap();
assert!(output.status.success());
// printenv's output ends with a newline.
assert_eq!(
String::from_utf8_lossy(&output.stdout),
"parallel=4 noch'eck \"x\"\n"
);
}
}
+109
View File
@@ -0,0 +1,109 @@
//! POSIX-shell quoting for command strings assembled by the remote/chroot
//! execution contexts.
//!
//! Unlike [`super::local`] — which spawns programs directly through
//! `std::process::Command`, with no shell in between — the SSH, schroot and
//! unshare drivers ultimately hand a *string* to a shell (`ssh
//! channel.exec`, `sh -c`, `bash -c`). Every program name, argument,
//! path or environment value interpolated into such a string must be
//! quoted, or shell metacharacters (`;`, `|`, `&`, quotes, `$`, backticks,
//! globs, whitespace, ...) are reinterpreted by the shell: at best the
//! command breaks, at worst it executes injected input.
/// Quote `s` for safe interpolation into a POSIX shell command line.
///
/// The result is `s` wrapped in single quotes, with every embedded single
/// quote replaced by the standard `'\''` sequence (close the quoting, an
/// escaped literal quote, reopen). Whatever the input contains — spaces,
/// newlines, `"`, `'`, `$`, backticks, globs, `;` — the shell parses the
/// result back into exactly `s` as a single word. The empty string becomes
/// `''` (one empty argument, not zero arguments).
///
/// Use this for *every* value interpolated into a shell command string:
/// programs, arguments, `cd` targets, `env` assignments (both key and
/// value) and paths. It is safe (though redundant) to quote values that are
/// known to need no quoting.
pub(crate) fn shell_quote(s: &str) -> String {
let mut quoted = String::with_capacity(s.len() + 2);
quoted.push('\'');
for c in s.chars() {
if c == '\'' {
quoted.push_str("'\\''");
} else {
quoted.push(c);
}
}
quoted.push('\'');
quoted
}
#[cfg(test)]
mod tests {
use super::shell_quote;
#[test]
fn plain_word() {
assert_eq!(shell_quote("plain"), "'plain'");
}
#[test]
fn spaces_stay_one_word() {
assert_eq!(shell_quote("parallel=4 nocheck"), "'parallel=4 nocheck'");
assert_eq!(
shell_quote(" leading and trailing "),
"' leading and trailing '"
);
}
#[test]
fn embedded_single_quotes() {
assert_eq!(shell_quote("it's"), "'it'\\''s'");
assert_eq!(shell_quote("''"), r"''\'''\'''");
}
#[test]
fn double_quotes_and_metacharacters() {
assert_eq!(
shell_quote("say \"hi\" $HOME `id` ; | & * ?"),
"'say \"hi\" $HOME `id` ; | & * ?'"
);
}
#[test]
fn dollar_and_backtick_do_not_expand() {
assert_eq!(shell_quote("$HOME"), "'$HOME'");
assert_eq!(shell_quote("$(rm -rf /)"), "'$(rm -rf /)'");
assert_eq!(shell_quote("`touch /tmp/pwned`"), "'`touch /tmp/pwned`'");
}
#[test]
fn empty_string() {
assert_eq!(shell_quote(""), "''");
}
#[test]
fn unicode_preserved() {
assert_eq!(shell_quote("héllo→wörld ✓"), "'héllo→wörld ✓'");
}
#[test]
fn newlines_preserved() {
assert_eq!(shell_quote("a\nb"), "'a\nb'");
}
/// The definitive check: a real shell must parse the quoted string back
/// into the original value as a single argument, without expanding or
/// executing anything inside it.
#[cfg(unix)]
#[test]
fn round_trips_through_sh() {
let tricky = "a'b\"c $HOME `echo pwned` ; | & \n x*y";
let output = std::process::Command::new("sh")
.arg("-c")
.arg(format!("printf '%s' {}", shell_quote(tricky)))
.output()
.unwrap();
assert!(output.status.success());
assert_eq!(String::from_utf8_lossy(&output.stdout), tricky);
}
}
+108 -51
View File
@@ -2,6 +2,7 @@
/// Context driver: Copies over SFTP with ssh2, executes commands over ssh2 channels
use super::api::{ContextDriver, LineSink, Stream};
use super::capture::pump;
use super::shell::shell_quote;
use log::debug;
use ssh2;
use std::fs;
@@ -53,6 +54,41 @@ pub struct SshDriver {
pub port: Option<u16>,
}
impl SshDriver {
/// Build the remote shell command line: `export` assignments for `env`,
/// an optional `cd` to `cwd`, then `program` with its `args`.
///
/// The line is executed verbatim by the remote login shell through
/// `channel.exec`, so every component is POSIX-shell-quoted (see
/// [`shell_quote`]): metacharacters in arguments, paths or environment
/// values can neither break out of their word nor be expanded by the
/// remote shell.
fn build_command_line(
env: &[(String, String)],
cwd: Option<&str>,
program: &str,
args: &[String],
) -> String {
let mut cmd_line = String::new();
for (key, value) in env {
cmd_line.push_str(&format!(
"export {}={}; ",
shell_quote(key),
shell_quote(value)
));
}
if let Some(dir) = cwd {
cmd_line.push_str(&format!("cd {} && ", shell_quote(dir)));
}
cmd_line.push_str(&shell_quote(program));
for arg in args {
cmd_line.push(' ');
cmd_line.push_str(&shell_quote(arg));
}
cmd_line
}
}
impl ContextDriver for SshDriver {
fn ensure_available(&self, src: &Path, dest_root: &str) -> io::Result<PathBuf> {
let sess = connect_ssh(&self.host, self.user.as_deref(), self.port)?;
@@ -106,22 +142,7 @@ impl ContextDriver for SshDriver {
// Construct command line with env vars
// TODO: No, use ssh2 channel.set_env
let mut cmd_line = String::new();
for (key, value) in env {
cmd_line.push_str(&format!(
"export {}='{}'; ",
key,
value.replace("'", "'\\''")
));
}
if let Some(dir) = cwd {
cmd_line.push_str(&format!("cd {} && ", dir));
}
cmd_line.push_str(program);
for arg in args {
cmd_line.push(' ');
cmd_line.push_str(arg); // TODO: escape
}
let cmd_line = Self::build_command_line(env, cwd, program, args);
debug!("Executing SSH command: {}", cmd_line);
@@ -152,23 +173,8 @@ impl ContextDriver for SshDriver {
let sess = connect_ssh(&self.host, self.user.as_deref(), self.port)?;
let mut channel = sess.channel_session().map_err(io::Error::other)?;
// Construct command line with env vars (same escaping as `run`)
let mut cmd_line = String::new();
for (key, value) in env {
cmd_line.push_str(&format!(
"export {}='{}'; ",
key,
value.replace("'", "'\\''")
));
}
if let Some(dir) = cwd {
cmd_line.push_str(&format!("cd {} && ", dir));
}
cmd_line.push_str(program);
for arg in args {
cmd_line.push(' ');
cmd_line.push_str(arg); // TODO: escape
}
// Construct command line with env vars (same quoting as `run`)
let cmd_line = Self::build_command_line(env, cwd, program, args);
debug!("Executing SSH command (captured): {}", cmd_line);
@@ -200,23 +206,8 @@ impl ContextDriver for SshDriver {
let sess = connect_ssh(&self.host, self.user.as_deref(), self.port)?;
let mut channel = sess.channel_session().map_err(io::Error::other)?;
// Construct command line with env vars
let mut cmd_line = String::new();
for (key, value) in env {
cmd_line.push_str(&format!(
"export {}='{}'; ",
key,
value.replace("'", "'\\''")
));
}
if let Some(dir) = cwd {
cmd_line.push_str(&format!("cd {} && ", dir));
}
cmd_line.push_str(program);
for arg in args {
cmd_line.push(' ');
cmd_line.push_str(arg); // TODO: escape
}
// Construct command line with env vars (same quoting as `run`)
let cmd_line = Self::build_command_line(env, cwd, program, args);
channel.exec(&cmd_line).map_err(io::Error::other)?;
@@ -266,7 +257,11 @@ impl ContextDriver for SshDriver {
let sess = connect_ssh(&self.host, self.user.as_deref(), self.port)?;
let mut channel = sess.channel_session().map_err(io::Error::other)?;
// TODO: use sftp
let cmd = format!("cp -a {:?} {:?}", src, dest);
let cmd = format!(
"cp -a {} {}",
shell_quote(&src.to_string_lossy()),
shell_quote(&dest.to_string_lossy())
);
debug!("Executing remote copy: {}", cmd);
channel.exec(&cmd).map_err(io::Error::other)?;
channel.wait_close().map_err(io::Error::other)?;
@@ -360,3 +355,65 @@ impl SshDriver {
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::SshDriver;
/// Program, arguments and the `cd` target must each be a single,
/// quoted word; `$` in the cwd must not be expanded.
#[test]
fn command_line_quotes_program_args_and_cwd() {
let line = SshDriver::build_command_line(
&[],
Some("/tmp/some dir/$HOST"),
"make",
&["install".to_string(), "PREFIX=/opt/my app".to_string()],
);
assert_eq!(
line,
"cd '/tmp/some dir/$HOST' && 'make' 'install' 'PREFIX=/opt/my app'"
);
}
/// Env keys and values are quoted too (values used to be escaped by
/// hand, keys and everything else not at all).
#[test]
fn command_line_quotes_env_keys_and_values() {
let line = SshDriver::build_command_line(
&[(
"DEB_BUILD_OPTIONS".to_string(),
"parallel=4 nocheck".to_string(),
)],
None,
"dpkg-buildpackage",
&["-us".to_string(), "-uc".to_string()],
);
assert_eq!(
line,
"export 'DEB_BUILD_OPTIONS'='parallel=4 nocheck'; 'dpkg-buildpackage' '-us' '-uc'"
);
}
/// The definitive check: a real shell must execute the assembled line
/// exactly as intended — one argument through, one env value verbatim —
/// even when both contain quotes, spaces and `$`.
#[cfg(unix)]
#[test]
fn command_line_survives_shell_parsing() {
let dir = tempfile::tempdir().unwrap();
let line = SshDriver::build_command_line(
&[("OPT".to_string(), "a b'c \"$d\"".to_string())],
Some(dir.path().to_str().unwrap()),
"printenv",
&["OPT".to_string()],
);
let output = std::process::Command::new("sh")
.arg("-c")
.arg(&line)
.output()
.unwrap();
assert!(output.status.success());
assert_eq!(String::from_utf8_lossy(&output.stdout), "a b'c \"$d\"\n");
}
}
+76 -11
View File
@@ -1,4 +1,5 @@
use super::api::{Context, ContextCommand, ContextDriver, LineSink};
use super::shell::shell_quote;
use log::debug;
use std::fs;
use std::io;
@@ -484,21 +485,85 @@ impl UnshareDriver {
// Build the bash command: set up /dev/pts and run the program
// /proc should already be bind-mounted from the host before entering the namespace
let program_args = args
.iter()
.map(|a| format!("\"{a}\""))
.collect::<Vec<_>>()
.join(" ");
cmd.arg("--")
.arg("bash")
.arg("-c")
.arg(format!(
"mkdir -p /dev/pts; mount -t devpts devpts /dev/pts 2>/dev/null || true; touch /dev/ptmx; mount --bind /dev/pts/ptmx /dev/ptmx 2>/dev/null || true; {} {}",
program,
program_args
));
.arg(build_namespace_script(program, args));
cmd
}
}
/// Build the shell script executed by `bash -c` inside the user namespace:
/// bring up `/dev/pts`, then run `program` with `args`.
///
/// The script is parsed by bash, so the program and every argument are
/// POSIX-shell-quoted (see [`shell_quote`]): quotes, `$`, backticks or
/// whitespace inside them can neither split the command into different
/// words nor trigger expansion. (Previously arguments were wrapped in
/// unescaped double quotes, so a `"` in an argument broke out and
/// `$`/backticks still expanded.)
fn build_namespace_script(program: &str, args: &[String]) -> String {
let mut script = String::from(
"mkdir -p /dev/pts; mount -t devpts devpts /dev/pts 2>/dev/null || true; touch /dev/ptmx; mount --bind /dev/pts/ptmx /dev/ptmx 2>/dev/null || true; ",
);
script.push_str(&shell_quote(program));
for arg in args {
script.push(' ');
script.push_str(&shell_quote(arg));
}
script
}
#[cfg(test)]
mod tests {
use super::build_namespace_script;
fn tail_after_devpts_setup(script: &str) -> &str {
script
.rsplit_once("|| true; ")
.map(|(_, rest)| rest)
.unwrap()
.trim_end()
}
/// Program and arguments must each be a single, quoted word at the end
/// of the `/dev/pts` setup script.
#[test]
fn script_quotes_program_and_args() {
let script = build_namespace_script(
"make",
&[
"install".to_string(),
"a b".to_string(),
"PREFIX=/opt/my app".to_string(),
],
);
assert_eq!(
tail_after_devpts_setup(&script),
"'make' 'install' 'a b' 'PREFIX=/opt/my app'"
);
}
/// An argument containing a double quote must not break out of the
/// script (arguments used to be wrapped in unescaped `"`), and `$`/
/// backticks must stay literal for bash.
#[test]
fn script_neutralizes_quotes_and_expansions() {
let script = build_namespace_script(
"echo",
&["$(touch /tmp/pwned) `id` \"; rm -rf /\"".to_string()],
);
assert_eq!(
tail_after_devpts_setup(&script),
"'echo' '$(touch /tmp/pwned) `id` \"; rm -rf /\"'"
);
}
/// Empty arguments must survive as one empty word (`''`), not vanish.
#[test]
fn script_preserves_empty_args() {
let script = build_namespace_script("prog", &[String::new(), "x".to_string()]);
assert_eq!(tail_after_devpts_setup(&script), "'prog' '' 'x'");
}
}
+119 -49
View File
@@ -1,24 +1,19 @@
use crate::context::Context;
use log::debug;
use std::collections::HashMap;
use std::error::Error;
use std::sync::Arc;
/// Set environment variables for cross-compilation
pub fn setup_environment(
env: &mut HashMap<String, String>,
arch: &str,
ctx: Arc<Context>,
) -> Result<(), Box<dyn Error>> {
let dpkg_architecture = String::from_utf8(
ctx.command("dpkg-architecture")
.arg("-a")
.arg(arch)
.output()?
.stdout,
)?;
/// Parse 'dpkg-architecture' output (KEY=value lines) into a set of
/// environment variables. Unexpected lines (e.g. warnings on stderr leaking
/// into stdout) are skipped instead of causing a failure.
fn parse_dpkg_architecture_output(output: &str, env: &mut HashMap<String, String>) {
let env_var_regex = regex::Regex::new(r"(?<key>.*)=(?<value>.*)").unwrap();
for l in dpkg_architecture.lines() {
let capture = env_var_regex.captures(l).unwrap();
for l in output.lines() {
let Some(capture) = env_var_regex.captures(l) else {
debug!("Skipping unexpected dpkg-architecture output line: '{l}'");
continue;
};
let key = capture.name("key").unwrap().as_str().to_string();
let value = capture.name("value").unwrap().as_str().to_string();
@@ -28,6 +23,45 @@ pub fn setup_environment(
env.insert("CROSS_COMPILE".to_string(), format!("{value}-"));
}
}
}
/// Set environment variables for cross-compilation
pub fn setup_environment(
env: &mut HashMap<String, String>,
arch: &str,
ctx: Arc<Context>,
) -> Result<(), Box<dyn Error>> {
let output = ctx
.command("dpkg-architecture")
.arg("-a")
.arg(arch)
.output()
.map_err(|e| {
format!(
"Failed to run 'dpkg-architecture -a {arch}': {e}. \
Is 'dpkg-dev' installed?"
)
})?;
if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr);
return Err(format!(
"'dpkg-architecture -a {}' failed with status: {}.{}",
arch,
output.status,
if stderr.trim().is_empty() {
String::new()
} else {
format!("\ndpkg-architecture output:\n{}", stderr.trim())
}
)
.into());
}
let dpkg_architecture = String::from_utf8(output.stdout)
.map_err(|e| format!("Invalid UTF-8 in dpkg-architecture output: {e}"))?;
parse_dpkg_architecture_output(&dpkg_architecture, env);
env.insert("DEB_BUILD_PROFILES".to_string(), "cross".to_string());
Ok(())
@@ -44,10 +78,19 @@ pub fn ensure_repositories(
let local_arch = crate::get_current_arch();
// Add target ('host') architecture
ctx.command("dpkg")
let status = ctx
.command("dpkg")
.arg("--add-architecture")
.arg(arch)
.status()?;
.status()
.map_err(|e| format!("Failed to run 'dpkg --add-architecture {arch}': {e}"))?;
if !status.success() {
return Err(format!(
"'dpkg --add-architecture {}' failed with status: {}",
arch, status
)
.into());
}
// Check if we are on Ubuntu
let os_release = String::from_utf8(ctx.command("cat").arg("/etc/os-release").output()?.stdout)?;
@@ -117,6 +160,7 @@ pub fn ensure_repositories(
}
let ports_entry = crate::apt::sources::SourceEntry {
enabled: true,
kind: crate::apt::sources::SourceKind::Deb,
components: vec![
"main".to_string(),
"restricted".to_string(),
@@ -125,43 +169,69 @@ pub fn ensure_repositories(
],
architectures: vec![arch.to_string()],
uri: "http://ports.ubuntu.com/ubuntu-ports".to_string(),
signed_by: None,
trusted: None,
suite: ports_suites,
// No origin: saved to the pkh-owned added-sources file
origin: None,
};
sources.push(ports_entry);
}
// Save the updated sources
// Try to save in DEB822 format first, fall back to legacy format
let deb822_path = "/etc/apt/sources.list.d/ubuntu.sources";
if ctx
.command("test")
.arg("-f")
.arg(deb822_path)
.status()?
.success()
{
// For DEB822 format, we need to reconstruct the file content
let mut content = String::new();
for source in &sources {
if !source.enabled {
continue;
}
content.push_str("Types: deb\n");
content.push_str(&format!("URIs: {}\n", source.uri));
content.push_str(&format!("Suites: {}\n", source.suite.join(" ")));
content.push_str(&format!("Components: {}\n", source.components.join(" ")));
content.push_str("Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg\n");
content.push_str(&format!(
"Architectures: {}\n",
source.architectures.join(" ")
));
content.push('\n');
}
ctx.write_file(std::path::Path::new(deb822_path), &content)?;
} else {
// Fall back to legacy format
crate::apt::sources::save_legacy(Some(ctx.clone()), sources, "/etc/apt/sources.list")?;
}
// Save the updated sources: each entry is written back to its origin
// file in its own format (keeping its own Signed-By and Enabled state),
// and the new ports entry goes to the pkh-owned added-sources file
crate::apt::sources::save(Some(ctx.clone()), sources)?;
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_parse_dpkg_architecture_output() {
let output = "DEB_BUILD_ARCH=amd64\n\
DEB_HOST_ARCH=arm64\n\
DEB_HOST_GNU_TYPE=aarch64-linux-gnu\n";
let mut env = HashMap::new();
parse_dpkg_architecture_output(output, &mut env);
assert_eq!(env.get("DEB_BUILD_ARCH").map(String::as_str), Some("amd64"));
assert_eq!(env.get("DEB_HOST_ARCH").map(String::as_str), Some("arm64"));
assert_eq!(
env.get("DEB_HOST_GNU_TYPE").map(String::as_str),
Some("aarch64-linux-gnu")
);
// Derived variable for the GNU type
assert_eq!(
env.get("CROSS_COMPILE").map(String::as_str),
Some("aarch64-linux-gnu-")
);
}
#[test]
fn test_parse_dpkg_architecture_output_skips_unexpected_lines() {
// Unexpected lines (warnings on stdout, empty lines) must be skipped
// instead of panicking
let output = "dpkg-architecture: warning: something odd happened\n\
\n\
DEB_HOST_GNU_TYPE=arm-linux-gnueabihf\n\
not an environment variable assignment\n";
let mut env = HashMap::new();
parse_dpkg_architecture_output(output, &mut env);
assert_eq!(
env.get("DEB_HOST_GNU_TYPE").map(String::as_str),
Some("arm-linux-gnueabihf")
);
assert_eq!(
env.get("CROSS_COMPILE").map(String::as_str),
Some("arm-linux-gnueabihf-")
);
assert_eq!(env.len(), 2);
}
}
+549 -20
View File
@@ -1,20 +1,303 @@
use crate::context::{self, Context, ContextConfig};
use crate::ui::deb::{DebUi, Phase};
use directories::ProjectDirs;
use std::any::Any;
use std::error::Error;
use std::fs;
use std::path::{Path, PathBuf};
use std::process::Command;
use std::sync::Arc;
use std::sync::Mutex;
use std::sync::atomic::{AtomicU64, Ordering};
use std::time::Duration;
use tar::Archive;
use xz2::read::XzDecoder;
// ---------------------------------------------------------------------------
// Process-global cleanup hooks
//
// On Ctrl-C, the SIGINT handler in `ui::deb` restores the terminal and then
// `libc::_exit(130)`s, skipping all destructors — including
// [`EphemeralContextGuard::drop`] — which leaks the freshly bootstrapped
// chroot under /tmp together with its bind-mounted /proc and any overlayfs
// mounts. To make interrupt-time cleanup possible anyway, resources register
// a self-contained cleanup hook here; the SIGINT handler drains and runs the
// registry right before exiting.
// ---------------------------------------------------------------------------
/// A boxed, send-safe cleanup hook body
type CleanupFn = Box<dyn Fn() + Send>;
/// A pending cleanup hook together with its registry id
struct CleanupHook {
id: u64,
f: CleanupFn,
}
/// Registry of cleanup hooks waiting to run at interrupt time
static CLEANUP_HOOKS: Mutex<Vec<CleanupHook>> = Mutex::new(Vec::new());
/// Source of the registry ids used to deregister a specific hook
static NEXT_CLEANUP_HOOK_ID: AtomicU64 = AtomicU64::new(1);
/// Register a hook to be run by [`run_cleanup_hooks`] (i.e. when the process
/// is interrupted), returning a guard whose drop deregisters the hook again
fn register_cleanup_hook(f: CleanupFn) -> CleanupHookGuard {
let id = NEXT_CLEANUP_HOOK_ID.fetch_add(1, Ordering::Relaxed);
CLEANUP_HOOKS.lock().unwrap().push(CleanupHook { id, f });
CleanupHookGuard(id)
}
/// RAII handle to a registered cleanup hook: dropping it (or an explicit
/// [`CleanupHookGuard::deregister`]) removes the hook from the registry so
/// the interrupt path can no longer run it
struct CleanupHookGuard(u64);
impl CleanupHookGuard {
/// Registry id of the hook (used to filter the registry in tests)
#[cfg(test)]
fn id(&self) -> u64 {
self.0
}
/// Remove the hook from the registry; returns whether it was still pending
fn deregister(&mut self) -> bool {
deregister_cleanup_hook(self.0)
}
}
impl Drop for CleanupHookGuard {
fn drop(&mut self) {
deregister_cleanup_hook(self.0);
}
}
/// Remove a hook from the registry; returns whether it was still pending
fn deregister_cleanup_hook(id: u64) -> bool {
let mut hooks = CLEANUP_HOOKS.lock().unwrap();
let len_before = hooks.len();
hooks.retain(|hook| hook.id != id);
hooks.len() != len_before
}
/// Drain and run every registered cleanup hook exactly once
///
/// Called from the SIGINT handler right before the process exits. Draining
/// uses `try_lock` with a bounded retry instead of a blocking lock: if the
/// signal interrupted the main thread while it held [`CLEANUP_HOOKS`] (inside
/// register/deregister), blocking on the same non-recursive mutex from the
/// handler would deadlock the process. Timing out therefore skips cleanup
/// (leaking, as before this registry existed) rather than hanging.
pub(crate) fn run_cleanup_hooks() {
run_drained_hooks(drain_cleanup_hooks());
}
/// Take every pending hook out of the registry, waiting at most ~1s for the
/// registry lock (see [`run_cleanup_hooks`] for why this must not block forever)
fn drain_cleanup_hooks() -> Vec<CleanupHook> {
const RETRIES: usize = 200;
const RETRY_DELAY: Duration = Duration::from_millis(5);
for _ in 0..RETRIES {
if let Ok(mut hooks) = CLEANUP_HOOKS.try_lock() {
return std::mem::take(&mut *hooks);
}
std::thread::sleep(RETRY_DELAY);
}
log::error!("Timed out waiting for the cleanup hook registry; skipping interrupt cleanup");
Vec::new()
}
/// Run drained hooks one by one, isolating panics so that one failing hook
/// cannot skip the remaining ones
fn run_drained_hooks(hooks: Vec<CleanupHook>) {
for CleanupHook { id, f } in hooks {
// Hooks are arbitrary user code; assert unwind safety so they can be
// run inside a catching context
if let Err(panic) = std::panic::catch_unwind(std::panic::AssertUnwindSafe(f)) {
log::error!("Cleanup hook {id} panicked: {}", panic_message(&panic));
}
}
}
/// Best-effort message extraction from a panic payload
fn panic_message(panic: &(dyn Any + Send)) -> String {
if let Some(s) = panic.downcast_ref::<&str>() {
(*s).to_string()
} else if let Some(s) = panic.downcast_ref::<String>() {
s.clone()
} else {
"non-string panic payload".to_string()
}
}
// ---------------------------------------------------------------------------
// Interrupt-time chroot cleanup
// ---------------------------------------------------------------------------
/// Interrupt-time cleanup of an ephemeral chroot: unmount every host-side
/// mount at or below `chroot_path` (the /proc bind mount, any overlay mounts)
/// and then remove the directory tree.
///
/// Unlike [`EphemeralContextGuard::drop`], this deliberately does NOT go
/// through the context manager, the ephemeral context's driver (whose
/// `cleanup()` unmounts the tracked overlays) or the base context's command
/// builder: the signal may arrive while the interrupted thread holds any of
/// those mutexes, and re-locking them from the signal handler would deadlock.
/// Instead it only reads /proc/mounts and spawns umount/rm directly.
///
/// It also differs from `drop` in that it removes the chroot regardless of
/// the build result: the build was aborted, and leaving a still-mounted
/// chroot behind is exactly the leak this hook exists to prevent.
///
/// Best-effort by design: if a child process still holds a mount busy or
/// privilege escalation is unavailable, individual steps fail; failures are
/// logged (pointing at `pkh prune` for the leftovers) and never panic.
fn sigint_cleanup_chroot(chroot_path: &Path) {
let is_root = unsafe { libc::geteuid() } == 0;
// Unmount children before parents: /proc/mounts lists mounts roughly in
// creation order, so walk it in reverse
let mounts = host_mounts_under(chroot_path);
for mount_point in mounts.into_iter().rev() {
if unmount_path(&mount_point, is_root) {
log::debug!(
"Unmounted {} during interrupt cleanup",
mount_point.display()
);
} else {
log::error!(
"Failed to unmount {} during interrupt cleanup; \
run `pkh prune` once the mount is free",
mount_point.display()
);
}
}
// Remove the chroot tree itself (tolerates a missing directory)
let status = privileged_command("rm", is_root)
.arg("-rf")
.arg(chroot_path)
.status();
match status {
Ok(status) if status.success() => {
log::debug!(
"Removed chroot {} during interrupt cleanup",
chroot_path.display()
);
}
Ok(status) => {
log::error!(
"Failed to remove chroot {} during interrupt cleanup \
(rm exited with {status}); run `pkh prune`",
chroot_path.display()
);
}
Err(e) => {
log::error!(
"Failed to run rm for chroot {} during interrupt cleanup: {e}; run `pkh prune`",
chroot_path.display()
);
}
}
}
/// Build a `Command` for `program`, wrapped in non-interactive sudo when not
/// running as root: interrupt cleanup must never block on a password prompt,
/// so without cached credentials the command fails fast and is logged instead
fn privileged_command(program: &str, is_root: bool) -> Command {
if is_root {
Command::new(program)
} else {
let mut cmd = Command::new("sudo");
cmd.arg("-n").arg(program);
cmd
}
}
/// Unmount `path`, falling back to a lazy unmount if the first attempt fails
/// because something still holds the mount busy (e.g. an interrupted child
/// that has not exited yet); returns whether the mount is gone
fn unmount_path(path: &Path, is_root: bool) -> bool {
if privileged_command("umount", is_root)
.arg(path)
.status()
.is_ok_and(|s| s.success())
{
return true;
}
privileged_command("umount", is_root)
.arg("-l")
.arg(path)
.status()
.is_ok_and(|s| s.success())
}
/// Collect the host-side mount points at or below `base`, in /proc/mounts
/// order (empty if /proc/mounts cannot be read)
fn host_mounts_under(base: &Path) -> Vec<PathBuf> {
let mut mounts = Vec::new();
let Ok(mounts_text) = fs::read_to_string("/proc/mounts") else {
return mounts;
};
// Compare against the canonical path: /proc/mounts shows resolved paths,
// while the chroot path may go through a symlinked TMPDIR
let base = base.canonicalize().unwrap_or_else(|_| base.to_path_buf());
for line in mounts_text.lines() {
let mut fields = line.split_whitespace();
let (Some(_device), Some(mount_point)) = (fields.next(), fields.next()) else {
continue;
};
let path = PathBuf::from(unescape_mount_field(mount_point));
if path.starts_with(&base) && !mounts.contains(&path) {
mounts.push(path);
}
}
mounts
}
/// Decode the octal escapes /proc/mounts uses in its path fields
/// (`\040` for space, `\011` for tab, `\012` for newline, `\134` for backslash)
fn unescape_mount_field(field: &str) -> String {
let bytes = field.as_bytes();
let mut out = Vec::with_capacity(bytes.len());
let mut i = 0;
while i < bytes.len() {
if bytes[i] == b'\\'
&& i + 4 <= bytes.len()
&& bytes[i + 1..i + 4]
.iter()
.all(|b| (b'0'..=b'7').contains(b))
&& let Ok(value) = u8::from_str_radix(&field[i + 1..i + 4], 8)
{
out.push(value);
i += 4;
} else {
out.push(bytes[i]);
i += 1;
}
}
String::from_utf8_lossy(&out).into_owned()
}
/// An ephemeral unshare context guard that creates and manages a temporary chroot environment
/// for building packages with unshare permissions.
pub struct EphemeralContextGuard {
previous_context: String,
/// The ephemeral build context this guard created (an unshare context
/// bound to the chroot, parented on the base context). Held explicitly so
/// cleanup and the build itself never depend on the process-global
/// "current" context, which concurrent builds swap for their own.
ephemeral_ctx: Arc<Context>,
/// The context that was current (globally) when this guard was created,
/// restored on drop. Saving the handle instead of a config name is what
/// keeps concurrent builds from restoring over each other.
previous_context: Arc<Context>,
chroot_path: PathBuf,
build_succeeded: bool,
base_ctx: Arc<Context>,
/// Registration of the interrupt-time cleanup hook; deregistered when
/// this guard drops, so the hook can never fire after the normal cleanup
cleanup_hook: Option<CleanupHookGuard>,
}
impl EphemeralContextGuard {
@@ -31,7 +314,11 @@ impl EphemeralContextGuard {
base_ctx: Arc<Context>,
ui: Option<Arc<DebUi>>,
) -> Result<Self, Box<dyn Error>> {
let current_context_name = context::manager().current_name();
// Save the globally-installed context so Drop can restore exactly
// this handle: concurrent builds install their own ephemeral
// overrides, so the only safe restoration value is the one observed
// before this guard swapped anything in.
let previous_context = context::current();
// Create a temporary directory for the chroot
let chroot_path_str = base_ctx.create_temp_dir()?;
@@ -44,24 +331,78 @@ impl EphemeralContextGuard {
chroot_path.display()
);
// Download and extract the chroot tarball
Self::download_and_extract_chroot(series, arch, &chroot_path, base_ctx.clone(), &ui)
.await?;
// Register the interrupt-time cleanup hook before any heavy work: if
// the user hits Ctrl-C during bootstrap or the build itself, the
// SIGINT handler unmounts and removes the chroot through this hook
// (see `sigint_cleanup_chroot`). This only works for a local base
// context: the hook must be self-contained (stored path + direct
// umount/rm subprocesses) and cannot go through `base_ctx`, whose
// driver mutex may be held by the interrupted thread. For remote or
// nested bases the chroot lives elsewhere, and leftovers stay
// handled by `pkh prune` as before.
let cleanup_hook = if matches!(base_ctx.config, ContextConfig::Local) {
Some(register_cleanup_hook(Box::new({
let chroot_path = chroot_path.clone();
move || sigint_cleanup_chroot(&chroot_path)
})))
} else {
log::debug!(
"Base context is not local; skipping interrupt-time cleanup registration for {}",
chroot_path.display()
);
None
};
// Switch to an ephemeral context to build the package in the chroot
context::manager().set_current_ephemeral(Context::new(ContextConfig::Unshare {
path: chroot_path.to_string_lossy().to_string(),
parent: Some(current_context_name.clone()),
}));
// Download and extract the chroot tarball
if let Err(e) =
Self::download_and_extract_chroot(series, arch, &chroot_path, base_ctx.clone(), &ui)
.await
{
// The guard (and its Drop) never materializes on this path, so
// stop tracking the chroot for interrupt cleanup; as before, a
// failed bootstrap leaves its partial directory in place.
drop(cleanup_hook);
return Err(e);
}
// Switch to an ephemeral context to build the package in the chroot.
// The parent is the base context itself (the one that bootstrapped
// the chroot), wired through `with_parent` instead of a config-name
// lookup, so an explicit non-current base (e.g. ssh) is used for
// everything that runs inside the chroot. The Arc stays in the
// guard: the build and the cleanup use it directly.
let ephemeral_ctx = Arc::new(Context::with_parent(
ContextConfig::Unshare {
path: chroot_path.to_string_lossy().to_string(),
// The real parent is bound below via `with_parent`; the
// config field is only used for contexts read from the
// persisted configuration.
parent: None,
},
base_ctx.clone(),
));
context::manager().set_current_ephemeral(ephemeral_ctx.clone());
Ok(Self {
previous_context: current_context_name,
previous_context,
ephemeral_ctx,
chroot_path,
build_succeeded: false,
base_ctx,
cleanup_hook,
})
}
/// The ephemeral build context created by this guard
///
/// Callers must take the context from here rather than from
/// [`crate::context::current()`]: the process-global is a shared swap
/// slot that another concurrent build may have re-pointed at its own
/// chroot, while this handle is guaranteed to be this guard's context.
pub fn context(&self) -> Arc<Context> {
Arc::clone(&self.ephemeral_ctx)
}
async fn download_and_extract_chroot(
series: &str,
arch: Option<&str>,
@@ -370,21 +711,33 @@ impl EphemeralContextGuard {
impl Drop for EphemeralContextGuard {
fn drop(&mut self) {
// Deregister the interrupt-time cleanup hook first: the normal
// cleanup below takes care of the chroot, so the hook must not fire
// afterwards. (If a SIGINT arrived mid-drop and the hook is already
// running concurrently, deregistration simply does not find it —
// both paths are individually idempotent and failure-tolerant.)
if let Some(mut cleanup_hook) = self.cleanup_hook.take() {
cleanup_hook.deregister();
}
log::debug!("Cleaning up ephemeral context ({:?})...", self.chroot_path);
// Clean up any overlay mounts before resetting the context.
// This must happen while the ephemeral context is still current so its
// driver is accessible. The actual unmount commands run via the parent
// Clean up any overlay mounts before resetting the context. This
// explicitly targets the context this guard created — never
// `context::current()`, which a concurrent build may have re-pointed
// at its own chroot. The actual unmount commands run via the parent
// (base) context, so they work regardless.
let ephemeral_ctx = context::current();
if let Err(e) = ephemeral_ctx.cleanup() {
if let Err(e) = self.ephemeral_ctx.cleanup() {
log::warn!("Failed to clean up overlay mounts: {}", e);
}
// Reset to normal context
if let Err(e) = context::manager().set_current(&self.previous_context) {
log::error!("Failed to restore context {}: {}", self.previous_context, e);
}
// Restore the context that was current when this guard was created,
// not whatever is globally current at drop time (another concurrent
// build's override may be installed there). This only swaps the
// in-memory handle: the persisted configuration still names the
// context selected by the user, as `set_current_ephemeral` never
// touches it.
context::manager().set_current_ephemeral(self.previous_context.clone());
// Remove chroot directory only if build succeeded
if self.build_succeeded {
@@ -453,3 +806,179 @@ impl Drop for EphemeralContextGuard {
}
}
}
#[cfg(test)]
mod cleanup_registry_tests {
use super::*;
use std::sync::atomic::AtomicUsize;
/// Serializes these tests: they drain the process-global registry, and
/// unrelated tests (e.g. live end-to-end builds) may hold registrations
/// concurrently that must be neither run nor lost. Poison-proof: a test
/// failing while holding the lock must not cascade into the others.
static TEST_LOCK: Mutex<()> = Mutex::new(());
fn test_lock() -> std::sync::MutexGuard<'static, ()> {
TEST_LOCK
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
}
/// Drain the registry and take out only the hooks with the given ids,
/// putting everything else back so unrelated registrations (e.g. hooks of
/// live end-to-end builds running concurrently) stay pending
fn take_hooks(ids: &[u64]) -> Vec<CleanupHook> {
let drained = drain_cleanup_hooks();
let mut mine = Vec::new();
let mut others = Vec::new();
for hook in drained {
if ids.contains(&hook.id) {
mine.push(hook);
} else {
others.push(hook);
}
}
CLEANUP_HOOKS.lock().unwrap().extend(others);
mine
}
/// Register a hook that counts its invocations
fn counting_hook() -> (CleanupHookGuard, Arc<AtomicUsize>) {
let counter = Arc::new(AtomicUsize::new(0));
let seen = counter.clone();
let guard = register_cleanup_hook(Box::new(move || {
seen.fetch_add(1, Ordering::SeqCst);
}));
(guard, counter)
}
/// Hooks run in registration order, and draining means each hook runs
/// exactly once even across repeated cleanup passes.
#[test]
fn hooks_run_once_in_registration_order() {
let _serial = test_lock();
let log = Arc::new(Mutex::new(Vec::new()));
let mut guards = Vec::new();
let mut ids = Vec::new();
for name in ["hook-a", "hook-b", "hook-c"] {
let log = log.clone();
// The returned guard must stay alive: dropping it deregisters
let guard = register_cleanup_hook(Box::new(move || log.lock().unwrap().push(name)));
ids.push(guard.id());
guards.push(guard);
}
// Only our own hooks are extracted; they run in registration order
let mine = take_hooks(&ids);
assert_eq!(mine.len(), ids.len());
run_drained_hooks(mine);
assert_eq!(*log.lock().unwrap(), vec!["hook-a", "hook-b", "hook-c"]);
// Draining removed them: a second pass runs nothing again
assert!(take_hooks(&ids).is_empty());
assert_eq!(*log.lock().unwrap(), vec!["hook-a", "hook-b", "hook-c"]);
drop(guards);
}
/// A panicking hook is contained by the runner: it neither aborts the
/// process nor skips the hooks registered around it.
#[test]
fn panicking_hook_does_not_skip_the_others() {
let _serial = test_lock();
let (before, ran_before) = counting_hook();
let boom = register_cleanup_hook(Box::new(|| panic!("cleanup exploded")));
let (after, ran_after) = counting_hook();
let ids = [before.id(), boom.id(), after.id()];
run_drained_hooks(take_hooks(&ids));
assert_eq!(ran_before.load(Ordering::SeqCst), 1);
assert_eq!(ran_after.load(Ordering::SeqCst), 1);
}
/// Explicit deregistration removes the hook: it is no longer drained and
/// never runs; a second deregistration reports it as already gone.
#[test]
fn deregistered_hook_never_runs() {
let _serial = test_lock();
let (mut guard, ran) = counting_hook();
assert!(guard.deregister());
assert!(!guard.deregister());
assert!(take_hooks(&[guard.id()]).is_empty());
assert_eq!(ran.load(Ordering::SeqCst), 0);
}
/// Dropping the registration guard deregisters the hook implicitly.
#[test]
fn dropping_the_guard_deregisters_the_hook() {
let _serial = test_lock();
let id;
let ran;
{
let (guard, counter) = counting_hook();
id = guard.id();
ran = counter;
drop(guard);
}
assert!(take_hooks(&[id]).is_empty());
assert_eq!(ran.load(Ordering::SeqCst), 0);
}
/// /proc/mounts path fields use octal escapes for whitespace and
/// backslashes; anything else must be kept verbatim.
#[test]
fn mount_field_unescaping_decodes_octal_escapes() {
assert_eq!(unescape_mount_field("/mnt/plain"), "/mnt/plain");
assert_eq!(
unescape_mount_field("/mnt/with\\040space"),
"/mnt/with space"
);
assert_eq!(unescape_mount_field("/mnt/with\\011tab"), "/mnt/with\ttab");
assert_eq!(unescape_mount_field("back\\134slash"), "back\\slash");
// Not an escape sequence: kept verbatim
assert_eq!(unescape_mount_field("back\\9slash"), "back\\9slash");
assert_eq!(unescape_mount_field("trailing\\"), "trailing\\");
}
/// Interrupt cleanup of a path that has no mounts and does not exist must
/// be a harmless no-op (no panic, nothing left behind).
#[test]
fn sigint_cleanup_of_missing_chroot_is_a_noop() {
let dir = tempfile::tempdir().unwrap();
let missing = dir.path().join("no-such-chroot");
sigint_cleanup_chroot(&missing);
assert!(!missing.exists());
}
/// A real directory with no mounts under it is simply removed. Skipped
/// when non-root without working non-interactive sudo, since removal then
/// legitimately fails (and is only logged).
#[test]
fn sigint_cleanup_removes_an_unmounted_directory() {
let is_root = unsafe { libc::geteuid() } == 0;
if !is_root
&& !privileged_command("true", false)
.status()
.is_ok_and(|s| s.success())
{
return;
}
let dir = tempfile::tempdir().unwrap();
let chroot = dir.path().join("chroot");
std::fs::create_dir_all(chroot.join("rootfs")).unwrap();
std::fs::write(chroot.join("rootfs").join("file.txt"), "data").unwrap();
sigint_cleanup_chroot(&chroot);
assert!(!chroot.exists());
}
}
+31 -25
View File
@@ -5,7 +5,7 @@ use crate::deb::find_dsc_file;
use crate::ui::deb::{DebUi, Phase};
use crate::ui::logfmt::QuiltClassifier;
use log::warn;
use std::collections::HashMap;
use std::collections::{BTreeMap, HashMap};
use std::error::Error;
use std::path::{Path, PathBuf};
use std::sync::Arc;
@@ -107,10 +107,15 @@ pub async fn build(
let new_source = crate::apt::sources::SourceEntry {
enabled: true,
kind: crate::apt::sources::SourceKind::Deb,
components: vec!["main".to_string()],
architectures: architectures.clone(),
signed_by: None,
trusted: None,
suite: suites,
uri: base_url,
// No origin: saved to the pkh-owned added-sources file
origin: None,
};
sources.push(new_source);
modified = true;
@@ -159,7 +164,9 @@ pub async fn build(
}
if modified {
apt::sources::save_legacy(Some(ctx.clone()), sources, "/etc/apt/sources.list")?;
// Each entry is written back to its origin file in its own format;
// new PPA entries go to the pkh-owned added-sources file
apt::sources::save(Some(ctx.clone()), sources)?;
// Download and import PPA keys for all added PPAs
for (user, ppa_name) in added_ppas {
@@ -413,10 +420,6 @@ fn generate_upload_metadata(
env: &HashMap<String, String>,
ctx: &Arc<Context>,
) -> Result<(PathBuf, PathBuf), Box<dyn Error>> {
let changelog_path = Path::new(package_dir).join("debian/changelog");
let changelog_content = ctx.read_file(&changelog_path)?;
let entry = crate::debian::parse_changelog_entry_from_str(&changelog_content)?;
// Build architecture: the machine inside the build context.
let build_arch = ctx
.command("dpkg")
@@ -433,34 +436,37 @@ fn generate_upload_metadata(
build_arch.clone()
};
// Vendor resolution inside the context (falls back to the host view).
// Vendor resolution inside the context (falls back to the host view);
// shared `Vendor:`/`Origin:` parsing with the source-build path.
let vendor = ctx
.read_file(Path::new("/etc/dpkg/origins/default"))
.ok()
.and_then(|content| {
for line in content.lines() {
if let Some(v) = line.strip_prefix("Vendor:") {
let v = v.trim();
if !v.is_empty() {
return Some(v.to_string());
}
}
}
None
})
.and_then(|content| crate::build::env::vendor_from_origins_content(&content))
.unwrap_or_else(crate::build::env::current_vendor);
let profiles = crate::build::env::resolve_build_profiles(&[], &vendor);
let source_date_epoch = env
.get("SOURCE_DATE_EPOCH")
.and_then(|v| v.parse::<i64>().ok())
.unwrap_or(entry.timestamp);
// The recorded profiles must describe what the build actually ran with:
// the DEB_BUILD_PROFILES exported to the build steps ('cross' for cross
// builds), else the vendor defaults.
let profiles = match env.get("DEB_BUILD_PROFILES") {
Some(value) => value
.split(',')
.map(|p| p.trim().to_string())
.filter(|p| !p.is_empty())
.collect(),
None => crate::build::env::resolve_build_profiles(&[], &vendor),
};
// Record exactly the environment exported to the build steps
// (DEB_BUILD_OPTIONS with the real parallel count and 'nocheck', LANG=C,
// SOURCE_DATE_EPOCH, cross DEB_* variables, ...), not values recomputed
// from host state; buildinfo_environment filters out non-dpkg variables.
let exported_env: BTreeMap<String, String> =
env.iter().map(|(k, v)| (k.clone(), v.clone())).collect();
let opts = crate::build::binary::BinaryMetadataOptions {
profiles,
vendor,
parallel: crate::build::env::num_parallel(),
source_date_epoch,
exported_env,
build_arch,
host_arch,
};
+15 -10
View File
@@ -1,5 +1,7 @@
mod cross;
mod ephemeral;
/// Ephemeral (per-build) unshare contexts, including the process-global
/// cleanup-hook registry drained by the SIGINT handler
pub(crate) mod ephemeral;
mod local;
use crate::context::{self, Context};
@@ -132,14 +134,17 @@ async fn build_binary_package_impl(
None
};
let result = async {
// Get the build context - either the ephemeral context or the base context
let build_ctx = if mode == BuildMode::Local {
context::current()
} else {
base_ctx.clone()
};
// Determine the build context explicitly: for Local builds it is the
// ephemeral context the guard just created (taken from the guard itself,
// never from the process-global, which concurrent builds may have
// re-pointed at their own chroot); otherwise the base context is used
// directly.
let build_ctx = match guard.as_ref() {
Some(g) => g.context(),
None => base_ctx.clone(),
};
let result = async {
// Prepare build directory
let build_root = build_ctx.create_temp_dir()?;
@@ -392,7 +397,7 @@ mod tests {
log::info!("Successfully pulled package {}", package);
// Create a fresh local context for this test
let ctx = Arc::new(Context::new(crate::context::ContextConfig::Local));
let ctx = Arc::new(Context::new(crate::context::ContextConfig::Local).unwrap());
// Change directory to the package directory
let cwd =
@@ -573,7 +578,7 @@ mod tests {
let temp_dir = tempfile::tempdir().unwrap();
let pkg_dir = create_indep_cross_test_source(temp_dir.path());
let ctx = Arc::new(Context::new(crate::context::ContextConfig::Local));
let ctx = Arc::new(Context::new(crate::context::ContextConfig::Local).unwrap());
crate::deb::build_binary_package(
Some("arm64"),
+136 -5
View File
@@ -12,7 +12,9 @@ use std::path::Path;
///
/// Values are stored with continuation-line breaks as `\n` and without the
/// leading whitespace of continuation lines. Serialization re-adds a single
/// leading space in front of every continuation line, matching dpkg output.
/// leading space in front of every continuation line, matching dpkg output;
/// blank lines inside a value are encoded as ` .` (and decoded back) so they
/// survive a write/parse round-trip.
#[derive(Debug, Clone, Default, PartialEq)]
pub struct Paragraph {
fields: Vec<(String, String)>,
@@ -66,7 +68,9 @@ impl Paragraph {
///
/// Comment lines (starting with `#`) are ignored. Blank lines separate
/// paragraphs. Continuation lines must start with a space or a tab; exactly
/// one leading space (or tab) is stripped from the stored value.
/// one leading space (or tab) is stripped from the stored value, and a
/// continuation whose content is a lone `.` decodes to an empty line
/// (dpkg's encoding for blank lines inside field values).
pub fn parse_paragraphs(input: &str) -> Vec<Paragraph> {
let mut paragraphs = Vec::new();
let mut current = Paragraph::new();
@@ -89,7 +93,14 @@ pub fn parse_paragraphs(input: &str) -> Vec<Paragraph> {
// Continuation line
if line.starts_with(' ') || line.starts_with('\t') {
let content = line.strip_prefix(' ').unwrap_or(line);
// Exactly one leading space or tab is stripped.
let content = line
.strip_prefix(' ')
.or_else(|| line.strip_prefix('\t'))
.unwrap_or(line);
// dpkg encodes a blank line inside a value as a lone `.` after
// the leading whitespace; mirror that on read.
let content = if content == "." { "" } else { content };
if let Some(field) = &last_field
&& let Some((_, v)) = current
.fields
@@ -147,14 +158,65 @@ pub fn write_paragraph(p: &Paragraph) -> String {
}
for line in lines {
out.push('\n');
out.push(' ');
out.push_str(line);
if line.is_empty() {
// dpkg encodes a blank line inside a value as ` .`; writing a
// bare continuation line would be mistaken for a paragraph
// separator on re-parse and silently drop the rest.
out.push_str(" .");
} else {
out.push(' ');
out.push_str(line);
}
}
out.push('\n');
}
out
}
/// Return the signed body of a clearsigned message, as a slice of `text`.
///
/// If `text` starts with the OpenPGP clearsigned-marker line, the armor
/// header block (the `Hash: ...` line and any `Comment:` lines, up to and
/// including the blank line that closes the header) is skipped, and the
/// result is cut at the `-----BEGIN PGP SIGNATURE-----` marker so the
/// signature trailer is dropped as well. This keeps the armor metadata from
/// being parsed as deb822 fields (`Hash:` would otherwise land in the first
/// stanza and `Comment:` in the last one).
///
/// Input that is not clearsigned is returned unchanged, so callers can apply
/// this unconditionally before parsing.
pub fn strip_clearsigned_armour(text: &str) -> &str {
const BEGIN_SIGNED: &str = "-----BEGIN PGP SIGNED MESSAGE-----";
const BEGIN_SIGNATURE: &str = "-----BEGIN PGP SIGNATURE-----";
if !text.starts_with(BEGIN_SIGNED) {
return text;
}
// Walk past the armor headers to the blank line that precedes the body.
let mut body = text;
loop {
match body.split_once('\n') {
Some((line, remainder)) => {
body = remainder;
// An empty line ends the armor header block (`\r` covers a
// CRLF-terminated blank line).
if line.is_empty() || line == "\r" {
break;
}
}
// Malformed armor: no body at all.
None => return "",
}
}
// Cut off the signature block, if present.
match body.find(BEGIN_SIGNATURE) {
Some(i) => &body[..i],
None => body,
}
}
#[cfg(test)]
mod tests {
use super::*;
@@ -197,6 +259,75 @@ mod tests {
assert_eq!(reparsed[0].get("Description"), Some(value));
}
#[test]
fn blank_lines_survive_roundtrip() {
let mut p = Paragraph::new();
p.set("Description", "a\n\nb");
let text = write_paragraph(&p);
// dpkg encoding: a blank line inside a value is written as ` .`.
assert_eq!(text, "Description: a\n .\n b\n");
// parse -> write -> parse must not lose data.
let reparsed = parse_paragraphs(&text);
assert_eq!(reparsed[0].get("Description"), Some("a\n\nb"));
assert_eq!(
parse_paragraphs(&write_paragraph(&reparsed[0]))[0].get("Description"),
Some("a\n\nb")
);
}
#[test]
fn parse_lone_dot_continuation_is_blank_line() {
let paras = parse_paragraphs("Description:\n a\n .\n b\n");
assert_eq!(paras[0].get("Description"), Some("\na\n\nb"));
}
#[test]
fn tab_continuation_strips_exactly_one_tab() {
let paras = parse_paragraphs("Description: a\n\tb\n\t\tdeep\n");
assert_eq!(paras[0].get("Description"), Some("a\nb\n\tdeep"));
}
#[test]
fn strip_armour_extracts_signed_dsc_body() {
let signed = "\
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Format: 3.0 (native)
Source: hello
Binary: hello
Architecture: any
Version: 1.0-1
Checksums-Sha256:
abc 100 hello_1.0.tar.gz
-----BEGIN PGP SIGNATURE-----
iQEcBAABCgAGBQJabcdAAoJEL abc
-----END PGP SIGNATURE-----
";
let body = strip_clearsigned_armour(signed);
assert!(body.starts_with("Format:"));
assert!(!body.contains("SIGNATURE"));
let paras = parse_paragraphs(body);
assert_eq!(paras.len(), 1);
// The armor `Hash:` header must not land in the stanza...
assert!(paras[0].get("Hash").is_none());
assert_eq!(paras[0].get("Source"), Some("hello"));
// ...and the signature trailer must not contribute a `Comment:` field.
assert!(paras[0].get("Comment").is_none());
assert_eq!(
paras[0].get("Checksums-Sha256"),
Some("\nabc 100 hello_1.0.tar.gz")
);
}
#[test]
fn strip_armour_passes_unsigned_text_through() {
let plain = "Source: hello\nVersion: 1.0\n";
assert_eq!(strip_clearsigned_armour(plain), plain);
}
#[test]
fn set_replaces_case_insensitive() {
let mut p = Paragraph::new();
+58 -4
View File
@@ -187,12 +187,16 @@ pub fn parse_simple(dep: &str, build_dep: bool) -> Result<PkgRelation, String> {
let constraint = match (caps.get(5), caps.get(6)) {
(Some(op), Some(version)) => {
// The deprecated single-character spellings `<` and `>` were
// "confusingly defined" (Debian Policy §7) to mean earlier-or-equal
// and later-or-equal, i.e. `<=` and `>=`; dpkg still accepts them
// with those non-strict semantics.
let relation = match op.as_str() {
"<<" | "<" => Relation::Lt,
"<=" => Relation::Le,
"<<" => Relation::Lt,
"<" | "<=" => Relation::Le,
"=" => Relation::Eq,
">=" => Relation::Ge,
">>" | ">" => Relation::Gt,
">" | ">=" => Relation::Ge,
">>" => Relation::Gt,
other => return Err(format!("invalid relation '{other}' in '{dep}'")),
};
let version = DebianVersion::parse(version.as_str())
@@ -1024,6 +1028,56 @@ mod tests {
assert!(Deps::parse("foo:native", &opts("amd64", &[])).is_ok());
}
/// The deprecated single-character operators `<` and `>` mean `<=` and
/// `>=` (Debian Policy §7, dpkg behavior), and canonicalize on output.
#[test]
fn legacy_single_char_relations() {
let rel = |dep: &str| {
parse_simple(dep, true)
.unwrap()
.constraint
.unwrap()
.relation
};
assert_eq!(rel("foo (<< 1.0)"), Relation::Lt);
assert_eq!(rel("foo (< 1.0)"), Relation::Le);
assert_eq!(rel("foo (<= 1.0)"), Relation::Le);
assert_eq!(rel("foo (= 1.0)"), Relation::Eq);
assert_eq!(rel("foo (>= 1.0)"), Relation::Ge);
assert_eq!(rel("foo (> 1.0)"), Relation::Ge);
assert_eq!(rel("foo (>> 1.0)"), Relation::Gt);
// Rendering always uses the canonical modern spellings.
for (dep, rendered) in [
("foo (< 1.0)", "foo (<= 1.0)"),
("foo (> 1.0)", "foo (>= 1.0)"),
("foo (<< 1.0)", "foo (<< 1.0)"),
("foo (>> 1.0)", "foo (>> 1.0)"),
] {
assert_eq!(parse_simple(dep, true).unwrap().output(), rendered);
}
}
/// `foo (< 1.0)` is satisfied by an installed `foo 1.0` (the legacy
/// operator is non-strict); `foo (< 0.9)` is not.
#[test]
fn legacy_single_char_evaluation() {
let mut facts = Facts::new("amd64", "amd64");
facts.add_installed("foo", "1.0", "amd64", "no");
let o = |s: &str| parse_simple(s, true).unwrap();
assert_eq!(facts.evaluate_relation(&o("foo (< 1.0)")), Some(true));
assert_eq!(facts.evaluate_relation(&o("foo (<= 1.0)")), Some(true));
assert_eq!(facts.evaluate_relation(&o("foo (< 0.9)")), Some(false));
assert_eq!(facts.evaluate_relation(&o("foo (<< 1.0)")), Some(false));
assert_eq!(facts.evaluate_relation(&o("foo (> 1.0)")), Some(true));
assert_eq!(facts.evaluate_relation(&o("foo (>= 1.0)")), Some(true));
assert_eq!(facts.evaluate_relation(&o("foo (> 1.1)")), Some(false));
assert_eq!(facts.evaluate_relation(&o("foo (>> 1.0)")), Some(false));
}
/// Ported from dpkg `t/Dpkg_Deps.t`: architecture reduction.
#[test]
fn arch_reduction() {
+3 -1
View File
@@ -24,6 +24,8 @@ pub use changelog::{
parse_previous_version_from_str,
};
pub use checksums::{Entry as ChecksumEntry, FileChecksums};
pub use control::{ControlInfo, Paragraph, parse_paragraphs, write_paragraph};
pub use control::{
ControlInfo, Paragraph, parse_paragraphs, strip_clearsigned_armour, write_paragraph,
};
pub use files::{FilesEntry, FilesList};
pub use version::DebianVersion;
+158 -26
View File
@@ -3,6 +3,7 @@ use lazy_static::lazy_static;
use serde::Deserialize;
use std::error::Error;
use std::path::Path;
use std::time::Duration;
#[derive(Debug, Clone)]
/// Information about a specific distribution series
@@ -13,8 +14,8 @@ pub struct SeriesInformation {
pub codename: String,
/// Series version as numbers
pub version: Option<String>,
/// Series creation date
pub created: NaiveDate,
/// Series creation date (absent if missing or invalid in the CSV data)
pub created: Option<NaiveDate>,
/// Series release date
pub release: Option<NaiveDate>,
/// Series end-of-life date
@@ -42,7 +43,45 @@ struct Data {
const DATA_YAML: &str = include_str!("../distro_info.yml");
lazy_static! {
static ref DATA: Data = serde_yaml::from_str(DATA_YAML).unwrap();
// The YAML is include_str!'d at compile time and statically valid; if it
// ever failed to parse it would be a build-time bug that cannot be
// recovered from at runtime, so panicking here is acceptable.
static ref DATA: Data = serde_yaml::from_str(DATA_YAML)
.expect("built-in distro_info.yml data is statically valid and must parse");
// Shared HTTP client used for all outgoing plain requests: timeouts keep
// a hanging remote (connect or transfer) from stalling pkh indefinitely.
static ref HTTP_CLIENT: reqwest::Client = reqwest::Client::builder()
.connect_timeout(Duration::from_secs(10))
.timeout(Duration::from_secs(30))
.build()
.expect("building the shared HTTP client with static options cannot fail");
}
/// Shared HTTP client with a connect timeout (10s) and a total request
/// timeout (30s), to be used for all outgoing plain HTTP(S) requests
pub(crate) fn http_client() -> &'static reqwest::Client {
&HTTP_CLIENT
}
/// Parse an optional '%Y-%m-%d' date from a CSV cell, warning instead of
/// panicking on invalid remote data
fn parse_optional_date(value: Option<&str>, series: &str, field: &str) -> Option<NaiveDate> {
value.and_then(
|date_str| match NaiveDate::parse_from_str(date_str, "%Y-%m-%d") {
Ok(date) => Some(date),
Err(e) => {
log::warn!(
"Invalid '{}' date '{}' for series '{}': {}. Ignoring the date.",
field,
date_str,
series,
e
);
None
}
},
)
}
fn parse_series_csv(content: &str) -> Result<Vec<SeriesInformation>, Box<dyn Error>> {
@@ -79,24 +118,39 @@ fn parse_series_csv(content: &str) -> Result<Vec<SeriesInformation>, Box<dyn Err
let mut series_info_list = Vec::new();
for result in rdr.records() {
let record = result?;
let series = record.get(series_idx).unwrap().to_string();
let codename = record.get(codename_idx).unwrap().to_string();
let record = match result {
Ok(record) => record,
Err(e) => {
log::warn!("Skipping malformed series CSV row: {}", e);
continue;
}
};
// Rows missing essential identification fields are skipped: they
// cannot be used nor reported meaningfully. Dates, on the other
// hand, are all optional in the model, so a bad date keeps the row.
let Some(series) = record.get(series_idx).filter(|s| !s.is_empty()) else {
log::warn!(
"Skipping series CSV row without a 'series' value: {:?}",
record
);
continue;
};
let Some(codename) = record.get(codename_idx).filter(|s| !s.is_empty()) else {
log::warn!(
"Skipping series CSV row for series '{}' without a 'codename' value",
series
);
continue;
};
let version = record.get(version_idx).map(|s| s.to_string());
let created = record
.get(created_idx)
.map(|date_str| NaiveDate::parse_from_str(date_str, "%Y-%m-%d").unwrap())
.unwrap();
let release = record
.get(release_idx)
.map(|date_str| NaiveDate::parse_from_str(date_str, "%Y-%m-%d").unwrap());
let eol = record
.get(eol_idx)
.map(|date_str| NaiveDate::parse_from_str(date_str, "%Y-%m-%d").unwrap());
let created = parse_optional_date(record.get(created_idx), series, "created");
let release = parse_optional_date(record.get(release_idx), series, "release");
let eol = parse_optional_date(record.get(eol_idx), series, "eol");
series_info_list.push(SeriesInformation {
series,
codename,
series: series.to_string(),
codename: codename.to_string(),
version,
created,
release,
@@ -134,7 +188,9 @@ pub async fn get_ordered_series(dist: &str) -> Result<Vec<SeriesInformation>, Bo
)
})?
} else {
reqwest::get(series_info.network.as_str())
http_client()
.get(series_info.network.as_str())
.send()
.await?
.text()
.await?
@@ -203,9 +259,13 @@ pub async fn get_dist_from_series(series: &str) -> Result<String, Box<dyn Error>
Err(format!("Unknown series: {}", series).into())
}
/// Get the package pockets available for a given distribution
/// Get the package pockets available for a given distribution, in search order
///
/// Example: get_dist_pockets(ubuntu) => ["proposed", "updates", ""]
/// The main archive ('') comes first so that a search without an explicit
/// pocket prefers the released archive over its pockets; development pockets
/// (e.g. '-proposed') come last.
///
/// Example: get_dist_pockets(ubuntu) => ["", "updates", "security", "proposed"]
pub fn get_dist_pockets(dist: &str) -> Result<Vec<String>, Box<dyn Error>> {
let dist_data = DATA.dist.get(dist).ok_or_else(|| {
format!(
@@ -216,8 +276,8 @@ pub fn get_dist_pockets(dist: &str) -> Result<Vec<String>, Box<dyn Error>> {
})?;
let mut pockets = dist_data.pockets.clone();
// Explicitely add 'main' pocket, which is just the empty string
pockets.push("".to_string());
// Explicitely add 'main' pocket, which is just the empty string, first
pockets.insert(0, "".to_string());
Ok(pockets)
}
@@ -234,7 +294,7 @@ pub fn get_sources_url(base_url: &str, series: &str, pocket: &str, component: &s
/// Get the archive base URL for a distribution
///
/// Example: ubuntu => http://archive.ubuntu.com/ubuntu
/// Example: ubuntu => https://archive.ubuntu.com/ubuntu
pub fn get_base_url(dist: &str) -> Result<String, Box<dyn Error>> {
DATA.dist
.get(dist)
@@ -320,7 +380,7 @@ pub async fn get_components(
let url = get_release_url(base_url, series, pocket);
log::debug!("Fetching Release file from: {}", url);
let content = reqwest::get(&url).await?.text().await?;
let content = http_client().get(&url).send().await?.text().await?;
for line in content.lines() {
if line.starts_with("Components:")
@@ -355,7 +415,9 @@ pub async fn get_debian_series_number(series: &str) -> Result<Option<String>, Bo
)
})?
} else {
reqwest::get(series_info.network.as_str())
http_client()
.get(series_info.network.as_str())
.send()
.await?
.text()
.await?
@@ -391,6 +453,76 @@ pub async fn get_debian_series_number(series: &str) -> Result<Option<String>, Bo
mod tests {
use super::*;
#[test]
fn test_parse_series_csv_malformed_rows() {
// A short row (missing 'codename') is skipped, a row with an invalid
// 'created' date is kept without a date, and invalid 'release'/'eol'
// dates become None: none of this may panic on remote data
let csv_data = "series,codename,version,created,release,eol\n\
noble,Noble N,24.04,2023-10-26,2024-04-25,2029-04-25\n\
lonely\n\
badbad,Bad B,1.0,not-a-date,2020-01-01,also-bad\n\
sid,sid,unstable,1999-01-01,,\n";
let series = parse_series_csv(csv_data).unwrap();
// Rows are returned most recent first (the parser reverses the list),
// with the malformed 'lonely' row skipped entirely
let names: Vec<&str> = series.iter().map(|s| s.series.as_str()).collect();
assert_eq!(names, vec!["sid", "badbad", "noble"]);
let noble = &series[2];
assert_eq!(noble.codename, "Noble N");
assert_eq!(noble.version.as_deref(), Some("24.04"));
assert_eq!(
noble.created,
Some(NaiveDate::from_ymd_opt(2023, 10, 26).unwrap())
);
assert_eq!(
noble.release,
Some(NaiveDate::from_ymd_opt(2024, 4, 25).unwrap())
);
assert_eq!(
noble.eol,
Some(NaiveDate::from_ymd_opt(2029, 4, 25).unwrap())
);
let badbad = &series[1];
assert_eq!(badbad.created, None);
assert_eq!(
badbad.release,
Some(NaiveDate::from_ymd_opt(2020, 1, 1).unwrap())
);
assert_eq!(badbad.eol, None);
}
#[test]
fn test_get_dist_pockets_order() {
// Without an explicit pocket, packages are searched in this order:
// main archive first, then updates, security, and proposed last
let pockets = get_dist_pockets("ubuntu").unwrap();
assert_eq!(
pockets,
vec![
"".to_string(),
"updates".to_string(),
"security".to_string(),
"proposed".to_string()
]
);
let pockets = get_dist_pockets("debian").unwrap();
assert_eq!(
pockets,
vec![
"".to_string(),
"updates".to_string(),
"security".to_string(),
"proposed-updates".to_string()
]
);
}
#[tokio::test]
async fn test_get_debian_series() {
let series = get_ordered_series_name("debian").await.unwrap();
+6 -6
View File
@@ -47,7 +47,7 @@ fn main() {
)
.arg(arg!(-v --version <version> "Target package version").required(false))
.arg(arg!(--archive "Only use the archive to download package source, not git").required(false))
.arg(arg!(--ppa <ppa> "Download the package from a specific PPA").required(false))
.arg(arg!(--ppa <ppa> "Download the package from a specific PPA (format: user/ppa_name)").required(false))
.arg(arg!(--repository <url> "Download the package from an external flat repository, given as its full suite URL (e.g. https://pkg.noctalia.dev/deb/resolute/)").required(false)
.conflicts_with("ppa"))
.arg(arg!(-p --pocket <pocket> "Target package distribution pocket (updates, security, proposed)").required(false))
@@ -152,14 +152,14 @@ fn main() {
let (pb, progress_callback) = pkh::ui::create_progress_bar(&multi);
// Convert PPA to base URL if provided
let base_url = ppa.and_then(|ppa_str| {
let base_url = ppa.map(|ppa_str| {
// PPA format: user/ppa_name
let parts: Vec<&str> = ppa_str.split('/').collect();
if parts.len() == 2 {
Some(pkh::package_info::ppa_to_base_url(parts[0], parts[1]))
} else {
None
if parts.len() != 2 || parts[0].is_empty() || parts[1].is_empty() {
error!("Invalid PPA format: '{}'. Expected: user/ppa_name", ppa_str);
std::process::exit(1);
}
pkh::package_info::ppa_to_base_url(parts[0], parts[1])
});
// Since pull is async, we need to block on it
+284 -81
View File
@@ -5,6 +5,7 @@ use std::io::Read;
use xz2::read::XzDecoder;
use crate::ProgressCallback;
use crate::apt::release::{self, VerifiedRelease};
use crossterm::style::Stylize;
use log::{debug, warn};
@@ -17,7 +18,7 @@ use log::{debug, warn};
/// # Returns
/// * The base URL for the PPA (e.g., "https://ppa.launchpadcontent.net/user/ppa_name/ubuntu/")
pub fn ppa_to_base_url(user: &str, name: &str) -> String {
format!("http://ppa.launchpadcontent.net/{}/{}/ubuntu", user, name)
format!("https://ppa.launchpadcontent.net/{}/{}/ubuntu", user, name)
}
fn check_launchpad_repo_sync(package: &str) -> Result<Option<String>, String> {
@@ -189,82 +190,98 @@ impl Iterator for DebianSources {
type Item = PackageStanza;
fn next(&mut self) -> Option<Self::Item> {
let stanza = self.splitted_sources.next()?;
// Iterate over stanzas in a loop: package-less (blank) stanzas are
// skipped without recursion, so that a crafted index with many
// consecutive blank stanzas cannot blow the stack
loop {
let stanza = self.splitted_sources.next()?;
// Parse stanza into a hashmap of strings, the fields
let mut fields: HashMap<String, String> = HashMap::new();
let mut current_key = String::new();
// Parse stanza into a hashmap of strings, the fields
let mut fields: HashMap<String, String> = HashMap::new();
let mut current_key = String::new();
for line in stanza.lines() {
if line.is_empty() {
for line in stanza.lines() {
if line.is_empty() {
continue;
}
if line.starts_with(' ') || line.starts_with('\t') {
// Continuation line
if let Some(val) = fields.get_mut(&current_key) {
val.push('\n');
val.push_str(line.trim());
}
} else if let Some((key, value)) = line.split_once(':') {
current_key = key.trim().to_string();
fields.insert(current_key.clone(), value.trim().to_string());
}
}
let Some(package) = fields.get("Package") else {
// Skip empty stanza
continue;
}
};
let package = package.to_string();
if line.starts_with(' ') || line.starts_with('\t') {
// Continuation line
if let Some(val) = fields.get_mut(&current_key) {
val.push('\n');
val.push_str(line.trim());
}
} else if let Some((key, value)) = line.split_once(':') {
current_key = key.trim().to_string();
fields.insert(current_key.clone(), value.trim().to_string());
}
}
// A stanza without a version is malformed remote data: skip it
// rather than panicking
let Some(version) = fields.get("Version") else {
debug!(
"Skipping malformed stanza for package '{}' without a 'Version' field",
package
);
continue;
};
let version = version.to_string();
let pkg = fields.get("Package");
if pkg.is_none() {
// Skip empty stanza
return self.next();
}
// Parse package files.
// Prefer the strongest available checksum field: Checksums-Sha256,
// then Checksums-Sha512, then the legacy 'Files' (MD5) field.
// Some archives (e.g. the Ubuntu development series) no longer ship
// Checksums-Sha256, so falling back is required to keep working.
let mut files = Vec::new();
let (checksum_field, algo) = if fields.contains_key("Checksums-Sha256") {
("Checksums-Sha256", ChecksumAlgo::Sha256)
} else if fields.contains_key("Checksums-Sha512") {
("Checksums-Sha512", ChecksumAlgo::Sha512)
} else {
("Files", ChecksumAlgo::Md5)
};
if let Some(checksums) = fields.get(checksum_field) {
for line in checksums.lines() {
let parts: Vec<&str> = line.split_whitespace().collect();
if parts.len() >= 3 {
files.push(FileEntry {
checksum: parts[0].to_string(),
size: parts[1].parse().unwrap_or(0),
name: parts[2].to_string(),
checksum_algo: algo,
});
// Parse package files.
// Prefer the strongest available checksum field: Checksums-Sha256,
// then Checksums-Sha512, then the legacy 'Files' (MD5) field.
// Some archives (e.g. the Ubuntu development series) no longer ship
// Checksums-Sha256, so falling back is required to keep working.
let mut files = Vec::new();
let (checksum_field, algo) = if fields.contains_key("Checksums-Sha256") {
("Checksums-Sha256", ChecksumAlgo::Sha256)
} else if fields.contains_key("Checksums-Sha512") {
("Checksums-Sha512", ChecksumAlgo::Sha512)
} else {
("Files", ChecksumAlgo::Md5)
};
if let Some(checksums) = fields.get(checksum_field) {
for line in checksums.lines() {
let parts: Vec<&str> = line.split_whitespace().collect();
if parts.len() >= 3 {
files.push(FileEntry {
checksum: parts[0].to_string(),
size: parts[1].parse().unwrap_or(0),
name: parts[2].to_string(),
checksum_algo: algo,
});
}
}
}
// Parse Vcs-Git field: it may contain just a URL, or URL followed by -b <branch>
// e.g., "https://salsa.debian.org/science-team/paraview.git -b debian/latest"
let vcs_git = fields.get("Vcs-Git").map(|vcs| {
// Split on whitespace and take the first part (the URL)
// The URL should not contain spaces, so this is safe
vcs.split_whitespace().next().unwrap_or(vcs).to_string()
});
return Some(PackageStanza {
package,
version,
directory: fields.get("Directory").cloned().unwrap_or_default(),
format: fields
.get("Format")
.cloned()
.unwrap_or_else(|| "1.0".to_string()),
vcs_git,
vcs_browser: fields.get("Vcs-Browser").cloned(),
files,
});
}
// Parse Vcs-Git field: it may contain just a URL, or URL followed by -b <branch>
// e.g., "https://salsa.debian.org/science-team/paraview.git -b debian/latest"
let vcs_git = fields.get("Vcs-Git").map(|vcs| {
// Split on whitespace and take the first part (the URL)
// The URL should not contain spaces, so this is safe
vcs.split_whitespace().next().unwrap_or(vcs).to_string()
});
Some(PackageStanza {
package: fields.get("Package").unwrap().to_string(),
version: fields.get("Version").unwrap().to_string(),
directory: fields.get("Directory").cloned().unwrap_or_default(),
format: fields
.get("Format")
.cloned()
.unwrap_or_else(|| "1.0".to_string()),
vcs_git,
vcs_browser: fields.get("Vcs-Browser").cloned(),
files,
})
}
}
@@ -315,29 +332,86 @@ async fn get(
// If using a custom base URL (PPA), disable VCS lookup to force archive download
let from_ppa = base_url != distro_base_url;
let components = crate::distro_info::get_components(&base_url, series, pocket).await?;
// Authenticate the metadata of the suite before trusting any index
// fetched from it: the signed Release file (InRelease, or Release plus
// detached Release.gpg) is verified against the archive keyring (or the
// PPA signing key), and each Sources index downloaded below is
// checksum-checked against it before parsing.
let suite = if pocket.is_empty() {
series.to_string()
} else {
format!("{series}-{pocket}")
};
let suite_url = format!("{base_url}/dists/{suite}");
let keyring_source = if from_ppa {
release::KeyringSource::Ppa {
base_url: base_url.clone(),
}
} else {
release::KeyringSource::Distro {
series: series.to_string(),
}
};
debug!("Verifying the Release file of: {}", suite_url);
let verified = match release::verify_suite(&suite_url, keyring_source, true).await {
Ok(release::Verification::Available(verified)) => verified,
Ok(release::Verification::Unavailable { .. }) => {
// The suite does not exist: same outcome as before verification
// existed (probing callers simply try the next pocket/series)
return Err(format!("No Release file found for suite '{suite}' at {base_url}").into());
}
Ok(release::Verification::KeyringUnavailable { reason, .. }) => {
// An existing suite whose keys are unavailable must not be
// trusted, but this is not tampering: report and let callers
// move on
return Err(reason.into());
}
Err(e) => return Err(e.into()),
};
let components = verified.components();
if components.is_empty() {
return Err(format!("Components not found in the Release file of '{suite_url}'").into());
}
debug!("Found components: {:?}", components);
// Collect the failures of individual fetch attempts so that, if the
// package is not found, the final error explains what actually went
// wrong instead of misleadingly claiming a plain 'not found'
let mut fetch_errors: Vec<String> = Vec::new();
for component in components {
let url = crate::distro_info::get_sources_url(&base_url, series, pocket, &component);
debug!("Fetching sources from: {}", url);
let response = match reqwest::get(&url).await {
let response = match crate::distro_info::http_client().get(&url).send().await {
Ok(resp) => resp,
Err(e) => {
debug!("Failed to fetch {}: {}", url, e);
fetch_errors.push(format!("{suite}/{component}: {}", e));
continue;
}
};
if !response.status().is_success() {
debug!("Failed to fetch {}: status {}", url, response.status());
fetch_errors.push(format!("{suite}/{component}: HTTP {}", response.status()));
continue;
}
let compressed_data = response.bytes().await?;
// The index must match the checksums listed in the signed Release
// file: this is what closes the 'substituted index with matching
// artifact checksums' man-in-the-middle attack
let suite_rel_path = format!("{component}/source/Sources.gz");
verified
.verify_file(&suite_rel_path, &compressed_data)
.map_err(release::VerifyError)?;
debug!(
"Downloaded Sources.gz for {}/{}/{}",
dist, series, component
@@ -366,9 +440,14 @@ async fn get(
}
}
let details = if fetch_errors.is_empty() {
String::new()
} else {
format!(" (last errors: {})", fetch_errors.join("; "))
};
Err(format!(
"Package '{}' not found in {}/{}",
package_name, dist, series
"Package '{}' not found in {}/{}{}",
package_name, dist, series, details
)
.into())
}
@@ -385,6 +464,11 @@ async fn find_package(
) -> Result<PackageInfo, Box<dyn Error>> {
let series_list = crate::distro_info::get_ordered_series_name(dist).await?;
// Collect the failures of the individual series/pocket probes so that,
// if nothing is found, the final error summarizes what went wrong
// (e.g. network errors, HTTP statuses) instead of a bare 'not found'
let mut attempt_errors: Vec<String> = Vec::new();
for (i, series) in series_list.iter().enumerate() {
if let Some(cb) = progress {
cb("", &format!("Checking {}...", series), i, series_list.len());
@@ -428,14 +512,39 @@ async fn find_package(
}
return Ok(info);
}
Err(_e) => {
Err(e) => {
// A Release verification failure is a security error,
// not a missing package: abort the search instead of
// silently probing other series/pockets
if e.downcast_ref::<release::VerifyError>().is_some() {
return Err(e);
}
// Remember the failure for the final error message, and
// keep probing the other series/pockets
let suite = if p.is_empty() {
series.clone()
} else {
format!("{series}-{p}")
};
attempt_errors.push(format!("{}: {}", suite, e));
continue;
}
}
}
}
Err(format!("Package '{}' not found.", package_name).into())
// Keep only the last few attempts so the message stays readable
if attempt_errors.len() > 5 {
let drain_to = attempt_errors.len() - 5;
attempt_errors.drain(..drain_to);
}
let details = if attempt_errors.is_empty() {
String::new()
} else {
format!(" (last errors: {})", attempt_errors.join("; "))
};
Err(format!("Package '{}' not found.{}", package_name, details).into())
}
/// Fetch the 'Release' file at the root of a flat repository, and return its suite name
@@ -445,7 +554,7 @@ async fn find_package(
/// read from the 'Codename' field, falling back to 'Suite'.
async fn get_flat_repo_series(base_url: &str) -> Result<String, Box<dyn Error>> {
let url = format!("{}/Release", base_url.trim_end_matches('/'));
let response = reqwest::get(&url).await?;
let response = crate::distro_info::http_client().get(&url).send().await?;
if !response.status().is_success() {
return Err(format!(
"No Release file at '{}' (HTTP {}) - is '{}' the suite URL of a flat repository? \
@@ -471,15 +580,39 @@ async fn get_flat_repo_series(base_url: &str) -> Result<String, Box<dyn Error>>
/// Fetch the sources index of a flat repository
///
/// Flat repositories are free to serve any compressed variant of the index
/// (or an uncompressed one), so try the usual candidates in turn.
async fn get_flat_repo_sources(base_url: &str) -> Result<Vec<u8>, Box<dyn Error>> {
/// (or an uncompressed one), so try the usual candidates in turn. When the
/// repository published a Release file, each index candidate is
/// checksum-verified against it (failing on mismatch, since the artifact
/// hashes would come from the index itself).
async fn get_flat_repo_sources(
base_url: &str,
verified: Option<&VerifiedRelease>,
) -> Result<Vec<u8>, Box<dyn Error>> {
let base = base_url.trim_end_matches('/');
let mut errors = Vec::new();
for name in ["Sources.xz", "Sources.gz", "Sources"] {
let url = format!("{base}/{name}");
match reqwest::get(&url).await {
match crate::distro_info::http_client().get(&url).send().await {
Ok(response) if response.status().is_success() => {
return Ok(response.bytes().await?.to_vec());
let data = response.bytes().await?.to_vec();
// Some flat Release files list their entries with a './' prefix
if let Some(release) = verified {
let mismatch = match release.verify_file(name, &data) {
Ok(()) => None,
Err(first) => release
.verify_file(&format!("./{name}"), &data)
.err()
.map(|_| first),
};
if let Some(e) = mismatch {
return Err(
format!("Verification of the repository index failed: {e}").into()
);
}
}
return Ok(data);
}
Ok(response) => errors.push(format!("{}: HTTP {}", url, response.status())),
Err(e) => errors.push(format!("{}: {}", url, e)),
@@ -505,6 +638,11 @@ async fn get_flat_repo_sources(base_url: &str) -> Result<Vec<u8>, Box<dyn Error>
/// Packages from external repositories are always downloaded from the
/// repository itself; the 'Vcs-Git' of the stanza is never used, as it may
/// point to an arbitrary source.
///
/// Third-party flat repositories have no known signing key: when they
/// publish a Release file its checksums are enforced on the index (with a
/// hard error on mismatch), but the absence of a verifiable signature only
/// produces a warning, preserving the previous behavior for such repos.
pub async fn lookup_repository(
package: &str,
version: Option<&str>,
@@ -521,8 +659,35 @@ pub async fn lookup_repository(
);
}
// Attempt to authenticate the repository. In non-strict mode only the
// absence of a Release file (or of a way to verify it) is tolerated:
// tampering evidence — an invalid or malformed signature — is a hard
// error even for third-party repositories.
let verified = match release::verify_suite(
repo_url.trim_end_matches('/'),
release::KeyringSource::None,
false,
)
.await
{
Ok(release::Verification::Available(verified)) => Some(verified),
Ok(release::Verification::Unavailable { .. }) => None,
Ok(release::Verification::KeyringUnavailable { reason, .. }) => {
warn!("Release verification of repository {repo_url} failed: {reason}");
None
}
Err(e) => return Err(e.into()),
};
// Prefer the suite name from the (fetched, possibly verified) Release
// file; fall back to a dedicated fetch when there is none
let resolved_series = if let Some(s) = series {
s.to_string()
} else if let Some(codename) = verified
.as_ref()
.and_then(|v| v.field("Codename").or_else(|| v.field("Suite")))
{
codename.to_string()
} else {
get_flat_repo_series(repo_url).await?
};
@@ -536,7 +701,7 @@ pub async fn lookup_repository(
);
}
let sources = get_flat_repo_sources(repo_url).await?;
let sources = get_flat_repo_sources(repo_url, verified.as_ref()).await?;
let stanza = parse_sources(&sources, package, version)?
.ok_or_else(|| format!("Package '{package}' not found in repository {repo_url}"))?;
@@ -747,6 +912,44 @@ Directory: pool/main/h/hello
assert_eq!(info.version, "1.0");
}
#[test]
fn test_parse_sources_many_blank_stanzas() {
// A crafted index with many consecutive package-less stanzas must be
// iterated without recursion: 100k blank stanzas would overflow the
// stack with the old recursive 'return self.next()' implementation
let mut data = String::new();
for _ in 0..100_000 {
data.push_str("Not-Really-Package: x\n\n");
}
data.push_str("Package: hello\nVersion: 1.0\n");
let info = parse_sources(data.as_bytes(), "hello", None)
.unwrap()
.unwrap();
assert_eq!(info.package, "hello");
assert_eq!(info.version, "1.0");
}
#[test]
fn test_parse_sources_stanza_without_version() {
// A stanza with a 'Package' but no 'Version' field is malformed
// remote data: it must be skipped rather than panic
let data = "Package: noversion\nDirectory: pool/main/n/noversion\n\n\
Package: hello\nVersion: 1.0\n";
let info = parse_sources(data.as_bytes(), "hello", None)
.unwrap()
.unwrap();
assert_eq!(info.package, "hello");
assert_eq!(info.version, "1.0");
assert!(
parse_sources(data.as_bytes(), "noversion", None)
.unwrap()
.is_none()
);
}
#[tokio::test]
async fn test_find_package_fallback() {
// python2.7 is in bullseye but not above
+381 -30
View File
@@ -10,12 +10,18 @@
//! bind-mounted `/proc` and overlay filesystems, so they require careful
//! unmounting before they can be removed.
//! - **Cached chroot tarballs** (`~/.cache/pkh/*-buildd.tar.xz`) and their
//! **stale download lockfiles** (`~/.cache/pkh/*.lock`).
//! - **The shared apt keyring directory** (`/tmp/pkh-keyrings`), used by
//! **stale download lockfiles** (`~/.cache/pkh/*.lock` untouched for longer
//! than [`LOCK_STALE_AFTER`]; younger lockfiles may belong to a concurrent
//! `pkh` run and are left alone).
//! - **The apt keyring cache directories** (`pkh-keyrings` and the per-uid
//! `pkh-keyrings-<uid>` under the system temp directory), used by
//! mmdebstrap runs.
//! - **Build logs** (`~/.cache/pkh/logs/deb-*.log`) written by `pkh deb`. By
//! default only logs beyond a small retention window (the newest
//! [`KEEP_LOGS`] are kept) are removed; pass [`PruneOptions::all`] to remove
//! - **Build logs** under `~/.cache/pkh/logs/` written by `pkh deb`:
//! `deb-<package>-<version>-<timestamp>.log` (binary builds),
//! `build-<package>-<version>-<timestamp>.log` (source builds) and the
//! pre-identity placeholder `pkh-<timestamp>.log`. By default only logs
//! beyond a small retention window (the newest [`KEEP_LOGS`] by embedded
//! timestamp are kept) are removed; pass [`PruneOptions::all`] to remove
//! them all.
//!
//! The [`prune()`] function discovers and removes all of the above. By
@@ -27,7 +33,9 @@
use std::fs;
use std::path::{Path, PathBuf};
use std::process::Command;
use std::time::{Duration, SystemTime};
use chrono::NaiveDateTime;
use directories::ProjectDirs;
/// Options controlling the prune operation.
@@ -62,12 +70,21 @@ impl PruneReport {
/// Number of newest build logs kept when `--all` is not passed.
pub(crate) const KEEP_LOGS: usize = 10;
/// Lockfiles younger than this are not pruned: a recently touched
/// `<tarball>.lock` may belong to a concurrent `pkh` run, whose chroot
/// tarball download uses the lockfile's mere existence as a mutual-exclusion
/// signal (see `deb::ephemeral`). Deleting such a lockfile mid-download would
/// let two builds corrupt the shared tarball cache. Only lockfiles untouched
/// for at least this long are considered stale.
const LOCK_STALE_AFTER: Duration = Duration::from_secs(24 * 60 * 60);
/// One discovered artifact that prune can act on.
#[derive(Debug, Clone)]
enum Artifact {
/// A residual build/chroot directory under the system temp dir.
TempDir(PathBuf),
/// The shared apt keyring directory (`/tmp/pkh-keyrings`).
/// The apt keyring cache directory (`pkh-keyrings*` under the system
/// temp dir).
KeyringDir(PathBuf),
/// A stale chroot tarball download lockfile.
LockFile(PathBuf),
@@ -105,11 +122,24 @@ pub fn cache_dir() -> Option<PathBuf> {
ProjectDirs::from("com", "pkh", "pkh").map(|d| d.cache_dir().to_path_buf())
}
/// Check whether a directory entry name is the apt keyring cache directory,
/// i.e. the legacy shared `pkh-keyrings` name or the per-uid
/// `pkh-keyrings-<uid>` used by current pkh versions.
fn is_keyring_dir_name(name: &str) -> bool {
if name == "pkh-keyrings" {
return true;
}
match name.strip_prefix("pkh-keyrings-") {
Some(uid) => !uid.is_empty() && uid.bytes().all(|b| b.is_ascii_digit()),
None => false,
}
}
/// Check whether a directory entry name is a residual pkh build directory,
/// i.e. it matches `pkh-<digits>` or `pkh-<digits>-<digits>`.
///
/// This deliberately rejects the `pkh-keyrings` and `pkh-build-*` names so they
/// are not mistaken for residual build chroots.
/// This deliberately rejects the `pkh-keyrings*` and `pkh-build-*` names so
/// they are not mistaken for residual build chroots.
fn is_residual_temp_dir(name: &str) -> bool {
let Some(rest) = name.strip_prefix("pkh-") else {
return false;
@@ -125,6 +155,68 @@ fn is_residual_temp_dir(name: &str) -> bool {
}
}
/// Check whether a file name is a pkh build log, i.e. it ends in `.log` and
/// starts with one of the prefixes used by `pkh deb` logging: `deb-` (binary
/// builds), `build-` (source builds) or `pkh-` (the pre-identity placeholder
/// log that is renamed once the build identity is known).
fn is_log_file_name(name: &str) -> bool {
name.ends_with(".log")
&& (name.starts_with("deb-") || name.starts_with("build-") || name.starts_with("pkh-"))
}
/// Length of the `YYYYMMDDTHHMMSS` UTC timestamp embedded in log file names.
const LOG_TIMESTAMP_LEN: usize = 15;
/// Parse the UTC timestamp embedded in a build log file name.
///
/// Log names embed the timestamp as their final dash-separated component
/// (e.g. `deb-pkg-1.0-20260101T000000.log`, `build-src-1.0-20260101T000000.log`,
/// `pkh-20260101T000000.log`). Returns `None` when the name carries no
/// parseable timestamp.
fn embedded_log_timestamp(path: &Path) -> Option<SystemTime> {
let name = path.file_name()?.to_str()?;
let stem = name.strip_suffix(".log")?;
let candidate = stem.rsplit_once('-')?.1;
if candidate.len() != LOG_TIMESTAMP_LEN {
return None;
}
let ts = NaiveDateTime::parse_from_str(candidate, "%Y%m%dT%H%M%S").ok()?;
Some(ts.and_utc().into())
}
/// Recency of a build log, used to order logs for retention: later values are
/// more recent.
///
/// The timestamp embedded in the file name is authoritative. When the name
/// carries no parseable timestamp the file's modification time is used
/// instead, and finally the Unix epoch, so that undateable logs sort as the
/// oldest ones.
fn log_recency(path: &Path) -> SystemTime {
if let Some(ts) = embedded_log_timestamp(path) {
return ts;
}
fs::metadata(path)
.and_then(|m| m.modified())
.unwrap_or(SystemTime::UNIX_EPOCH)
}
/// Check whether a lockfile is safe to prune, i.e. it has not been modified
/// for at least [`LOCK_STALE_AFTER`].
///
/// A freshly touched lockfile may belong to a live concurrent `pkh` run, so
/// it must be left alone. If the modification time cannot be read the lock is
/// treated as ancient and pruned (best-effort cleanup of an unreadable
/// leftover); a modification time in the future (clock skew) counts as fresh.
fn is_stale_lock(path: &Path) -> bool {
let mtime = fs::metadata(path)
.and_then(|m| m.modified())
.unwrap_or(SystemTime::UNIX_EPOCH);
match mtime.elapsed() {
Ok(age) => age >= LOCK_STALE_AFTER,
Err(_) => false,
}
}
/// Unescape the octal escape sequences used in `/proc/mounts` mount points
/// (e.g. `\040` for space, `\011` for tab, `\012` for newline, `\134` for
/// backslash).
@@ -189,7 +281,7 @@ fn discover_artifacts(temp_dir: &Path, cache_dir: Option<&Path>) -> Vec<Artifact
}
if is_residual_temp_dir(&name) {
artifacts.push(Artifact::TempDir(path));
} else if name == "pkh-keyrings" {
} else if is_keyring_dir_name(&name) {
artifacts.push(Artifact::KeyringDir(path));
}
}
@@ -202,7 +294,11 @@ fn discover_artifacts(temp_dir: &Path, cache_dir: Option<&Path>) -> Vec<Artifact
let path = entry.path();
let name = entry.file_name().to_string_lossy().into_owned();
if name.ends_with(".lock") {
artifacts.push(Artifact::LockFile(path));
// Only stale lockfiles are prune-able; a fresh one may belong
// to a live concurrent pkh run (see [`is_stale_lock`]).
if is_stale_lock(&path) {
artifacts.push(Artifact::LockFile(path));
}
} else if name.ends_with("-buildd.tar.xz") {
artifacts.push(Artifact::Tarball(path));
}
@@ -216,7 +312,7 @@ fn discover_artifacts(temp_dir: &Path, cache_dir: Option<&Path>) -> Vec<Artifact
for entry in entries.flatten() {
let path = entry.path();
let name = entry.file_name().to_string_lossy().into_owned();
if path.is_file() && name.starts_with("deb-") && name.ends_with(".log") {
if path.is_file() && is_log_file_name(&name) {
artifacts.push(Artifact::LogFile(path));
}
}
@@ -228,8 +324,12 @@ fn discover_artifacts(temp_dir: &Path, cache_dir: Option<&Path>) -> Vec<Artifact
/// Compute the set of build log paths that should be removed for the given
/// options: all of them with `--all`, otherwise every log except the
/// [`KEEP_LOGS`] newest ones (filenames embed timestamps, so lexicographic
/// order is chronological).
/// [`KEEP_LOGS`] newest ones.
///
/// Recency is the timestamp embedded in each log's file name (falling back to
/// the file modification time when the name cannot be parsed; see
/// [`log_recency()`]). Log names sort by package and version first, so plain
/// lexicographic order is *not* chronological and must not be used here.
fn removable_logs(artifacts: &[Artifact], all: bool) -> std::collections::HashSet<&Path> {
let mut logs: Vec<&Path> = artifacts
.iter()
@@ -238,7 +338,7 @@ fn removable_logs(artifacts: &[Artifact], all: bool) -> std::collections::HashSe
_ => None,
})
.collect();
logs.sort_unstable();
logs.sort_by_key(|p| log_recency(p));
if all {
return logs.into_iter().collect();
@@ -396,6 +496,18 @@ mod tests {
use std::fs;
use tempfile::tempdir;
/// Set a file's modification time (test helper).
fn set_mtime(path: &Path, mtime: SystemTime) {
let f = fs::OpenOptions::new().write(true).open(path).unwrap();
f.set_times(std::fs::FileTimes::new().set_modified(mtime))
.unwrap();
}
/// A modification time safely older than [`LOCK_STALE_AFTER`].
fn old_mtime() -> SystemTime {
SystemTime::now() - LOCK_STALE_AFTER - Duration::from_secs(3600)
}
#[test]
fn test_is_residual_temp_dir() {
assert!(is_residual_temp_dir("pkh-1700000000"));
@@ -404,6 +516,7 @@ mod tests {
// Keyrings and unshare work dirs must NOT match.
assert!(!is_residual_temp_dir("pkh-keyrings"));
assert!(!is_residual_temp_dir("pkh-keyrings-1000"));
assert!(!is_residual_temp_dir("pkh-build-1700000000"));
// Non-numeric / malformed names.
@@ -415,6 +528,83 @@ mod tests {
assert!(!is_residual_temp_dir("other-123"));
}
#[test]
fn test_is_keyring_dir_name() {
// Legacy shared name and current per-uid names are matched.
assert!(is_keyring_dir_name("pkh-keyrings"));
assert!(is_keyring_dir_name("pkh-keyrings-0"));
assert!(is_keyring_dir_name("pkh-keyrings-1000"));
// Everything else is not.
assert!(!is_keyring_dir_name("pkh-keyrings-"));
assert!(!is_keyring_dir_name("pkh-keyrings-abc"));
assert!(!is_keyring_dir_name("pkh-keyrings-1000-2"));
assert!(!is_keyring_dir_name("keyrings"));
assert!(!is_keyring_dir_name("pkh-1000"));
}
#[test]
fn test_is_log_file_name() {
// The three naming shapes produced by ui/deb.rs.
assert!(is_log_file_name("deb-hello-1.0-20260101T000000.log"));
assert!(is_log_file_name("build-src-1.0-20260101T000000.log"));
assert!(is_log_file_name("pkh-20260101T000000.log"));
// Everything else is not.
assert!(!is_log_file_name("deb-hello-1.0-20260101T000000.log.gz"));
assert!(!is_log_file_name("not-a-build.txt"));
assert!(!is_log_file_name("deb.txt"));
assert!(!is_log_file_name("logs.txt"));
assert!(!is_log_file_name("deb-"));
assert!(!is_log_file_name("other-thing.log"));
}
#[test]
fn test_log_recency_orders_by_embedded_timestamp() {
// The embedded timestamp decides, even when it disagrees with
// lexicographic order (zzz sorts after aaa).
let old = Path::new("/logs/deb-zzz-1.0-20200101T000000.log");
let new = Path::new("/logs/deb-aaa-2.0-20260101T000000.log");
assert!(log_recency(old) < log_recency(new));
// All three prefixes participate in the same timeline.
let build = Path::new("/logs/build-src-3.0-20230101T000000.log");
let placeholder = Path::new("/logs/pkh-20250101T000000.log");
assert!(log_recency(old) < log_recency(build));
assert!(log_recency(build) < log_recency(placeholder));
assert!(log_recency(placeholder) < log_recency(new));
}
#[test]
fn test_log_recency_falls_back_to_mtime() {
let temp = tempdir().unwrap();
// No parseable timestamp in the name, so the mtime decides.
let path = temp.path().join("deb-weird-name.log");
fs::write(&path, "log").unwrap();
let mtime = SystemTime::UNIX_EPOCH + Duration::from_secs(1_700_000_000);
set_mtime(&path, mtime);
assert_eq!(log_recency(&path), mtime);
// A missing file with an unparseable name sorts as the oldest.
assert_eq!(
log_recency(&PathBuf::from("/logs/deb-nope-nope.log")),
SystemTime::UNIX_EPOCH
);
}
#[test]
fn test_is_stale_lock() {
let temp = tempdir().unwrap();
let fresh = temp.path().join("fresh.tar.lock");
let stale = temp.path().join("stale.tar.lock");
fs::write(&fresh, "lock").unwrap();
fs::write(&stale, "lock").unwrap();
set_mtime(&stale, old_mtime());
assert!(!is_stale_lock(&fresh));
assert!(is_stale_lock(&stale));
}
#[test]
fn test_unescape_mountpath() {
assert_eq!(unescape_mountpath("/tmp/simple"), "/tmp/simple");
@@ -455,8 +645,9 @@ none /tmp/other proc rw 0 0
// Residual build dirs.
fs::create_dir_all(temp_path.join("pkh-1700000000")).unwrap();
fs::create_dir_all(temp_path.join("pkh-1700000001-2")).unwrap();
// Keyring dir.
// Keyring dirs (legacy shared name and per-uid name).
fs::create_dir_all(temp_path.join("pkh-keyrings")).unwrap();
fs::create_dir_all(temp_path.join("pkh-keyrings-1000")).unwrap();
// Non-matching entries that should be ignored.
fs::create_dir_all(temp_path.join("pkh-build-1700000000")).unwrap();
fs::create_dir_all(temp_path.join("other-dir")).unwrap();
@@ -465,36 +656,42 @@ none /tmp/other proc rw 0 0
// Cache entries.
fs::write(cache_path.join("noble-buildd.tar.xz"), "tarball").unwrap();
fs::write(cache_path.join("noble-amd64-buildd.tar.xz"), "tarball").unwrap();
// Only a stale lockfile is discovered; a fresh one could belong to a
// live concurrent run and must be ignored.
fs::write(cache_path.join("noble-buildd.tar.lock"), "lock").unwrap();
set_mtime(&cache_path.join("noble-buildd.tar.lock"), old_mtime());
fs::write(cache_path.join("noble-fresh.tar.lock"), "lock").unwrap();
fs::write(cache_path.join("stray.txt"), "ignore me").unwrap();
// Build logs.
// Build logs: binary, source, and the pre-identity placeholder.
let logs_dir = cache_path.join("logs");
fs::create_dir_all(&logs_dir).unwrap();
fs::write(logs_dir.join("deb-hello-20260101T000000.log"), "log").unwrap();
fs::write(logs_dir.join("deb-hello-1.0-20260101T000000.log"), "log").unwrap();
fs::write(logs_dir.join("build-src-1.0-20260101T000000.log"), "log").unwrap();
fs::write(logs_dir.join("pkh-20260101T000000.log"), "log").unwrap();
fs::write(logs_dir.join("not-a-build.txt"), "ignore me").unwrap();
let artifacts = discover_artifacts(temp_path, Some(cache_path));
let mut temp_dirs = 0;
let mut keyring = false;
let mut keyring_dirs = 0;
let mut locks = 0;
let mut tarballs = 0;
let mut logs = 0;
for a in &artifacts {
match a {
Artifact::TempDir(_) => temp_dirs += 1,
Artifact::KeyringDir(_) => keyring = true,
Artifact::KeyringDir(_) => keyring_dirs += 1,
Artifact::LockFile(_) => locks += 1,
Artifact::Tarball(_) => tarballs += 1,
Artifact::LogFile(_) => logs += 1,
}
}
assert_eq!(temp_dirs, 2);
assert!(keyring);
assert_eq!(keyring_dirs, 2);
assert_eq!(locks, 1);
assert_eq!(tarballs, 2);
assert_eq!(logs, 1);
assert_eq!(logs, 3);
}
#[test]
@@ -505,14 +702,19 @@ none /tmp/other proc rw 0 0
let logs_dir = cache_path.join("logs");
fs::create_dir_all(&logs_dir).unwrap();
// Create KEEP_LOGS + 3 logs; the 3 oldest should be pruned by default
// Create KEEP_LOGS + 3 logs across all naming shapes; the 3 oldest
// (by embedded timestamp) should be pruned by default.
let total = KEEP_LOGS + 3;
let mut names = Vec::new();
for i in 0..total {
fs::write(
logs_dir.join(format!("deb-pkg-20260101T{:06}.log", i)),
"log",
)
.unwrap();
let ts = format!("20260101T{:06}", i);
let name = match i % 3 {
0 => format!("deb-pkg-1.0-{ts}.log"),
1 => format!("build-src-2.0-{ts}.log"),
_ => format!("pkh-{ts}.log"),
};
fs::write(logs_dir.join(&name), "log").unwrap();
names.push(name);
}
let report = prune_in(
@@ -525,12 +727,18 @@ none /tmp/other proc rw 0 0
)
.unwrap();
let removed_logs = report
let mut removed_logs: Vec<String> = report
.removed
.iter()
.filter(|p| p.starts_with(&logs_dir))
.count();
assert_eq!(removed_logs, 3);
.map(|p| p.file_name().unwrap().to_string_lossy().into_owned())
.collect();
removed_logs.sort();
// Exactly the three oldest logs (i = 0, 1, 2) were removed.
let mut expected: Vec<String> = names[..3].to_vec();
expected.sort();
assert_eq!(removed_logs, expected);
let remaining = fs::read_dir(&logs_dir).unwrap().count();
assert_eq!(remaining, KEEP_LOGS);
@@ -554,6 +762,68 @@ none /tmp/other proc rw 0 0
assert_eq!(fs::read_dir(&logs_dir).unwrap().count(), 0);
}
#[test]
fn test_prune_log_retention_orders_by_timestamp_not_name() {
let temp = tempdir().unwrap();
let cache = tempdir().unwrap();
let cache_path = cache.path();
let logs_dir = cache_path.join("logs");
fs::create_dir_all(&logs_dir).unwrap();
// KEEP_LOGS new logs whose names sort lexicographically FIRST
// (package "aaa"), and older logs whose names sort LAST ("zzz" and a
// pkh- placeholder). Retention must follow the embedded timestamps:
// the aaa logs are kept and the lexicographically-later old logs are
// pruned, the opposite of what name ordering would do.
for i in 0..KEEP_LOGS {
fs::write(
logs_dir.join(format!("deb-aaa-1.0-20210101T{:06}.log", i)),
"log",
)
.unwrap();
}
let old_names = [
"deb-zzz-2.0-20200101T000000.log".to_string(),
"deb-zzz-2.0-20200101T000001.log".to_string(),
"build-zzz-3.0-20200101T000000.log".to_string(),
"pkh-20200101T000000.log".to_string(),
];
for name in &old_names {
fs::write(logs_dir.join(name), "log").unwrap();
}
let report = prune_in(
temp.path(),
Some(cache_path),
PruneOptions {
dry_run: false,
all: false,
},
)
.unwrap();
let removed_logs: Vec<String> = report
.removed
.iter()
.filter(|p| p.starts_with(&logs_dir))
.map(|p| p.file_name().unwrap().to_string_lossy().into_owned())
.collect();
let mut removed_sorted = removed_logs.clone();
removed_sorted.sort();
let mut expected = old_names.clone();
expected.sort();
assert_eq!(removed_sorted, expected);
for i in 0..KEEP_LOGS {
assert!(
logs_dir
.join(format!("deb-aaa-1.0-20210101T{:06}.log", i))
.exists()
);
}
assert_eq!(fs::read_dir(&logs_dir).unwrap().count(), KEEP_LOGS);
}
#[test]
fn test_prune_dry_run_keeps_artifacts() {
let temp = tempdir().unwrap();
@@ -564,8 +834,13 @@ none /tmp/other proc rw 0 0
let dir = temp_path.join("pkh-1700000000");
fs::create_dir_all(&dir).unwrap();
fs::create_dir_all(temp_path.join("pkh-keyrings")).unwrap();
fs::create_dir_all(temp_path.join("pkh-keyrings-1000")).unwrap();
fs::write(cache_path.join("noble-buildd.tar.xz"), "tarball").unwrap();
// A stale lockfile is reported; a fresh one may belong to a live
// concurrent run and is left out entirely.
fs::write(cache_path.join("noble-buildd.tar.lock"), "lock").unwrap();
set_mtime(&cache_path.join("noble-buildd.tar.lock"), old_mtime());
fs::write(cache_path.join("noble-live.tar.lock"), "lock").unwrap();
let report = prune_in(
temp_path,
@@ -587,12 +862,25 @@ none /tmp/other proc rw 0 0
.iter()
.any(|p| p == &temp_path.join("pkh-keyrings"))
);
assert!(
report
.removed
.iter()
.any(|p| p == &temp_path.join("pkh-keyrings-1000"))
);
assert!(
report
.removed
.iter()
.any(|p| p == &cache_path.join("noble-buildd.tar.lock"))
);
// The fresh lockfile is not reported and survives the dry run.
assert!(
!report
.removed
.iter()
.any(|p| p == &cache_path.join("noble-live.tar.lock"))
);
// Tarballs require --all, so not reported here.
assert!(
!report
@@ -604,8 +892,10 @@ none /tmp/other proc rw 0 0
// Nothing was actually removed.
assert!(dir.exists());
assert!(temp_path.join("pkh-keyrings").exists());
assert!(temp_path.join("pkh-keyrings-1000").exists());
assert!(cache_path.join("noble-buildd.tar.xz").exists());
assert!(cache_path.join("noble-buildd.tar.lock").exists());
assert!(cache_path.join("noble-live.tar.lock").exists());
}
#[test]
@@ -619,7 +909,10 @@ none /tmp/other proc rw 0 0
let cache_path = cache.path();
fs::create_dir_all(temp_path.join("pkh-keyrings")).unwrap();
fs::create_dir_all(temp_path.join("pkh-keyrings-1000")).unwrap();
fs::write(cache_path.join("noble-buildd.tar.lock"), "lock").unwrap();
set_mtime(&cache_path.join("noble-buildd.tar.lock"), old_mtime());
fs::write(cache_path.join("noble-fresh.tar.lock"), "lock").unwrap();
fs::write(cache_path.join("noble-buildd.tar.xz"), "tarball").unwrap();
let report = prune_in(
@@ -639,6 +932,12 @@ none /tmp/other proc rw 0 0
.iter()
.any(|p| p == &temp_path.join("pkh-keyrings"))
);
assert!(
report
.removed
.iter()
.any(|p| p == &temp_path.join("pkh-keyrings-1000"))
);
assert!(
report
.removed
@@ -647,7 +946,10 @@ none /tmp/other proc rw 0 0
);
assert!(!temp_path.join("pkh-keyrings").exists());
assert!(!temp_path.join("pkh-keyrings-1000").exists());
assert!(!cache_path.join("noble-buildd.tar.lock").exists());
// The fresh lockfile may belong to a live run and is preserved.
assert!(cache_path.join("noble-fresh.tar.lock").exists());
// Tarball is preserved when --all is not set.
assert!(cache_path.join("noble-buildd.tar.xz").exists());
}
@@ -661,6 +963,7 @@ none /tmp/other proc rw 0 0
fs::write(cache_path.join("noble-buildd.tar.xz"), "tarball").unwrap();
fs::write(cache_path.join("noble-amd64-buildd.tar.xz"), "tarball").unwrap();
fs::write(cache_path.join("noble-buildd.tar.lock"), "lock").unwrap();
set_mtime(&cache_path.join("noble-buildd.tar.lock"), old_mtime());
let report = prune_in(
temp.path(),
@@ -696,6 +999,54 @@ none /tmp/other proc rw 0 0
assert!(!cache_path.join("noble-buildd.tar.lock").exists());
}
#[test]
fn test_prune_keeps_fresh_lockfile_removes_stale_one() {
// The chroot tarball download uses the lockfile's existence as a
// mutual-exclusion signal, so a lockfile fresh enough to belong to a
// live concurrent pkh run must survive pruning.
let temp = tempdir().unwrap();
let cache = tempdir().unwrap();
let cache_path = cache.path();
let fresh = cache_path.join("noble-live.tar.lock");
let stale = cache_path.join("noble-abandoned.tar.lock");
fs::write(&fresh, "lock").unwrap();
fs::write(&stale, "lock").unwrap();
set_mtime(&stale, old_mtime());
let report = prune_in(
temp.path(),
Some(cache_path),
PruneOptions {
dry_run: false,
all: false,
},
)
.unwrap();
assert!(report.removed.iter().any(|p| p == &stale));
assert!(!report.removed.iter().any(|p| p == &fresh));
assert!(fresh.exists());
assert!(!stale.exists());
// Dry-run reports the same: only the stale lockfile.
fs::write(&stale, "lock").unwrap();
set_mtime(&stale, old_mtime());
let report = prune_in(
temp.path(),
Some(cache_path),
PruneOptions {
dry_run: true,
all: false,
},
)
.unwrap();
assert!(report.removed.iter().any(|p| p == &stale));
assert!(!report.removed.iter().any(|p| p == &fresh));
assert!(fresh.exists());
assert!(stale.exists());
}
#[test]
fn test_prune_removes_user_writable_temp_dirs() {
// Residual build dirs created by the tests are user-writable, so the
+192 -7
View File
@@ -1,6 +1,7 @@
use std::cmp::min;
use std::error::Error;
use std::os::unix::fs::symlink;
use std::path::Component;
use std::path::Path;
use std::path::PathBuf;
@@ -209,6 +210,28 @@ where
continue;
}
// Security: `Entry::unpack` performs no path sanitization, so extract
// the entry only if its path is confined to the destination directory.
// An absolute path or a path containing '..' would let a malicious (or
// deeply malformed) tarball write files anywhere outside 'dest' (path
// traversal). Refuse such entries with an error rather than skipping
// them silently, so the problem is not hidden.
let escapes_dest = relative.components().any(|component| {
matches!(
component,
Component::Prefix(_) | Component::RootDir | Component::ParentDir
)
});
if escapes_dest {
return Err(format!(
"Refusing to extract '{}': archive entry path is absolute or \
contains '..' and would escape the destination directory '{}'",
relative.display(),
dest.display()
)
.into());
}
let dest_path = dest.join(&relative);
// Create parent directories if needed
@@ -287,8 +310,14 @@ async fn download_file_checksum(
target_dir: &Path,
progress: ProgressCallback<'_>,
) -> Result<(), Box<dyn Error>> {
// Download with reqwest
let response = reqwest::get(url).await?;
// Download with the shared client (connect timeout). Large orig tarballs
// can legitimately take longer than the client's default total timeout,
// so use a generous per-request timeout for streaming downloads
let response = crate::distro_info::http_client()
.get(url)
.timeout(std::time::Duration::from_secs(30 * 60))
.send()
.await?;
if !response.status().is_success() {
return Err(format!("Failed to download '{}' : {}", url, response.status()).into());
}
@@ -299,7 +328,12 @@ async fn download_file_checksum(
let mut index = 0;
// Target file: extract file name from URL
let filename = Path::new(url).file_name().unwrap().to_str().unwrap();
let filename = Path::new(url)
.file_name()
.and_then(|f| f.to_str())
.ok_or_else(|| {
format!("Could not determine a file name from URL '{url}' to download the package file")
})?;
let path = target_dir.join(filename);
let mut file = File::create(path)?;
@@ -627,10 +661,17 @@ pub async fn pull(
// we target the development branch, i.e. the default branch
// Only use Ubuntu-specific branch naming if the VCS is from Launchpad
let is_launchpad_vcs = url.contains("launchpad.net");
let branch_name = if crate::distro_info::get_ordered_series_name(package_info.dist.as_str())
.await?[0]
!= *series
{
let series_list =
crate::distro_info::get_ordered_series_name(package_info.dist.as_str()).await?;
let latest_series = series_list.first().ok_or_else(|| {
format!(
"No series information available for distribution '{}', \
cannot determine its development series to select the git branch. \
The 'distro-info' package provides this data.",
package_info.dist
)
})?;
let branch_name = if latest_series != series {
if package_info.dist == "ubuntu" && is_launchpad_vcs {
Some(format!("{}/{}", package_info.dist, series))
} else {
@@ -834,4 +875,148 @@ mod tests {
async fn test_pull_paraview_ubuntu_end_to_end() {
test_pull_package_end_to_end("paraview", Some("noble"), None, None).await;
}
/// Build a minimal uncompressed ustar archive from (name, data) entries.
///
/// Raw header blocks are crafted instead of using `tar::Builder` because
/// the builder itself refuses entry names containing '..' or absolute
/// paths, which is exactly what the traversal tests need to exercise.
fn build_tar(entries: &[(&str, &[u8])]) -> Vec<u8> {
let mut out = Vec::new();
for (name, data) in entries {
let mut block = [0u8; 512];
block[..name.len()].copy_from_slice(name.as_bytes());
block[100..108].copy_from_slice(b"0000644\0");
block[124..136].copy_from_slice(format!("{:011o}\0", data.len()).as_bytes());
block[136..148].copy_from_slice(b"00000000000\0");
block[156] = b'0'; // regular file
block[257..263].copy_from_slice(b"ustar\0");
block[263..265].copy_from_slice(b"00");
// Checksum: sum of the header bytes with the checksum field
// (bytes 148..156) taken as spaces
let mut checksum: u32 = 0;
for (i, byte) in block.iter().enumerate() {
checksum += if (148..156).contains(&i) {
u32::from(b' ')
} else {
u32::from(*byte)
};
}
block[148..154].copy_from_slice(format!("{checksum:06o}").as_bytes());
block[154] = 0;
block[155] = b' ';
out.extend_from_slice(&block);
out.extend_from_slice(data);
let padding = (512 - (data.len() % 512)) % 512;
out.extend_from_slice(&vec![0u8; padding]);
}
// End-of-archive marker: two zero-filled blocks
out.extend_from_slice(&[0u8; 1024]);
out
}
#[test]
fn test_extract_tar_rejects_parent_dir_traversal() {
let temp_dir = tempfile::tempdir().unwrap();
let tar_path = temp_dir.path().join("malicious.orig.tar");
std::fs::write(
&tar_path,
build_tar(&[("good.txt", b"ok"), ("../evil.txt", b"pwned")]),
)
.unwrap();
let dest = temp_dir.path().join("dest");
let result = extract_tar_archive(&tar_path, &dest, None, |f| f);
let err = match result {
Ok(_) => panic!("extraction of a traversal archive should fail"),
Err(e) => e,
};
assert!(
err.to_string().contains("../evil.txt"),
"error should name the offending entry, got: {err}"
);
// Nothing may be written outside of the destination directory
assert!(!temp_dir.path().join("evil.txt").exists());
// Legitimate entries preceding the malicious one are still extracted
assert_eq!(
std::fs::read_to_string(dest.join("good.txt")).unwrap(),
"ok"
);
}
#[test]
fn test_extract_tar_rejects_absolute_path() {
let temp_dir = tempfile::tempdir().unwrap();
let tar_path = temp_dir.path().join("malicious.orig.tar");
std::fs::write(
&tar_path,
build_tar(&[
("good.txt", b"ok"),
("/pkh_test_absolute_escape.txt", b"pwned"),
]),
)
.unwrap();
let dest = temp_dir.path().join("dest");
let result = extract_tar_archive(&tar_path, &dest, None, |f| f);
let err = match result {
Ok(_) => panic!("extraction of an absolute-path archive should fail"),
Err(e) => e,
};
assert!(
err.to_string().contains("pkh_test_absolute_escape.txt"),
"error should name the offending entry, got: {err}"
);
// Nothing may be written at the filesystem root
assert!(!Path::new("/pkh_test_absolute_escape.txt").exists());
assert_eq!(
std::fs::read_to_string(dest.join("good.txt")).unwrap(),
"ok"
);
}
#[test]
fn test_extract_tar_archive_normal_entries() {
let temp_dir = tempfile::tempdir().unwrap();
let tar_path = temp_dir.path().join("normal.orig.tar");
// Raw names: 'tar::Builder' would normalize away the './' prefix
std::fs::write(
&tar_path,
build_tar(&[
("hello-1.0/file.txt", b"hello"),
("./debian/rules", b"#!/m"),
]),
)
.unwrap();
let dest = temp_dir.path().join("dest");
let extracted = extract_tar_archive(&tar_path, &dest, None, |f| f).unwrap();
assert_eq!(
std::fs::read_to_string(dest.join("hello-1.0/file.txt")).unwrap(),
"hello"
);
assert_eq!(
std::fs::read_to_string(dest.join("debian/rules")).unwrap(),
"#!/m"
);
assert!(extracted.in_place);
assert!(
extracted.files.contains(
&dest
.join("hello-1.0/file.txt")
.to_string_lossy()
.to_string()
)
);
assert!(
extracted
.files
.contains(&dest.join("debian/rules").to_string_lossy().to_string())
);
}
}
+16
View File
@@ -563,6 +563,22 @@ extern "C" fn on_sigint(_sig: libc::c_int) {
{
eprintln!("\nInterrupted — full log: {}", path.display());
}
// Run the registered cleanup hooks (currently: unmount and remove the
// ephemeral build chroot, see `deb::ephemeral::sigint_cleanup_chroot`),
// then exit with the conventional 130 status.
//
// Like the terminal restoration above, this is NOT strictly
// async-signal-safe: it locks a mutex, spawns subprocesses and does I/O.
// That is a deliberate tradeoff, no worse than the rest of this handler:
// exiting immediately would skip all destructors and leak the chroot
// together with its bind-mounted /proc and overlay mounts. The hooks are
// self-contained (they only touch stored paths and spawn umount/rm
// directly), so they cannot deadlock on a lock the interrupted thread
// might have held; the hook registry itself is only ever taken with
// try_lock plus a bounded retry for the same reason. Note that SIGINT
// stays blocked for the duration of the handler, so a second Ctrl-C will
// not interrupt a slow cleanup — send SIGTERM/SIGKILL if it ever hangs.
crate::deb::ephemeral::run_cleanup_hooks();
// SAFETY: raw exit bypassing destructors, intended in a signal handler
unsafe {
libc::_exit(130);