put: degrade to the anonymous FTP queue when the SSH transport fails
CI / build (push) Successful in 3m0s
CI / test (push) Skipped
CI / snap (push) Successful in 6m6s

pkh put only spoke SFTP to the PPA queue, so a failure of the SSH
transport itself (TCP, banner exchange) failed the upload even though
dput happily pushes the same files: its plain ppa: profile goes over
the anonymous FTP queue of ppa.launchpad.net, the same destination
over another port.

Classify the SSH connection failures: Transport (the connection never
came up: resolution, TCP, banner or key exchange) degrades to that FTP
queue — the upload order (payload first, .changes last), the
reverse-order DELE cleanup of a failed upload and the per-chunk
progress reporting all mirror the SFTP path, sharing cleanup_list.
Refused failures (host key not accepted, no matching authentication)
stay errors: silently switching transport would bypass the refusal.

The FTP client is suppaftp's blocking stream, with the time bounds it
does not carry by itself: the control channel's reads and writes, the
data channel's writes and connect (through a custom passive stream
builder), and the NAT workaround for PASV replies announcing an
unroutable address. The queue endpoints (host, port) join
data/launchpad.yml next to the SFTP ones, and the FTP transport is
covered by unit tests against an in-process fake queue plus a live
control-channel handshake with the real server (ignored, network).
This commit is contained in:
2026-09-21 14:34:04 +02:00
parent ac19fd9d65
commit 9e0b6a37a6
6 changed files with 624 additions and 62 deletions
+95 -54
View File
@@ -1,17 +1,12 @@
//! Native upload of built source packages (`pkh put`): the dput
//! replacement. Resolves the upload target, discovers and validates the
//! `.changes` file and its artifacts, then pushes them over SFTP with
//! host-key verification and an upload record preventing accidental
//! duplicate uploads.
//!
//! Payload files are uploaded first and the `.changes` file last, like
//! dput does, so a partially uploaded set cannot be picked up by the
//! server-side queue processors. A run that fails mid-upload removes its
//! already-uploaded files from the incoming queue (best effort), so a
//! retried upload starts from a clean queue; a failed upload is never
//! recorded in the upload log, so a re-run replays every file.
//! PPA upload: the dput replacement. Resolves the upload target, discovers
//! and validates the `.changes` file and its artifacts, then pushes them
//! over SFTP with host-key verification and an upload record preventing
//! accidental duplicate uploads. When the SSH connection itself never
//! comes up, the upload degrades to the anonymous FTP queue — the
//! transport dput's plain `ppa:` profiles use — through [`ftp`].
pub mod changes;
pub mod ftp;
pub mod ssh;
pub mod target;
@@ -139,11 +134,6 @@ async fn put_impl(opts: &PutOptions<'_>) -> Result<(), Box<dyn std::error::Error
.into());
}
opts.view
.message(&format!("Connecting to {login}@{host}:{port}..."));
let session = ssh::connect(&host, port, &login, &ssh_config, opts.prompter)?;
let sftp = ssh::sftp(&session)?;
// Payload first, the .changes file last (like dput), so the server-side
// queue processor can never pick up an incomplete upload
let dir = changes_path.parent().unwrap_or_else(|| Path::new("."));
@@ -157,48 +147,50 @@ async fn put_impl(opts: &PutOptions<'_>) -> Result<(), Box<dyn std::error::Error
.and_then(|n| n.to_str())
.ok_or_else(|| format!("invalid .changes path: {}", changes_path.display()))?
.to_string();
uploads.push((changes_path.clone(), changes_name.clone()));
uploads.push((changes_path.clone(), changes_name));
let incoming = target.incoming.trim_end_matches('/');
// Remote names of the files uploaded so far, in upload order (the
// .changes last). A run failing mid-upload removes these from the
// write-only incoming queue before returning: the uploaded payloads
// would otherwise linger in the queue area forever, and a .changes
// truncated by a failed close could even be picked up by the scanner.
let mut uploaded: Vec<String> = Vec::new();
for (path, name) in &uploads {
let size = match path.metadata() {
Ok(metadata) => metadata.len(),
Err(e) => {
// Nothing was attempted for this file: only what earlier
// iterations uploaded needs removing
let error: Box<dyn std::error::Error> =
format!("cannot stat '{}': {}", path.display(), e).into();
cleanup_partial_upload(&sftp, incoming, &uploaded, None, &host);
return Err(error);
}
};
let remote = format!("{incoming}/{name}");
let label = format!("Uploading {name}");
let view = opts.view;
let on_progress = |uploaded: u64| view.progress(&label, uploaded as usize, size as usize);
let result = ssh::upload_file(&sftp, path, &remote, &host, &on_progress);
if let Err(e) = result {
// The failed file itself joins the cleanup: its remote `create`
// may have succeeded before the failure, leaving a partial — or,
// on a failed close, a truncated — file behind
cleanup_partial_upload(&sftp, incoming, &uploaded, Some(name), &host);
return Err(e);
// The upload queue is SFTP first, degrading to the anonymous FTP
// queue when the SSH connection itself never comes up (name
// resolution, TCP, banner or key exchange): dput pushes PPAs over
// that FTP queue by default, so it is the interoperability-tested
// fallback. A server that answers but refuses the upload (host key
// not accepted, no matching key) stays an error: silently switching
// transport would bypass the refusal.
opts.view
.message(&format!("Connecting to {login}@{host}:{port}..."));
let transfer = match ssh::connect(&host, port, &login, &ssh_config, opts.prompter) {
Ok(session) => sftp_transfer(&session, &uploads, incoming, &host, opts.view),
Err(ssh::ConnectFailure::Transport(e)) => {
log::warn!("SSH transport to {host}:{port} failed: {e}");
let (ftp_host, ftp_port) = launchpad::ppa_ftp_queue();
opts.view.message(&format!(
"Falling back to the anonymous FTP queue on {ftp_host}:{ftp_port} \
(dput's upload method)..."
));
ftp::upload_queue(
&ftp_host,
ftp_port,
incoming,
&uploads,
&|name, uploaded, total| {
opts.view.progress(
&format!("Uploading {name}"),
uploaded as usize,
total as usize,
);
},
)
}
uploaded.push(name.clone());
}
Err(ssh::ConnectFailure::Refused(e)) => return Err(e),
};
transfer?;
// Recorded only once the whole upload succeeded: the log backs the
// duplicate-upload guard, and a failed upload must not count as
// uploaded (a re-run replays every file — `sftp.create` truncates, so
// replaying is safe).
// uploaded (a re-run replays every file — `sftp.create` truncates and
// the FTP `STOR` overwrites, so replaying is safe).
record_upload(&upload_log_path()?, &record)?;
info!(
@@ -212,6 +204,54 @@ async fn put_impl(opts: &PutOptions<'_>) -> Result<(), Box<dyn std::error::Error
Ok(())
}
/// Push `uploads` over SFTP: payload files first, the `.changes` last
/// (like dput), so the server-side queue processor can never pick up an
/// incomplete upload. Best-effort removal of a partial upload, mirroring
/// the FTP transport's `DELE` cleanup ([`ftp::upload_queue`]).
fn sftp_transfer(
session: &ssh2::Session,
uploads: &[(PathBuf, String)],
incoming: &str,
host: &str,
view: &dyn BuildView,
) -> Result<(), Box<dyn std::error::Error>> {
let sftp = ssh::sftp(session)?;
// Remote names of the files uploaded so far, in upload order (the
// .changes last). A run failing mid-upload removes these from the
// write-only incoming queue before returning: the uploaded payloads
// would otherwise linger in the queue area forever, and a .changes
// truncated by a failed close could even be picked up by the scanner.
let mut uploaded: Vec<String> = Vec::new();
for (path, name) in uploads {
let size = match path.metadata() {
Ok(metadata) => metadata.len(),
Err(e) => {
// Nothing was attempted for this file: only what earlier
// iterations uploaded needs removing
let error: Box<dyn std::error::Error> =
format!("cannot stat '{}': {}", path.display(), e).into();
cleanup_partial_upload(&sftp, incoming, &uploaded, None, host);
return Err(error);
}
};
let remote = format!("{incoming}/{name}");
let label = format!("Uploading {name}");
let on_progress = |uploaded: u64| view.progress(&label, uploaded as usize, size as usize);
let result = ssh::upload_file(&sftp, path, &remote, host, &on_progress);
if let Err(e) = result {
// The failed file itself joins the cleanup: its remote `create`
// may have succeeded before the failure, leaving a partial — or,
// on a failed close, a truncated — file behind
cleanup_partial_upload(&sftp, incoming, &uploaded, Some(name), host);
return Err(e);
}
uploaded.push(name.clone());
}
Ok(())
}
/// The remote names to attempt removing after a failed upload: everything
/// already uploaded plus, when set, `failed` (the file whose upload just
/// failed — its remote `create` may have succeeded before the failure,
@@ -219,8 +259,9 @@ async fn put_impl(opts: &PutOptions<'_>) -> Result<(), Box<dyn std::error::Error
/// so a `.changes` is removed before the payloads it references and the
/// queue scanner never observes the payload set shrinking under a
/// still-present `.changes`. Pure so the ordering decision is testable
/// without a server; the network side is [`cleanup_partial_upload`].
fn cleanup_list(uploaded: &[String], failed: Option<&str>) -> Vec<String> {
/// without a server; the network sides are [`cleanup_partial_upload`] and
/// the FTP transport's `DELE` loop ([`ftp`]).
pub(crate) fn cleanup_list(uploaded: &[String], failed: Option<&str>) -> Vec<String> {
let mut names: Vec<String> = uploaded.to_vec();
if let Some(failed) = failed {
names.push(failed.to_string());