put: degrade to the anonymous FTP queue when the SSH transport fails
pkh put only spoke SFTP to the PPA queue, so a failure of the SSH transport itself (TCP, banner exchange) failed the upload even though dput happily pushes the same files: its plain ppa: profile goes over the anonymous FTP queue of ppa.launchpad.net, the same destination over another port. Classify the SSH connection failures: Transport (the connection never came up: resolution, TCP, banner or key exchange) degrades to that FTP queue — the upload order (payload first, .changes last), the reverse-order DELE cleanup of a failed upload and the per-chunk progress reporting all mirror the SFTP path, sharing cleanup_list. Refused failures (host key not accepted, no matching authentication) stay errors: silently switching transport would bypass the refusal. The FTP client is suppaftp's blocking stream, with the time bounds it does not carry by itself: the control channel's reads and writes, the data channel's writes and connect (through a custom passive stream builder), and the NAT workaround for PASV replies announcing an unroutable address. The queue endpoints (host, port) join data/launchpad.yml next to the SFTP ones, and the FTP transport is covered by unit tests against an in-process fake queue plus a live control-channel handshake with the real server (ignored, network).
This commit is contained in:
+95
-54
@@ -1,17 +1,12 @@
|
||||
//! Native upload of built source packages (`pkh put`): the dput
|
||||
//! replacement. Resolves the upload target, discovers and validates the
|
||||
//! `.changes` file and its artifacts, then pushes them over SFTP with
|
||||
//! host-key verification and an upload record preventing accidental
|
||||
//! duplicate uploads.
|
||||
//!
|
||||
//! Payload files are uploaded first and the `.changes` file last, like
|
||||
//! dput does, so a partially uploaded set cannot be picked up by the
|
||||
//! server-side queue processors. A run that fails mid-upload removes its
|
||||
//! already-uploaded files from the incoming queue (best effort), so a
|
||||
//! retried upload starts from a clean queue; a failed upload is never
|
||||
//! recorded in the upload log, so a re-run replays every file.
|
||||
//! PPA upload: the dput replacement. Resolves the upload target, discovers
|
||||
//! and validates the `.changes` file and its artifacts, then pushes them
|
||||
//! over SFTP with host-key verification and an upload record preventing
|
||||
//! accidental duplicate uploads. When the SSH connection itself never
|
||||
//! comes up, the upload degrades to the anonymous FTP queue — the
|
||||
//! transport dput's plain `ppa:` profiles use — through [`ftp`].
|
||||
|
||||
pub mod changes;
|
||||
pub mod ftp;
|
||||
pub mod ssh;
|
||||
pub mod target;
|
||||
|
||||
@@ -139,11 +134,6 @@ async fn put_impl(opts: &PutOptions<'_>) -> Result<(), Box<dyn std::error::Error
|
||||
.into());
|
||||
}
|
||||
|
||||
opts.view
|
||||
.message(&format!("Connecting to {login}@{host}:{port}..."));
|
||||
let session = ssh::connect(&host, port, &login, &ssh_config, opts.prompter)?;
|
||||
let sftp = ssh::sftp(&session)?;
|
||||
|
||||
// Payload first, the .changes file last (like dput), so the server-side
|
||||
// queue processor can never pick up an incomplete upload
|
||||
let dir = changes_path.parent().unwrap_or_else(|| Path::new("."));
|
||||
@@ -157,48 +147,50 @@ async fn put_impl(opts: &PutOptions<'_>) -> Result<(), Box<dyn std::error::Error
|
||||
.and_then(|n| n.to_str())
|
||||
.ok_or_else(|| format!("invalid .changes path: {}", changes_path.display()))?
|
||||
.to_string();
|
||||
uploads.push((changes_path.clone(), changes_name.clone()));
|
||||
uploads.push((changes_path.clone(), changes_name));
|
||||
|
||||
let incoming = target.incoming.trim_end_matches('/');
|
||||
|
||||
// Remote names of the files uploaded so far, in upload order (the
|
||||
// .changes last). A run failing mid-upload removes these from the
|
||||
// write-only incoming queue before returning: the uploaded payloads
|
||||
// would otherwise linger in the queue area forever, and a .changes
|
||||
// truncated by a failed close could even be picked up by the scanner.
|
||||
let mut uploaded: Vec<String> = Vec::new();
|
||||
|
||||
for (path, name) in &uploads {
|
||||
let size = match path.metadata() {
|
||||
Ok(metadata) => metadata.len(),
|
||||
Err(e) => {
|
||||
// Nothing was attempted for this file: only what earlier
|
||||
// iterations uploaded needs removing
|
||||
let error: Box<dyn std::error::Error> =
|
||||
format!("cannot stat '{}': {}", path.display(), e).into();
|
||||
cleanup_partial_upload(&sftp, incoming, &uploaded, None, &host);
|
||||
return Err(error);
|
||||
}
|
||||
};
|
||||
let remote = format!("{incoming}/{name}");
|
||||
let label = format!("Uploading {name}");
|
||||
let view = opts.view;
|
||||
let on_progress = |uploaded: u64| view.progress(&label, uploaded as usize, size as usize);
|
||||
let result = ssh::upload_file(&sftp, path, &remote, &host, &on_progress);
|
||||
if let Err(e) = result {
|
||||
// The failed file itself joins the cleanup: its remote `create`
|
||||
// may have succeeded before the failure, leaving a partial — or,
|
||||
// on a failed close, a truncated — file behind
|
||||
cleanup_partial_upload(&sftp, incoming, &uploaded, Some(name), &host);
|
||||
return Err(e);
|
||||
// The upload queue is SFTP first, degrading to the anonymous FTP
|
||||
// queue when the SSH connection itself never comes up (name
|
||||
// resolution, TCP, banner or key exchange): dput pushes PPAs over
|
||||
// that FTP queue by default, so it is the interoperability-tested
|
||||
// fallback. A server that answers but refuses the upload (host key
|
||||
// not accepted, no matching key) stays an error: silently switching
|
||||
// transport would bypass the refusal.
|
||||
opts.view
|
||||
.message(&format!("Connecting to {login}@{host}:{port}..."));
|
||||
let transfer = match ssh::connect(&host, port, &login, &ssh_config, opts.prompter) {
|
||||
Ok(session) => sftp_transfer(&session, &uploads, incoming, &host, opts.view),
|
||||
Err(ssh::ConnectFailure::Transport(e)) => {
|
||||
log::warn!("SSH transport to {host}:{port} failed: {e}");
|
||||
let (ftp_host, ftp_port) = launchpad::ppa_ftp_queue();
|
||||
opts.view.message(&format!(
|
||||
"Falling back to the anonymous FTP queue on {ftp_host}:{ftp_port} \
|
||||
(dput's upload method)..."
|
||||
));
|
||||
ftp::upload_queue(
|
||||
&ftp_host,
|
||||
ftp_port,
|
||||
incoming,
|
||||
&uploads,
|
||||
&|name, uploaded, total| {
|
||||
opts.view.progress(
|
||||
&format!("Uploading {name}"),
|
||||
uploaded as usize,
|
||||
total as usize,
|
||||
);
|
||||
},
|
||||
)
|
||||
}
|
||||
uploaded.push(name.clone());
|
||||
}
|
||||
Err(ssh::ConnectFailure::Refused(e)) => return Err(e),
|
||||
};
|
||||
transfer?;
|
||||
|
||||
// Recorded only once the whole upload succeeded: the log backs the
|
||||
// duplicate-upload guard, and a failed upload must not count as
|
||||
// uploaded (a re-run replays every file — `sftp.create` truncates, so
|
||||
// replaying is safe).
|
||||
// uploaded (a re-run replays every file — `sftp.create` truncates and
|
||||
// the FTP `STOR` overwrites, so replaying is safe).
|
||||
record_upload(&upload_log_path()?, &record)?;
|
||||
|
||||
info!(
|
||||
@@ -212,6 +204,54 @@ async fn put_impl(opts: &PutOptions<'_>) -> Result<(), Box<dyn std::error::Error
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Push `uploads` over SFTP: payload files first, the `.changes` last
|
||||
/// (like dput), so the server-side queue processor can never pick up an
|
||||
/// incomplete upload. Best-effort removal of a partial upload, mirroring
|
||||
/// the FTP transport's `DELE` cleanup ([`ftp::upload_queue`]).
|
||||
fn sftp_transfer(
|
||||
session: &ssh2::Session,
|
||||
uploads: &[(PathBuf, String)],
|
||||
incoming: &str,
|
||||
host: &str,
|
||||
view: &dyn BuildView,
|
||||
) -> Result<(), Box<dyn std::error::Error>> {
|
||||
let sftp = ssh::sftp(session)?;
|
||||
|
||||
// Remote names of the files uploaded so far, in upload order (the
|
||||
// .changes last). A run failing mid-upload removes these from the
|
||||
// write-only incoming queue before returning: the uploaded payloads
|
||||
// would otherwise linger in the queue area forever, and a .changes
|
||||
// truncated by a failed close could even be picked up by the scanner.
|
||||
let mut uploaded: Vec<String> = Vec::new();
|
||||
|
||||
for (path, name) in uploads {
|
||||
let size = match path.metadata() {
|
||||
Ok(metadata) => metadata.len(),
|
||||
Err(e) => {
|
||||
// Nothing was attempted for this file: only what earlier
|
||||
// iterations uploaded needs removing
|
||||
let error: Box<dyn std::error::Error> =
|
||||
format!("cannot stat '{}': {}", path.display(), e).into();
|
||||
cleanup_partial_upload(&sftp, incoming, &uploaded, None, host);
|
||||
return Err(error);
|
||||
}
|
||||
};
|
||||
let remote = format!("{incoming}/{name}");
|
||||
let label = format!("Uploading {name}");
|
||||
let on_progress = |uploaded: u64| view.progress(&label, uploaded as usize, size as usize);
|
||||
let result = ssh::upload_file(&sftp, path, &remote, host, &on_progress);
|
||||
if let Err(e) = result {
|
||||
// The failed file itself joins the cleanup: its remote `create`
|
||||
// may have succeeded before the failure, leaving a partial — or,
|
||||
// on a failed close, a truncated — file behind
|
||||
cleanup_partial_upload(&sftp, incoming, &uploaded, Some(name), host);
|
||||
return Err(e);
|
||||
}
|
||||
uploaded.push(name.clone());
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// The remote names to attempt removing after a failed upload: everything
|
||||
/// already uploaded plus, when set, `failed` (the file whose upload just
|
||||
/// failed — its remote `create` may have succeeded before the failure,
|
||||
@@ -219,8 +259,9 @@ async fn put_impl(opts: &PutOptions<'_>) -> Result<(), Box<dyn std::error::Error
|
||||
/// so a `.changes` is removed before the payloads it references and the
|
||||
/// queue scanner never observes the payload set shrinking under a
|
||||
/// still-present `.changes`. Pure so the ordering decision is testable
|
||||
/// without a server; the network side is [`cleanup_partial_upload`].
|
||||
fn cleanup_list(uploaded: &[String], failed: Option<&str>) -> Vec<String> {
|
||||
/// without a server; the network sides are [`cleanup_partial_upload`] and
|
||||
/// the FTP transport's `DELE` loop ([`ftp`]).
|
||||
pub(crate) fn cleanup_list(uploaded: &[String], failed: Option<&str>) -> Vec<String> {
|
||||
let mut names: Vec<String> = uploaded.to_vec();
|
||||
if let Some(failed) = failed {
|
||||
names.push(failed.to_string());
|
||||
|
||||
Reference in New Issue
Block a user