spec: update spec
CI / Check, test, lint (push) Failing after 11m1s

- Default `p -- <cmd>` now streams logs via SSH (like tail -f)
   - Ctrl+C detaches from stream; job keeps running on worker
   - Add `-d/--detach` flag to start job without streaming
   - Remove `p attach` command (use `p logs -f` instead)
   - Remove p-agent daemon; jobs launched via nohup over SSH
   - Simplify worker requirements: only rsync needed (no tmux, no agent)
   - Jobs managed via ad-hoc SSH: kill $(cat pid), tail -f output.log
This commit is contained in:
2026-06-06 02:12:27 +02:00
parent 883114e2a3
commit 92132bc37a
+53 -91
View File
@@ -1,14 +1,15 @@
# `p` — push jobs to worker # `p` — push jobs to worker
A small Rust CLI utility to push command-line jobs to remote worker machines, A small Rust CLI utility to push command-line jobs to remote worker machines,
with directory sync, job management, and attach/detach support. with directory sync, job management, and log streaming.
## Motivation ## Motivation
The common developer workflow of "run this build/test/script on a more powerful The common developer workflow of "run this build/test/script on a more powerful
remote machine" currently requires manually chaining `rsync`, `ssh`, and `tmux`. remote machine" currently requires manually chaining `rsync` and `ssh` with
a way to keep the job alive in the background (e.g. `nohup`, `tmux`).
`p` wraps that entire flow into a single ergonomic command, while adding proper `p` wraps that entire flow into a single ergonomic command, while adding proper
job tracking, log capture, and attach/detach mechanics. job tracking and log capture.
## Core Concepts ## Core Concepts
@@ -26,13 +27,6 @@ directory. Each job has:
- Start time, end time, exit code - Start time, end time, exit code
- Captured output log - Captured output log
### p-agent
A small Rust (?) binary, automatically uploaded and started by `p` on first use
of a worker. It manages job lifecycle, log capture, and status tracking on
the worker side. The user never manually installs or configures it.
`p` checks the agent version on each connection and re-uploads if outdated.
Communication happens over SSH port forwarding — no extra open ports needed.
## CLI Reference ## CLI Reference
### Running jobs ### Running jobs
@@ -41,14 +35,19 @@ Communication happens over SSH port forwarding — no extra open ports needed.
p -- <command> p -- <command>
``` ```
Sync the current directory to the default worker and run `<command>` on it. Sync the current directory to the default worker and run `<command>` on it.
Attaches to the job's tmux session immediately. `Ctrl+B D` detaches without Streams the job's output directly to the terminal (like `tail -f`). This feels
killing the job. `Ctrl+C` sends SIGINT to the running process (standard behavior). like running the command locally.
When the job finishes, the session stays open and displays: - `Ctrl+C` detaches from the log stream — the job keeps running on the worker.
Use `p logs -f <job-id>` to resume watching.
- Use `p stop <job-id>` to kill a running job.
- If the network connection drops, the job keeps running on the worker.
Use `p logs -f <job-id>` to resume watching.
When the job finishes, `p` prints the exit code and exits:
``` ```
--- Job done [exit 0]. Press any key to detach. --- [Job done: exit 0]
``` ```
This lets the user read final output before returning to their shell.
``` ```
p <worker> -- <command> p <worker> -- <command>
@@ -59,7 +58,14 @@ Same, but targets a specific named worker.
p [-n | --no-sync] -- <command> p [-n | --no-sync] -- <command>
``` ```
Run `<command>` on the worker without syncing the current directory first. Run `<command>` on the worker without syncing the current directory first.
Useful for commands that need no local files (e.g. `p -n -- htop`). Useful for commands that need no local files.
```
p [-d | --detach] -- <command>
```
Run `<command>` and immediately detach — do not stream output to the terminal.
The job starts on the worker and `p` prints the job ID. Useful for fire-and-forget
jobs. Use `p logs -f <job-id>` to watch later.
### Job management ### Job management
@@ -71,19 +77,12 @@ completed jobs (done, failed, stopped).
Shows: ID (short), worker, original CWD, command, status, duration. Shows: ID (short), worker, original CWD, command, status, duration.
Style inspired by `docker ps` / `lxc list`. Style inspired by `docker ps` / `lxc list`.
```
p attach <job-id>
```
Re-attach to the tmux session of a running job. Supports partial IDs.
Behaves identically to the initial attach: `Ctrl+B D` detaches, and if the job
has already finished the "press any key" screen is shown.
Only works on **running** jobs. For finished jobs, use `p logs`.
``` ```
p logs <job-id> p logs <job-id>
``` ```
Print the captured output of a job (running or finished). Supports `-f` to Print the captured output of a job (running or finished). Supports `-f` to
follow a running job's output without attaching to its TTY. follow a running job's output in real-time. `Ctrl+C` detaches without stopping
the job.
``` ```
p stop <job-id> p stop <job-id>
@@ -148,62 +147,40 @@ Set the default worker.
- No automatic sync-back after job completion. Use `p pull` to retrieve - No automatic sync-back after job completion. Use `p pull` to retrieve
specific artifacts. specific artifacts.
## Attach / Detach Mechanics ## Execution Model
Jobs run inside a `tmux` session on the worker. `p` attaches to the session No persistent agent daemon is needed. Jobs are launched and managed via
immediately after starting the job. ad-hoc SSH commands:
### Status bar 1. `p -- <command>` syncs the directory, then runs via SSH:
The tmux session has a custom status bar showing: ```
``` nohup sh -c '<command> 2>&1 | tee output.log; echo $? > exitcode' & echo $! > pid
p-<short-id> beefy make [running] 0:02:14 ```
``` 2. The client streams `output.log` in real-time over a separate SSH connection.
Fields: job short-ID, worker name, command (truncated), status, elapsed time. 3. `Ctrl+C` closes the SSH stream — the job keeps running.
4. `p stop <job-id>` runs `kill $(cat pid)` over SSH.
5. `p logs -f <job-id>` tails the log file over SSH.
6. `p ls` reads the local job DB and SSH-polls to reconcile state when needed.
### Key bindings while attached > **Worker requirements:** `rsync` must be available on the worker.
| Key | Effect |
|---|---|
| `Ctrl+B D` | Detach from session. Job keeps running. |
| `Ctrl+C` | Sends SIGINT to the foreground process (standard terminal behavior). |
### On job completion
When the job's process exits, `run.sh` writes the exit code and then displays:
```
--- Job done [exit 0]. Press any key to detach. ---
```
The tmux session stays open (`remain-on-exit on` for the window) so the user
can scroll through final output. Pressing any key detaches the client and
returns to the local shell. `p` then reads the exit code and prints a summary.
### `p attach` on a finished job
If the job has already finished and the tmux session is still open (user has
not yet pressed a key), `p attach` reconnects to the "press any key" screen.
Once the key is pressed, the session closes. For a fully-closed session, use
`p logs` instead.
> **Worker requirements:** `tmux` and `rsync` must be available on the worker
> (standard on most Linux systems). The `p-agent` binary is auto-uploaded by `p`.
--- ---
## Job Status & Notification ## Job Status & Tracking
The **p-agent** runs as a lightweight background process on the worker The client maintains a local job database (`~/.local/share/p/jobs/<uuid>.json`).
(started automatically, not a system service). It: `p ls` reads from this local store for fast output.
- Manages job launch and tmux session creation ### State reconciliation
- Tees output to `output.log` When a job is running, the client periodically checks if `exitcode` exists on
- Writes `exitcode` on completion the worker. If the client was offline or the connection dropped, the next
- Notifies the client over the SSH reverse tunnel when a job finishes `p ls` SSH-polls workers to reconcile state. Jobs with unknown status are
marked accordingly.
The client maintains a local job database (`~/.local/share/p/jobs/<uuid>.json`) ### Connection drops during streaming
mirroring job state. `p ls` reads from this local store (fast, no SSH), If the SSH connection drops while `p` is streaming output, `p` exits with an
updated in real time while attached, and via agent notifications otherwise. error message showing the job ID. The job continues running on the worker.
Resume watching with `p logs -f <job-id>`.
### Degraded mode (agent unreachable / client was offline)
If the client missed a completion notification, `p ls` marks affected jobs as
`unknown`. The next `p ls` SSH-polls all workers with known-running jobs to
reconcile state.
## Worker-side Layout ## Worker-side Layout
@@ -211,8 +188,6 @@ All data lives under `~/.p/` on the worker (no root access required).
``` ```
~/.p/ ~/.p/
bin/
p-agent # auto-uploaded by p, versioned
jobs/ jobs/
<uuid>/ <uuid>/
cmd # command string cmd # command string
@@ -221,7 +196,8 @@ All data lives under `~/.p/` on the worker (no root access required).
started_at # unix timestamp started_at # unix timestamp
output.log # combined stdout+stderr, always captured output.log # combined stdout+stderr, always captured
exitcode # written on completion; absent = still running exitcode # written on completion; absent = still running
tmux_session # tmux session name (e.g. "p-<short-uuid>") pid # process ID of the running job
workdirs/ workdirs/
<uuid>/ # rsync'd copy of client CWD for this job <uuid>/ # rsync'd copy of client CWD for this job
``` ```
@@ -252,23 +228,9 @@ b004f123 cloud ~/scripts ./bench.sh done [1] 0:00:47
## Open Questions ## Open Questions
- **Worker arch detection**: `p-agent` must be compiled for the worker's
architecture. Options: (a) ship common targets and detect via SSH, (b)
compile on the worker if a Rust toolchain is present, (c) require user to
specify arch in worker config.
Maybe we could also implement the agent core in the form of a shell script?
At least the entry point, which could do some detection and install or something.
We will see on implementation what works... Small rust binary seems nice,
but we want support for amd64, aarch64 and riscv64.
- **Multiple jobs from the same CWD**: each gets its own `workdirs/<uuid>/`, - **Multiple jobs from the same CWD**: each gets its own `workdirs/<uuid>/`,
so they're fully isolated. This may use significant disk space — `p rm` so they're fully isolated. This may use significant disk space — `p rm`
should prompt to clean up. should prompt to clean up.
- **Non-Linux workers**: tmux availability and path conventions may differ on - **Non-Linux workers**: path conventions may differ on macOS workers. Out of
macOS workers. Out of scope for now. scope for now.
- **Ctrl+C → detach** (future): it would be nicer if Ctrl+C detached the
session instead of sending SIGINT to the job, matching the spirit of the
tool. This requires per-session tmux key table configuration and is deferred.