ssh-long-running-job

Files (1)

• SKILL.md

SKILL.md (first lines)

# SSH Long-Running Job

Workflow runbook for "this command will run for hours and must survive
the MCP server, my conversation, and a flaky link". Pattern: run the
command in a detached `tmux` session on the remote, record the session
name + log path in [agent host-notes](../../skills/ssh-host-notes-append/SKILL.md),
poll until done. The motivating case is a multi-TB `rsync`, but the
same shape fits long `restic` backups, big `pg_dump`/`pg_restore`, mass
`apt` upgrades on slow boxes, dataset downloads, etc.

`ssh_exec_run_streaming` covers "minutes, not seconds" -- this runbook
covers "hours-to-days", where the job MUST be decoupled from the SSH
channel and the MCP process.

## Default-on cheatsheet rejection (since v1.9.0)

`ssh_exec_run` refuses commands that have a native MCP tool -- see
`skills/ssh-exec-run/SKILL.md`. The native-tool flow below avoids
that. Composite scripts (where the script IS the artefact) opt out
via `SSH_EXEC_ALLOW_CHEATSHEET_PATTERNS=true` at the operator level.

Every `ssh_exec_run` call in this runbook is a `tmux` / `tail` /
`grep` / `cat` / `command -v` invocation -- none of those have
native MCP wrappers and none are cheatsheet-matched. The launch
command (Section 2) wraps the long-running job inside `tmux new -d`;
that wrapper IS the long-job artefact (its detachment + rc-file
contract is what makes resume possible), so it intentionally runs
as a single exec under one audit correlation id.

## Why tmux and not the streaming tool

| Concern | `ssh_exec_run_streaming` | This runbook (tmux + notes) |
|---|---|---|
| Per-call timeout | bounded by `timeout` arg | none -- job outlives every call |
| Survives MCP restart | only with Redis docket | yes, unconditionally |
| Survives network blip | no (`ConnectError` aborts the stream) | yes |
| Resumable | no | yes (rsync `--partial`, restic checkpoints) |
| Future conversation can find it | no | yes (notes auto-injected on `ssh_host_ping`) |
| Live progress to the client | streaming chunks | poll `tail` of the log |

If any row in column 3 doesn't matter for your case, use the streaming
tool -- it's simpler.

## Sequence

1. Precheck source host (read)
2. Launch in detached tmux + log to disk (dangerous: `ssh_exec_run`)
3. Record job in host-notes (low-access: `ssh_host_notes_append`)
4. Poll until tmux session is gone (read)
5. Resume from a future session (read + low-access)
6. Verify exit code + finalize note (low-access)
7. Failure recovery (dangerous)

## 1. Precheck source host

The "host" the MCP server connects to is wherever the long-running
command *runs from* -- for an rsync push, that's the source. Confirm:

```python
# tmux installed?
ssh_exec_run(host="nas01", command="command -v tmux", timeout=10)
# disk for the log file
ssh_host_disk_usage(host="nas01")
# nothing already in flight under the same name (avoid clobbering a
# prior run)
ssh_host_notes(host="nas01")  # look for an existing [long-run] line
ssh_exec_run(host="nas01", command="tmux ls 2>/dev/null", timeout=10)
```

If tmux isn't installed, stop and ask the operator -- this runbook
does not silently fall back to `nohup` or `screen` because the
polling commands below are tmux-specific. (Replace the launch +
poll lines if you really need `screen`; the rest of the shape is
identical.)

## 2. Launch in detached tmux

Pick a stable, dated session name -- it doubles as the handle for
every subsequent poll AND the slug the future-you will grep for in
host-notes. Format: `<task>-<UTC-date>-<short-tag>`.

```python
from shlex import quote as shlex_quote

job = "xfer-2026-05-19-rsync-nas-to-dst"   # session name + log slug
log = f"/var/log/{job}.log"
rc  = f"/var/log/{job}.rc"

# wrap the command so its exit code lands in a known file when the
# session exits. without this you can tell "it's done" but not
# "it succeeded".
inner = (
    f"rsync -aP --partial --append-verify --log-file={log} "
    f"/data/ user@dst01:/data/ ; echo $? > {rc}"
)

ssh_exec_run(
    host="nas01",
    command=f"tmux new -d -s {job} {shlex_quote(inner)}",
    timeout=15,
)
```

Key flags:

- `tmux new -d -s <name>` -- detached; returns immediately, tmux owns
  the process. No PID juggling needed; the session name IS the handle.
- `--partial --append-verify` (rsync) -- a network blip or full restart
  resumes from the last good byte instead of retransferring TB.
- `--log-file=...` (rsync) -- progress is preserved even if no one is
  watching. Don't rely on `tmux capture-pane` for completeness; the
  scrollback is bounded.
- `echo $? > <rc>` -- the success signal. `tmux has-session` only
  tells you the session is GONE, not whether the job succeeded; an
  rsync that aborted with rc=23 (partial transfer) and one that
  finished cleanly look identical from tmux's perspective.

The MCP `ssh_exec_run` call returns in milliseconds because tmux
detached. The rsync is now uncoupled from the MCP channel.

From Nightreaver/python-ssh-mcp, GPL-3.0.

More from Nightreaver/python-ssh-mcp