ssh-os-upgrade

Files (1)

• SKILL.md

SKILL.md (first lines)

# SSH OS Upgrade

Choreography for taking a host across a major OS boundary (e.g. Ubuntu
jammy -> noble) without ad-hoc improvisation. Read-only verification
calls are LLM-driven; package mutations and `do-release-upgrade` itself
are operator-driven inside `tmux` so they survive the inevitable
mid-upgrade SSH session loss.

Not for: minor security updates (unattended-upgrades handles those, no
choreography needed); kernel-only updates within the same release
(`apt upgrade && reboot`); container / app upgrades that don't touch
the OS.

## Default-on cheatsheet rejection (since v1.9.0)

`ssh_exec_run` refuses commands that have a native MCP tool -- see
`skills/ssh-exec-run/SKILL.md`. The native-tool flow below avoids
that. Composite scripts (where the script IS the artefact) opt out
via `SSH_EXEC_ALLOW_CHEATSHEET_PATTERNS=true` at the operator level.

This runbook is operator-driven: Section 6 (`do-release-upgrade`),
Sections 4 and 8 (`tmux new -d -s ... 'apt update && apt upgrade ...
| tee'`), and the Section 5 docker-stack-stop loop are all
multi-step bash scripts that the operator runs from their own SSH
session (not via `ssh_exec_run`). Where the runbook does call
native MCP tools (Section 1's snapshot via `ssh-host-snapshot`,
Section 7's `ssh_host_ping` / `ssh_host_info` / `ssh_host_alerts`,
package mutations via `ssh_apt_*`, service mutations via
`ssh_systemctl_*`), use those wrappers directly.

If you DO drive any step from the LLM via `ssh_exec_run`
(e.g. catching up the current release with `apt update && apt upgrade`
inside tmux), the wrapped command is a composite artefact -- the
tmux+apt+dpkg-options+tee bundle MUST run atomically under one
audit correlation id to behave correctly. Set
`SSH_EXEC_ALLOW_CHEATSHEET_PATTERNS=true` for that opt-out, then
prefer `ssh_apt_upgrade` for non-tmux-wrapped catch-ups instead.

## Sequence

1. Pre-flight + snapshot
2. Cleanup orphans
3. Hold app packages
4. Catch up current release (apt upgrade, **tmux-wrapped**)
5. Stop service stacks (internal -> mid -> edge, edge-proxy LAST)
6. `do-release-upgrade` interactively (operator-driven, tmux)
7. Post-reboot verification
8. Remap third-party APT sources + final `apt upgrade`
9. Bring services back up (edge-proxy LAST, again)
10. Post-snapshot + diff vs pre-upgrade snapshot

## 1. Pre-flight + snapshot

- Capture via [ssh-host-snapshot](../ssh-host-snapshot/SKILL.md) with
  label `pre-<target-codename>` (e.g. `pre-noble`). Pull off-box per
  that runbook's Section 4 -- a snapshot only on the host being
  upgraded is the wrong place for it.
- Verify the host's backup system (bareos / restic / borg) is actually
  running and the most recent job completed cleanly.
- Confirm rescue-mode access at the hosting provider (Hetzner Robot,
  Hetzner Cloud console, etc.). That is the Plan B if `do-release-
  upgrade` breaks boot.
- Read `ssh_host_notes` for the host. Per-host quirks (forbidden
  packages, custom port bindings, edge-proxy dependency order) live
  there, not in this generic runbook.

**What the standard snapshot does NOT cover** -- worth knowing because
the OS upgrade can land you in territory the snapshot can't restore by
itself:

- `/boot/grub/grub.cfg` and `/boot/grub/i386-pc/*` -- the snapshot
  captures `/etc/default/grub` and `/etc/grub.d/*` (so `update-grub`
  can regenerate `grub.cfg`), but the generated config itself and the
  grub modules aren't in the tarball.
- The MBR boot code (stage 1 + stage 1.5 in the MBR gap). Not
  filesystem-resident. If grub-install corrupts it, only Hetzner
  rescue + `dd` from a backup recovers.

If you anticipate touching the bootloader (Section 6 GRUB prompts
might force this), capture these extras as a mini-snapshot **before**
the change -- see Section 6's "If you hit GRUB prompts" subsection.

## 2. Cleanup orphans

Stuff that makes the upgrade noisier than it needs to be:

- Exited compose containers from removed services: `docker ps -a
  --filter status=exited` -> `docker rm <name>` once confirmed not
  referenced in any compose file.
- dpkg `rc`-state packages (removed-but-conffiles-remain):
  `dpkg -l | awk '/^rc/ {print $2}'` -> `dpkg --purge <pkg>`. They can
  reactivate dependencies during the upgrade if anything pulls them
  back transitively.
- Failed systemd units: `systemctl --failed` -> resolve or mask before
  proceeding.

Per-host traps (e.g. "do not let apt re-pull apache2") belong in
`ssh_host_notes` -- check before purging.

## 3. Hold app packages

`do-release-upgrade` will move every package available in its target
dist's repos. If you don't want application stacks to move at the same
time, hold them now. From the LLM use the dedicated wrapper:

```python
ssh_apt_mark(host="...", action="hold",
             packages=["postgresql-14", "mariadb-server"])
```

Operator-side equivalent (run from the operator's own root shell):

```bash
apt-mark hold <package> [<package>...]
```

Worth holding:

- Database engines (postgres, mariadb, mongodb) where major-version

From Nightreaver/python-ssh-mcp, GPL-3.0.

More from Nightreaver/python-ssh-mcp