ssh-docker-incident-response

Files (1)

• SKILL.md

SKILL.md (first lines)

# Docker Incident Response

When a container is dead, looping, OOMing, or the host is out of disk, follow
this sequence. Uses only `safe`/`read` tier tools by default; low-access +
dangerous steps are explicitly marked and guarded.

## Default-on cheatsheet rejection (since v1.9.0)

`ssh_exec_run` refuses commands that have a native MCP tool -- see
`skills/ssh-exec-run/SKILL.md`. The native-tool flow below avoids
that. Composite scripts (where the script IS the artefact) opt out
via `SSH_EXEC_ALLOW_CHEATSHEET_PATTERNS=true` at the operator level.

Every docker action below uses an `ssh_docker_*` wrapper; the only
`ssh_exec_run` mention in this runbook is a prose reference to
`ss -tlnp` for port-conflict diagnosis, which has no native wrapper
and is not cheatsheet-matched.

## 1. Get the full container inventory

```python
ssh_docker_ps(host="docker1", all_=True)
```

`all_=True` includes stopped containers -- a missing container is often the
whole diagnosis. Read the list:

- Container you expect is **missing** -> it died hard (see Section 3 to
  find out why) or it was never started (check `ssh_docker_images` and the
  compose file).
- Container is **restarting** / stuck in `Created` -> restart loop; go to
  Section 3.
- Container is `Up` but the service is still broken -> go to Section 4.

## 2. Triage host-level resource pressure + recent events

Container symptoms often reflect host-level problems. In parallel:

- `ssh_host_disk_usage` -- Docker blows up hard when `/var/lib/docker`
  partition is > 90% full. Look for the mount hosting Docker's data-root.
- `ssh_host_alerts` -- one call for disk / load / memory against the host's
  configured thresholds.
- `ssh_docker_stats(host=...)` -- one-shot CPU/memory/net/block I/O per
  container. The container eating 95% CPU or 100% memory is usually the
  culprit, even if it's not the one the user reported.
- `ssh_docker_events(host=..., since="30m")` -- the daemon's own log of
  what just happened. Often tells you the whole story in 30 seconds:
  OOM kills surface as `event=oom`, crashes as `event=die` with an exit
  code attribute, surprise image pulls as `type=image event=pull`.
  Narrow with `filters=["container=<name>"]` once you know the suspect.

If disk is the bottleneck, jump to Section 6.

## 3. Root-cause a failing container

For the suspect container, pull its metadata + logs in parallel:

```python
ssh_docker_inspect(host="docker1", target="nginx", kind="container")
ssh_docker_logs(host="docker1", container="nginx", tail=200)
```

Read `inspect` output first -- it tells you **why** the container is in
its current state without you having to parse logs:

- `State.Status` -- `exited` / `restarting` / `dead` / `running`.
- `State.ExitCode` -- non-zero means the process died on its own.
  - `137` -> SIGKILL, usually OOM (cross-check with `State.OOMKilled: true`).
  - `143` -> SIGTERM, clean shutdown requested (so the orchestrator killed it).
  - `1` / `2` -> application-level error; check logs.
- `State.Health.Status` -- `healthy` / `unhealthy` / `starting`. If
  unhealthy, `State.Health.Log[-1].Output` has the last healthcheck output.
- `RestartCount` -- high number + recent `State.StartedAt` = restart loop.
- `HostConfig.RestartPolicy.Name` -- `no` / `on-failure` / `always` /
  `unless-stopped`. Tells you whether Docker is trying to recover.
- `Mounts[]` -- bind mounts / volumes. A missing source directory is a
  common silent failure.

Then `ssh_docker_logs` to see what the container itself logged. Start with
`tail=200`; raise to 10000 only if the truncation is eating the error.
Prefer `since="10m"` when you know roughly when the problem started:

```python
ssh_docker_logs(host="docker1", container="nginx", since="15m", tail=5000)
```

## 4. Running-but-broken container

Container is `Up` but the service is unreachable / slow / wrong.

- `ssh_docker_top(host="docker1", container="nginx")` -- what's actually
  running inside? PID 1 should match the image's entrypoint. If PID 1 is
  a shell / init wrapper waiting on a child, the real process died and
  the restart policy didn't catch it.
- `ssh_docker_inspect ... "network"` -- check the container's network
  mode, published ports, and IP. A container on the wrong network or with
  no published port explains "reachable from host, unreachable from
  outside".
- `ssh_docker_stats` -- sustained 100% CPU or near-limit memory -> the
  workload is saturated, restart won't help; scaling or limits config is
  the fix.

## 5. Compose-stack failures

If the project is compose-managed:

```python
ssh_docker_compose_ps(host="docker1", compose_file="/opt/app/docker-compose.yml")
ssh_docker_compose_logs(host="docker1", compose_file="/opt/app/docker-compose.yml",
                        tail=200)
```

`compose_ps` shows each service's state (`running` / `exit 1` / ...). When
a service keeps failing to come up, the failure is almost always one of:

- Dependency service isn't healthy -- `depends_on` with `condition:
  service_healthy` waits but can time out.
- Volume mount references a host path that doesn't exist -- `docker
  inspect` shows the bind mount source; `ssh_sftp_stat` against that path

From Nightreaver/python-ssh-mcp, GPL-3.0.

More from Nightreaver/python-ssh-mcp