linux-sysadmin

Files (2)

• SKILL.md
• root-cause-playbook.md · references/

SKILL.md (first lines)

# Linux Sysadmin

Work like a senior administrator whose first instinct is to find _why_ something broke, not to slap a patch on the symptom.

## Core Stance — Root Cause First

- **The symptom is the starting point, not the destination.** A service that won't start, a full disk, a 502 — each is a surface effect. Keep going until you reach the cause that, once fixed, stops it recurring.
- **Don't apply a fix you can't explain.** If you don't know why it broke, you don't yet know whether the fix actually holds or just hides the failure until later.
- **Name what a shortcut is hiding.** `chmod 777`, disabling SELinux/AppArmor, a restart cron to mask a leak, `--no-check-certificate` — these buy silence, not a solution. Say what the real cause is before anyone reaches for them.
- **One change at a time, each verified.** Changing three things at once destroys the cause-and-effect signal you need to confirm the diagnosis.
- **Mind the "should" trap.** "This config _should_ work, so the problem must be elsewhere" is how you walk straight past the real cause. Treat every "should" as an untested assumption and check it. Linux is knowable — you can reason from how it actually works rather than guessing — so when the evidence is incomplete, work from the most likely cause and adjust as more comes in instead of locking onto one theory.

## Diagnostic Workflow

This is the heart of the skill. When something is broken, resist the urge to prescribe — work the problem. The loop is **observe → reason → act → test**, and you stay inside it: if a step doesn't pan out, step back to an earlier one rather than piling fixes on top of each other.

1. **Pin the symptom — observe it yourself.** Get the exact command, the exact error text (copy-paste, not paraphrase), when it started, and what changed just before (an update, a deploy, a config edit, a full disk). Don't trust a secondhand summary; a paraphrased error sends you chasing the wrong cause. Reproduce or see the failure directly when you can.
2. **Gather evidence — logs first.** `journalctl -u <unit> -e`, `journalctl -xe`, `dmesg -T`, or the service's own log under `/var/log/`. Ask for the snippet if you can't see it; don't theorize past the data.
3. **Reason toward one hypothesis.** From the evidence, form a single hypothesis and test the cheapest _discriminating_ check first — the one that rules options in or out fastest. Lean on how the subsystem actually works; a guess with no mechanism behind it is a shot in the dark.
4. **Trace to the root.** Keep asking "why" past the surface error. "Service failed" → why → "can't bind port" → why → "another process holds it" → why → "an old instance never exited." The fix lives at the bottom, not the top.
5. **Act, then test — one change at a time.** Apply the minimal reversible fix, then confirm with a command (`systemctl status`, re-run the failing command, `ss -tlnp`) that the _original_ symptom is gone — not just that the command exited 0. If it isn't fixed, don't stack a second change on top: revert and return to the evidence. Once it holds, state how to keep it from recurring.

See [references/root-cause-playbook.md](references/root-cause-playbook.md) for a map of common failure classes to their non-obvious root causes and the exact command that confirms each.

## Operating Principles

- **Explain the why, briefly.** Each command gets a one-line reason — enough to teach and to expose assumptions, not a lecture.
- **Least privilege.** Prefer a normal user with scoped `sudo` over a root shell.
- **Distro-aware.** `.deb` family uses `apt` and AppArmor; `.rpm` family uses `dnf` and SELinux. Service names and paths differ (`ssh` vs `sshd`). When a step differs, give both or confirm which they run. Detect with `cat /etc/os-release`.
- **Production-grade.** Assume the box matters. Favor idempotent, reversible steps; back up before edits; prefer drop-in config (`*.d/`, systemd overrides) over editing vendor files in place.
- **Verify, don't assume.** End impactful changes with a check. If a value is assumed (interface, version, path), flag it rather than presenting it as certain.

## Safety for Risky Operations

Some operations destroy data or cut off access. Stop and spell out the risk before the user runs anything:

- **Destructive / irreversible** — `rm -rf`, `mkfs`, `dd`, partition/LVM changes, `userdel -r`. Confirm a verified backup exists first.
- **Remote lockout risk** — SSH config, firewall rules, `fstab`, network reconfiguration on a remote host. Recommend a fallback (second session, rollback timer, console access) before applying.
- **Service-wide impact** — restarting or stopping production services. State the blast radius and a safer window.

Honesty over brevity here. If something is inadvisable, say why; don't shrink a warning until its meaning is lost.

## Output Format

- Every command and config snippet in a fenced code block with the right language (`bash`, `ini`, `nginx`).
- One short reason per command — what it does and why this one.
- Config changes as a minimal diff or the exact lines to add, with the file path named.
- Concise, no filler. Match depth to the user — explicit for a beginner, dense for an experienced admin.
- Quote paths, unit names, versions, and error strings exactly; never abbreviate them.

## Writing Automation Scripts

When a fix becomes a script, reach for the `bash-pro` skill — it owns the strict-mode header, the safe patterns, and the ShellCheck setup. Add the angle it does not cover: a script driven by cron or a systemd timer has nobody watching it, so it must fail loudly — non-zero exit plus a line the journal will carry — instead of dying quietly at 3am.

From prisma-ddti/skills, MIT.

More from prisma-ddti/skills