infra-automation

Files (1)

• SKILL.md

SKILL.md (first lines)

# Infrastructure Automation

Reusable patterns for SSH-based operations across Arcana servers.

## Server Inventory

| Name | Public IP | Tailscale IP | SSH |
|------|-----------|-------------|-----|
| www | <www-public-ip> | <www-tailscale-ip> | `ssh root@<www-host>` |
| prod | <prod-public-ip> | <prod-tailscale-ip> | `ssh root@<prod-host>` |
| db | <db-public-ip> | <db-tailscale-ip> | `ssh root@<db-host>` |
| trading | <trading-public-ip> | <trading-tailscale-ip> | `ssh root@<trading-host>` |

> Always verify current IPs against your own private infrastructure inventory (never hardcode real addresses in this shipped skill) before use.

## Post-Provision Checklist — Public IP

Any server inventory record (this table, a project's `space.yml`, or equivalent) for a host with a routable public address MUST have that address recorded before the provisioning task is closed. Do not leave the field `null`/blank "for later" once a routable address exists.

**Why.** A server bootstrapped without its public IP recorded reads as unreachable/dark to anyone consulting the inventory later — the address exists and is reachable, but the record gives no evidence of that, so an operator has no way to distinguish "not yet provisioned" from "provisioned, IP not written down". The gap surfaces only when someone happens to `ssh root@<ip>` from a stale note elsewhere and is surprised the host answers.

**Rule.** As the last step of provisioning (or first step of onboarding an existing host into the inventory), confirm the host has a routable public interface (`ip -4 addr show` on the host, or the cloud provider's dashboard) and write the address into the inventory record in the same commit/change as the rest of the provisioning artefacts.

**Optional lint.** A pre-commit or CI check MAY flag an inventory record where a public-interface field is `null`/empty while the host is reachable at a known address — treat this as a checklist reminder, not a hard gate, since some hosts are intentionally NAT-only.


## SSH Batch Execute

> **Bootstrap once** (Datarim § Security Mandate S1, host-key verification):
> add the host key to `~/.ssh/known_hosts` on the operator machine before any
> batch SSH automation. Document the exact bootstrap event in
> `documentation/infrastructure/known-hosts-rotation.md`.
>
> ```bash
> for host in <HOST_LIST>; do
>   ssh-keyscan -H "$host" >> ~/.ssh/known_hosts
> done
> ```

**Mesh-health pre-check.** Before running a fleet sweep or batch command, verify
Tailscale reachability for each node. Unreachable nodes fall back to public IP or
are deferred — never silently skipped without a log entry:

```bash
# nosec-extract
for host in <MESH_IP_LIST>; do
  if ping -c1 -W2 "$host" >/dev/null 2>&1; then
    echo "mesh-ok: $host"
  else
    echo "mesh-unreachable: $host — falling back to public IP or deferring" >&2
  fi
done
```

Record the reachability result in the sweep log before proceeding.

Run a command on all (or selected) servers (relies on default
`StrictHostKeyChecking=ask` — bootstrap above pre-populates `known_hosts` so the
prompt never fires; an unknown host fails fast in batch mode):

```bash
# nosec-extract
for host in <HOST_LIST>; do
  echo "=== $host ==="
  ssh -o BatchMode=yes -o ConnectTimeout=5 "deploy@$host" "<COMMAND>" 2>&1 | head -20
done
```

**Flags:** `-o BatchMode=yes` disables interactive prompts (fails fast on
unknown host or missing key). `-o ConnectTimeout=5` prevents hanging on
unreachable hosts. Use `deploy@` user with narrow `sudo` rules; reserve `root@`
for one-shot bootstrap with a logged EOL date.

<!-- security:counter-example -->
# UNSAFE — bypasses host-key verification, never use in shipped recipes:
ssh -o StrictHostKeyChecking=no -o BatchMode=yes "$host" "<COMMAND>"
<!-- /security:counter-example -->

## Ping Matrix (Tailscale)

Test NxN connectivity across all devices in mesh:

```bash
declare -A TSIP=([www]="$WWW_TS_IP" [prod]="$PROD_TS_IP" [db]="$DB_TS_IP" [trading]="$TRADING_TS_IP")
declare -A PUBIP=([www]="$WWW_PUB_IP" [prod]="$PROD_PUB_IP" [db]="$DB_PUB_IP" [trading]="$TRADING_PUB_IP")

for SRC in www prod db trading; do
  printf "%-10s" "$SRC"
  for DST in www prod db trading; do
    [ "$SRC" = "$DST" ] && { printf "%-14s" "—"; continue; }
    LAT=$(ssh -o BatchMode=yes root@${PUBIP[$SRC]} \
      "ping -c 2 -W 3 ${TSIP[$DST]} 2>/dev/null | awk -F'/' '/min\/avg/{print \$5}'" 2>/dev/null)
    [ -n "$LAT" ] && printf "%-14s" "✅ ${LAT}ms" || printf "%-14s" "❌"
  done; echo
done
```

## Tailscale Enrolment (auth-key liveness)

An auth-key's `Expires:` date in the stored credentials is NOT proof of
liveness — a key can be revoked/deleted in the admin before its expiry, and
`tailscale up` then fails `invalid key: API key does not exist`. Do not assume
a dated creds entry is usable.

Recovery is autonomous when an API access token is stored: generate a fresh
key and approve routes over the Tailscale API, no admin-UI step required.

```bash
# Generate a fresh reusable tagged auth-key (from stored tskey-api-... token):
curl -s -X POST -H "Authorization: Bearer $TS_API" -H "Content-Type: application/json" \
  https://api.tailscale.com/api/v2/tailnet/-/keys \
  -d '{"capabilities":{"devices":{"create":{"reusable":true,"ephemeral":false,"preauthorized":true,"tags":["tag:server"]}}},"expirySeconds":7776000,"description":"<task-id> enrolment"}'
# (description must be alphanumeric-ish — parentheses/special chars → "invalid characters")

# Approve an exit-node's advertised routes (replaces admin-UI "approve exit node"):
DEVID=$(curl -s -H "Authorization: Bearer $TS_API" https://api.tailscale.com/api/v2/tailnet/-/devices \
  | python3 -c "import sys,json;print([d['id'] for d in json.load(sys.stdin)['devices'] if d['hostname']=='<name>'][0])")
curl -s -X POST -H "Authorization: Bearer $TS_API" -H "Content-Type: application/json" \
  "https://api.tailscale.com/api/v2/device/$DEVID/routes" -d '{"routes":["0.0.0.0/0","::/0"]}'

From Arcanada-one/datarim, MIT.

More from Arcanada-one/datarim