bip-scout

Files (1)

• SKILL.md

SKILL.md (first lines)

# /bip-scout

Check remote server availability and resource usage.
Runs `bip scout` to collect metrics, presents a human-readable summary, and answers follow-up questions by reasoning over the data.

## Usage

```
/bip-scout [question]
```

**Arguments:**
- `[question]` — Optional natural-language question (e.g., "which server has free GPUs?", "where should I run a training job?")

If no question is given, show a full summary of all servers.

## Workflow

### Step 1: Collect Server Metrics

Run `bip scout` to get JSON output:

```bash
bip scout
```

**If the command fails**, check for common issues:
- Missing `servers.yml`: "No servers.yml found — see `bip scout --help` for config location."
- SSH authentication failure: Report the error and suggest checking SSH agent and `~/.ssh/config`.
- All servers offline, or several at once: **do NOT report an outage — verify one host by direct `ssh <host> uptime` first.**
  `bip scout` reports a pre-auth SSH throttle as `OFFLINE` rather than as an error, so a burst of scouts produces a false fleet-wide outage.
  Observed 2026-08-31: five hosts (orca01, orca03, orca04, orca05, conatus) all reported `OFFLINE`; every one answered a direct `ssh` seconds later.
  Cause confirmed 2026-09-03, having been inferred in the 08-31 case: scout reported orca01, orca03, orca04 and quokka `offline` with `handshake failed ... connection reset by peer` against `snail.fhcrc.org`; direct `ssh` seven minutes later found **orca01 at load 0.59 and orca03 at load 0.06 — the two idlest machines reachable**, and both were immediately given real jobs.
  So the failure is not merely "scout is sometimes wrong": it is **biased toward reporting exactly the hosts you most want**, because a scout sweep hits them in the same burst that triggers the throttle. Taking it at its word cost nothing only because someone checked.
  Only report an outage once a direct `ssh` also fails; a genuine one will fail that too, so the check costs a second and the alternative is standing down a fleet that is up.
  Once confirmed, if `ListAgents` has a session named `infra` (`/bip-infra`), send it the report too.

### Step 2: Parse JSON Output

The JSON output has this structure:

```json
{
  "servers": [
    {
      "name": "servername",
      "status": "online",
      "metrics": {
        "cpu_percent": 45.2,
        "memory_percent": 62.1,
        "load_avg_1min": 2.3,
        "load_avg_5min": 1.8,
        "load_avg_15min": 1.5,
        "gpus": [
          {
            "utilization_percent": 85,
            "memory_used_mb": 30720,
            "memory_total_mb": 49152,
            "users": ["alice"]
          }
        ],
        "top_users": [
          {"user": "alice", "cpu_percent": 45.2},
          {"user": "bob", "cpu_percent": 12.1}
        ]
      }
    },
    {
      "name": "deadserver",
      "status": "offline",
      "error": "connection timed out"
    }
  ]
}
```

### Step 3: Present Results

**If no question was asked**, present a summary table showing:
- All servers grouped by status (online first, offline last)
- For online servers: CPU%, Memory%, per-GPU utilization and memory
- For offline servers: note them as unreachable
- Highlight servers with low utilization (CPU < 20%, no busy GPUs) as good candidates

**If a question was asked**, reason over the JSON data to answer it.
Common question types:

| Question Type | How to Answer |
|---------------|---------------|
| "Which server has free GPUs?" | Find online servers where GPU utilization < 50% and GPU memory has headroom |
| "Where should I run a training job?" | Find servers with lowest GPU utilization and most free GPU memory |
| "Is server X available?" | Check that specific server's status and metrics |
| "What's the GPU memory on Y?" | Report per-GPU memory used/total for that server |
| "Which servers are idle?" | Find servers with CPU < 20% and memory < 50% |
| "Who is using server X?" | Report top_users with their CPU percentages (100% = one core), and each GPU's `users` |

A GPU with no `users` is free only if the server's `metrics` has no `gpu_owners_unknown: true`. That flag means the owner listing failed, so every GPU on that server shows no users whether or not it is in use.

**Formatting guidelines:**
- Use a markdown table for multi-server summaries
- Convert GPU memory from MB to GB for readability (divide by 1024)
- Show percentages with one decimal place
- For GPU-focused questions, show per-GPU breakdown (not just averages)

### Step 4: Offer Follow-Up

After presenting results, briefly note that the user can ask follow-up questions like:
- "Which of these has the most free GPU memory?"
- "Run my job on the least-loaded GPU server"

## Running After Scouting

Once you've picked a host, three rules keep remote runs from silently breaking or clobbering a neighbor:

- **Never write `cd` in an ssh command string — pin the directory as a flag.**
  This is a *hard mechanical rule*, not advice: composing `ssh host "cd /long/path && <cmd>"`, the `cd /long/path && ` prefix gets silently dropped from what actually runs, so the command executes in `$HOME` (→ "not a git repository", the wrong project resolved, etc.).
  This recurs *every session* and prose warnings have not stopped it — the fix is structural: make the working directory an un-droppable argument.
  - **Single command → use the tool's dir flag:** `git -C <dir> …`, `uv run --directory <dir> …`, `make -C <dir> …`, `tar -C <dir> …`, `snakemake --directory <dir> …`.
  - **Multi-step → write a script on the remote and run *that*.**

From matsen/bipartite, MIT.

More from matsen/bipartite