remote-cluster-agent

Files (47)

• .gitignore
• MUTAGEN.md
• Makefile
• README.md
• README.zh-CN.md
• SKILL.md
• VERSION
• agent.py · cluster-agent/
• cmd_agent.go · cmd/rca/
• cmd_batch.go · cmd/rca/
• cmd_config.go · cmd/rca/
• cmd_connect.go · cmd/rca/
• cmd_cp.go · cmd/rca/
• cmd_daemon_register.go · cmd/rca/
• cmd_exec.go · cmd/rca/
• cmd_nodes.go · cmd/rca/
• main.go · cmd/rca/
• architecture.html · docs/
• architecture.png · docs/
• go.mod
• go.sum
• connection.go · internal/agent/
• connection_test.go · internal/agent/
• protocol.go · internal/agent/
• sshcmd.go · internal/agent/
• sshcmd_test.go · internal/agent/
• client.go · internal/client/
• client_test.go · internal/client/
• config.go · internal/config/
• config_test.go · internal/config/
• migrate.go · internal/config/
• migrate_test.go · internal/config/
• handlers.go · internal/daemon/
• handlers_test.go · internal/daemon/
• monitor.go · internal/daemon/
• monitor_test.go · internal/daemon/
• pool.go · internal/daemon/
• pool_test.go · internal/daemon/
• server.go · internal/daemon/
• transfer.go · internal/daemon/
• transfer_test.go · internal/daemon/
• types.go · internal/protocol/
• types_test.go · internal/protocol/
• com.rca.daemon.plist · launchd/
• mutagen-setup.sh
• cluster-health.md · reference/
• mutagen-troubleshooting.md · reference/

SKILL.md (first lines)

# remote-cluster-agent

Local control plane for remote GPU clusters: a Go daemon + `rca` CLI. All commands go through the daemon's Unix socket to a cluster-side Python `agent.py` over persistent SSH, ~0.1s/command latency.

## What to do when triggered

1. **Check if `rca` is installed**: run `which rca`.
   - If `not found` → run the first-time install flow (below), then continue to step 2.
2. **Check daemon status**: run `rca daemon status`.
   - If `daemon request failed` → run `rca daemon start`.
3. **Check node connectivity**: run `rca nodes`.
   - If the target node is dead → `rca connect <node>`. Still dead → re-establish whatever SSH tunneling / VPN / jump host you depend on, then retry.
   - Second round still dead → read `rca daemon logs -f` for error detail.
4. Run the user's command.

**Don't write raw `ssh` commands**, always use `rca exec` / `rca batch`.

### First-time install (auto, no user action needed beyond SSH info)

When `which rca` returns not found, execute in order:

```bash
cd <skill_dir>
make install              # builds rca into ~/go/bin (requires Go 1.21+)
rca config init           # migrates legacy ~/.config/remote-cluster-agent/*.md, or generates blank template
```

Then **ask the user** for each node's SSH command, and write them into `~/.config/rca/config.toml` under `[nodes.*]`. Example:

```toml
[nodes.train]
ssh = "ssh gpu-train"

[nodes.eval]
ssh = "ssh -p 2222 gpu-eval"
dir = "/home/user/project"
agent_path = "/shared/.agent/agent.py"
```

Optional per-node overrides: `dir` (default working directory) and `agent_path` (where `agent.py` lives on that node). Both fall back to globals if unset.

Finally start and verify:

```bash
rca daemon start
rca daemon status         # confirm running
rca nodes                 # confirm nodes connected
rca agent check           # if agent missing → rca agent deploy
```

## Common operations

### Single-node exec / heredoc

```bash
rca exec -n train "nvidia-smi"
rca exec -n train -d /home/user/project "git pull"
rca exec -t 600 -n train "python train.py"

# heredoc (bypasses shell escaping, recommended for multi-line or special chars)
rca exec --stdin -n train <<'EOF'
cd /home/user/project
python -c "import json; print(json.dumps({'k':'v \"q\"'}))"
EOF
```

### Multi-node parallel (batch)

```bash
rca batch "nvidia-smi | head -20"
rca batch -n train,eval "df -h /home"
rca batch --json "hostname" | jq -r '.results[] | "\(.node): \(.output)"'
```

### Node status / connection management

```bash
rca nodes                   # current connection state
rca nodes --check           # deep ping (with latency)
rca nodes --health          # latency history (monitor-tracked)
rca connect train           # manually reconnect a dead node
rca disconnect train        # actively close a node's connection
```

### File transfer (cp)

`rca cp` moves files through the agent's JSON-Lines channel (base64-encoded, 50 MB/file limit). Works on any SSH setup — no separate SCP/rsync path needed.

```bash
rca cp train:/home/user/logs/train.log ./
rca cp ./config.yaml train:/home/user/project/config.yaml
rca cp -r train:/home/user/checkpoints/exp07 ./local-copy/
```

For very large files (multi-GB checkpoints), use shared filesystem or object storage — `rca cp` peaks around 2–3 MB/s.

### Streaming / long tasks

```bash
rca exec -s -n train "tail -f /var/log/train.log"
rca exec -s -n train "python train.py"
```

### Cluster health inspection

Triggered when the user says "cluster status", "which node is free", "cluster health", "GPU usage". Use `rca batch` for parallel sampling:

```bash
rca nodes --check
rca batch "nvidia-smi --query-gpu=name,memory.used,memory.total,utilization.gpu --format=csv"
rca batch "df -h /home | tail -1"
rca batch "uptime && tmux ls 2>/dev/null | wc -l"
```

Summarize into a table and recommend the most idle node. See `reference/cluster-health.md` for detailed probe commands, parse rules, and report format.

## Reading files vs running commands

`rca exec` is for **executing commands** (starting training, killing processes, checking process state, installing deps) — not reading file contents.

From jiahao-shao1/sjh-skills, MIT.

More from jiahao-shao1/sjh-skills