generate-profile

Files (1)

• SKILL.md

SKILL.md (first lines)

# Generate Profiling Trace

End-to-end workflow: launch server → validate accuracy → run profiling client → collect traces → analyze. Produces trace files and an analysis Excel report ready for `/profile` or `/compare-kernels`.

## Step 0: Gather context

### 0a: Detect the active SGLang installation

Before doing anything else, determine which SGLang repo is in use. There may be multiple SGLang directories on the system (e.g., `/sgl-workspace/sglang`, `$HOME/sglang`). **Always use the one that's currently installed in the active Python environment.**

```bash
SGLANG_ROOT=$(python3 -c "import sglang, pathlib; print(pathlib.Path(sglang.__file__).resolve().parents[2])")
echo "Active SGLang root: $SGLANG_ROOT"
```

This resolves the path from the **installed** package, so it works regardless of which copy is active. Use `$SGLANG_ROOT` for all subsequent paths (benchmarks, scripts, etc.) instead of hardcoded paths like `$HOME/sglang` or `/sgl-workspace/sglang`.

Verify it looks correct:
```bash
ls "$SGLANG_ROOT/benchmark/gsm8k/bench_sglang.py"
```

### 0b: Resolve from the model card first (card-driven)

**Resolve everything you can from the registry before asking the user.** From
`$ARGUMENTS` (or the server script path) infer the model, then read
[[../../memory/models/INDEX.md]] and the matching card under `memory/models/`.
The card provides: **server script, client script, port, accuracy threshold,
output directory convention, and profiling detail-module/instance.**

- Resolve the **workload** via [[../../memory/workflows/workloads.md]] — default
  `canonical-8k` (IL8192/OL1024); use `diag-1k` (1024/1024) for a cheap capture.
- Resolve the **output directory** from the card, NOT the DSv4 default. e.g.
  qwen35-mxfp4 → `$HOME/qwen3.5-mxfp4/<label>/`; DSv4 → `$HOME/dsv4/<label>/`.
  Never save Qwen artifacts under `~/dsv4/`.

Only `AskUserQuestion` for what the card does **not** resolve — typically just the
**profile label** (short name like `baseline`, `after_fix`). If everything else is
resolved and a label is derivable (e.g. git branch), proceed without asking at all.
Ask for server/client/threshold/model only when the model is unknown or unresolvable.

Fields to end up with:
1. **Server launch script** — from card, or ask
2. **Client benchmark script** — from card, or ask
3. **Profile label** — ask if not derivable
4. **Model name** — from card resolution (do NOT default to DSv4)
5. **Accuracy threshold** — from card (0.88 DSv4, 0.92 qwen35-mxfp4); ask only if unknown
6. **Profile by stage?** — default yes (separate prefill/decode traces). **Exception:** if the goal is *module-level* trace analysis on a model whose decode is CUDA-graph-replayed (e.g. qwen35-mxfp4, whose `*-DECODE` trace collapses to `CudaGraphReplay` with no per-module detail — see its card), capture a **combined** trace instead (no `--profile-by-stage`).

Parse the server script to extract the **port** (look for `--port <N>` in the launch command).

## Step 1: Start the server

### 1a: Check if server is already running

```bash
curl -s -o /dev/null -w "%{http_code}" http://localhost:<PORT>/health
```

If `200`, ask the user whether to:
- **Reuse** the running server (skip to Step 2)
- **Restart** it (kill and relaunch)

### 1b: Launch the server in background

```bash
bash <server_script>
```

### 1c: Wait for server readiness

Poll until ready — run this in the **foreground**, not background:

```bash
for i in $(seq 1 120); do
  status=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:<PORT>/health 2>/dev/null)
  if [ "$status" = "200" ]; then
    echo "Server ready after ${i} attempts"
    exit 0
  fi
  sleep 10
done
echo "Server failed to start after 1200s"
exit 1
```

If the server doesn't start within ~20 minutes, **STOP** and report the failure.

## Step 2: Accuracy Test

Run the GSM8K accuracy benchmark using the detected `$SGLANG_ROOT`:

```bash
cd "$SGLANG_ROOT" && python3 benchmark/gsm8k/bench_sglang.py --num-questions 200 --parallel 100 --num-shots 5 --port <PORT>
```

Note: we use fewer questions (200) than full validation (2000) since this is a sanity check, not a full accuracy run.

**Interpret the results:**
- Parse the output for the accuracy score
- Compare against the threshold (e.g., > 0.88 for DSv4)
- If accuracy is **below** the threshold:
  - **STOP** — report the failure immediately
  - Show the accuracy number and sample outputs
  - Do NOT continue to profiling
- If accuracy **passes**: record the score and proceed

## Step 3: Run the Profiling Client

### 3a: Prepare the profiling client command

Take the client benchmark script and modify it for profiling:

1. **Add `--profile`** flag to the `sglang.bench_serving` command
2. **Change `num_prompts`** to `num_prompts=$((max_concurrency * 2))` — profiling only needs a short run
3. **Add `--profile-by-stage`** if the user opted for stage-separated traces (default: yes)
4. **Add `--profile-output-dir <OUTPUT_DIR>/<profile_label>`** to save traces to the labeled directory, where `<OUTPUT_DIR>` is the card-resolved per-model dir (Step 0b) — e.g. `$HOME/qwen3.5-mxfp4` for qwen35-mxfp4, `$HOME/dsv4` for DSv4. Do NOT hardcode `$HOME/dsv4` for non-DSv4 models.

Show the modified profiling command to the user and confirm before running.

From yichiche/agent-box, MIT.

More from yichiche/agent-box