pr-ci-watch

Files (9)

• SKILL.md
• auth.py
• cron-claude-sweep.sh
• cron-sweep.sh
• dashboard.py
• history.py
• perf.py
• serve_dashboard.sh
• watch.py

SKILL.md (first lines)

# Watch PRs, re-run only the CI that deserves it

Two kinds of toil this removes:

1. **NVIDIA CI goes red for reasons unrelated to the PR** — GPU busy after
   teardown, profiler teardown decode error, a GSM8K threshold flake, a
   `cutlass-dsl` pin bug. The fix is "re-run the failed group", but only after
   someone confirms it is not a real bug.
2. **A PR silently goes `CONFLICTING`** and sits there because nobody told the
   author.

This skill is the loop around [`/ci-analysis`](../ci-analysis/SKILL.md) (which
does the triage) and [`/pr-conflict-fix`](../pr-conflict-fix/SKILL.md) (which
fixes a conflict on demand). It contributes the watchlist, the schedule, the
memory of what has already been done, and somewhere to look at it all.

Read `_shared/repo-config.md` for the `GH_TOKEN=""` rule. Repo is
`sgl-project/sglang`; `--repo` exists but the CI scope filter is tuned for it.

## Invocation

```
/pr-ci-watch add <pr url|number>… [--high] [--note "..."]
/pr-ci-watch remove <pr url|number>…
/pr-ci-watch list | status
/pr-ci-watch sweep [--track high|regular|all] [--pr N…] [--apply] [--force]
/pr-ci-watch triage <pr>…   # /ci-analysis + apply-verdict for PRs already swept
/pr-ci-watch auto <pr> [--triage on|off] [--rerun on|off]   # per-PR switches
/pr-ci-watch update-branch --pr N [--apply]   # GitHub's Update branch
/pr-ci-watch report         # the <P0><CI clear><PR…> status block
/pr-ci-watch pause | resume # kill switch; a paused sweep is a no-op
/pr-ci-watch dashboard [--port 8812]
/pr-ci-watch arm            # register the two cron tracks
```

Entries render as a markdown bullet with the perf line nested under it —
`- <P0><CI clear><PR41134>Title` / `  - TPOT 6.3% improvement`. The rich flavour
is a real nested `<ul>`, so a paste into Teams lands as a proper list; the plain
flavour uses `- ` and two-space indent, which Teams also turns into bullets.
In the rich flavour, `CI clear` is wrapped in `<b>`, so that token is bold on
the page and stays bold in Teams. Other tokens are not.

Line 2 of each entry is the PR's **own performance claim**, extracted from its
body by `perf.py` — not the CI triage reason. It reads the markdown result
tables two ways: metric-in-first-cell (`| TPOT median | 2.60 | 2.48 | -4.6% |`)
and metric-in-header (`| … | Δ TTFT | … | Δ |`, where a bare Δ column inherits
the metric to its left). Only *signed* percentages count as deltas — an unsigned
one beside them is a standard error. Across several benchmark tables it reports
the **median**, preferring rows the PR marks `median`/`p50`, so a multi-shape
sweep is not cherry-picked.

Which metrics: **TPOT and TTFT when the PR states both**, otherwise up to two of
TPOT / TTFT / E2E / total throughput. Direction is per metric — `-4.6% TPOT` is
an improvement, `-2.9% throughput` is a regression. Reciprocal restatements
(`Interactivity (1 / TPOT)`) and accuracy tables are excluded; counting the
former would report one result as both an improvement and a regression.

A delta column is recognised by `DELTA_HEADER` — `Δ`, `delta`, `change`,
`diff`, `%`. **`change` is load-bearing**: a header of `tok/s change | TPOT
change` is the same table as `… | Δ`, and while the word was missing the whole
table fell through to the metric-in-first-cell path, where the first cell is a
concurrency number and nothing matches — so a PR with perfectly good numbers
reported none at all.

When no table yields a metric, `_prose()` is tried as a last resort: a sentence
like *"median TPOT drops **from 3.50 to 2.78** ms at concurrency 1"* has real
e2e numbers, just absolute and unpercented, so nothing above sees them. It is
scoped tightly — the metric name must be in the same sentence **and before**
the `from A to B`, or "from 4 to 8" in a sentence about concurrency becomes a
100% regression — and it only fills metrics the tables were silent on, because
a table stating its own delta is the PR's considered claim. It reads one pair
per metric mention, so a sentence quoting two shapes contributes the first.

**A blank second line means the PR states no percentage delta, not that it is
slow.** Four shapes legitimately produce nothing, and they are not bugs:

| Shape | Example |
|---|---|
| Bug/crash fix with descriptive tables only | before/after behaviour, log lines, no numbers |
| Absolute numbers with no delta stated | `Conc \| 4 \| 8 \| …` rows of raw tok/s — the extractor reports the PR's *claim*, it does not compute deltas itself |
| Accuracy-only tables | excluded by `EXCLUDE` on purpose |
| Kernel microbenchmarks only | `TP4 speedup 2.30x`, `µs` columns — no e2e metric is named, and a kernel win is not an e2e perf claim (see `memory/workflows/` on claiming perf only on `canonical-8k`) |

A table whose metric is named only in the **prose above it** (`| Batch size |
TP4 aiter | TP4 this PR | TP4 change |`, under a paragraph saying "decode step
time … in ms") still yields nothing from the table itself — headers and first
cells are all that is read. That shape is why the prose fallback exists.

Override a poor extraction with `watch.py set <pr> --note "..."`; check one
without a sweep via `python3 perf.py <pr>`. If a PR you expect numbers from
shows none, run that first — it prints what was collected and tells you whether
the PR is silent or the extractor missed it.

The report token comes from the same `ci_verdict` the table shows (`CI clear` /
`CI running` / `CI red`), with `code-fix`, `merge-main` and `wait-upstream`
overriding it because they say more than the raw state. Row order is shared via
`row_order()` — the table and the block cannot drift apart.

**`triage`** is Phase B on demand: for each PR named, run
[`/ci-analysis`](../ci-analysis/SKILL.md), reduce its table to one action, and
record it with `apply-verdict --apply`. The dashboard's "Waiting on triage" panel
generates this command for you to paste.

Everything runs through `watch.py` in this directory. **No mutation happens
without `--apply`** — a bare `sweep` prints exactly what it would do and touches
nothing outward-facing.

## Tabs

Each PR carries a free-text `group` — a model name, `debug`, whatever you want
to manage separately. The dashboard shows one tab per group (plus **All**), and
the tab filters the table, the triage panel **and the status block**, so you can
paste a standup entry for one workstream without editing it by hand.

Set a group by typing in the **Group** column — a plain text field, no
dropdown; an existing name moves the PR to that tab, a new one creates it.
(It had a datalist; the marker that adds costs real width in a column this
narrow, and typing the name was the only thing it ever did.) Or from the CLI:

From yichiche/agent-box, MIT.

More from yichiche/agent-box