ci-analysis

Files (2)

• SKILL.md
• known_failure.py

SKILL.md (first lines)

# CI Analysis

Systematically triage PR CI: identify real failures, attribute them to the PR or not, check main for fixes, and deliver an actionable merge verdict.

## When to Use

- User pastes a GitHub Actions job/run URL (`.../actions/runs/.../job/...?pr=...`)
- User asks: "is this CI failure from my change?", "check CI for PR #N", `/ci-analysis`
- User wants the structured report format with merge verdict, failure tables, and fix steps

## Phase 1: Gather Context (parallel)

Run these concurrently:

```bash
# PR metadata + all checks
gh pr view <N> --repo <owner/repo> --json title,state,files,headRefName,baseRefName,statusCheckRollup,commits

# All jobs in the workflow run (extract run_id from job URL)
gh run view <RUN_ID> --repo <owner/repo> --json name,status,conclusion,headBranch,jobs,url

# Compact check table
gh pr checks <N> --repo <owner/repo>
```

From the job URL, extract:
- `owner/repo`, `run_id`, `job_id`, `pr=N`
- Job name (runner, shard, partition)

Record **changed files** from `gh pr view --json files` — this is the attribution baseline.

## Phase 2: Pull Failure Evidence

For every **failed** job (start with the linked job, then all failed siblings):

```bash
# Job metadata
gh api repos/<owner>/<repo>/actions/jobs/<JOB_ID> \
  --jq '{name: .name, conclusion: .conclusion, started: .started_at, completed: .completed_at, steps: [.steps[] | {name: .name, conclusion: .conclusion}]}'

# Logs — grep for signal, not noise
gh api repos/<owner>/<repo>/actions/jobs/<JOB_ID>/logs 2>/dev/null | rg \
  "Test passed|Test failed|Failed test files|❌|✅|FAILED |AssertionError|UnicodeDecodeError|Traceback|short test summary|Fast-fail|SIGTERM|CUDA error|Memory access fault|Process completed with exit code" \
  | head -80
```

For each failed job, extract:
1. **Exact failed test file(s)** — look for `Failed test files`, `❌ Test failed:`, `FAILED`
2. **Error type + message** — first root-cause line, not cascade SIGTERM
3. **Test sequence** — `Running test:` / `Test passed:` to see if failure is isolated
4. **Shard assignment** — `AITER_TEST=`, `--auto-partition-id`, partition model entries
5. **Fast-fail** — if job skipped/failed in <30s with "Fast-fail: root cause job(s): ..."

Batch failed jobs in one shell loop when there are many shards.

## Phase 2.5: Root Cause Analysis (mandatory)

**Do this before attributing failures.** Many red jobs never ran tests — they were killed by upstream fast-fail. Fixing or analyzing a cascade job wastes time.

### Step 1: Classify every failed job

Pull the failed **step** name from job metadata:

```bash
gh api repos/<owner>/<repo>/actions/jobs/<JOB_ID> \
  --jq '{name: .name, conclusion: .conclusion, duration_sec: ((.completed_at | fromdateiso8601) - (.started_at | fromdateiso8601)), failed_step: ([.steps[] | select(.conclusion == "failure") | .name] | first)}'
```

Assign each failed job to exactly one bucket:

| Bucket | How to recognize | Action |
|--------|------------------|--------|
| **Root cause** | Failed step is `Run test` / actual work step; ran ≥1 min; has real test error in logs | Analyze test failure; may be PR-related |
| **Cascade (fast-fail)** | Failed step is `Run ./.github/actions/check-pr-test-health` or log contains `Fast-fail: skipping — root cause job(s):` | **Do not analyze this job's tests** — find root job(s) from log |
| **Rollup gate** | Names like `finish`, `pr-test-finish`, `Standard Test Results`, `call-gate / pr-gate` | Report as aggregation only |
| **Skipped downstream** | `conclusion: skipped` because upstream red | Not a failure to fix |

### Step 2: `check-pr-test-health` = cascade, not root cause

When the failed step is **`Run ./.github/actions/check-pr-test-health`**:

> This job **never actually ran tests**. CI killed it early because an upstream job in the same workflow (or a sibling workflow the health check watches) already failed. **Find and fix the root-cause job first** — rerunning or debugging this job alone will not help.

Extract the cited root job(s) from logs:

```bash
gh api repos/<owner>/<repo>/actions/jobs/<JOB_ID>/logs 2>/dev/null | rg \
  "Fast-fail|root cause job|check-pr-test-health|skipping — root"
```

Typical log lines:
- `Fast-fail: skipping — root cause job(s): base-c-test-4-gpu-b200 / base-c-test-4-gpu-b200 (2)`
- `Fast-fail: root cause job(s): wait-for-base-b, base-b-test-1-gpu-small (4)`

Then open **those** job IDs and analyze their `Run test` step failures.

### Step 3: Build the root-cause tree (internal only)

Use this tree **during analysis** to trace cascades back to root jobs. **Do not include cascade or rollup jobs in the user-facing report** — only root-cause jobs go in **Root cause summary**.

```
Root cause (1–N independent):
  └─ base-c-test-4-gpu-b200 (2)  →  test_collectives.py  →  CUDA devices busy
```

If the user linked a cascade job, mention in one sentence that it never ran tests and redirect to the root job — do not list cascade/rollup job names as separate fix items.

**Count rule:** "N executed failures" = root-cause jobs only. Cascade + rollup jobs are **never** counted as additional PR regressions and **never** appear in Root cause summary.

### Step 4: Cross-workflow root causes

Some health checks watch jobs in **other workflow runs** on the same PR (e.g. sglang `call-gate`, `wait-for-stage-b-amd`). If `check-pr-test-health` fails but no root job is named in the same run:

```bash
# All failed checks across the PR
gh pr checks <N> --repo <owner/repo> 2>&1 | rg "fail"

# Find jobs that ran tests (long duration, failed at "Run test")
gh pr view <N> --repo <owner/repo> --json statusCheckRollup \

From yichiche/agent-box, MIT.

More from yichiche/agent-box