wk-buildkite

Files (23)

• README.md
• SKILL.md
• 2026-05-12_no-token-curl-workaround.md · references/
• 2026-05-12_prefer-bk-over-npx-webfetch.md · references/
• 2026-05-13_native-step-outcome.md · references/
• 2026-06-09_build-view-positional-build-number.md · references/
• 2026-06-09_mirror-not-pullthrough.md · references/
• 2026-06-11_only-own-branches.md · references/
• 2026-06-12_foreground-poll-loop-interrupted.md · references/
• 2026-06-23_announce-ci-status-on-monitor.md · references/
• 2026-06-23_non-required-checks-dont-gate.md · references/
• 2026-07-15_rebuild-fallback-mutation-token.md · references/
• 2026-07-24_rollup-not-per-job-evidence.md · references/
• 2026-07-27_absent-step-renders-as-pass.md · references/
• 2026-07-27_job-executor-error-is-infra.md · references/
• 2026-08-22_separate-trigger-from-cause.md · references/
• 2026-08-25_premature-should-pass-claim.md · references/
• 2026-08-26_artifact-download-retry-fallback.md · references/
• 2026-08-26_query-only-retry-fallback.md · references/
• 2026-08-26_stale-base-not-infra-flake.md · references/
• build-job-states.md · references/
• json-warning-prefix.md · references/
• url-retrieval-vs-browser-open.md · references/

SKILL.md (first lines)

# Buildkite

Use the `bk` CLI for all CI tasks instead of GitHub tools.

## Pre-flight

- **HARD RULE: investigate only your own branches.** Verify the branch is one you created before debugging; unfamiliar →
  stop:
  ```bash
  gh pr view "$PR" --json headRefName,author --jq '.headRefName + " (@" + .author.login + ")"'
  ```
- **HARD RULE:** Use the local `bk` CLI for every inspection — never `npx` or `WebFetch` on Buildkite URLs when `bk` is
  available. Verify once: `command -v bk >/dev/null || { echo "bk CLI not on PATH"; exit 1; }`
- **HARD RULE:** Before any `bk` command, verify auth: `bk build view -p <pipeline> -b main --json 2>&1 | head -3`.
- **HARD RULE: strip warning prefix on every `--json | jq` pipe** — `grep -v '^Warning:'`.
- URL pasted → parse pipeline/build from path, use `bk build view`. Specific build → positional:
  `bk build view -p <pipeline> <build-number> --json`. On `bk build view`, `-b` = `--branch`; on `bk job log`, `-b` =
  `--build-number`.
- Pipeline slug: `bk build view --json 2>&1 | jq -r '.pipeline.slug'`. Build URL: `jq -r '.web_url'`; use `-w`/`--web`
  only when the user explicitly asks to open the browser.
- Downloads: write to `/tmp/agent/buildkite/<build_number>/<job_id>/<filename>`; `mkdir -p` first.

## Auth errors

Any `bk` command returning 401/403/scope-missing/`Error: GET https://api.buildkite.com` → **stop**; tell the user to run
`bk auth login`. Never configure auth, create tokens, or extract tokens from config for curl workarounds. Exception:
`Mutation operations are not allowed` is NOT a stop-case → try `bk build rebuild` first. ≥2 consecutive rebuilds with
identical error → classify as infra-side, stop diagnosing.

## Checking build status

```bash
bk build view -p <pipeline> -b <branch> --json 2>&1 | grep -v '^Warning:' | \
  jq '{number: .number, state: .state, finished: .finished_at, \
       jobs: [.jobs[] | select(.state | IN("failed","broken","canceled","timed_out")) | \
              {name: .name, state: .state, exit_status: .exit_status}]}'
```

- Current branch: canonical query above. Specific build: positional build number, filter `.type == "script"` jobs.
- **Rollup is not per-job evidence** — use per-job view for specific job claims:
  ```bash
  bk build view -p <pipeline> <build-number> --json 2>&1 | grep -v '^Warning:' | \
    jq -r --arg s '<step-name>' '.jobs[] | select(.name // "" | test($s)) | "\(.name) \(.state) exit=\(.exit_status)"'
  ```
- **No matching job = unwired gate, not a pass.** Guard name with `// ""` (null-named jobs abort `test()`). Count 0 →
  grep repo for wiring, report unwired.
- States: [references/build-job-states.md](references/build-job-states.md). `broken` jobs are almost never root cause —
  investigate `failed` first.
- Upstream step status inside a build: `buildkite-agent step get "outcome" --step "<step-key>"`
  (`passed|failed|soft_failed|timed_out|broken`); never use sentinel files or artifact searches as proxies (crash before
  cleanup → no signal).

## HARD RULE: never predict build outcomes

Never state a build will pass without fetching the actual result. After every push, fetch and report state.

## Investigating failures

**HARD RULE: check branch currency before diagnosing failures** — `git rev-list --count origin/main..HEAD`; branch
behind main → sync first, rebuild second.

1. Overall status → canonical query (failed, broken, canceled, timed-out jobs).
2. Fetch + analyze logs: `bk job log <job-uuid> -p <pipeline> -b <build-number> --no-timestamps 2>&1`. Raw REST logs →
   strip ANSI escapes AND `_bk;t=<epoch-ms>` markers first.
3. **HARD RULE: classify infra vs. code before attributing a red build to the diff** — read the log body; infra markers
   (retry, never touch the diff):
   `Error setting up job executor`, `job_executor_error`, `exit code -1`,
   `Running agent environment hook` failures, agent lost/terminated.
   Retry single job: `bk job retry <job-uuid>`; retry succeeds = classification done.
   - **A `canceled` job is unexplained, not infra** — read its log and the job that issued the cancel before retrying
     (a fail-closed gate can `buildkite-agent build cancel`, so a real failure renders as `canceled`).
4. Check if pre-existing: canonical query against `-b main`.
5. **Report trigger AND cause separately** — "Build #N triggered by your merge;
   failure in step X is pre-existing/infra, not your diff."

## Monitoring after push

```bash
bk build view -p <pipeline> -b <branch> --json 2>&1 | grep -v '^Warning:' | jq -r '.state'
```

- **HARD RULE: never foreground-poll** (no `until`/`while` on `bk build view`) — run one check, report state; background
  with `run_in_background: true` if needed.
- **HARD RULE: a monitoring announcement states URL + failing step + next action — never just "monitoring in
  background."**
- **HARD RULE: a background watch loop must prove each poll parsed.** Run it under explicit `bash -c` with pipeline and
  build as separate positional args (never word-split a combined `$p` — zsh does not split it); print a first-poll
  line and a line on every state change; empty/unparsable state → print and exit non-zero, never re-poll silently. (A
  parse that fails without output starves the failure branch: the loop reads "still running" while the build is red.)
  ```bash
  bash -c 'prev=; n=0; while :; do n=$((n+1))
    s=$(bk build view -p "$1" "$2" --json 2>&1 | grep -v "^Warning:" | jq -r ".state" 2>/dev/null) || s=
    [[ -z "$s" || "$s" == null ]] && { echo "poll $n: UNPARSABLE state"; exit 2; }
    [[ "$s" != "$prev" ]] && echo "poll $n: $s"; prev=$s
    case $s in passed) exit 0;; failed|canceled|blocked) exit 1;; esac; sleep 30; done' _ "$PIPELINE" "$BUILD"
  ```
- Non-required checks do not gate merge — gate on
  `gh pr checks --json name,state,required | jq 'select(.required==true)'`.

## Cancelling and retrying

- Cancel from within: `buildkite-agent build cancel` (no token). From outside: REST `PUT .../builds/{n}/cancel` with
  `write_builds` token.
- Retry: take the first that works: 1) `bk job retry <job-uuid> --yes --no-pager` (no pipeline flag; needs REST-mutation
  token) 2) GraphQL-only → `bk build rebuild <n> -p <pipeline> -y` 3) mutation unavailable + infra failure →
  `git commit --allow-empty -m "ci: retry infra failure"`. Escalate to `bk auth login` only if rebuild also fails.
- `bk artifacts download` forwards the bearer token to S3 presigned URLs → dual-auth rejection: download from the web UI
  instead.

## CI pipeline config

- New env var: set at **every** forwarding layer or it is silently dropped — `env:` in `pipeline.yml`, plugin `env:`
  array in `pipeline.rb`, `environment:` in `docker-compose.yml`, `ENV` in `Dockerfile`. Add
  `expect(config['env']).to include('VAR')` in pipeline specs.
- Mirrored image: internal mirrors may be pre-seeded, not pull-through — verify the exact repo exists before pinning;
  absent → reuse an existing pipeline image + install toolchain.

---

From whizzzkid/skills, MIT.

More from whizzzkid/skills