ci-debugging

Files (1)

• SKILL.md

SKILL.md (first lines)

# CI Debugging — Find the Root Cause

Debug CI failures by comparing passing vs failing commits, never by weakening the test.

## The core rule

**Fix the root cause, never weaken the test.**

When a fix seems to work, verify it actually fixed the right thing. If a test
is flaky or "fails sometimes", the failure is telling you something about the
code — not about the test.

## The 5-step procedure

1. **Get the failing commit's full log.**
   ```bash
   gh run view <run-id> --log | grep -E "FAIL|Error|✗" | head -20
   # identify the exact step and line
   ```

2. **Compare passing vs failing commits.**
   ```bash
   git log --oneline <passing>..<failing>
   git diff <passing>..<failing> -- <suspect-file>
   ```
   A "silent failure" (process killed before writing output) gives you zero
   error info — the only signal is which commit changed the behavior.

3. **Reproduce locally in isolation** (not by guessing).
   Use the repo's test harness (`dts` / `scripts/dts.sh`) to run the exact
   failing command inside a throwaway container, so the host environment
   can't mask the bug.

4. **Fix the root cause, then re-run the original test.**
   If your fix "works" but the test had to be weakened, you fixed the wrong
   thing.

5. **Verify with a clean test run** — not just the one case that failed.
   Run the full suite; a root-cause fix should not require relaxing other
   tests.

## Step 0: Ensure logs are captured *before* debugging

If probes redirect to `/dev/null`, there is nothing to debug with.
Before any debugging, verify the test suite captures failure output
to files that get uploaded as CI artifacts.

See the `ci-log-capture` skill for the pattern: replace
`>/dev/null 2>&1` with file captures to `${CI_LOGS_DIR}` so that:
- On **success**: clean `[PASS]` in CI log, full output sits quietly in artifact
- On **failure**: `[FAIL]` + the full captured body is `cat`-ed to stderr
  (visible in CI log immediately) AND persisted to artifact for later analysis

### The `run_and_log` helper pattern

```sh
# Add after CI_LOGS_DIR setup:
run_and_log() {
    local test_name="$1"; shift
    local log_file="${CI_LOGS_DIR}/${test_name}.log"
    if "$@" >"${log_file}" 2>&1; then
        log_info "${test_name} — OK"
        return 0
    else
        log_error "${test_name} — FAILED (see ${log_file})"
        cat "${log_file}"   # print to CI log for immediate visibility
        return 1
    fi
}
```

### Which probes to capture

| Category | Examples | Why |
|----------|----------|-----|
| **Version checks** | `hermes --version`, `9router --version` | Confirms binary installed correctly |
| **Health/model checks** | `/v1/models` curl probes | Shows HTTP status, model count |
| **Chat completion** | Chat completion endpoints | Shows **400/401/403 body** from upstream OC bug |
| **Preconfig** | 9Router/OmniRoute login/setup | Shows which step failed in the chain |

All captured files must match the artifact upload path (`path: ${{ env.CI_LOGS_DIR }}/`)
so they're downloaded on failure. See `ci-log-capture` skill.

## Silent failure patterns (hardest to debug)

| Symptom | Likely cause | How to find it |
|---------|--------------|----------------|
| Process exits 0 but no output written | Process killed early (OOM, timeout) before flush | Compare exit code + check for killed processes in CI log |
| Test "passes" but artifact is empty | `if-no-files-found: warn` masks empty upload | Check artifact size/content, not just job status |
| Fail only in CI, never locally | NGINX/CI PATH differs, env var unset, runner image change | Compare `env` output between local and CI |
| Fail only sometimes (flaky) | Race condition, retry logic, provider rate-limit | Look for "retry" loops and timeouts; bump tolerance only if truly ephemeral |
| `rm -rf` reports success but `verify CLEAN` fails with leftover mount | Docker volume mount (`-v host:container/path`) blocks removal — `rm -rf` cannot delete a mountpoint | Check `mount` output for `$HOME` in failure logs; unmount with `umount -l` plus `sudo umount -l` fallback (container runs as uid 1000) before retrying removal |
| `--keep-config` test fails "removed X (should preserve)" | Install never created X because its setup is disabled by default | Ensure install creates empty placeholder dirs (`mkdir -p ~/.mnemon ~/.ollama ~/.9router`) even when disabled so the uninstall contract has something to preserve |
| Uninstall test deletes real `~/.mnemon`/`~/.hermes` on host/codespace | Test ran with real `$HOME` instead of isolated temp dir — codespace mnemon is the memory layer, deleting it kills the session's model proxy | Always run uninstall/purge tests with `HOME=$(mktemp -d)` and clean up; never `rm -rf` with real `$HOME` in tests or scripts |
| CLI env var silently ignored (e.g. `TAILSCALE_SOCKET` ignored by v1.104.1) `fails to connect` on custom socket | `bin --socket` works but `ENV=... bin status` dials default path | Probe with both forms live: `ENV=... bin status` vs `bin --socket=... status`; if only the flag succeeds, fix all probes/wrappers to use `--socket` and ship a `bin/ts` wrapper that always passes it — never rely on the env var |
| `--socket=~/path` dials literal `~` and fails | `~` only expands at word start or after `:` in `PATH`-like assignments; `--flag=~/...` is not an assignment | Always use `$HOME` or `$MINIONS_HOME` in flags and wrappers (`--socket="$HOME/.minions/..."`), and use `${MINIONS_HOME:-$HOME/.minions}` in wrappers so CI/`ssh`/`sudo` resolve correctly |
| `var=$(failing_cmd); rc=$?` error handler never runs and log stays empty | `set -e` aborts at a failing command substitution before the next line, so the `[FAIL] + tail` branch is dead code | Wrap the capture: `set +e; var=$(cmd 2>&1); rc=$?; set -e` so the handler can print diagnostics |
| `mkdir` fails "Read-only filesystem" or "Permission denied" on a path that is a symlink | Dir symlink `lib/ollama -> /tmp/cache:ro` makes `mkdir -p lib/ollama` walk through the ro mount; or the base image pre-creates `$HOME` owned by root and a conditional `[ -d $home ] or chown` skipped the fix | Mount cache outside the tree (`/tmp/cache:ro`) and symlink files not dirs (`lib/ollama/ollama -> /tmp/cache/ollama`); always `mkdir -p $home && chown` unconditionally |
| Cache miss causes re-download then a different downstream failure | Version string carries `v` prefix (`v0.40.0` in deps.yaml) but cache dir is `ollama-0.40.0-...` — un-stripped key leaves `OLLAMA_HOST_CACHE` empty so the container mounts nothing | Normalize before use: `version=${version#v}` and map arch aliases before building `OLLAMA_HOST_CACHE` |
| `bash install.sh \| tee log` reports success while install OOMed | Pipe exit is `tee`'s 0 when `pipefail` is off, masking left-side failure — downstream keep-config then sees missing dot dirs | Prefix the pipe: `set -o pipefail; cmd \| tee log` so failure propagates |
| `npm install` OOM `JavaScript heap out of memory` in constrained container | `NODE_OPTIONS=--max-old-space-size=512` too small for large package (e.g. `omniroute@3.8.x`) under DTS/CI limits | Raise heap in shared `install_npm_package` wrapper (512 → 2048) |
| `.tar.zst` extract fails `stdin: not in gzip format` or `zstd required` | DTS container lacks `zstd` and `tar -xzf` handles gzip only — `.tar.zst` needs `zstd -dc \| tar -xf` or `tar --zstd -xf` | Install `zstd` in DTS prereqs and fix `lib/download.sh` `.tar.zst` branch to use `tar --zstd -xf` fallback; add `/tmp/ollama-cache/ollama` host-cache copy fast-path in `lib/ollama.sh` to skip re-download on reinstall |
| `Dependency drift (advisory)` fails but overall CI still green | Drift is `continue-on-error: true` — the job fails to surface `OUTDATED` but does not block merge; easy to ignore until 4-file sync invariant breaks `version-sync` | Run `check-updates.sh --json` to identify the outdated pin, then `bump.sh DEP=VERSION --open-pr` + `sync-readme-badges.sh` to fix all 4 files (see `version-single-source` skill) |
| `self-check.sh --ci` exit 1 treated as failure after reinstall | `--ci` exits 1 for warnings (1 warning, 0 critical) and 2 for critical — the pre-reinstall health check tolerates warnings; a new strict `or fail` after reinstall breaks on advisory warnings | Fail only on critical: `set +e; bash self-check.sh --ci; SC=$?; set -e; [ "$SC" -ge 2 ] && fail` — tolerate `SC=1` |
| `DTS Integration` cancelled at `timeout-minutes: 60` mid `stop.sh` (all 8 tests PASS) | Single DTS run ~38min; 3-retry loop needs ~115min headroom worst case — wall timeout kills mid-test and reports `cancelled`, not a code defect | Bump `dts-integration.timeout-minutes` to 75 (or 90 for 3 retries); verify via `gh api .../runs/<id>/jobs` duration |
| `Real Install` `npm 404 pi-ai-1.1.0.tgz` while PI pinned `1.0.4` | `pi-coding-agent@1.0.4` deps `^1.0.4` float to `pi-ai@1.1.0`; registry propagation can 404 transiently | Bump PI via `bump.sh PI=1.1.0` (+ badges) so install pins `1.1.0` directly; avoid fixing via rerun alone |
| `verify-knowledge.sh` F5 reports `Unknown function: F5` after printing `WARN: .devcontainer references found` | `echo "$hits" \| head -20` always SIGPIPEs (exit 141) when hits exceed 20 lines; with `set -euo pipefail` the pipeline fails, and `type -t "$fn" && "$fn" \|\| { echo Unknown; exit 1; }` misattributes any function failure as a missing function | Append `\|\| true` after `head -20` and split dispatch into `if ! type -t "$fn"; then echo Unknown; exit 1; fi; "$fn"` so real failures surface with their own output |
| `boot.sh` aborts silently right after `Starting Tailscale (kernel...)` with no error; second run passes | `set -e` + `$(stat -c %u "$SOCK" 2>/dev/null \|\| stat -f %u "$SOCK" 2>/dev/null)` — both `stat` branches fail (exit 1) on missing socket, so the substitution exits 1 and `set -e` kills the script. Second run hits `pgrep` guard, skips Tailscale entirely and masks the bug | Append `\|\| true` inside the substitution: `$(stat -c %u "$SOCK" 2>/dev/null \|\| stat -f %u "$SOCK" 2>/dev/null \|\| true)` and test emptiness; audit the class (`grep -rn 'stat -c'`), fix wrapper templates and `self-check.sh` too |
| First boot fails, second boot passes with `already running -- skipping` | Warm-start guard (`pgrep` / `already installed`) skips the failing cold-start path — bug only reproduces on cold start | Repro must force cold state (kill daemon, remove socket/pid) and run without `set -x` masking; diff the two boot logs to find which step was skipped |

## Anti-patterns

- **Weakening the test to make it pass** — changes the contract, hides the bug.
- **Silencing output with `>/dev/null` without understanding why** — every
  silenced probe should have a log capture path (see `ci-log-capture` skill).
- **Skipping the health-check step** — if the stack isn't fully up, the check
  that "works" is meaningless.
- **"It passed locally"** — always verify in the actual CI environment.

From gitricko/.minions, MIT.

More from gitricko/.minions