audit-runs

Files (3)

• SKILL.md
• scans-design.tsv
• scans-implement.tsv

SKILL.md (first lines)

**MANDATORY: Follow the complete shared lifecycle contract in `${CLAUDE_PLUGIN_ROOT}/skills/shared/run-lifecycle.md` with declared skill `audit-runs`.**

# audit-runs

**MANDATORY: READ ENTIRE FILE before composing user-facing prose: `$PWD/skills/shared/readability-style.md`.**

Audit recently-merged larch run logs for the selected skill (`--skill=design|implement`) for anomalies. The current implement registry covers EXON/OOS/missing-file/NS-retry/self-deploying checks; the current design registry is intentionally narrower and ships cache-freshness plus guideline-assessment coverage. Always file a chain-of-history audit-report issue; record bug-issue candidates as proposals at scan time and act on them only after explicit user direction in chat.

This is a **dev-only** operator skill (`.claude/skills/`). It is NOT shipped with the plugin.

## Usage

```
/larch:audit-runs --skill <name> [<verbal-description>] [--repo owner/name] [--allow-concurrent]
```

(Plugin slash-alias: `/audit-runs …` may also resolve to this skill depending on marketplace wiring; prefer `/larch:audit-runs` when unsure.)

### Args

- `--skill <name>` (**required**): which skill’s run logs to audit. Closed enum: `design` or `implement`. Reject missing or out-of-enum values before any side effect (no GitHub call, no tmpdir). Parse from argv before preflight.
- `<verbal-description>` (optional positional): when present, describes which PRs to audit. When omitted or empty, treat as `since last audit` (same error paths as that form). Supported forms:
  - `last N PRs` — N most-recently-merged PRs targeting `main`
  - `since last audit` — PRs merged after the prior audit report's `audited_pr_range.last`; error if no prior report exists or no new PRs have merged (do NOT file an empty report)
  - `since <ISO8601-instant>` — PRs merged after that instant (same interpretable forms as GitHub `mergedAt`: `…Z` or explicit `±HH:MM` offset). This filter is **not** tied to the Pacific wall-clock convention used for audit report titles and `audit_timestamp`.
  - `#N` or `PR #N` — exactly one PR
  - Default when empty: treat as `since last audit` (requires a prior audit-report issue; continues to error if no prior report or malformed frontmatter — same as the explicit `since last audit` form)
- `--repo <owner/name>`: target repo. Default: `character-ai/larch`
- `--allow-concurrent`: override the shared 5-minute `audit-report` concurrency guard (not skill-scoped)
- **Removed flag**: `--no-fix-issues` is not supported. If any token in the skill argv is exactly `--no-fix-issues`, refuse immediately with a clear usage error (flag removed); do not proceed or silently ignore it.

## Pre-flight

After parsing and validating `--skill` into `$SKILL`:

```bash
PREFLIGHT_OUT=$("${CLAUDE_PLUGIN_ROOT}/scripts/larch.sh" audit-runs preflight \
  --skill "$SKILL" --repo "<owner/name>" [--allow-concurrent])
```

Read `PREFLIGHT_OK`, `REASON`, and (on success) exactly one whole-line
`CORPUS_ROOT`. Preflight synchronizes immutable inputs exactly once, refuses a
partial or unsynchronized corpus, and no later audit step contacts storage.

## Bugs-backlog advisory

After `PREFLIGHT_OK=true` and before `resolve-prs`, run the advisory once:

```bash
NUDGE_RC=0
NUDGE_OUT=$("${CLAUDE_PLUGIN_ROOT}/scripts/larch.sh" audit-runs bugs-backlog-nudge \
  --repo "<owner/name>" \
  --root "$PWD" 2>"$TMPDIR/bugs-backlog-nudge.err") || NUDGE_RC=$?
```

If stdout is non-empty, print it to chat as advisory text only and retain it for early-exit paths. If the command fails, print a short non-fatal advisory failure, including stderr when helpful, and continue the audit. Do not include this output in the filed audit report body, YAML frontmatter, `compute-counters` input, or scan NDJSON.

## Verbal-Description Resolution

```bash
RESOLVE_OUT=$("${CLAUDE_PLUGIN_ROOT}/scripts/larch.sh" audit-runs resolve-prs \
  --skill "$SKILL" --repo "<owner/name>" [--verbal-description "<verbal-description>"])
```

Read `PR_LIST`, `PR_COUNT`, `IMPLICIT_SINCE_LAST_AUDIT`, `PRIOR_REPORT_NUMBER`, `RESOLVED_ECHO`, and `ERROR` from stdout. Fail-fast when `ERROR` is non-empty; print any non-empty `NUDGE_OUT`, then print `ERROR` to the user. Print `RESOLVED_ECHO` before scanning. Contract: `scripts/larch.sh audit-runs resolve-prs`.

## Scan Registry

The scan list is externalized in `.claude/skills/audit-runs/scans-$SKILL.tsv` (one row per scan: `name`, `type`, `pattern`, `expected_outcome`, `severity`). JSON emission, category filtering, and scan implementations live in `crates/larch-cli/src/audit_runs_commands.rs` behind `scripts/larch.sh audit-runs scan-run`; Rust unit and parity coverage owns the wire. **Adding a scan** requires coordinated updates: (1) a new `scans-$SKILL.tsv` row in the relevant per-skill registry, (2) matching Rust scan logic, (3) any Rust counter wiring when the scan feeds cumulative totals, (4) this `SKILL.md` scan table if the operator-facing baseline changes, and (5) hermetic Rust coverage for the new NDJSON shape and counter path. **Plan fidelity**: substantive changes to that surface (new counters, new cumulative YAML keys, or registry-wide behavior) should be tracked in their own issue/PR when they go beyond a routine scan-row + test update. Routine `changelog-rebase-conflicts` / `changelog_rebase_conflicts` / `ns_retries_cursor_specialist` wiring is part of this skill’s maintained baseline, not an ad-hoc add-on. **Operator parity with run-log audit-title hygiene on `main`**: audit-title and search-exclusion work assumes run logs and issue titles stay aligned with the same `^\[(Run Logs Audit |Implement Run Logs Audit |Design Run Logs Audit ).* Report\]` title regex used by the audit-report writer; the pre-lock `"${CLAUDE_PLUGIN_ROOT}/scripts/larch.sh" git check-main-sync` probe (KV stdout: `SYNC_STATUS=not-main|ok|blocked|probe-error`, plus optional `AHEAD_COUNT` and `ERROR`) uses the locally cached `origin/main` ref (no fetch). If the probe fails with `SYNC_STATUS=probe-error`, operators must refresh the cached ref with a manual `git fetch origin main` before locking. This is the same freshness requirement as the audit-report title migration. Treat main-sync as a first-class preflight gate next to audit-title hygiene, not an undocumented side effect.

Read the registry at runtime:
```bash
SCANS_TSV="$PWD/.claude/skills/audit-runs/scans-$SKILL.tsv"
```

### Scans (baseline — see `scans-implement.tsv` / `scans-design.tsv` for the machine-readable registry)

Implement currently uses the full table below. Design currently uses `cache-freshness` and `guideline-assessment` unless/until `scans-design.tsv` grows additional rows.

| Scan | What | Where |
|---|---|---|
| Required-file presence | Compare against `docs/run-logs-required-files.tsv` (NDJSON `result` is `pass` / `fail` / `skip` / `error`) | run-log root |
| EXON misclassification | `\| FINDING_.* \| 0 \| 0 \| [1-9]+ \|.*\| rejected \|` | `round-*/voting-tally.md` |
| OOS category mangle | plan-review **accepted** rows only: non-empty `category` not in `{code-quality, risk-integration, correctness, architecture, security}` (code-review accepted prose categories are ignored by design) | `review-findings-full.jsonl` |
| NS-retry sidecars | `reviewer_signals[].ns_retry_reason` when present; legacy `*-ns-retry*` file fallback when the concise carrier is missing (`result:"skip"` only when both are absent) | `round-*/round-meta.json` (+ legacy `round-*/`) |
| Cursor CI stall causes | `cursor-ci-stall-*.json` sidecars: informational histogram of `.channel` values (pass when none) | `round-*/` |
| Codex round-1 adherence | round 2+ panel-manifest should not contain `tool=codex` | `round-N/panel-manifest.ndjson` |
| Codex generalist waste | `reviewer_signals` entry for `codex-generalist-output.txt` is `NO_ISSUES_FOUND` only AND timing > 120s (`result:"skip"` when carrier missing) | `round-1/round-meta.json` + `timing-report.json` |
| Execution-issues categories | non-Warnings entries in `execution-issues.ndjson` | `execution-issues.ndjson` |
| Cache freshness | `manifest.json::larch_version` vs latest plugin version (`result: informational` when the run lags current; empty `larch_version` remains `fail`; other rows may emit `skip`/`error`) | `manifest.json` |
| Guideline assessment | published Gate C guideline assessment is present and non-empty when the artifact exists (`result:"informational"` when absent) | `architectural-guideline-assessment.md` |
| Changelog rebase/conflicts (heuristic) | `execution-issues.ndjson` bodies mentioning changelog + rebase/conflict | `execution-issues.ndjson` |
| Coder tool | `CODER_TOOL` field | `round-*/coder.env` |
| Trailing-content NO_ISSUES_FOUND | `reviewer_signals[].first_pass_trailing_content == true` (`result:"skip"` when carrier missing; legacy `*-first-pass.txt` no longer primary) | `round-*/round-meta.json` |
| OOS silent drop | accepted non-security `### OOS_` blocks vs filed GitHub URLs, Inline-triage commit lines, and rejected-OOS markers in `oos-issues.ndjson` | `oos-accepted-*.md`, `oos-issues*.ndjson`, `oos-issues-created.md`, typed commit history of the run-log repo root |

## Scanning

```bash
# lint-consecutive-bash: ok map output feeds per-run scan loop
# Map each PR to its run-log directory
RUN_MAP_TSV=$("${CLAUDE_PLUGIN_ROOT}/scripts/larch.sh" audit-runs map-runs \
  --skill "$SKILL" --pr-list "$PR_LIST" --repo "<owner/name>" \
  --log-root "$CORPUS_ROOT/$SKILL")
# TSV: pr_number<TAB>run_id<TAB>started_at<TAB>larch_version<TAB>closes_issue
```

Then for each PR row in the TSV:

```bash
"${CLAUDE_PLUGIN_ROOT}/scripts/larch.sh" audit-runs scan-run \
  --skill "$SKILL" \
  --run-dir "$CORPUS_ROOT/$SKILL/<RUN_ID>" \
  --pr <PR_NUM> \
  --scans-tsv "$SCANS_TSV" \
  --required-files-tsv "$PWD/docs/run-logs-required-files.tsv" \
  --current-version "<latest-plugin-version>" \
  > "$TMPDIR/scan-results-<PR_NUM>.ndjson"
```

From character-ai/larch, MIT.

More from character-ai/larch