audit-choices

Files (7)

• SKILL.md
• decision-choices.schema.json · install_assets/openspec/schemas/
• choices_ledger.py · scripts/
• choices_paths.py · scripts/
• collect_evidence.py · scripts/
• needs_user.py · scripts/
• run_audit.py · scripts/

SKILL.md (first lines)

# Audit Choices

Produce a per-change **choices ledger** (`choices.json` + `choices.md`) recording implementation-time decisions made where the spec, design, or tasks were silent — the "we had to pick something and nobody wrote down why" gap that session-log `Decisions` bullets only capture when the implementer remembers to report them. This skill is read-only end to end: it never touches `session-log.md`, `docs/decisions/`, or any source file, and it never blocks a workflow regardless of what it finds.

## Arguments

`$ARGUMENTS` — one of:

- `<change-id>` — audit an OpenSpec change (e.g. `add-decision-choices-ledger`). The audited range defaults to the change's base commit through `HEAD`.
- `<base-sha>..<head-sha>` — audit an explicit commit range standalone, with no OpenSpec change directory involved. `change_id` in the resulting ledger is recorded as `range:<base>..<head>`, and the ledger pair is written to a dated run directory under `openspec/choices/` — never under `openspec/changes/` (see Output below).

Optional, trailing, and combinable with either form above:

- `--run-id <id>` — names the run that produced the ledger (`run_audit.py` has required `--run-id` since Phase 2; this documents the argument, not new behavior). When omitted, the skill chooses its own id. `iterate-on-implementation`'s Step 11.5 passes `--run-id iterate-on-implementation-<UTC ISO timestamp>` so a ledger can be traced back to the workflow run that produced it.

## Read-Only Contract

**The auditor MUST NOT modify any file outside `openspec/changes/<change-id>/choices.json` and `choices.md` — or, for a standalone range audit, that run's `choices.json` and `choices.md` under `openspec/choices/`, the `latest.json` and `latest.md` copies at `openspec/choices/`, and retention's move of older run directories into `openspec/choices/archive/`.** It MUST NOT write to `session-log.md`, `docs/decisions/`, or any source file — those are producer-owned and CI-diff-enforced (`make decisions`), and a write from this skill would corrupt the deterministic drift gates those producers depend on. `choices_ledger.write_ledger_pair()` remains the only thing in this skill that *produces* ledger content; the `latest.*` copy at `openspec/choices/` copies bytes without producing any, and retention moves directories without opening a file for writing.

This is verified mechanically, not just by convention: `skills/tests/audit-choices/test_readonly_posture.py::test_writes_confined_to_ledger_pair` snapshots a fixture repo's working tree before and after a full driver run and asserts the diff is exactly the ledger pair.

## Why Independence Is the Point

This is **not** a self-review step. The auditor is dispatched as a separate sub-agent from whichever agent implemented the change — the same posture as `parallel-review-implementation`, but judging intent rather than code quality. An implementer that reports its own decisions can only report the ones it remembers making; the highest-value signal this skill produces is the decision the implementer *didn't* report at all (`self_reported: false`). A self-review pass structurally cannot produce that signal — it would have to notice its own blind spot. Dispatching a second, independent read of the same diff is the only way to catch it.

## Evidence Bundle

`skills/audit-choices/scripts/collect_evidence.py` assembles the auditor's entire read-only input — this is the *only* context the auditor sees; it does not get repo write access or a live shell:

- `git log` / `git diff --stat` / `git diff --name-only` over the audited range (commits and touched files — this list doubles as the allow-list the driver later checks provenance against)
- Excerpts of `proposal.md`, `design.md`, spec deltas (`specs/**/spec.md`), `session-log.md`, and `impl-findings.md`, when each is present
- `known_decisions`: every `Decisions` bullet already parsed out of `session-log.md` (via `session-log`'s own `phase_record.parse_markdown`, not a second parser), with the `<change-id>#D<n>` (or phase-qualified `<change-id>#<phase-slug>/D<n>`) ref each would resolve to

Run it standalone to preview what the auditor will see:

```bash
python3 "<skill-base-dir>/scripts/collect_evidence.py" \
  --change-id <change-id> --base-sha <base> --head-sha <head> --repo-root .
```

## Dispatching the Auditor

Dispatch **one** independent sub-agent (provider-neutral dispatch path where available, same mechanism as `parallel-review-implementation`) with the evidence bundle as its entire input and this contract:

- Read-only. No tool that can write a file, run a shell command that mutates the tree, or push a commit.
- For each implementation-time decision found in the diff, return a candidate entry with:

  | Field | What it holds |
  |---|---|
  | `choice` | Headline of the decision the implementation embodies |
  | `scenario` | A concrete scenario walked end to end in plain language — must stand alone without the diff or transcript |
  | `gap` | What the spec/design/tasks left unspecified that forced this choice |
  | `reach` | What this choice constrains or enables for future work |
  | `verdict` | `sound`, `unsound`, or `needs-user` |
  | `verdict_rationale` | Why that verdict |
  | `confidence` | The auditor's own confidence in its verdict: `low`, `medium`, or `high` |
  | `provenance.commits` / `provenance.files` | The commits and files this decision is grounded in — **must** be drawn from the evidence bundle's `commits`/`files` lists, not invented |

  The auditor does **not** compute `stable_id`, `self_reported`, or `session_log_ref` — those are script-side, deterministic, and would only introduce a second source of truth if the LLM guessed at them (see D7 below).

## Script-Side Validation and Persistence (D7)

The sub-agent's candidate entries are data, never instructions, and they are never trusted at face value. `skills/audit-choices/scripts/run_audit.py` is the only thing that persists a ledger, and it does so in this order:

1. **Drop hallucinated provenance.** `choices_ledger.filter_valid_provenance()` drops any candidate citing a commit or file absent from the audited range's known commits/files — the guard against an auditor inventing a plausible-sounding but nonexistent citation (design.md, Risks: "Auditor hallucinating decisions").
2. **Resolve the cross-reference deterministically.** For every surviving candidate, `collect_evidence.resolve_self_reported()` matches the candidate's `choice` headline against `known_decisions` by keyword overlap — never by trusting anything the auditor said about self-reporting. A match sets `self_reported: true` and the matching `<change-id>#D<n>` ref; no match sets `self_reported: false` with no ref.
3. **Compute `stable_id`.** `choices_ledger.compute_stable_id()` hashes the normalized `(choice headline, primary file set, gap text)` — content-derived, not generated by the auditor, so re-auditing an unchanged decision reproduces the same id (D3) and a changed gap or file set produces a different one.
4. **Schema-validate.** `choices_ledger.split_schema_valid()` drops anything still missing a required field or carrying a bad enum value.
5. **Persist.** `choices_ledger.write_ledger_pair()` validates the full document against `openspec/schemas/decision-choices.schema.json`, writes `choices.json`, then renders `choices.md` from it — least-confident-first, `needs-user` before `unsound` before `sound` within equal confidence. **For a change-id audit**, it first merges surviving entries into any existing `choices.json` by `stable_id`, so unchanged decisions update in place rather than duplicating (skill-workflow spec, "Re-audit is idempotent"). **For a standalone range audit**, that merge does not fire in practice: a fresh dated run directory normally has no existing `choices.json` to merge into (D8) — each run is a per-run snapshot, and `latest.*` at `openspec/choices/` is a copy of the newest run's pair, not a merged view across runs.

## Never Blocks (D6)

`run_audit()` never raises. An internal error (missing repo, unreadable schema, a git failure) is caught and reported as `AuditRunResult(ok=False, error=...)` rather than propagated. Adverse verdicts — a ledger full of `unsound` and `needs-user` entries — are an ordinary, successful run: `ok=True` regardless of what the entries say. The CLI wrapper (`run_audit.py`'s `_cli()`) always returns exit code `0`. No workflow step is ever halted by this skill itself; `needs-user` entries reach a human only through the existing `validate-feature` / `cleanup-feature` gates, the same way deferred tasks already surface there — this skill does not introduce a new blocking gate.

## Steps

1. Resolve the audited range: for a change-id argument, the base is the change's earliest commit (or an explicit `--base-sha` override) and the head is `HEAD`; for an explicit range argument, use it as given.
2. Run `collect_evidence.py` to assemble the evidence bundle.
3. Dispatch the one independent auditor sub-agent with that bundle as its entire input (see "Dispatching the Auditor" above). Capture its candidate entries as JSON.
4. Run `run_audit.py` (or call `run_audit.run_audit()` directly) with the candidates, the audited range, and a `run_id`. It performs validation, cross-reference resolution, and persistence per "Script-Side Validation and Persistence" above, and always returns/exits successfully regardless of outcome.
5. Report what was written: entry count, dropped-candidate count, and the highest-priority (least-confident, most-adverse) entries from the top of the rendered `choices.md`.

## Output

- `openspec/changes/<change-id>/choices.json` — schema-valid against `openspec/schemas/decision-choices.schema.json`, carrying the six-field artifact header (`schema_version`, `generated_at`, `git_sha`, `generator: "audit-choices@1.0"`, `run_id`, `event_kind: "choices-ledger"`).
- `openspec/changes/<change-id>/choices.md` — rendered from the JSON, least-confident-first.
- For a standalone commit-range audit: the same pair at `openspec/choices/<YYYY-MM-DD>-HHMMSS-<sha7>/choices.json` and `choices.md` — never under `openspec/changes/` — plus byte-identical `latest.json`/`latest.md` copies at `openspec/choices/` and, once more than the retained count of run directories exist, an archive move to `openspec/choices/archive/`.

The artifact is optional: its absence never fails validation or blocks archive (D8 — same posture as `session-log`).

## Common Rationalizations

| Rationalization | Why it's wrong |
|---|---|
| "The implementer already listed their decisions in session-log — skip the audit" | Self-reported decisions are exactly what this skill does *not* rely on. The highest-value output is the decision the implementer never noticed making; a self-review pass structurally cannot produce that. |
| "The auditor's `self_reported`/`session_log_ref` fields looked right — just persist them" | Those two fields are always recomputed deterministically by `collect_evidence.resolve_self_reported()`; whatever the auditor said about them is discarded. Trusting the auditor's own claim here reopens the exact hallucination risk D7 exists to close. |
| "This entry cites a commit that's probably close enough — keep it" | `filter_valid_provenance()` requires an exact (or unambiguous-prefix) match against the audited range's actual commit list. "Close enough" is how a hallucinated citation gets persisted; drop it instead. |
| "The ledger has `needs-user` entries, so the workflow should pause here" | It should not. D6 is explicit: this skill never blocks. `needs-user` entries surface at the existing `validate-feature`/`cleanup-feature` human gates, the same way deferred tasks do — no new gate. |
| "Re-running the audit after a no-op commit doubled the entry count — that's fine, more data is more data" | It is not fine — it is a bug, **for a change-id audit**. Unchanged decisions must keep the same `stable_id` and `merge_entries()` must update that slot in place, not duplicate it. **A standalone range audit is different by design**: each run gets its own dated directory (D8), so a second run over an unchanged range producing a second directory with the same `stable_id`s inside it is the expected per-run-snapshot behavior, not a bug — `latest.*` moves to the newest run either way. |

## Red Flags

- Any file under `git status --short` after a run other than `openspec/changes/<change-id>/choices.json` and `choices.md` — or, for a standalone range audit, that run's `choices.json`/`choices.md` under `openspec/choices/`, the `latest.json`/`latest.md` copies at `openspec/choices/`, and retention's move into `openspec/choices/archive/` — the read-only contract has been violated.
- A `choices.json` entry whose `provenance.commits` or `provenance.files` is not a subset of the audited range's actual `git log`/`git diff` output.
- A `choices.json` where `self_reported: true` but no `session_log_ref` is present, or vice versa — the two are supposed to travel together.
- Two entries in `choices.md` with the same `choice` text and different `stable_id`s after a re-audit of an unchanged decision — the content-derived hash isn't actually content-derived.
- `choices.md` where a `high`-confidence entry appears before a `low`-confidence one, or a `sound` verdict appears before a `needs-user` verdict at the same confidence — the ranking invariant is the entire point of "least-confident-first."
- A non-zero exit code from anything in `skills/audit-choices/scripts/` — every entry point in this skill is required to exit 0.

## Verification

1. Run `cd skills && uv run pytest tests/audit-choices/ -q` and confirm all tests pass, including `test_readonly_posture.py`'s working-tree-snapshot assertion.
2. Run `grep -rniE 'anthropic|openai|import claude' "<skill-base-dir>/scripts/"` and confirm it finds nothing — no LLM SDK import anywhere in this skill's Python (host-assisted invariant).
3. Validate a produced `choices.json` against `openspec/schemas/decision-choices.schema.json` with `jsonschema.Draft202012Validator` and confirm it passes, including the six-field header.
4. Confirm `choices.md`'s first entry is a lowest-confidence entry and no entry precedes one of strictly lower confidence.
5. **For a change-id audit**, re-run the audit over an unchanged range and confirm `choices.json`'s entry count and every `stable_id` are unchanged — no duplicates. **For a standalone range audit**, re-run over the same range at a later time and compare the two run directories' ledgers instead: confirm the second run lands in its own new directory, the first directory's ledger is byte-unchanged, and the two ledgers' entries carry the same `stable_id`s for the same decisions (D8) — there is no single file to compare against itself.
6. Confirm the driver's exit code is `0` even when the candidate set includes `unsound` and `needs-user` verdicts, and even when an internal error is forced (e.g. an unreadable repo root).

From jankneumann/agentic-coding-tools, MIT.

More from jankneumann/agentic-coding-tools