agent-output-audit

Files (10)

• SKILL.md
• audit-report-template.md · assets/
• issue-template.md · assets/
• ai-implementation-audit.md · references/
• checklist.md · references/
• e2e-coverage.md · references/
• flaky-triage.md · references/
• independent-evaluator-protocol.md · references/
• project-signals.md · references/
• discover-project-contract.py · scripts/

SKILL.md (first lines)

# Agent Output Audit

Independent verification of AI-implemented work. The skill that asks: *"Did the implementing agent actually do what `task_NN.md` says it did?"* — not *"Would a real user succeed at this product?"* (that's `qa-execution`).

## Required Reading Router

Match your task to the row. Read the listed files **in full before** producing output. They are not appendices — they are load-bearing. Inline content in this SKILL.md is a pointer, not a substitute.

| Task                                                                 | MUST read                                                                                |
| -------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| Discovering install/lint/test/build/start commands (Step 1)          | `references/project-signals.md`                                                          |
| Deciding E2E support and classifying coverage (Step 1)               | `references/e2e-coverage.md`                                                             |
| Building the audit scope checklist (Step 2)                          | `references/checklist.md`                                                                |
| Holding independent-evaluator stance on AI tasks (Step 3)            | `references/independent-evaluator-protocol.md`                                           |
| Scanning test diffs for AI hygiene red flags (Step 4)                | `references/ai-implementation-audit.md`                                                  |
| Diagnosing a test that passed on retry without a code change         | `references/flaky-triage.md`                                                             |

## Reference Index

- `references/project-signals.md` — Heuristics for picking install/lint/test/build/start commands across ecosystems when the repo lacks an umbrella gate.
- `references/e2e-coverage.md` — Taxonomy for `existing-e2e` / `needs-e2e` / `manual-only` / `blocked` and how to detect harness support.
- `references/checklist.md` — Audit checklist by category: contract discovery, baseline, task audit, AI hygiene, flaky detection, quality gates.
- `references/ai-implementation-audit.md` — Red Flag scanners (RF-1..RF-6), Requirement→Test mapping, verdict matrix for completed tasks.
- `references/independent-evaluator-protocol.md` — What counts (and doesn't count) as evidence; transcript classification (`genuine-failure` / `grader-bug` / `ambiguous-task` / `bypass-exploit`).
- `references/flaky-triage.md` — Taxonomy, diagnosis protocol, and quarantine workflow for retry-passes-without-code-change failures.

## Required Inputs

- **audit-output-path** (optional): Directory where audit artifacts (bugs, audit report, evidence) are stored. When provided, create the directory if it does not exist and use it for all audit outputs. When omitted, fall back to repository conventions or `/tmp/agent-output-audit-<slug>`.

## Procedures

**Step 1: Discover the Repository Verification Contract**

1. Read root instructions, repository docs, and CI/build files before running commands.
2. Execute `python3 scripts/discover-project-contract.py --root .` to surface candidate install, verify, build, test, lint, start commands, and E2E signals.
3. **STOP. Read `references/project-signals.md` in full before picking commands when discovery surfaces more than one plausible gate or the repo mixes ecosystems.**
4. **STOP. Read `references/e2e-coverage.md` in full before classifying any flow.**
5. Prefer repository-defined umbrella commands such as `make verify`, `just verify`, or CI entrypoints over language-default commands.
6. Resolve the audit artifact directory. If the user provided an `audit-output-path` argument, use it. Otherwise use repository conventions, falling back to `/tmp/agent-output-audit-<slug>`. Create the `audit/` subdirectory; store all bugs and reports under `<audit-output-path>/audit/`.
7. **Detect Compozy mode.** If `.compozy/tasks/<slug>/` exists, record the slug and switch into Compozy-aware audit:
   - Read `state.yaml` **(read-only — never write to it; `scripts/update-state.py` owns mutation per the cy-codex-loop contract)**.
   - Read `_techspec.md` (deliverable source of truth) and `_tasks.md` (task roster) when present.
   - List every `task_NN.md` and capture its frontmatter `status:` value (allowed: `pending`, `in_progress`, `completed`). When `task_NN.md` frontmatter disagrees with `state.yaml`, treat frontmatter as the source of truth.
   - Note the canonical memory slot `.compozy/tasks/<slug>/memory/qa-execution.md` — Step 4 writes audit notes there before any status flips.

**Step 2: Run the Baseline Verification Gate**

1. Install dependencies with the repository-preferred command.
2. Run the canonical verification gate once before any audit work. Execute in fastest-first order: lint and type-check, then build, then unit tests, then integration tests.
3. If the E2E command is separate from the umbrella gate, decide whether to run it now or after runtime prerequisites are ready, then record that plan explicitly.
4. If the baseline fails, read the first failing output carefully and determine whether it is pre-existing or introduced by current work before moving on.
4a. **Flaky-failure protocol.** When a baseline command fails, before classifying as pre-existing or new, run the failing test in isolation 3-5 times on the same SHA. If it passes at least once without code changes, classify as `flaky-suspect`, record in `audit-report.md` under `SUITE HEALTH SNAPSHOT` (test name, attempts, retry outcome, suspected category), and **do NOT promote to PASS via retry**. **STOP. Read `references/flaky-triage.md` in full before assigning a suspected category or proposing a quarantine.**

**Step 3: Audit Task Implementations (Compozy mode and any AI-implemented tasks)**

Skip this step only when no task, phase, PRD, tech spec, or implementation-plan artifacts exist.

1. **STOP. Read `references/independent-evaluator-protocol.md` in full before forming any task verdict.** Tripwire summary: never accept the implementing agent's transcript, success message, or memory note as evidence. In Compozy mode, read the implementing agent's `.compozy/tasks/<slug>/memory/<phase>.md` artifacts and classify anomalies (`genuine-failure` / `grader-bug` / `ambiguous-task` / `bypass-exploit`) in the `Errors / Corrections` section of `memory/qa-execution.md` **before** judging the task.
2. Read each `task_NN.md` and its body. Summarize each task into a Task Implementation Matrix (column names mirror cy-codex-loop frontmatter):
   - `task_path` (e.g., `.compozy/tasks/<slug>/task_07.md`)
   - `declared_status` — literal frontmatter `status:` value
   - `title`, `type`, `complexity`, `dependencies` — mirrored from frontmatter
   - `techspec_deliverable` — linked section in `_techspec.md` when present
   - Requirements, subtasks, checklist items, success criteria, dependent files
   - `implementation_evidence` — files, modules, routes, commands, migrations, seeds, tests
   - `verification_evidence` — commands executed, exit codes, output summaries
   - `qa_verdict` — `PASS` | `PARTIAL` | `FAIL` | `REOPEN` | `BLOCKED` (distinct from `declared_status`)
   - `ai_audit_findings` — red flag IDs that fired in Step 4 with verdict
   - `action` — `none` | `fixed` | `reopened-frontmatter` | `BUG-NNN.md filed`
   - `linked_bugs` — BUG IDs
3. Do not treat a task `declared_status`, checked checkbox, memory note, or prior agent summary as proof. Verify every completed or claimed-complete task against actual files, public behavior, automated tests, and acceptance criteria.
4. Classify each task with `qa_verdict`:
   - `PASS`: every material requirement and success criterion has implementation and fresh verification evidence.
   - `PARTIAL`: implementation exists but one or more non-critical requirements, tests, or evidence are missing.
   - `FAIL`: claimed behavior does not work or a critical requirement is absent.
   - `REOPEN`: the source `task_NN.md` has `status: completed` in frontmatter but the QA verdict is `PARTIAL` or `FAIL`.
   - `BLOCKED`: audit cannot continue because a concrete prerequisite is missing.

**Step 4: AI Test-Hygiene Scan (RF-1..RF-6)**

1. **STOP. Read `references/ai-implementation-audit.md` in full before scanning the test diff of any task with `declared_status: completed`.** That file owns the Red Flag scanners (RF-1..RF-6), the Requirement→Test mapping rules, and the verdict matrix.
2. Run the scans against the diff since the task baseline (`git log --follow <test_file>`, `git diff <baseline_sha>..HEAD`).
3. Emit verdict `FAIL` automatically when scanners detect:
   - Weakened assertions on P0/P1 Success Criterion (RF-2).
   - `.skip` / `.only` / `xit` / `t.Skip` inserted in the diff (RF-1).
   - Mocks inserted in tests whose corresponding TC declared `External Dependencies` as Integration/E2E (RF-3).
   - Snapshot drift on P0/P1 with no requirement-change justification (RF-4).
4. Record findings in the Task Implementation Matrix column `ai_audit_findings` and in the per-task block of `audit-report.md`.
5. Apply the Requirement → Test mapping table from `references/ai-implementation-audit.md`. For every Success Criterion in `task_NN.md` (frontmatter or body) and every linked bullet in `_techspec.md`, find the corresponding test by name, reference, or assertion content. Mark each criterion `covers` / `weak` / `missing`. A checked item or `status: completed` without a `covers` row is an audit failure.

**Step 5: Reopen, File Bugs, Write Memory**

1. Mark incomplete completed tasks as `REOPEN` in the matrix.
2. **In Compozy mode**, write audit notes to `.compozy/tasks/<slug>/memory/qa-execution.md` using the canonical sections required by cy-codex-loop: `Objective Snapshot`, `Important Decisions`, `Learnings`, `Files / Surfaces`, `Errors / Corrections`, `Ready for Next Run`. This file must be written **before** any `task_NN.md` frontmatter is flipped (memory-precedes-status invariant).
3. Edit the offending `task_NN.md` frontmatter `status:` back to `pending` (or `in_progress` if salvageable). **Never write to `state.yaml`** — cy-codex-loop's `update-state.py` owns mutation; frontmatter wins because the next iteration reconciles from it.
4. File `BUG-<num>.md` under `<audit-output-path>/audit/issues/` using `assets/issue-template.md`. Include:
   - The task path under `Reopens task:`.
   - The failed Success Criterion under `Summary:`.
   - The original strict assertion (when RF-2 fired) under `Root cause:`.
   - The red flag ID and verdict under `Automation Follow-up:` notes.
   - The transcript anomaly classification (when applicable) under `Related:`.
5. When the missing work is a bounded root-cause fix inside the audit scope, you may implement it, add regression coverage, and rerun the task proof. Otherwise reopen the task — do not silently pass it.

**Step 6: Quality Gates Verdict**

1. Re-run the canonical verification gate from scratch after the last code change made during the audit.
2. Compile the Quality Gates section of `audit-report.md`. Each gate is `PASS` / `FAIL` / `N/A`:
   - Flaky rate <2% in canonical suite.
   - Zero `FAIL` from AI test-hygiene audit on P0/P1 tasks.
   - Zero `Critical` / `High` issues open.
   - Coverage delta ≥ baseline (no regression).
   - Zero unresolved `flaky-suspect` on P0 flows.
3. A `FAIL` on any gate blocks an unconditional PASS verdict for the run.

**Step 7: Write the Audit Report**

1. Summarize the audit using `assets/audit-report-template.md` and write the report to `<audit-output-path>/audit/audit-report.md`.
2. Mandatory sections:
   - **Claim / Command / Exit code / Verdict** per command executed in Step 2 and Step 6.

From haryelramalho/skills, MIT.

More from haryelramalho/skills