findings-audit

Files (3)

• SKILL.md
• open-ended-mode.md · reference/
• output-schemas.md · reference/

SKILL.md (first lines)

# /findings-audit — Audit findings against external evidence

Reality-check a project's empirical findings against external evidence the
project has already collected. The audit is **asymmetric by design**: it looks
for counter-evidence to load-bearing interpretations, not confirmation.
Pro-evidence is over-selected (interpretations were authored having seen
supportive evidence) — what tests them is contradictions.

External evidence comes in two families and four source types:

| Family    | Source type        | Typical home                                  | Bias                                              |
|-----------|--------------------|-----------------------------------------------|---------------------------------------------------|
| Narrative | `news`             | `references/news/stories.csv`                 | Over-samples unusual / contested / extreme events |
| Narrative | `interview`        | `references/interviews/*.md`                  | Subject self-selection; contested-case salience   |
| Narrative | `court-opinion`    | `references/cases/*.md`                       | Appellate / published-decision over-representation |
| Aggregate | `aggregate-report` | `references/reports/*.pdf`, `references/cnj/` | Methodology-bound; definitional gaps              |

The procedure is the same in shape but **adjudication and falsification
thresholds differ by type** — see §5 and §6.

Two modes:

- **Targeted (default).** For each finding, derive or accept a falsification
  pattern ("what evidence would contradict this?"), then scan each source
  type for matches.
- **Open-ended (`--open-ended`).** Find evidence patterns that no current
  finding or hypothesis addresses. Surfaces blind spots and new-hypothesis
  candidates. Slower; run quarterly or after a corpus expansion.

## When to invoke

- "sanity check findings against the news / interviews / CNJ reports"
- before drafting a section that leans heavily on a single interpretation
- after a corpus refresh (new articles fetched, new CNJ release, new transcripts)

Don't use it for:

- collecting new evidence (that's `/anecdotes` for news; manual for the rest)
- checking prose against scripts (that's `/validate-section`)
- pure literature lookup — that's `WebFetch`/research

## Args

```
/findings-audit [--findings <path>] [--corpus <path>...] [--mode targeted|open-ended]
                [--finding <slug-or-heading>] [--types news,interview,court-opinion,aggregate-report]
                [--full] [--max-rows N] [--auto]
```

- `--findings` — findings doc to audit. Default: discovered (§1).
- `--corpus` — one or more evidence sources. Default: discovered (§2).
- `--mode` — `targeted` (default) or `open-ended`.
- `--finding` — restrict to one finding (heading slug or phrase).
- `--types` — restrict to a subset of source types. Default: all that exist.
- `--full` — ignore `last_audited_sha` cache.
- `--max-rows` — cap rows examined per flag (cost guardrail).
- `--auto` — skip the §0 confirmation gate. Use for autonomous / scheduled runs.

**Default behavior on bare invocation (`/findings-audit` with no args):**
discover findings doc and sources, derive a plan, **present the plan and ask
the user to confirm or edit before any classification work runs**. See §0.

## Procedure

### 0. Plan and confirm (default; skip with `--auto`)

On bare invocation, run §1, §2, and §3 first as a *plan-only pass* — no
classification, no LLM cost — then present the plan and wait for the user.

The plan should include:

```
Findings audit — plan

Findings doc:    <path> (<N> entries; sha <short>)
Mode:            targeted   [pass --mode open-ended for residual sweep]
Cache:           last audit on <date> at sha <short>; <K> findings changed,
                 <M> sources changed since   [or: no prior audit]

Sources detected:
  [x] news               <N> rows         <path>
  [x] aggregate-report   <N> documents    <paths>
  [ ] interview          (none found)
  [ ] court-opinion      (none found)

Restrict to one finding?   no  [pass --finding <slug> to restrict]
Candidates to read (after keyword/theme filter):
  news               <N1> rows across <K> findings
  aggregate-report   <N2> tables across <K> findings
  ...
  Total:             <N> snippets (warn if >300; see Cost guardrails)

Derived falsification patterns (per finding × source family):
  - <finding-slug-1>:
      narrative:  "..."
      aggregate:  "..."
  - <finding-slug-2>:
      narrative:  "..."
      aggregate:  "..."   [or: n/a — finding has no numeric form]
  ...
```

Then ask:

> Proceed as shown? Or edit:
> (1) findings doc, (2) mode, (3) source types,
> (4) restrict to one finding, (5) falsification patterns, (6) cancel.

If the user picks an edit option, revise that piece and re-show the plan.
Loop until the user confirms or cancels.

If `--auto` is passed, skip this section and proceed to §4.

If args were partially specified (e.g. `--mode open-ended` but no
`--findings`), fill in defaults via discovery, show the plan, and confirm
once. Don't re-prompt on individual fields the user already pinned.

If discovery comes up empty for a load-bearing input (no findings doc;
no evidence sources at all), stop here and ask — don't show a plan you

From hsigstad/research-kit, MIT.

More from hsigstad/research-kit