audit-agent-tooling

Files (3)

• SKILL.md
• preflight-corpus.mjs · scripts/
• check-preflight.mjs · tests/

SKILL.md (first lines)

# Audit agent tooling

Treat session transcripts as measurements, not anecdotes. The output is a
small set of falsifiable rules, each with its population, command, result, and
bound. This is not an agent ranking or a productivity score.

Follow the `efficient-tool-use` skill for rules already earned. Its
topic references are the durable rulebook; do not rediscover or silently
overwrite an entry.

## 1. Freeze the corpus

Record every transcript path, harness, session ID, file hash when stable, and
read time. Live sessions change between reads, so a count without its read
time is not reproducible. State the corpus and time window in every table.

State the active window explicitly in every census, including whether `--since`
was supplied or defaulted. Before reading the corpus, preflight it for the
subject under audit:

- a selected file or store with zero recognized subject records in the active
  window is refused, with the observed record-type census and the window;
- a mixed store reports which record types were recognized and which were
  ignored;
- an intentionally empty corpus requires an explicit opt-in, so a legitimate
  zero is distinguishable from an instrument that never saw the subject.

A zero from a store whose subject records were never reached is a failed
measurement, not evidence of absence.

Use the focused boundary before reading a corpus:

```sh
node skills/audit-agent-tooling/scripts/preflight-corpus.mjs \
  --since <ISO> [--until <ISO>] <file-or-directory>...
```

The command emits the active window, recognized and ignored record-type
censuses, and the per-file counts. It exits `1` with `REFUSED` when the
selected corpus has zero conversation records in the window. `--allow-empty`
is the explicit exception; it prints the opt-in in its receipt. Its focused
cases and seeded removal-red are in
`skills/audit-agent-tooling/tests/check-preflight.mjs`.

Keep transcript source files read-only. Do not paste large transcript
payloads into the report; reference the source and show only the relevant
records or normalized rows.

## 2. Parse records, not text

Extract assistant tool-call records and pair each with its tool result by ID.
Count actual calls, not strings that mention a call. If the transcript format
does not expose IDs or a complete record type, say so and reduce the claim.

Use a small, rerunnable parser when the corpus has more than one session.
Keep the parser next to the report or in the owning repository so another
reader can reproduce the census.

Searching before reading is a corpus-level lesson: search for the target
statement first, then read only the matching region.

## 3. Classify by operation, then control for conflation

At minimum, separate:

- direct structured tool calls from shell calls;
- reads used to answer a targeted question;
- search-before-read;
- inline interpreters and ad-hoc parsers;
- poll loops;
- output shaping such as `head`, `tail`, `cut`, and `fold`;
- repeated invocation shapes after normalizing arguments;
- skill loads and other explicit procedure activation.

Do not attribute an outcome to syntax when one command or subcommand explains
it. Split the data by command identity before naming a shape rule. Replicate a
candidate rule on another corpus when one is available.

Bucket non-zero or error results by their structured code. Semantic families
such as "safe hold" and "transport failure" have different recovery paths and
must not be collapsed under one word.

## 4. Keep the instrument inside its claim

`is_error` measures crashes and refusals. It does not measure wrong answers. A
low crash rate cannot establish that a pattern is safe, and a successful
command cannot establish that its result is correct.

If correctness is part of the question, use an instrument that checks the
result against a known answer. State the residual risk when the corpus has no
correctness signal.

For a guard, extension, or wrapper, measure three separate things:

1. **Reach:** which harnesses, seats, and sessions can load it.
2. **Activation:** a real round trip, not only installation or source.
3. **Scope:** what it can change and what remains outside its boundary.

A guard that cannot reach the failure path does not protect it, even when the
guard itself works.

## 5. Convert the finding into the right artifact

- Repeated invocation shape: save the operation as a named routine, wrapper,
  or command. The gap is often "the tool exists; the invocation shape does
  not."
- Numeric or set-valued answer parsed from text: expose a typed or structured
  result and make required arguments explicit. This removes the parsing step
  where silent errors occur.
- A high crash rate concentrated in one command: fix or document that
  command's contract instead of labeling the syntax unsafe.
- A low crash rate: do not promote it to a safety rule without a correctness
  check.
- A verified correction: replace the superseded rule in
  the matching `efficient-tool-use` topic and mark the correction in the
  report. Do not leave both versions active.

Add a rule to the cookbook only when it changes a future decision and its
evidence is reproducible. Keep single instances labeled as candidates until a

From emliunix/home.conf, GPL-3.0.

More from emliunix/home.conf