conformance

Files (3)

• SKILL.md
• report-template.md · references/
• rule-schema.md · references/

SKILL.md (first lines)

# /conformance

Measure a codebase against rules **someone else wrote**. The ruleset is the
authority; the code is the thing under test. Your job is to say — rule by rule,
with evidence — which rules are met, which are half-met, which are absent, and
which never applied in the first place.

Three guiding principles:

- **The ruleset is external and fixed.** You do not get to reinterpret a rule
  because the code is inconvenient. If the code diverges deliberately, that is a
  recorded *divergence with a rationale*, not a redefinition of the rule.
- **Evidence or silence.** Every verdict cites `path/file.ext:line`. A claim you
  cannot cite is a claim you do not make.
- **Coverage of the ruleset must be total.** Every rule is either audited or
  explicitly excluded with a reason. Silent drops are the failure mode that makes
  an audit worthless.

## The two tiers

This is the load-bearing distinction. Get it wrong and the skill produces
fiction.

| Tier | Produces | Applies to |
|---|---|---|
| **1 — Audit** | per-rule verdict + `file:line` evidence + gap | **every** ruleset, always |
| **2 — Executable** | a real, running test per rule | **only** rules the source states concretely enough to assert |

Tier 1 always runs. Tier 2 runs **only** where the ruleset supplies a concrete
expected value or an unambiguous mechanical predicate.

A rule is tier-2 eligible when the source gives you a number, a required
ordering, a state transition, or a yes/no predicate you can evaluate. Specs with
worked examples — "in this situation the answer is 700" — are gold: they are an
external authority that cannot drift with your code.

A rule is **tier 1 only** when it reads like *"ensure appropriate access
controls"* or *"the operator should act reasonably"*. There is nothing to
assert. Say so and move on.

> **Never invent an expected value.** If the ruleset does not state the answer,
> the rule is tier 1. Fabricating an expected value produces a test that asserts
> your guess and calls it compliance.

## Modes

Mode comes from the phrasing; there are no strict flags.

| Ask sounds like | Do | Output |
|---|---|---|
| `/conformance <spec>` (default) | full pipeline | rule file + audit + harness |
| "parse/ingest this spec" | ingest only | canonical rule file |
| "which of these apply to us" | ingest + scope | rule file with `applies` set |
| "audit us against <rules>" | scope + audit | `CONFORMANCE_AUDIT.md` |
| "turn the spec's examples into tests" | harness only | test module |
| "re-run the conformance audit" | re-audit | updated audit; verdict deltas called out |
| "write up the failures" | defect report | repo's defect/docs convention |

## Workflow

### 1. Ingest — source material to canonical rules

Convert whatever you were handed (PDF, HTML spec, markdown, RFC text, an
existing YAML) into one canonical rule file. Schema and worked example:
`references/rule-schema.md`.

Extract text mechanically rather than transcribing by eye (`pdftotext -layout`
for PDFs). Multi-column layouts interleave — read the whole extraction before
structuring it.

Preserve rule text **verbatim**. The rule text is quoted authority; your commentary
belongs in separate fields. Record the source file, version, and date.

**Verify coverage before moving on:** parse the file and assert the rule count and
numbering match the source. A rule silently lost at ingest is invisible forever
after.

### 2. Scope — which rules bind this system

Not every rule in a ruleset binds every implementation. Tag each rule:

- **`direct`** — the rule is mechanically checkable against this system.
- **`adapted`** — the rule solves a problem that exists here in a *different*
  form. It does not port verbatim, but the design question it raises is real and
  must be answered.
- **excluded** — the rule genuinely does not apply. Goes in an `excluded:` list
  with a one-line reason.

**Do not collapse `adapted` into excluded.** It is where the interesting work
lives — the rule whose literal mechanism is gone but whose intent still demands a
decision. Dropping those turns an audit into a checklist.

Record the inclusion criteria in the file so the filter is reviewable.

### 3. Audit — verdict plus evidence

For each in-scope rule, search the codebase and record:

- `implemented:` — `"yes"` / `"partial"` / `"no"` (**quoted** — see honesty rules)
- `evidence:` — `path:line — what is there`, omitted only when there is nothing
  to cite
- `gap:` — what is missing or divergent, omitted only when the verdict is `"yes"`

Search by **mechanism, not vocabulary**. The code will not use the ruleset's
words. Search for the behaviour: the arithmetic, the state transition, the
ordering, the guard.

An absence is a finding. If nothing implements a rule, say so and record how you
established the absence (the search that returned nothing).

### 4. Report

Write the audit into the repo's existing documentation folder — detect it, do not
assume `docs/`. Follow the repo's own conventions if it has a defect or audit
format already. Template: `references/report-template.md`.

The most valuable section is usually **not** the individual findings. It is the
coverage analysis: *would a fix for any of these make an existing test fail?* If
the answer is no, the suite is pinned at the wrong altitude, and that is a bigger

From folkengine/.claude, Apache-2.0.

More from folkengine/.claude