model-card

Files (12)

• SKILL.md
• datasheet_template.md · references/
• metric_dimensions.md · references/
• model_card_template.md · references/
• check_model_card_complete.py · scripts/
• DATASHEET.md · scripts/check_model_card_complete_challenge/fixture/complete/
• MODEL_CARD.md · scripts/check_model_card_complete_challenge/fixture/complete/
• MODEL_CARD.md · scripts/check_model_card_complete_challenge/fixture/incomplete/
• problem.md · scripts/check_model_card_complete_challenge/
• verify.sh · scripts/check_model_card_complete_challenge/
• skill.yml
• test_model_card_complete.sh · tests/

SKILL.md (first lines)

# Model-Card Skill

## Purpose

This skill produces the **documentation an engineer-built medical-imaging model must carry**: a
**Model Card** (intended use, out-of-scope use, training data, per-subgroup performance, caveats), a
**Datasheet** for its dataset (provenance, composition, collection, labelling, consent), and a
**METRIC-informed data-quality pass**. It fills the templates **from facts the user supplies** — it
never invents a number, a provenance detail, a consent status, or a licence — and ships a deterministic
gate that no required section is missing or left as an unfilled `[NEEDS INPUT]` placeholder.

It is the **reporting** seam of the model-engineering lane: after `/model-assessment` audits the design
and produces the numbers, this skill records them in a portable, auditable card that
`/write-paper` and `/check-reporting` consume. It mirrors `/version-dataset` structurally (generate +
deterministic verify).

## When to use
- A trained model needs a Model Card / Datasheet for a repo, Hugging Face card, or manuscript supplement.

## When NOT to use
- Auditing the validation design / metrics → `/model-assessment`.
- Versioning the dataset bytes → `/version-dataset`; tabular variable docs → `/generate-codebook`.
- Item-by-item reporting-guideline compliance of the manuscript → `/check-reporting`.
- Building / training the model → `/model-scaffold`.

## Workflow

### Phase 1 — Collect the facts
Gather, from the user / the model's developers: task + architecture + provenance + licence; intended use
and out-of-scope use; training and evaluation cohorts; the reference standard and inter-reader agreement;
overall and **per-subgroup** performance; data collection, consent, and de-identification. Anything not
supplied stays `[NEEDS INPUT]` — never guess.

### Phase 2 — Fill the Model Card
Copy `${CLAUDE_SKILL_DIR}/references/model_card_template.md` to `MODEL_CARD.md` and fill each section
from the facts. Keep the headings. Numbers come only from `/model-assessment` / executed results.

### Phase 3 — Fill the Datasheet
Copy `${CLAUDE_SKILL_DIR}/references/datasheet_template.md` to `DATASHEET.md` and fill the seven
question groups (Motivation, Composition, Collection, Preprocessing/Labeling, Uses, Distribution,
Maintenance).

### Phase 4 — METRIC data-quality pass
Walk `${CLAUDE_SKILL_DIR}/references/metric_dimensions.md` (completeness, correctness, consistency,
representativeness, timeliness, provenance, label provenance, fairness/coverage, leakage safety) and
record each finding in the Datasheet. Anything that affects the headline metric's validity is also a
`/model-assessment` finding — cross-check there.

### Phase 5 — Verify completeness (deterministic gate)
```bash
python3 ${CLAUDE_SKILL_DIR}/scripts/check_model_card_complete.py \
  --card MODEL_CARD.md --datasheet DATASHEET.md --strict
```
`MISSING_SECTION` / `EMPTY_REQUIRED_SECTION` / `UNFILLED_FIELD` must be zero before the card ships.
`UNFILLED_FIELD` names each field of a required section still left as `[NEEDS INPUT]` / `[VERIFY]`
(e.g. License or subgroup performance), even when a sibling field is filled. An explicit `N/A` /
`None` counts as an answer only as a field's whole value, and never in Intended Use, Training Data,
Evaluation Data, Metrics or Quantitative Analyses. Only the template tokens as written
(`[NEEDS INPUT ...]`, `[VERIFY]`, `[VERIFY: ...]`, upper case) count; markdown link text such as
`[verify the protocol](https://...)` or `[Verify][ref]`, and anything inside an HTML comment or a
fenced code block, is not a placeholder.
Known limits: a field left as a hand-written `TODO` / `TBD` / `<...>` / `XXXX` is caught only when
the whole section is unfilled; keep the template's bracketed `[NEEDS INPUT]` markers for open fields.

### Phase 6 — Hand off
Carry the card into `/write-paper` (the Methods / supplement reference it), `/check-reporting`
(CLAIM 2024 / TRIPOD+AI item audit of the manuscript), and `/self-review`.

## Anti-Hallucination

- **Never invent evaluation numbers, subgroup results, or dataset provenance.** Every figure comes from
  `/model-assessment` or the user's executed results; every provenance / consent / licence statement is
  user-confirmed. Unknown → `[NEEDS INPUT]`, which the gate flags.
- **Never mark a section complete without user-supplied content**, and never auto-fill a placeholder to
  pass the gate.
- **Never assert a licence or consent status the user did not confirm.**
- The gate checks **presence**, not truth — a complete card can still contain a wrong number; validity
  is `/model-assessment` and the human's responsibility.

## Deterministic gate
`scripts/check_model_card_complete.py` — verifies every required Model Card / Datasheet section is
present and non-empty (stdlib, network-free). Reproducible challenge:
`bash ${CLAUDE_SKILL_DIR}/scripts/check_model_card_complete_challenge/verify.sh`.

## Note on classification
Model Cards (Mitchell et al. 2019) and Datasheets (Gebru et al. 2021) are **documentation standards**,
not clinical reporting guidelines, so they live here as `references/` **templates** (uncounted), not in
`/check-reporting`'s counted checklist set — the same way `appraisal_tools/METRICS.md` is kept separate.
`/check-reporting` still owns the manuscript-level CLAIM 2024 / TRIPOD+AI item audit.

## Boundaries

```
model-assessment (audit design + metrics)
  └─ model-card (this skill: Model Card + Datasheet + METRIC pass, completeness-gated)
       └─ write-paper + check-reporting (manuscript) ; version-dataset (dataset bytes)
```

From Aperivue/medsci-skills, MIT.

More from Aperivue/medsci-skills