model-selection

Files (19)

• SKILL.md
• classification.md · references/
• detection.md · references/
• foundation_models.md · references/
• graph.md · references/
• index.md · references/
• segmentation.md · references/
• synthesis.md · references/
• check_model_provenance.py · scripts/
• clean.txt · scripts/check_model_provenance_challenge/expected/
• defect.txt · scripts/check_model_provenance_challenge/expected/
• unstated.txt · scripts/check_model_provenance_challenge/expected/
• dossier_clean.json · scripts/check_model_provenance_challenge/fixture/
• dossier_defect.json · scripts/check_model_provenance_challenge/fixture/
• dossier_unstated.json · scripts/check_model_provenance_challenge/fixture/
• problem.md · scripts/check_model_provenance_challenge/
• verify.sh · scripts/check_model_provenance_challenge/
• skill.yml
• test_model_provenance.sh · tests/

SKILL.md (first lines)

# Model-Selection Skill

Two questions, in this order. Phases 1–3 answer a literature question with a stable answer: which
architecture **family** suits the task. Phases 4–6 answer a provenance question: which concrete
**artifact** — repository, revision, checkpoint — will be run, and what its numbers may claim.
The skill writes decision notes and runs one stdlib gate; it never downloads, runs, trains or
benchmarks a model, and it describes archetypes, not a live SOTA leaderboard.

Elsewhere: building the repo → `/model-scaffold`; profiling data and planning preprocessing →
`/imaging-data`; validation design, metrics, uncertainty, explainability → `/model-assessment`;
documenting a model you built → `/model-card`; an LLM/MLLM, including benchmark contamination in
that setting → `/mllm-eval`; AI-vs-expert reader study → `/design-ai-benchmarking`.

## Workflow

### Phase 1 — Frame the question
State the **task** (classification / segmentation / detection / synthesis / transfer), the
**modality and dimensionality** (2-D vs 3-D volume), the **labelled-data scale** (events /
structures, not just images), **label availability** (lots / few / unlabelled pool), and the
constraints (class imbalance, small structures, interpretability, deployment compute).

### Phase 2 — Walk the decision tree, read the family card
Follow `${CLAUDE_SKILL_DIR}/references/index.md` (task → constraints → default pick). It routes to
one card; each gives the source paper, core idea, when-to-use, medical-imaging use, reference
implementation, and the typical validation setup for that class.

| Card | Families |
|---|---|
| `${CLAUDE_SKILL_DIR}/references/classification.md` | ResNet / DenseNet / EfficientNet / Inception / ConvNeXt / ViT / Swin / DeiT |
| `${CLAUDE_SKILL_DIR}/references/segmentation.md` | U-Net / 3-D U-Net / V-Net / Attention & Residual U-Net / nnU-Net (+ ResEnc, MedNeXt, STU-Net) / SegResNet / Swin-UNETR / Mask R-CNN |
| `${CLAUDE_SKILL_DIR}/references/detection.md` | nnDetection / Faster R-CNN + FPN / Mask R-CNN / RetinaNet / YOLO / DETR |
| `${CLAUDE_SKILL_DIR}/references/synthesis.md` | Pix2Pix / CycleGAN / SPADE / diffusion (DDPM, latent) / VAE / fastMRI reconstruction |
| `${CLAUDE_SKILL_DIR}/references/foundation_models.md` | SAM / MedSAM / MedSAM2 / nnInteractive / VISTA3D / TotalSegmentator / SegVol / BiomedCLIP / DINO / MAE / SimCLR / MoCo |
| `${CLAUDE_SKILL_DIR}/references/graph.md` | GCN / GraphSAGE / GAT / GIN / BrainGNN for connectomes and population graphs (PyTorch Geometric / DGL; not scaffolded by `/model-scaffold`) |

Never recommend an architecture for a modality or data scale it does not suit (a from-scratch ViT
on a few hundred images, 2-D slices for a volumetric structure) — the decision-tree constraints
exist to prevent exactly that. If asked for "the best" model, say the zoo is a curated archetype
map, not a current SOTA ranking.

### Phase 3 — Write the architecture decision note
Write `decisions/architecture_choice.md`: the task, the chosen architecture, its **source paper**
(mandatory — the Methods cite it), the reason against the constraints, the runner-up and why not,
and the matching `/model-scaffold` template. Never invent a benchmark number or paper claim: cite it
(verify via `/search-lit`), or write `[VERIFY]` and ask. A reference without a `/search-lit`-confirmed
DOI/PMID is marked `[UNVERIFIED - NEEDS MANUAL CHECK]`.

### Phase 4 — Write the model dossier for the concrete artifact
When a concrete candidate exists (a GitHub repo, a Hugging Face checkpoint, a paper's released
weights), write `model_dossier.json` recording what is **known**; leave unknowns unstated, never
guessed — the gate turns an absence into a finding.

```json
{
  "model": "OrganSeg-3D v2.5.1",
  "source": {"kind": "github", "url": "...", "version": "v2.5.1", "commit": "abc1234"},
  "licence": {"spdx": "Apache-2.0", "verified_from": "LICENSE at commit abc1234"},
  "intended_use": "research",
  "weights": {"pretrained": false},
  "task": {"model": "3d_ct_organ_segmentation", "study": "3d_ct_organ_segmentation"},
  "reported_validation": [{"dataset": "ExampleBench", "metric": "Dice", "source": "J Ex 2021"}],
  "developed_on": ["ExampleBench"],
  "evaluation_arms": [{"name": "external", "dataset": "OtherCohort-2026"}],
  "hardware": {"claimed": "any CUDA GPU", "verified_on": "GTX 1080 Ti", "verified": true}
}
```

Read each field from the artifact, not from memory:
- **Licence** — only from the licence file at the pinned revision, and record which file
  (`verified_from`). Never from a README badge, a model-card summary, or memory; an unstated
  licence is `LICENCE_UNSTATED`, never "probably MIT".
- **`developed_on`** — the paper's own account of where the method was built and tuned. It is the
  field people skip and the one the gate needs: a method that won a challenge was tuned on it.
- **`hardware.verified`** — true only after it actually ran. A support matrix is a different claim:
  a CUDA capability one compiler accepts may still be refused by another in the same stack.

### Phase 5 — Gate the dossier
```bash
python3 ${CLAUDE_SKILL_DIR}/scripts/check_model_provenance.py --dossier model_dossier.json \
    --out qc/model_provenance.json --strict
```
Stdlib-only and network-free: nothing is fetched and no licence resolved online.

| Verdict | Severity | Fires when |
|---|---|---|
| `BENCHMARK_PROVENANCE_CONFLICT` | Major | an evaluation arm uses a dataset the model was developed or tuned on |
| `EVAL_DATA_IN_TRAINING` | Major | an evaluation arm's dataset is inside the pretraining corpus |
| `LICENCE_UNSTATED` | Major | no licence recorded — not the same as a permissive one |
| `LICENCE_INCOMPATIBLE` | Major | non-commercial / research-only licence under commercial or deployment intent |
| `WEIGHTS_PROVENANCE_UNKNOWN` | Major | pretrained weights whose training corpus is not stated |
| `DEVELOPED_ON_UNSTATED` | Major | the `developed_on` key is absent, so the conflict check could not run |
| `EVALUATION_ARMS_UNSTATED` | Major | the `evaluation_arms` key is absent, so neither provenance check could run |
| `INTENDED_USE_UNSTATED` | Major | `intended_use` is absent, so licence compatibility could not be checked |
| `TASK_MISMATCH` | Minor | the model's task is not the study's task |
| `NO_VERSION_PIN` | Minor | no commit, tag or revision |
| `VALIDATION_UNREPORTED` | Minor | no reported validation (dataset + metric + source) |
| `HARDWARE_UNVERIFIED` | Minor | hardware support claimed but never executed |
| `LICENCE_UNVERIFIED` | Minor | a licence is named but not the file it was read from |

The gate flags a **relationship, not a reputation**: `developed_on: ExampleBench` passes cleanly as
long as no evaluation arm uses ExampleBench (the clean fixture proves this). Dataset names match as
**token sequences** with a small family-alias table — `MSD Task09 Spleen` matches `MSD`,
`MS Cohort 2026` does not — never by substring.

An explicit empty list (`"developed_on": []`) states "none" and is not a finding; an absent key is.
`intended_use` must be `research`, `commercial` or `clinical_deployment`, and `developed_on` /
`weights.trained_on` must be lists of strings; anything else is an input error (exit 2), never a pass.

**Known limits.** `LICENCE_INCOMPATIBLE` recognises non-commercial / research-only markers in the
licence string (`NC`, `non-commercial`, `research-only`, in SPDX or spaced form). A free-text licence
that restricts use without those markers (e.g. "academic use only"), or a copyleft licence under
closed deployment, is not detected: read the licence file and record the conclusion yourself.

### Phase 6 — Turn a Major into a study decision
A `BENCHMARK_PROVENANCE_CONFLICT` rarely means abandoning the model (it is often the best-engineered
option precisely because it was tuned hard). It changes **what the arm may claim**:
1. Report that arm as a demonstration that the pipeline runs end to end, not as evidence the method works.
2. Put the evidential weight on an arm whose data **post-dates** the model, and say so with dates.
3. State the conflict in Methods and Limitations rather than leaving a reviewer to find it.

From Aperivue/medsci-skills, MIT.

More from Aperivue/medsci-skills