model-assessment

Files (66)

• SKILL.md
• explainability_guide.md · references/
• metric_guide.md · references/
• metric_selection_grounding.md · references/
• metrics_manifest_schema.md · references/
• uncertainty_guide.md · references/
• validation_design.md · references/
• check_explainability_report.py · scripts/
• strong.txt · scripts/check_explainability_report_challenge/expected/
• weak.txt · scripts/check_explainability_report_challenge/expected/
• report_strong.json · scripts/check_explainability_report_challenge/fixture/
• report_weak.json · scripts/check_explainability_report_challenge/fixture/
• problem.md · scripts/check_explainability_report_challenge/
• verify.sh · scripts/check_explainability_report_challenge/
• check_metric_reporting.py · scripts/
• check_split_leakage.py · scripts/
• clean.txt · scripts/check_split_leakage_challenge/expected/
• leak.txt · scripts/check_split_leakage_challenge/expected/
• split_seed.txt · scripts/check_split_leakage_challenge/fixture/
• splits_clean.csv · scripts/check_split_leakage_challenge/fixture/
• splits_leak.csv · scripts/check_split_leakage_challenge/fixture/
• problem.md · scripts/check_split_leakage_challenge/
• verify.sh · scripts/check_split_leakage_challenge/
• check_uncertainty_reporting.py · scripts/
• strong.txt · scripts/check_uncertainty_reporting_challenge/expected/
• weak.txt · scripts/check_uncertainty_reporting_challenge/expected/
• uncertainty_strong.json · scripts/check_uncertainty_reporting_challenge/fixture/
• uncertainty_weak.json · scripts/check_uncertainty_reporting_challenge/fixture/
• problem.md · scripts/check_uncertainty_reporting_challenge/
• verify.sh · scripts/check_uncertainty_reporting_challenge/
• clf_bad.md · scripts/metric_reporting_challenge/fixture/
• clf_good.md · scripts/metric_reporting_challenge/fixture/
• det_good_wrapped.md · scripts/metric_reporting_challenge/fixture/
• det_no_iou.md · scripts/metric_reporting_challenge/fixture/
• generative_bad.md · scripts/metric_reporting_challenge/fixture/
• generative_good.md · scripts/metric_reporting_challenge/fixture/
• interactive_bad.md · scripts/metric_reporting_challenge/fixture/
• interactive_good.md · scripts/metric_reporting_challenge/fixture/
• manifest_clf_bad.json · scripts/metric_reporting_challenge/fixture/
• manifest_det_no_match.json · scripts/metric_reporting_challenge/fixture/
• manifest_seg_bad.json · scripts/metric_reporting_challenge/fixture/
• manifest_seg_good.json · scripts/metric_reporting_challenge/fixture/
• multiclass_bad.md · scripts/metric_reporting_challenge/fixture/
• seg_bad.md · scripts/metric_reporting_challenge/fixture/
• seg_good.md · scripts/metric_reporting_challenge/fixture/
• problem.md · scripts/metric_reporting_challenge/
• verify.sh · scripts/metric_reporting_challenge/
• skill.yml
• metrics_manifest.json · templates/
• clean_imageid_case.csv · tests/fixtures/
• clean_imageid_case3class.csv · tests/fixtures/
• clean_imageid_case4class.csv · tests/fixtures/
• clean_imageid_casecontrol.csv · tests/fixtures/
• clean_participant_site_id.csv · tests/fixtures/
• clean_patient_visit_no.csv · tests/fixtures/
• clean_subject_sex_code.csv · tests/fixtures/
• leak_imageid_case.csv · tests/fixtures/
• leak_studyid_mrn.csv · tests/fixtures/
• leak_subject.csv · tests/fixtures/
• noseed_clean.csv · tests/fixtures/
• …and 6 more

SKILL.md (first lines)

# Model-Assessment Skill

Assess a trained medical-imaging model (in-house, vendor or open-weights; segmentation,
classification or detection) in the order the evidence is built: **Part A** designs and audits the
validation study, **Part B** computes task-correct held-out metrics, **Part C** adds the
uncertainty / OOD / abstention layer a deployment claim needs, and **Part D** makes an
explainability analysis survive review. Run the parts the request needs (a Grad-CAM question starts
at Part D), but no metric headline is reported before the Part A split gate is green. Each part ends
in a stdlib gate whose verdict is reproduced from a file — never report a pass without running it.

Numbers come only from code executed on the supplied predictions, split table, or the researcher's
executed UQ/XAI code; if predictions or ground truth are missing, say so and stop. Integrate
MONAI / nnU-Net, MAPIE, captum, pytorch-grad-cam and pretrained OOD scorers by reference — never
reimplement them, never build or train the model, never run a model on real patient data.

Elsewhere: building/training → `/model-scaffold`; choosing or vetting the model →
`/model-selection`; data-stage preprocessing leakage → `/imaging-data`; paired model comparison /
added value over a baseline / decision curves / MRMC / ICC / calibration tables → `/analyze-stats`
(added value: `incremental_value.md`); AI-vs-expert reader rubric
and IRR → `/design-ai-benchmarking`; LLM/MLLM → `/mllm-eval`; general validity → `/design-study`;
classical-ML tabular calibration → `/radiomics-ml`; item-by-item reporting audit →
`/check-reporting`; a finished manuscript → `/self-review` or `/peer-review` (the MD0–MD11
`model_development.md` probe).

## Part A — Validation design

The rationale behind Phases 2–7 — the full leakage taxonomy, the internal-vs-external ladder,
comparator design, run variance, test-set sizing, the reporting map — is in
`${CLAUDE_SKILL_DIR}/references/validation_design.md` (load on demand). Verify citations via
`/search-lit` (confirmed DOI/PMID), else mark `[UNVERIFIED - NEEDS MANUAL CHECK]`; flag an uncertain
CLAIM 2024 / TRIPOD+AI / Metrics Reloaded item `[VERIFY]` and ask.

### Phase 1 — Task, intended-use horizon, analysis unit
State the task, the **intended-use horizon** (screening, triage, pre-procedure, post-hoc), the
**single headline metric** the conclusion leans on, and the **analysis unit** (per-patient /
per-lesion / per-image). Everything downstream is read against this; a per-lesion metric is never
reported as per-patient.

### Phase 2 — Leakage audit (run first)
Produce the emitted split-assignment table (`patient_id,split`) and run:

```bash
python3 ${CLAUDE_SKILL_DIR}/scripts/check_split_leakage.py \
  --splits <split_assignment.csv> --out qc/split_leakage.json --strict
```

`PATIENT_OVERLAP` (a patient in ≥ 2 partitions) and `MISSING_SEED` (an unreproducible split) are
Major, `SINGLE_PARTITION` Minor — proven by set arithmetic on the ID column the gate prints as
`id_col` (pass `--id-col` with the patient identifier when it is not the one; an auto-picked
column whose name is not patient-level, such as `image_id`, gets a Minor `ID_COL_NOT_PATIENT_LEVEL`). A design with patient overlap is never
approved. Then walk the leakage the table cannot show (Kapoor & Narayanan, *Patterns* 2023):
preprocessing fit before the split (normalisation, resampling, foundation-model embeddings, ComBat
harmonisation over the whole cohort — `/imaging-data` gates the declared pipeline), site / scanner /
burned-in-label shortcuts, and temporal leakage (a random split where future and past coexist).
The decisive question: *could any value used in training have been computed only with knowledge of
a test case?*

### Phase 3 — Validation tier
Classify honestly: apparent → internal random split → cross-validation → temporal → geographic /
external (different site, scanner, vendor) → multi-site external. Cross-validation and bootstrap are
development-time optimism corrections, **not** external validation. Flag a generalisability or
deployment claim that outruns the design, and "external validation" where the single external set
was used for tuning. Confirm the test set was touched **once** — no architecture search,
hyperparameter sweep, early stopping, or threshold choice read it.

### Phase 4 — Comparator
Clinical-only baseline, incremental value over an existing score, or reader comparison (hand the
rubric / inter-rater design to `/design-ai-benchmarking`).

### Phase 5 — Test-set sizing
Count **events per class** in the test set, not the cohort total: a sparse positive set gives a CI
spanning much of the usable range, and calibration needs roughly ≥ 100 events. Hand formal sizing to
`/calc-sample-size`.

### Phase 6 — Prospective evaluation and deployment-monitoring horizon
Retrospective external validation shows accuracy *transfers*, not that the model is safe and useful
in the workflow. For a clinical-use claim design the higher tier explicitly — silent / shadow
deployment → prospective comparative or impact study / RCT on a clinical endpoint →
post-deployment monitoring with recalibration-or-withdrawal triggers and subgroup audit
(`references/validation_design.md` §2b). Scope the claim to the tier reached: a retrospective
external study never claims deployment readiness or outcome benefit.

### Phase 7 — Reporting-guideline fit
Map via `/check-reporting`: **CLAIM 2024** (diagnostic imaging AI), **TRIPOD+AI** (prediction model),
**STARD-AI** (diagnostic accuracy), **PROBAST+AI** (risk of bias), and for a prospective/live
evaluation **DECIDE-AI** or **CONSORT-AI / SPIRIT-AI**.

## Part B — Held-out metrics

### Phase 8 — Compute task-correct metrics
Generate and **execute** evaluation code on the held-out predictions (Metrics Reloaded —
Maier-Hein & Reinke et al., *Nat Methods* 2024):
- **segmentation** — Dice/IoU **and** a boundary metric (HD95 / NSD), **per structure**, 95% CIs by
  **patient-level** bootstrap (resample patients, not pixels or slices);
- **classification** — **AUROC and AUPRC** with patient-level bootstrap CIs, sensitivity/specificity,
  and PPV/NPV **at the deployment prevalence**, never bare accuracy on a balanced set. Report AUPRC
  with the test-set prevalence, which is its no-skill value: AUPRC moves with prevalence, so a value
  from an enriched or case-control test set does not carry over to deployment or across datasets. For
  **multiclass**, state the aggregation (one-vs-rest / macro / micro / pairwise / Obuchowski);
- **detection** — **FROC / mAP** with the **IoU match criterion stated**. Lesions and false positives
  cluster within patients, so CIs come from a **patient-level bootstrap** (resample patients, carrying
  all their lesions and false positives), not a Wilson/binomial interval over lesions, which is too
  narrow; compare two detectors' FROC curves with **JAFROC** (RJafroc), not per-lesion tests;
- **interactive / promptable segmentation** (SAM2 / MedSAM2 / nnInteractive) — the segmentation
  metrics **plus** Dice-vs-interactions / number-of-clicks (NoC) to a target threshold,
  initial-vs-converged (or peak) Dice, and per-case interaction/inference time. With two arms
  (simulated prompting + human operator), record **protocol fidelity** — identical prompt types,
  stopping rule, target threshold, seeds — because arm-to-arm comparability is what lets the human arm
  validate the simulated one (human-arm design: `/design-study`);
- **generative / synthesis** — full-reference (MSE/RMSE/PSNR/SSIM) or no-reference (SNR/CNR, visual
  scores) quality **plus a downstream-task evaluation**: image quality is not clinical utility
  (Park et al., *Radiol Med* 2024);
- **time-to-event** discrimination (Harrell's C, time-dependent ROC) → `/analyze-stats`.

Report the headline as the **point estimate with a patient-level bootstrap 95% CI** over the test
cases: that is the uncertainty of the test-set estimate. Seed-to-seed SD across training runs is a
different quantity (training-run variability, usually smaller) — report it separately, over ≥ 5
runs, for a training-recipe or model-comparison claim, and never present it as the CI. A frozen
vendor or open-weights model has no training runs to vary; its uncertainty is the test-set CI. Add

From Aperivue/medsci-skills, MIT.

More from Aperivue/medsci-skills