design-study

Files (18)

• SKILL.md
• combine_models_ablation_design.md · references/
• dag_adjustment.md · references/
• multi_model_comparison_design.md · references/
• reader_elicitation_design.md · references/
• segmentation_failure_characterization_design.md · references/
• target_trial_emulation.md · references/
• venue_accept_recipe.md · references/
• butterfly.json · scripts/adjustment_set_challenge/fixture/
• confounder.json · scripts/adjustment_set_challenge/fixture/
• instrument.json · scripts/adjustment_set_challenge/fixture/
• mbias.json · scripts/adjustment_set_challenge/fixture/
• mediator.json · scripts/adjustment_set_challenge/fixture/
• proxy.json · scripts/adjustment_set_challenge/fixture/
• problem.md · scripts/adjustment_set_challenge/
• verify.sh · scripts/adjustment_set_challenge/
• adjustment_set_helper.py · scripts/
• skill.yml

SKILL.md (first lines)

# Design-Study Skill

## Core Review Questions

Always inspect:

1. The exact research question.
2. The analysis unit: patient, lesion, exam, study, phase, report.
3. The index date or decision point.
4. How inclusion and exclusion criteria are applied.
5. Any information leakage.
6. The reference standard or endpoint definition.
7. A clinically meaningful comparator.
8. The validation strategy.
9. The uncertainty reporting required.
10. The best-fitting reporting guideline.
11. Whether exposure/outcome/covariate **definitions are literature-grounded** or invented ad hoc from the data dictionary. If ad hoc, defer to `/define-variables` before drafting Methods.

---

## Standard Output

```text
## Study Design Review
Question: ...
Study type: ...
Analysis unit: ...
Index date / prediction timepoint: ...

### Strengths
- ...

### Major validity risks
1. ...
2. ...

### Minimal fixes
- ...

### Reporting fit
- Recommended guideline: ...

### Decision
- Ready for analysis / Needs redesign / Drafting can proceed with limitations
```

Cite a reference only with a `/search-lit`-confirmed DOI or PMID; mark any other `[UNVERIFIED - NEEDS MANUAL CHECK]`. Never invent a clinical definition, diagnostic criterion, or guideline recommendation: flag an unconfirmed one `[VERIFY]` and ask the user.

---

## Workflow

### Phase 1: Reconstruct the study

Extract from the protocol, draft, slides, tables, or notes: clinical problem, intended use case,
population, inputs, outputs, outcome definition, and timing of variable availability.

**Gate:** Present the reconstructed study summary (question, analysis unit, intended use) to the
user and confirm it before proceeding — a wrong reconstruction misdirects the entire validity review.

### Phase 2: Check structural validity

Read a reference only when its condition holds:

| File | Read it when |
|---|---|
| `references/dag_adjustment.md` | confounding control needs an explicit adjustment set |
| `references/target_trial_emulation.md` | the design emulates a target trial |
| `references/venue_accept_recipe.md` | a clinical DL / AI-validation study must decide **which venue tier the achievable design can be accepted at, and the one design move that reaches the tier above** (the bridge into `/find-journal`); skip when there is no publication-tier decision |
| `references/combine_models_ablation_design.md` | the model is built by **combining / adapting / fine-tuning existing models** (nnU-Net, TotalSegmentator, SAM/MedSAM, a pretrained backbone) and the comparator must be designed as an ablation; skip for a model trained de novo |
| `references/multi_model_comparison_design.md` | the contribution is **comparing several models / architectures head-to-head** and the comparison must be fair; skip for a single-model study (one model's ablation → the row above; AI-vs-human → `/design-ai-benchmarking`) |
| `references/segmentation_failure_characterization_design.md` | the claim is that a segmentation model is **clinically usable**, not that it scores well; skip when the endpoint is benchmark accuracy (metric choice → `/model-assessment`; abstention / risk–coverage → `/model-assessment`) |

#### A. Analysis unit

Look for mismatches: a patient-level claim from lesion-level analysis; an exam-level split with
patient overlap; phase-level samples treated as independent.

#### B. Leakage

Look for:
- postoperative features used for preoperative prediction
- normalization or thresholding performed before the data split
- repeated exams across train/test
- reader annotations derived from outcome information
- **input-text contamination for NLP/LLM extraction tasks**: if the input includes clinical history,
  indication, impression, prior diagnosis, or referral text, confirm it does not name or strongly imply
  the target label. If it does, the task is information retrieval under label leakage, not phenotype
  inference: redesign the input mask, report a sensitivity analysis excluding those fields, or reframe.
- **construct dependence** (a predictor that is a definitional component of the outcome): (i) an input
  that computes the outcome (fasting insulin and glucose when the outcome is HOMA-IR), or (ii) a
  near-tautological composite of the outcome's defining components (inflated, near-circular
  association). Test: "could this predictor be derived, in whole or part, from the outcome's definition
  or the same measurement?" If yes, exclude it, or keep it only as a labeled calibration probe.

#### F. Time origin & survivorship (incident / transition models)

For any time-to-event or incident/transition design, check before drafting:
- **Time origin per model.** Each incident model starts its at-risk clock at the correct origin.
  Watch for **immortal-time bias** (a span in which the event cannot occur, misattributed to one
  group) and **left-truncation / delayed entry**.
- **Mediator-ascertainment-window survivorship.** A "progressor" / transition label conditional on
  *surviving to* a later ascertainment (a second scan, a follow-up visit) is survivorship-biased; plan
  a landmark time or an explicit intermediate-state (multistate / illness-death) model.
- **Primary-analysis-set selection.** If the primary is not the full cohort, pre-specify why, from
  the missingness mechanism *relative to the outcome*: complete-case regression is unbiased when
  missingness is independent of the outcome given the covariates (even with a covariate MNAR) and
  generally biased when it depends on the outcome, including under MAR, where multiple imputation
  is the usual primary (Hughes et al., *Int J Epidemiol* 2019; logistic-regression odds ratios are
  the exception when missingness depends on the outcome alone). Never pick it for significance.
- A design that cannot yet answer these should say so — but at review time a Methods/Limitations
  admission that it was *"not formally assessed"* is escalated to MAJOR by the survival probe (S1).

#### C. Reference standard

Check who established ground truth, when, whether blinding was possible, and whether only a subset
had gold-standard verification. Then:
- **Construct ↔ nominal-definition match.** Restate each construct's nominal definition and confirm
  every included case satisfies it. An "incidentaloma" defined as an *indeterminate* finding must not

From Aperivue/medsci-skills, MIT.

More from Aperivue/medsci-skills