audit-scientific-claim

Files (3)

• SKILL.md
• claim-evidence-graph.schema.json · references/
• validate_claim_graph.py · scripts/

SKILL.md (first lines)

# Audit Scientific Claim

Build a claim-to-evidence graph using
`references/claim-evidence-graph.schema.json`.

## Assurance activities

Keep these activities separate; evidence for one does not satisfy another:

1. **Code verification** — was the intended algorithm implemented correctly?
2. **Solution verification** — was the mathematical problem solved accurately?
3. **Model validation** — is the model adequate for the stated context of use?
4. **Uncertainty quantification** — what uncertainty applies to the predicted
   quantity and how was it propagated or calibrated?

Every edge must name its activity, test, run, artifact SHA-256, acceptance
criterion, evidence state, reviewer decision, and decision source. Evidence
state is exactly `pass`, `fail`, `unknown`, `not-applicable`, or
`evidence-gap`; reviewer and workflow decisions use separate vocabularies.

## Workflow

1. Normalize claim wording, scope, context of use, and quantity of interest.
2. Add evidence edges without collapsing the four assurance activities.
3. Mark absent or inapplicable evidence explicitly.
4. Store every referenced artifact below an explicit evidence root using a
   relative path and its SHA-256. Reference each run through its contained
   run-record path and SHA-256; a caller-supplied run ID alone is not evidence.
   The run record must recompute to that `run_id` and its `artifacts` map must
   bind every claim edge's relative path to the same artifact SHA-256. Run
   `python3 scripts/validate_claim_graph.py --evidence-root <directory> <claim-graph.json>`.
   Validation rejects absolute paths, traversal, symlink escape, missing files,
   hash mismatches, modified run records, and artifacts unbound to their run.
5. Route findings to an accountable human reviewer or deterministic acceptance
   check. Agent analysis alone cannot produce an accepted/pass decision.

## Rules

- Do not upgrade correlation, low residual, or benchmark performance into model
  validation.
- Preserve contradictory evidence and rejected edges.
- Do not let the claim auditor make the final pass decision.

From dasobral/skills, MIT.

More from dasobral/skills