traceability-review

Files (2)

• SKILL.md
• pdf_extract.py

SKILL.md (first lines)

# Traceability Review

Audit a workspace document (report, manuscript, or notebook) with three checks.
You verify **traceability** — that claims trace to sources, data, and code —
not truth. Never state or imply that the document is error-free.

## PDF manuscripts — extract first, never guess

If the document is a **PDF**, do not read the raw bytes or infer its contents.
Run the bundled extractor first — it pulls the text plus concrete citation
identifiers and quantitative claims deterministically, so you audit real
identifiers, not ones recalled from memory:

```bash
python skills/traceability-review/pdf_extract.py MANUSCRIPT.pdf
```

It prints JSON: `{backend, pages, chars, citations:{dois,arxiv,pmids},
claims:[{kind,text,context}], text}`. Use `citations` as the input to Check 1,
`claims` as the input to Check 2, and `text` to locate figure references for
Check 3. If it returns `{"error": …}` (no PDF backend installed), say so plainly
and fall back to whatever text you can read — do not fabricate identifiers.

## Check 1 · Citation audit

1. Extract every citation identifier from the document: DOI (`10.xxxx/…`),
   arXiv id, PMID, or title + year when no identifier is given.
2. Resolve each against a public registry (no API key needed):
   - DOI: `curl -s "https://api.crossref.org/works/<doi>"`
   - arXiv: `curl -s "http://export.arxiv.org/api/query?id_list=<id>"`
   - PMID: `curl -s "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed&id=<pmid>&retmode=json"`
3. Findings:
   - `error` — the identifier does not resolve (HTTP 404 / empty result).
   - `warn` — it resolves, but the registry's title/authors/year clearly
     disagree with how the document cites it.
   - `warn` — network unavailable: report "could not verify (offline)" rather
     than skipping silently.

## Check 2 · Untraceable numbers

1. List the document's quantitative claims: statistics, percentages, sample
   sizes, effect sizes, p-values, model scores.
2. For each, look for its source inside the workspace: a data file, a code or
   notebook output, or an execution log that produces that value.
3. Finding: `warn` for any number with no traceable source. Quote the exact
   sentence in the evidence.

## Check 3 · Figure ↔ code consistency

1. Read `.pi-science/provenance.jsonl` in the workspace — one JSON record per
   line: `{path, version, ts, tool, content, …}`; `ts` is epoch seconds. It
   records every file version the agent wrote. The directory is hidden: read
   the file directly (`cat .pi-science/provenance.jsonl`) instead of relying
   on `ls`. Fall back to file mtimes only when the file is truly absent.
2. For each figure the document references:
   - Latest record `ts` for the figure file (fall back to file mtime when the
     figure has no record).
   - Latest record `ts` of the script/notebook that generates it — match by
     scanning record `content` and workspace code for the figure's filename.
3. Findings:
   - `warn` — the generating code has a newer version than the figure:
     "figure may be stale — regenerate it from the current code".
   - `warn` — a referenced figure has no provenance record and no matching
     workspace file.

## Output contract

End the reply with exactly one fenced block (the app renders it as reviewer
cards; keep it as the LAST thing in the message):

```review
{"findings":[{"level":"error","check":"citation","title":"DOI does not resolve","evidence":"10.9999/fake.2026 → Crossref 404"}],"note":"Traceability review — verified what could be traced. Absence of findings is not a guarantee of correctness."}
```

- `level`: `error` | `warn` | `ok` · `check`: `citation` | `number` | `figure`.
- One finding per issue; `ok` findings are allowed for confirmed traceable
  items worth stating explicitly.
- Evidence: the exact identifier / quoted sentence / file paths, plus what you
  observed.
- The note must never claim the document has no errors.

From Garhorne0813/pi-science, Apache-2.0.

More from Garhorne0813/pi-science