verify-refs

Files (33)

• SKILL.md
• claim_evidence_workflow.md · references/
• manual_checkpoint_guide.md · references/
• _claim_evidence.py · scripts/
• _quote_match.py · scripts/
• check_claim_fidelity.py · scripts/
• 10.1000_synthetic.oversight.md · scripts/claim_fidelity_challenge/fixture/fulltext/
• 10.1000_synthetic.stub.md · scripts/claim_fidelity_challenge/fixture/fulltext/
• manuscript_numbered.md · scripts/claim_fidelity_challenge/fixture/
• manuscript_shortsource.md · scripts/claim_fidelity_challenge/fixture/
• manuscript_supported.md · scripts/claim_fidelity_challenge/fixture/
• manuscript_unsupported.md · scripts/claim_fidelity_challenge/fixture/
• refs.bib · scripts/claim_fidelity_challenge/fixture/
• verify.sh · scripts/claim_fidelity_challenge/
• verify_cli.sh · scripts/
• verify_refs.py · scripts/
• skill.yml
• corporate_author.bib · tests/fixtures/
• pagination_placeholder.bib · tests/fixtures/
• test_audit_source_path.sh · tests/
• test_author_normalization.sh · tests/
• test_bib_entry_headers.sh · tests/
• test_bib_last_field.sh · tests/
• test_bibtex_et_al.sh · tests/
• test_claim_evidence.py · tests/
• test_corporate_author.sh · tests/
• test_doi_not_registered.sh · tests/
• test_fabricated_author.sh · tests/
• test_identifier_title.sh · tests/
• test_markdown_reference_heading.sh · tests/
• test_openalex_tier.sh · tests/
• test_pagination_placeholder.sh · tests/
• test_title_fallback_guard.sh · tests/

SKILL.md (first lines)

# Verify References (Audit-Only)

Audit an existing manuscript or bibliography for fabricated or mismatched references. This
skill **never writes** to `references/` or `manuscript/_src/refs.bib`: corrections flow through
`/lit-sync` (Zotero → Better BibTeX → `refs.bib`), and `/manage-project`'s
`scripts/validate_project_contract.py` flags any `references/*` file written here as drift. It
does not discover literature (`/search-lit`), and it never replaces a missing or bad citation
with a plausible alternative — replacements go through `/search-lit` or the user.

## Inputs

1. Manuscript or bibliography: `.md`, `.docx`, `.bib`, `.txt`, or `.tsv`.
2. Optional project root. Default: current working directory.
3. Optional flags: `--offline` (extract and classify without API calls), `--timeout N` (HTTP
   timeout seconds), `--strict`, `--no-openalex`.

For markdown manuscripts with pandoc `[@bibkey]` citations, run a citation-key check first
(`/manage-refs`'s `check_citation_keys.py`, or your reference manager's). It catches mis-keyed
cites; `verify_refs.py` catches fabricated metadata.

## Deterministic Script

Run the bundled script rather than verifying citations by memory:

```bash
python "${CLAUDE_SKILL_DIR}/scripts/verify_refs.py" manuscript/manuscript.md --project-root .
```

For hooks or quick manual runs, use the wrapper:

```bash
"${CLAUDE_SKILL_DIR}/scripts/verify_cli.sh" manuscript/manuscript.md --offline
```

Manual pre-submission strict run:

```bash
"${CLAUDE_SKILL_DIR}/scripts/verify_cli.sh" manuscript/index.qmd --strict
```

`--strict` forbids `--offline` and exits non-zero on any UNVERIFIED row. Read
`references/manual_checkpoint_guide.md` for when to run it and what to do per status.

Lookups run PubMed (PMID) → CrossRef (DOI; doi.org's handle registry when CrossRef answers
404) → OpenAlex, with a PubMed title search last. Every
`OK` row rests on DOI, PMID, CrossRef, or PubMed title evidence; a failed lookup is recorded as
`UNVERIFIED`, never silently passed. A title search, in either index, counts only when a returned
title matches the cited one (the same token-similarity guard), so a made-up title cannot earn `OK`
from a search that merely returned something.

**OpenAlex tier.** It recovers conference and non-DOI citations (NeurIPS / ICLR / ACL, common
in medical-AI manuscripts) that PubMed and CrossRef miss, and is called only when no author list
was obtained yet. It resolves by DOI, otherwise by a title search behind a token-similarity
guard so a fabricated title cannot earn `OK`. OpenAlex names have no family/given split, so they
support only an existence check and a tolerant first-author membership check — never the strict
positional or author-count MISMATCH, which stays with PubMed efetch / CrossRef. An OpenAlex miss
is `UNVERIFIED`, never `FABRICATED`. `--no-openalex` restricts verification to PubMed + CrossRef.

## Output Contract (v1.3.0)

`qc/reference_audit.json` (`schema_version` 4) is the only output; this skill is its sole writer
and writes no TSV and no `library.bib`.

- `records[]`: per reference, `status`, `note`, `evidence`, `cited_authors[]` /
  `actual_authors[]`, `cited_author_count` / `actual_author_count`. `status` is one of:
  - `OK`: a lookup confirmed the work (DOI, PMID, or a title search passing the similarity
    guard) and the authors agree.
  - `MISMATCH`: the work exists but the citation is wrong: its authors differ (Gate 4), or its
    DOI/PMID does not exist while a lookup still found the work (`note = "wrong identifier…"`).
  - `FABRICATED`: the cited identifier does not exist and nothing else found the work (details
    below).
  - `UNVERIFIED`: nothing confirmed or refuted it (a failed lookup, a DOI held only by another
    registration agency that no index confirmed, no identifier and no title match, a PubMed
    title-only match whose authors could not be compared or differ (`note = "title_only: …"`), or
    `--offline`).
- `counts` and `duplicate_findings[]` (Gate 5).
- `submission_safe`: no `FABRICATED`, no `MISMATCH`, and `duplicate_findings` empty.
  `fully_verified` additionally requires no `UNVERIFIED`.
- `submission_safe` tolerates `UNVERIFIED` rows because offline runs produce them; before a
  submission, resolve each one (confirm it by hand and say how, or remove the citation) — do not
  ship an `UNVERIFIED` reference as if it were checked.
- `source_sha256` and `audited_ref_ids`: what was audited. A later reader compares them with the
  current bib; a changed bib makes the audit stale.

## Workflow

1. Identify the input file and project root.
2. Run `scripts/verify_refs.py`.
3. Read `qc/reference_audit.json`. Every verdict comes from it; never mark a row `OK` yourself.
4. Report all `FABRICATED` and `MISMATCH` rows first (from `records[]`).
5. Report all `duplicate_findings[]` entries (verbatim PMID/DOI duplicates — cite renumbering
   required).
6. If `UNVERIFIED` rows remain, list them as manual checks and do not call the manuscript fully
   submission-safe. Check `note` on every row, whatever its status: `pagination_placeholder`
   (`e000–e000` / `in press` / `TBD` / `forthcoming`) needs the citation resolved before
   submission; `/self-review` Phase 2.5c decides whether any is a P0 blocker.
7. If the user needs a human-readable table, summarize from `records[]` in chat — do not write a
   TSV.

## Quality Gates

- Gate 1: stop submission if any row is `FABRICATED`.
- Gate 2: require user confirmation before accepting `UNVERIFIED` references.
- Gate 3: rerun after any reference edits.
- Gate 4 (full-author cross-check): the authoritative author list comes from PubMed
  `efetch.fcgi` (XML) when a PMID is present — preferred because CrossRef is unreliable for
  given names — with CrossRef and PubMed esummary as fallbacks. For BibTeX inputs every cited family name is
  compared index-by-index, and the cited and source author counts are compared, after
  normalizing case, diacritics, hyphen vs space, and name particles ("von", "van", "de"); one
  name may be a whole word of the other ("Garcia" / "Garcia-Lopez"), never a mere substring
  ("Li" / "Williams"). A row
  whose DOI/PMID resolves but whose authors differ at any index or in count becomes `MISMATCH`:
  `note = "first-author hallucination suspected"` for the first author,
  `note = "non-first-author hallucination or count mismatch"` for #2..#N or the count. A correct
  first author does not establish the rest of the list. Plain-text / TSV inputs, and lists that
  cannot be parsed confidently, degrade to the first-author check (skipped if even that is
  empty). A PubMed title-only match is `OK` only when the matched record's esummary authors were
  compared with the cited ones and agree; otherwise it stays `UNVERIFIED`. A declared
  truncation — BibTeX `and others`, or a `_audit_truncated = <N>` field — turns a

From Aperivue/medsci-skills, MIT.

More from Aperivue/medsci-skills