integrity-auditor

Files (22)

• LICENSE
• SKILL.md
• README.md · forensics_tools/
• bilingual_cn_geography.json · forensics_tools/
• channel_check.py · forensics_tools/
• decimal_match.py · forensics_tools/
• image_dup.py · forensics_tools/
• image_dup_orb.py · forensics_tools/
• magnitude_consistency.py · forensics_tools/
• panel_split.py · forensics_tools/
• requirements.txt · forensics_tools/
• xlsx_aggregate_consistency.py · forensics_tools/
• 00-incremental-execution.md · references/
• 01-image-evidence.md · references/
• 02-numerical-evidence.md · references/
• 02a-supplement-acquisition.md · references/
• 02b-ml-paper-arithmetic.md · references/
• 03-logical-evidence.md · references/
• 04-evidence-grading.md · references/
• 05-quality-gate.md · references/
• audit_report.md · templates/
• smoketest.sh · tests/

SKILL.md (first lines)

# Integrity Auditor

> **LFL 本地化说明（2026-10-10）**：源自 ai4s-research/ai4s-skills@744ab2049a7b52ae726c43313468fd518b330980（MIT，LICENSE 随附）。依赖事实：审计方法论（references/ 00–05）为核心、无外部依赖；forensics_tools/ 中 numpy / Pillow / openpyxl 本机已装，opencv-python 未装——涉及 ORB 图像查重的工具缺它时跳过并在报告中如实声明，不影响审计主体流程。

## Overview

Paper-integrity audit package. **Single stage, full quality from the start.** The agent reads each reference, then carries out three evidence tracks (image / numerical / logical) and produces a structured `audit_report.md` with Level 1–4 graded findings.

This skill ships no LLM SDK — it is the skill instructions, references, templates, and single-purpose `forensics_tools/` only.

The substantive work is decomposed into reference playbooks under `references/`:

| Reference | Topic |
|---|---|
| `references/00-incremental-execution.md` | how to do this without losing work: batches, persistence, resume — **read first** |
| `references/01-image-evidence.md` | image evidence: panel split, dup detection, rotate/flip alignment, Western-blot continuity |
| `references/02-numerical-evidence.md` | numerical evidence: n-consistency, mean/SD/SEM recompute, P-value sanity, decimal trail, Benford with caveats, deterministic-column-pair and last-digit chi-square sweepers, variance-reporting consistency |
| `references/02a-supplement-acquisition.md` | publisher CDN routes: how to get hi-res figures and source-data XLSXs even when the article PDF is paywalled |
| `references/02b-ml-paper-arithmetic.md` | ML / non-biology papers: arithmetic re-derivation of every quoted improvement against tabulated benchmark cells; leaderboard archive routes |
| `references/03-logical-evidence.md` | logical evidence: conclusion-chain compression, missing controls, replication gap |
| `references/04-evidence-grading.md` | 4-level finding grading + reviewable-evidence format (DOI / figure-id / pointer / transformation / requested raw data) |
| `references/05-quality-gate.md` | self-check before delivery |

Also:

- `templates/audit_report.md` — report skeleton the agent fills.
- `forensics_tools/image_dup.py` — perceptual-hash (dHash + aHash) duplicate detector for figure / panel PNGs. Single-purpose pure-Python utility (Pillow only). Catches untransformed dups.
- `forensics_tools/image_dup_orb.py` — ORB feature-matching duplicate detector with horizontal-flip augmentation. Catches **transformed** dups (rotation / flip / crop / brightness change) that perceptual hashing misses. Pair with `image_dup.py`: use phash first, escalate to ORB when phash distance is suspicious-but-inconclusive (16–60 range). Deps: OpenCV + NumPy.
- `forensics_tools/panel_split.py` — whitespace-gutter panel splitter. Pair with `image_dup.py` / `image_dup_orb.py` for cross-panel duplicate detection; whole-figure phash without panel splitting almost never finds anything.
- `forensics_tools/channel_check.py` — RGB channel-content classifier (DAPI / Flag / Merge / other) for fluorescence sub-images. Catches within-panel label swaps (e.g., a "DAPI" sub-image that is actually a Merge); cross-panel phash cannot catch this class.
- `forensics_tools/decimal_match.py` — cross-cell last-N-decimal matching sweeper for source-data XLSX. Detects fabrication where many distinct values share trailing decimal patterns (Kang Tiebang whistleblower class). Single-purpose pure-Python utility (openpyxl only). See `references/02-numerical-evidence.md` Check 1.5.
- `forensics_tools/magnitude_consistency.py` — supplement-text vs source-data XLSX unit/scale consistency. Catches unit-confusion (TWh vs GWh, mM vs µM, MHz vs Hz, etc.) and order-of-magnitude transcription errors via entity-overlap + literal-value matching across a generic SI-prefix-aware unit taxonomy covering energy / power / mass / length / area / volume / time / voltage / current / frequency / pressure / concentration / amount / force / dose / genomics-bp / CO2 / currency. Cross-family pairs (e.g., kV vs TWh) are automatically rejected. Pair with `bilingual_cn_geography.json` (or your own JSON map) for cross-language entity matching. Empirical baseline: Hu et al. 2026 Nature Tongyu county 1000× unit error. See `references/02-numerical-evidence.md` Check 1.6.
- `forensics_tools/xlsx_aggregate_consistency.py` — cross-XLSX same-quantity sum/row consistency. Detects when two source-data tables in the same paper purport to carry the same aggregate quantity but disagree by a small systematic margin (e.g., Hu et al. 2026 Nature MOESM3 vs MOESM6 1110.78 vs 1103.89 TWh, 0.62 percent diff, all 31 provinces same sign). Reuses the unit taxonomy from `magnitude_consistency.py`. Empirical baseline same paper Level 1 finding. See `references/02-numerical-evidence.md` Check 1.7.
- `tests/smoketest.sh` — < 30-second pre-commit gate. Compiles every script, runs every `--help` (catches argparse `%` bugs), and runs positive + negative controls for `decimal_match`, `magnitude_consistency`, and `xlsx_aggregate_consistency`. Run before every change.
- See `forensics_tools/README.md` for the design rule that distinguishes utility scripts from forbidden "skeleton → enrich" orchestration, and for the recommended pipeline.

**Read the relevant reference _before_ writing, not after.** The full audit does not fit in a single turn — `references/00-incremental-execution.md` is the only execution mode that completes.

## When to Use

- User hands you a paper (PDF / DOI / arXiv ID) and asks whether the figures / data / logic are trustworthy.
- User wants a quality gate on outputs from a local paper-writer run (slug-based).
- Reviewer / investigator wants a reviewable-evidence document to forward to authors or an integrity body.

## When NOT to Use

- User wants to write a paper → `paper-writer`.
- User wants to build / run an experiment → `experiment-suite`.
- User wants a literature survey → `literature-survey`.
- User wants a verdict ("is this fraud?") — this skill produces evidence and grading, never verdicts.

## Workflow

### Step 1 — Identify the input and set up the run

Detect input mode:

| Mode | Trigger | Acquisition |
|---|---|---|
| **PDF path** | local `*.pdf` argument | use directly |
| **DOI / arXiv ID** | `10.xxxx/...`, `arXiv:NNNN.NNNNN` | `WebFetch` landing page; record DOI / arXiv URL and abstract; download PDF if open-access |
| **Local paper-writer slug** | matches `output/paper-writer/<slug>/latest/` | use slug directly; also read `output/experiment-suite/<slug>/latest/` if present |

Compute slug:

```bash
TITLE_OR_ID="<input identifier>"
SLUG=$(python3 -c "import re,hashlib,sys; t=sys.argv[1]; n=re.sub(r'[\\s_]+','-',re.sub(r'[^\\w\\s-]','',t.lower().strip())).strip('-')[:40].rstrip('-'); h=hashlib.sha1(t.encode()).hexdigest()[:8]; print(f'{n}-{h}')" "$TITLE_OR_ID")
# For local-slug mode, set SLUG to the existing paper-writer slug directly.
TS=$(date +%Y-%m-%d_%H%M%S)
RUN=output/integrity-auditor/$SLUG/$TS

mkdir -p "$RUN/findings/image" "$RUN/findings/numerical" "$RUN/findings/logical"
ln -sfn "$TS" "output/integrity-auditor/$SLUG/latest"
```

In commands below `$RUN` = `output/integrity-auditor/<slug>/latest`.

### Step 2 — Gather materials into the run

**Open `references/02a-supplement-acquisition.md` first if the input is a DOI / arXiv ID / external PDF reference.** Many publishers (Nature / Springer being the canonical example) gate the article body behind a paywall while serving the supplementary tables, source data, and hi-res figures on **open CDN endpoints**. Never conclude "paywalled, audit blocked" before trying the supplement / figure CDN routes.

Acquisition order (try every step; do not stop early):

1. **Article HTML landing page** — usually open even when PDF is gated. `curl -sL -A "Mozilla/5.0" "<article URL>" -o $RUN/paper.html`. This page typically embeds the abstract, all main-figure captions, and direct CDN URLs for every supplementary file and high-resolution figure.
2. **Harvest CDN URLs from the HTML**: `grep -oE 'https?://[^"]*(MOESM|Fig[0-9]_HTML|mmc|MEDIA)\.(?:pdf|xlsx|docx|png)' $RUN/paper.html | sort -u`. Download every distinct URL into `$RUN/supplementary/` and `$RUN/figures_hires/`.
3. **Author Correction PDF** — `curl -sL -A "Mozilla/5.0" "<correction URL>.pdf" -o $RUN/correction.pdf` then `file $RUN/correction.pdf` to confirm it is a real PDF (not HTML auth wall).
3a. **CRITICAL — the correction's own supplementary** — fetch the correction's landing HTML, grep for its `MOESM` supplementary URLs (separate DOI namespace from the parent article), download every one. These often contain the **pre-correction** originals of replaced figures. See `references/02a-supplement-acquisition.md` for the exact recipe. Empirical baseline: the Wang Ping 2025 Nature audit only surfaced its image-track Level 4 finding because the correction's `MOESM1_ESM.pdf` carried the original (pre-correction) Fig 2 and Extended Data Figs 7, 10.
4. **Article PDF** — `curl -sL -A "Mozilla/5.0" "<article URL>.pdf" -o $RUN/paper.pdf` then `file $RUN/paper.pdf`. If gated, record that fact in the manifest and continue with whatever the earlier steps acquired.
5. **For a local paper-writer slug only**: read the matching `results.json`, `data_contract.md`, `figures/manifest.json`, `bibliography.bib`.

Write `$RUN/input_manifest.md` listing every artefact actually acquired. At minimum it must enumerate what was downloaded and what was attempted-but-blocked.

Extract structured material:

```bash
# Text (every numeric claim, caption, figure reference will be greppable later)
pdftotext -layout "$PDF" "$RUN/paper.txt"

# Panels (one .png / .ppm per embedded raster image)
mkdir -p "$RUN/panels"
pdfimages -all "$PDF" "$RUN/panels/page"
N_RASTER=$(ls "$RUN/panels" 2>/dev/null | wc -l)

# Fallback for vector-figure papers (e.g., paper-writer outputs):
# pdfimages only extracts embedded raster images; a paper built from matplotlib
# vector PDFs through \includegraphics will yield zero panels here. In that case,
# render each page to a PNG so visual inspection is still possible.
if [ "$N_RASTER" -eq 0 ]; then
  pdftoppm -r 150 "$PDF" "$RUN/panels/page" -png
fi

# For a local paper-writer slug audit, also pull the production-side figure PDFs directly
# (these are the originals, before LaTeX embedding):
if [ "$INPUT_MODE" = "slug" ]; then
  mkdir -p "$RUN/figures_from_suite"
  cp output/experiment-suite/$SLUG/latest/figures/*.pdf "$RUN/figures_from_suite/" 2>/dev/null
  cp output/experiment-suite/$SLUG/latest/figures/manifest.json "$RUN/figures_from_suite/" 2>/dev/null
fi

From deyi2026/llm-first-loop, MIT.

More from deyi2026/llm-first-loop