evidence-chain-builder

Files (4)

• SKILL.md
• evals.json · evals/
• evidence-types.md · references/
• evidence_chain.py · scripts/

SKILL.md (first lines)

# Evidence Chain Builder

## Overview

`scripts/evidence_chain.py` splits a claim from its evidence and scores how well the
evidence carries it. Every evidence item is tagged with a source type, and the type
decides whether that item counts:

| type | counts as independently verifiable |
|---|---|
| `official` — a primary document, filing, standard, or the issuing body itself | yes |
| `paper` — peer-reviewed literature, or a dataset with a stated method | yes |
| `data` — raw measurements, logs, exports the user can re-run | yes |
| `internal` — the user's own notes, records, or an undocumented internal number | no |
| `assertion` — the claim restated, someone's opinion, or no source stated | no |

The score is the share of items that count, as a percentage. `internal` and
`assertion` are not "worth less" by taste — they are items nobody outside the
conversation can check, and that is the property the score is measuring.

Full taxonomy, edge cases and tagging rules: `references/evidence-types.md`.

## The one boundary that matters

**This tool does not decide whether a claim is true. It grades the evidence, and
nothing else.**

That boundary is the whole point. A grader that also pronounced on truth would
reproduce exactly the failure it exists to catch — a confident verdict unsupported
by anything. So the tool reports what can be checked and what cannot, and stops.

Two consequences to carry into your answer:

- A score of 100 does not mean the claim is correct. It means every item offered is
  the kind of thing that could be checked. Nobody has checked it yet.
- A low score does not mean the claim is false. It means the claim is **not
  established**, which is a different and more useful statement.

Never upgrade "not established" into "false", and never upgrade a high score into
"true". Both are the same mistake in opposite directions.

## Workflow

1. **Separate the claim from the evidence before running anything.** Write the claim
   as one sentence. If the user gave you three conclusions, that is three runs — a
   single chain will average unrelated things and hide the weak one.
2. **Tag every item, and tag honestly.** An item is `official` only if it names a
   document or issuing body that exists independently of the conversation. "Our
   records show" is `internal`. "Studies suggest" with no study is `assertion`. If
   you cannot tell what an item is, it is `assertion` until the user says otherwise —
   guessing a stronger type defeats the tool.
3. **Run the tool** (command below).
4. **Read the verdict band, not just the number.** The bands are defined below.
5. **Name the gap concretely.** Do not report "the evidence is weak". Report which
   items are unverifiable and what specifically would replace each one — "the
   failure-rate figure needs the QA report it came from, not a description of it".
6. **Report the score and the gap together.** A score without the unverifiable list
   invites the reader to treat the number as a quality grade, which it is not.

## Command

```bash
python3 scripts/evidence_chain.py \
  --claim "Device X failure rate is below 0.5%" \
  --evidences "2025 QA audit report@official:QA dept" \
              "no complaints on record@internal:support" \
              "peer-reviewed cohort study@paper:JAMA"
```

Evidence strings take an optional `@type:source` suffix; one argument per item, or
several joined with `||`. No suffix means `assertion`.

| flag | meaning |
|---|---|
| `--claim TEXT` | the claim to test |
| `--evidences E1 E2 ...` | evidence items, optionally `@type:source` |
| `--from FILE.json` | `{claim, evidences:[{text, source, type}]}` for longer chains |
| `--threshold N` | score needed to count as supported (default `50`) |
| `--subject TEXT` | what the claim is about, printed in the header |
| `--json` | machine-readable output, for gating or downstream use |
| `--fail-below-threshold` | exit `1` when not supported; default is exit `0` either way |

Exit codes: `0` the chain was scored, whatever the verdict; `2` usage or input
error. Non-zero-to-signal-unsupported is opt-in, because a caller that treats any
non-zero exit as "the command broke" would report a finding as an error.

Run `python3 scripts/evidence_chain.py --help` for the authoritative flag list.

## Reading the result

| condition | what you may say |
|---|---|
| no evidence supplied | nothing is established. There is no chain to stand on. |
| score below threshold | **Do not treat this claim as a conclusion.** Say which items are unverifiable. |
| score at or above threshold | Supported *to the stated threshold*. The sources still need a human check. |

The verdict text is deliberately phrased to survive being quoted out of context.
Keep that phrasing when you relay it; do not round it up to "verified".

Threshold defaults to 50 because a chain where the majority of the weight is
checkable is at least auditable. Raise it with `--threshold` when the claim will be
published, filed, or acted on — 80 for anything going to a regulator or a customer.
Say which threshold you used, since the verdict depends on it.

## What this cannot do

| asked for | status |
|---|---|
| Confirm a cited source exists | Not checked. The tool never fetches anything. |
| Detect a fabricated citation | Not possible here — a plausible-looking but invented reference scores as `paper` |
| Judge whether the claim is true | Out of scope by design; see the boundary above |
| Weigh study quality — sample size, bias, design | Not modelled. `paper` means "a paper was named", not "a good paper" |
| Rank sources against each other | Types are unordered beyond verifiable / not verifiable |
| Resolve conflicting evidence | Two sources that disagree both score; the conflict is yours to surface |
| Handle a claim with several parts | Run it per part; averaging hides the weak link |

State these limits plainly when they bite rather than padding the answer. If the user
needs source existence checked, that is a search task — do it as one, and treat the
result as a new evidence item, not as the same chain.

From Willian8021743/windows-ai-toolkit, MIT.

More from Willian8021743/windows-ai-toolkit