threadlight-ground

Files (4)

• SKILL.md
• ground-manifest.schema.json · references/
• ground.py · scripts/
• test_ground.py · tests/

SKILL.md (first lines)

# Threadlight Ground — prove the retrieval is safe, not just plausible

The **GROUND** leg: the step that turns "the agent cited something" into "the
citation is provably grounded, access-controlled, and honest about what it
doesn't know." `threadlight-ground` is a **coordinator**, not a retrieval or
evaluation engine — it never calls Foundry IQ, never issues a retrieval
query, and never runs an evaluator. It only ingests probe *results* an
operator already captured (an ACL probe run, a citation validation run, a
refusal probe run) and turns them into four findings.

> **Why "coordinator."** Provisioning and query planning stay with
> `foundry-iq` (Azure AI Search Knowledge Agent); quality/relevance scoring
> stays with `threadlight-evals`. This leg's job is narrower and stricter:
> given evidence someone already gathered, does it *prove* the knowledge
> source is safe to ground answers on? This is a **manual, live handoff** —
> `threadlight-auto` does not run live ACL/citation/refusal probes against a
> real agent for you. Running those probes against production or pilot data
> is an operator decision; this script only assesses evidence the operator
> already captured and supplies.

## What this skill does and does not do

- **Does:** sanitize a SPEC `knowledge_sources` declaration (whitelist-only —
  see `references/ground-manifest.schema.json`), assess ACL enforcement,
  citation grounding, and refusal behavior from caller-supplied probe runs,
  compute source freshness/coverage, and emit a schema-validated,
  atomically-written `specs/ground-manifest.json`.
- **Does NOT** call Foundry IQ, run a retrieval query, or invoke an
  evaluator. Every finding here is computed from evidence the caller already
  captured — a live probe is the operator's responsibility, not this
  script's.
- **Does NOT** score answer quality/relevance (`threadlight-evals`) or
  aggregate findings across every leg into a single go/no-go
  (`threadlight-production-ready` consumes GRD-001..004 for that).

## The four findings

| ID | Checks | `not-verified` when | `must-fix` when |
|---|---|---|---|
| `GRD-001` | ACL enforcement | no `permission_model: acl` source declared → trivial `pass`; an ACL source has **no runs carrying its `source_id`** (uncovered); fewer than two distinct principals; no explicit `expected_entitled` on a run (ambiguous); no entitled *and* unentitled probe for a source; a source's declared `acl_probe_principals` are not all probed | an **explicit** unentitled principal received **any** document outside its explicit `allowed_document_ids` — a subset is enough, and no allowlist means nothing is allowed. **A proven leak takes precedence over a coexisting coverage gap:** the finding is `must-fix` even when another run is ambiguous or a declared principal is unprobed, and a sibling source's `not-verified` never erases it |
| `GRD-002` | Citation grounding | a `citation_required` source has no non-vacuous citation evidence carrying its `source_id` — it needs at least one citation and at least one retrieved ID | any citation falls outside its run's retrieved set (`missing_from_retrieval`, sorted, de-duplicated) |
| `GRD-003` | Refusal behavior | a `refuse_when_unsupported` source has no refusal runs carrying its `source_id` (uncovered) | an executed probe for an unsupported query was **answered** instead of refused |
| `GRD-004` | Source freshness / coverage / baseline | no sources declared → trivial `pass`; a declared source has no covering run; a covered source's runs carry no valid RFC3339 `captured_at` (freshness unverifiable); no `retrieval_quality_baseline` was supplied | never `must-fix` — caps at `should-fix` (a covered source's oldest evidence is stale relative to its own `refresh_cadence`) |

`GRD-001..003` can reach `pass`, `must-fix`, or `not-verified`. `GRD-004` can
reach `pass`, `should-fix`, or `not-verified` — freshness/coverage gaps are
never escalated past `should-fix`, reserving `must-fix` for a *proven*
control failure. The manifest `status` is `partial` exactly when required
evidence is genuinely missing: a `not-verified` finding, an uncovered /
freshness-unverifiable source hidden behind an aggregated must-fix, or a
`must-fix` proven against **incomplete** ACL coverage (see below). An
**executed** `must-fix`/`should-fix` with **complete coverage** is still
complete evidence and never downgrades `status` on its own.

> **Negative evidence is never erased by a coverage gap.** `GRD-001` computes
> a proven leak from the **explicit** unentitled runs first, so an ambiguous
> run (no `expected_entitled`), too few principals, or an unprobed declared
> principal cannot suppress a leak proven elsewhere in the same source — and
> across sources the worst per-source status wins, so one source's
> `not-verified` never downgrades another source's `must-fix`. The coexisting
> gap is not discarded: it is preserved in a schema-safe `coverage_complete:
> false` detail flag (alongside `missing_principals` where available) so the
> envelope `status` is still `partial`. A `must-fix` with **complete**
> coverage omits the flag and remains `complete`.

> **Malformed shapes raise, they are never `not-verified`.** A missing/empty
> `source_id`, a `source_id` not in the declared inventory, a missing
> `document_ids`/`citations`/`retrieved_ids`/`query_id`, a non-string element,
> a non-boolean `expected_entitled`/`refused`, or a negative/non-finite
> telemetry value all raise `GroundEvidenceError` **before** any manifest is
> built, returned, or written. `not-verified` is reserved for genuinely
> *absent* evidence (an uncovered source, an ambiguous entitlement), never a
> *malformed* run.

## Source inventory is authoritative — coverage is never inferred

The `sources` argument **is** the SPEC-derived `knowledge_sources[]` input
contract (see `threadlight-design/references/speckit-template.md`), not
anything the CLI's `--project-root` supplies. Every declared source's
*enabled* control must be covered by evidence that carries **that source's
`source_id`**: ACL evidence when `permission_model == "acl"`, citation
evidence when `citation_required`, refusal evidence when
`refuse_when_unsupported`. Coverage for one source is never inferred from
another source's runs. A source that declares `acl_probe_principals` must
have every one of those principals probed. Every declared source id must be
**unique**: a duplicate id (e.g. a public twin that would otherwise mask an
ACL-protected source under the same id) raises `GroundEvidenceError` before
any grouping, assessment, or output.

## Explicit `expected_entitled` is required — no name heuristic

`assess_acl` classifies each ACL run only by an **explicit**
`expected_entitled` (or `entitled`) boolean on the run. There is no name
heuristic: a run without an explicit signal is *ambiguous* and yields
`not-verified`, never a guessed pass or fail. A source's ACL runs must
include at least one entitled (`true`) and one unentitled (`false`) probe.
An unentitled principal's `allowed_document_ids` is its allowlist of
legitimately-visible (e.g. public) documents; receiving anything outside it —
including a subset — is a `must-fix` leak, and **no allowlist means nothing
is allowed**. Only **explicit** unentitled runs contribute a leak, so an
ambiguous run never fabricates one — but neither can it hide a leak proven by
another run in the same source.

## Evidence shapes

```jsonc
// acl_runs[] — one per principal probed against a source. source_id and an
// explicit expected_entitled are REQUIRED; allowed_document_ids is the
// unentitled principal's allowlist (public docs it may legitimately receive).
{"principal": "entitled-analyst", "document_ids": ["doc-1", "doc-2"], "source_id": "policy-library",
 "expected_entitled": true, "captured_at": "2026-08-17T09:00:00+00:00"}
{"principal": "unentitled-guest", "document_ids": ["public-1"], "source_id": "policy-library",
 "expected_entitled": false, "allowed_document_ids": ["public-1"], "captured_at": "2026-08-17T09:00:00+00:00"}

// citation_runs[] — one per answer's citations vs. what was actually retrieved
{"citations": ["doc-1"], "retrieved_ids": ["doc-1", "doc-2"], "source_id": "policy-library",
 "captured_at": "2026-08-17T09:00:00+00:00"}

// refusal_runs[] — one per unsupported-query probe

From aiappsgbb/threadlight-skills, MIT.

More from aiappsgbb/threadlight-skills