threadlight-evals

Files (20)

• SKILL.md
• ab-comparison.md · references/
• dataset-shape.md · references/
• evals-manifest.schema.json · references/
• dataset.jsonl · references/fixtures/sample-manual/evals/
• spec.md · references/fixtures/sample-manual/specs/
• evals-report.md · references/fixtures/sample-scheduled/docs/
• champion-challenger.yaml · references/fixtures/sample-scheduled/evals/ab/
• continuous-eval.yaml · references/fixtures/sample-scheduled/evals/
• dataset.jsonl · references/fixtures/sample-scheduled/evals/
• 2026-01-01.json · references/fixtures/sample-scheduled/evals/runs/
• eval-alert.bicep · references/fixtures/sample-scheduled/infra/
• evals-manifest.json · references/fixtures/sample-scheduled/specs/
• spec.md · references/fixtures/sample-scheduled/specs/
• continuous_eval.py · references/fixtures/sample-scheduled/src/
• foundry-ce-wiring.md · references/
• judge-calibration.md · references/
• evals_check.py · scripts/
• test_evals_agentops_consumer.py · tests/
• test_evals_check.py · tests/

SKILL.md (first lines)

# Threadlight Evals — run continuous evals, then prove they ran

<!-- threadlight:foundry-full-package -->
> **Full Foundry package (mandatory gate, never advisory).** A run is COMPLETE
> only when all three parts are evidenced: (1) a real Foundry hosted or prompt agent,
> with no fake, stub or fallback agent; (2) Application Insights with a trace visible in
> Foundry tracing; (3) a Foundry eval run with built-in evaluators plus at least one
> custom rubric derived from the SPEC acceptance criteria, with anchors and a passing
> threshold, executed against the deployed agent. Continuous evaluation is optional (GA rules reject hosted agents). Record the evidence in
> `specs/foundry-package-manifest.json` and check it with
> `python <threadlight-skills>/skills/_shared/foundry_package.py --workspace <pilot-root>`. Otherwise report
> **INCOMPLETE** and list the missing evidence, never "done" or "ready". Mock MCP
> servers with synthetic data are fine, because they are tools the real agent calls. See
> [`_shared/foundry-only-agents.md`](../_shared/foundry-only-agents.md#full-foundry-package-mandatory-delivery-gate).

The **DISCOVER + GOVERN + IMPROVE** evals leg of `path2production` for AI agents
on Microsoft Foundry. `threadlight-production-ready` pillar 6 scores whether
continuous evals are scheduled, fresh, and alerting; this skill is the executable
leg that owns running and verifying that posture in the pipeline.

```
DESIGN → BUILD/DEPLOY → DISCOVER → PROTECT → [ GOVERN / IMPROVE ] → PRODUCTION-READY
                                            threadlight-evals
```

> **Why this skill exists.** A pilot can have strong unit tests, local smoke
> tests, and a green deployment gate, yet still regress silently after a model,
> prompt, retrieval, or tool-schema change. Previously the chain delegated eval
> work entirely to `microsoft-foundry`, and pillar 6 could only score whether a
> schedule appeared to exist. This skill makes evals a threadlight-owned leg:
> it detects offline datasets, live continuous evaluation wiring, threshold
> alerts, run freshness, and champion-challenger gates; then emits a manifest the
> scorecard can verify.

## What this skill covers

| Feature | Purpose | Output signal |
|---|---|---|
| **F1 Offline batch quality eval** | Thin-wrap the `microsoft-foundry` invoke+score path in the threadlight spine. Heavy evaluator authoring still belongs to `microsoft-foundry`; this leg owns running it and producing evidence. | `eval_scenarios_present`, `eval_datasets_present`, `run_history_present`, `latest_pass_rate_ok` |
| **F2 Online / continuous eval** | Wire Foundry Continuous Evaluation on live threads using `create_agent_evaluation(thread, run, evaluators=[...], app_insights_connection_string=...)`. Results land in Application Insights. | `schedule_present`, `online_eval_wired`, `latest_eval_run_fresh`, `alert_wired` |
| **F3 A/B champion–challenger** | Gate a model/prompt/tool swap by running the same eval dataset against champion and challenger before the swap. | `ab_comparison_present` |

## What this skill does NOT replace

| Concern | Use instead |
|---|---|
| Token-level content filtering, prompt shields, content safety at the model edge | Model guardrail / Azure AI Content Safety |
| Adversarial scanning and jailbreak probe campaigns | `threadlight-redteam` |
| Agent-runtime action policy, tool allow/deny, excessive-agency controls | `threadlight-govern` |
| Deep evaluator authoring, grading rubric design, dataset enrichment work | `microsoft-foundry` |
| Overall production-readiness scorecard | `threadlight-production-ready` |

## The contract — `specs/evals-manifest.json`

`scripts/evals_check.py` walks a pilot repo and emits a manifest whose
capability keys map directly to pillar 6 (`continuous-evals`) IDs:

| Capability key | Pillar ID | Meaning | Severity when missing |
|---|---:|---|---|
| `eval_scenarios_present` | `EVAL-001` | SPEC § 9 lists eval scenarios or `evals/` contains scenario files | must-fix |
| `eval_datasets_present` | `EVAL-002` | Held-out datasets are stored under `evals/` or `specs/evals/` | must-fix |
| `dataset_shape_ok` | `EVAL-003` | At least one JSON/JSONL row includes `tool_calls` and `tool_outputs` | should-fix |
| `thresholds_declared` | `EVAL-004` | Per-scenario `threshold`, `min_score`, or pass-rate threshold is declared | should-fix |
| `schedule_present` | `EVAL-005` | Plan A Foundry CE schedule or Plan B cron/ACA Job eval runner exists | must-fix |
| `run_history_present` | `EVAL-006` | Latest run output is committed under `evals/runs/*.json` or `docs/eval-runs/` | should-fix |
| `online_eval_wired` | `EVAL-101` | Code calls `create_agent_evaluation(...)` with Application Insights connection | should-fix; must-fix when Plan A is declared but unwired |
| `latest_eval_run_fresh` | `EVAL-102/103` | Latest scheduled run is within the freshness window (default 7 days) | should-fix if stale; not-verified if no history |
| `alert_wired` | `EVAL-104` | Alert or notification exists for eval threshold breach | must-fix |
| `latest_pass_rate_ok` | `EVAL-105` | Latest run pass rate meets the declared minimum | should-fix if below threshold |
| `ab_comparison_present` | `F3` | Champion-challenger comparison config/script exists before swaps | should-fix |

Status taxonomy is exactly: `pass`, `must-fix`, `should-fix`, `not-verified`,
`not-applicable`.

Verdict roll-up:

| Verdict | Meaning |
|---|---|
| `comprehensive` | No `must-fix`, `should-fix`, or `not-verified` findings. Offline, online, alerting, freshness, pass-rate, and A/B gate are all verified. |
| `partial` | No `must-fix`, but one or more advisory or unverified capabilities remain. |
| `offline-only` | Offline basics are present, but at least one hard requirement for continuous/online operation is missing. |
| `none` | Evals are absent or too incomplete to trust. |

The emitted manifest includes:
```json
{
  "schema": "threadlight-evals-manifest/v1",
  "tool_version": "0.1.0",
  "captured_at": "2026-06-22T16:00:00+00:00",
  "freshness_window_days": 7,
  "verdict": "comprehensive",
  "must_fix": [],
  "should_fix": [],
  "not_verified": [],
  "metrics": {
    "pass_rate": 0.91,
    "threshold": 0.8,
    "latest_run": "evals/runs/2026-01-01.json"
  },
  "capabilities": {
    "eval_scenarios_present": {
      "check_id": "EVAL-001",
      "status": "pass",
      "evidence": "specs/spec.md",
      "hint": null
    }
  }
}
```

The top-level `metrics` block surfaces the latest run's `pass_rate`, the
declared `threshold`, and the relative path to the `latest_run`. This is the
join key `threadlight-production-ready` reads (`metrics.pass_rate`) to render
the eval-quality column of its outcome-KPI scorecard — so eval quality flows
into the business-KPI view instead of staying locked inside the evals leg.
`pass_rate`/`latest_run` are `null` when no machine-readable run history exists.

## Usage

From aiappsgbb/threadlight-skills, MIT.

More from aiappsgbb/threadlight-skills