harness-effectiveness

Files (1)

• SKILL.md

SKILL.md (first lines)

# /dev-kit:harness-effectiveness — workflow-effectiveness scorecard

## What it does

Invokes the deterministic `lib.harness_effectiveness.build_report(root)` reducer and prints the resulting JSON. The reducer consumes structured TraceLog events (guard actions, first-write verification, repair events, control/treatment cohorts, measurement-integrity signals) already emitted by the harness; **no LLM judge runs**, no transcript replay, no case fixture. Missing evidence is reported as `INSUFFICIENT_EVIDENCE` per component — it is never converted to a passing or zero score.

This is the standalone counterpart to the table that `/dev-kit:evaluate` embeds at the bottom of `.dev-kit/eval-report.md`. The difference is intent: `/dev-kit:evaluate` runs the 12-case judge pass + the 4-component report together; `/dev-kit:harness-effectiveness` runs only the 4-component reducer (cheap, sub-second, no API spend).

## Inputs (resolved at runtime, NOT user args)

| Variable | Source |
|---|---|
| `ROOT` | `Path(".").resolve()` — the worker's current worktree |
| `EVENTS` | `lib.trace_log.read_events(root)` — every JSONL event the harness has emitted into the worktree's TraceLog |
| `COMPONENT_WEIGHTS` | `lib.harness_effectiveness.COMPONENT_WEIGHTS` — single source of truth |

No flags. The slash is 0-arg by design.

## Four components

| Component | Weight | What it scores |
|---|---:|---|
| `prevention_quality` | 0.25 | guard block-rate vs. ground-truth-labelled guard events |
| `first_pass_quality` | 0.25 | write → first-verification-pass rate |
| `recovery_quality` | 0.31 | median iterations to recover from a verification error |
| `measurement_integrity` | 0.19 | TraceLog event_id uniqueness, schema-version compliance, dedup; nested `stability` submetric (issue #663) reports agent / model / provider swap behaviour |

`learning_quality` (treatment-vs-control cohort divergence) was removed
because its evidence class is unreachable from current producers — the
cohort-tagging instrumentation isn't wired into the build loop, so the
component reported `INSUFFICIENT_EVIDENCE` permanently. The remaining
four shippable components carry the unit weight and are observable
from evidence already on the production path.

Each component returns:
```
{
  "score": <float|null>,            # null when coverage < 0.90 or no evidence
  "weight": <float>,                # from COMPONENT_WEIGHTS
  "status": "OK" | "DRIFT_WARNING" | "ROT" | "INSUFFICIENT_EVIDENCE",
  "coverage": <float 0..1>,
  "submetrics": { ... },
  "evidence_event_ids": [ ... ],
  "findings": [ "<one-line reason>" ],
  "config_version": "harness-effectiveness-v1"
}
```

`overall_score` is `null` when **no** component has a score (i.e.
when every component reports `INSUFFICIENT_EVIDENCE`). Otherwise it
is the weighted sum of the scored components' scores, with both the
numerator and the divisor restricted to the scored set so a
partially-scored corpus reports a meaningful overall instead of
collapsing to `None`.

## Incremental measurement envelope (PR #817)

The 4-component reducer above remains the canonical
`harness-effectiveness` contract. A complementary bounded envelope is
available from `lib.effectiveness_collection.collect(root)` and is
exposed by this skill under the `envelope` field of the printed JSON.
The envelope:

* Carries `schema_version: 1` and `contract_version:
  "effectiveness-collection-v1"`.
* Reports `counts.{enrolled,closed,unresolved,paired,missing_start,
  missing_terminal,conflicting_terminal,unexpected_units,success,
  distinct_units}`.
* Reports `ratios.coverage` (paired / closed) and `ratios.success`
  (completed / paired).
* Reports `readiness` ∈ `READY` / `INSUFFICIENT_EVIDENCE` / `DEGRADED` /
  `COLLECTION_ERROR` / `NO_OPPORTUNITY` / `UNSUPPORTED` /
  `COVERAGE_BELOW_POLICY`.
* Identifies its `origin` (`runtime` or `ci-probe`) and `retention_floor`
  (oldest cohort still in the journal).
* Lists `findings` and `bounded_errors` separately so the reducer can
  surface what is missing vs what is broken.

The envelope is read-only with respect to the existing
`harness_effectiveness` reducer. Callers that want a hard gate check
the envelope's `readiness == "READY"`; callers that want quality
scores use the 4-component report above.

## Stability submetric (issue #663)

Nested under `components.measurement_integrity.submetrics.stability`. It is a
fifth dimension that is *not* a top-level weight — the four-component
`overall_score` formula is unchanged. The submetric carries:

- `coverage` (float 0..1) — minimum of the five dimension sub-coverage ratios
- `score` (`None` when coverage < 0.90 or evidence is missing; otherwise a
  float in [0, 100]). `None` is distinct from `0.0` — the reducer never
  collapses missing evidence into a zero score.
- `status` — one of `OK` / `DRIFT_WARNING` / `ROT` / `INSUFFICIENT_EVIDENCE`
- `findings` — list of one-line reasons (e.g. "agent/provider/model
  identity not recorded on any event")
- `evidence_event_ids` — event_ids the submetric saw, sorted + deduped
- `submetrics` — per-dimension breakdown:
  - `agent_identity_coverage`: fraction of events carrying agent / provider / model
  - `replay_compatibility`: fraction of events with monotonic timestamps
  - `agent_provider_neutrality`: fraction of events free of agent/provider/model keys in evidence_ref
  - `gate_portability`: fraction of distinct event_types with at least one neutral event
  - `contract_test_pass_rate`: fraction of `contract.test` events with outcome=`passed`

A new `INSUFFICIENT_EVIDENCE` constant is exported from
`lib.harness_effectiveness`; consumer code should compare against the
constant instead of hardcoding the string. `build_report` also bumps
`schema_version` from 1 → 2 so consumers can detect the new submetric
without breaking the 4-component contract.

## Output

The full reducer JSON is printed to stdout (one line per field, pretty-printed). Exit code is 0 on every successful invocation — this skill does not gate; the gating decision belongs to the caller. The one-line status verdict is **always** present in the JSON; callers that want a hard fail condition can check `status == "ROT"` and exit 1 themselves.

```
$ /dev-kit:harness-effectiveness
{
  "schema_version": 4,
  "contract_version": "harness-effectiveness-v1",

From sh-ai-x/dev-harness-kit, MIT.

More from sh-ai-x/dev-harness-kit