harness-review

Files (1)

• SKILL.md

SKILL.md (first lines)

# Harness Review

Evaluate each overnight runner component against its original rationale using the 4-question rubric below. Produce a structured verdict for all 17 inventory components, then list candidates for removal or experimentation.

## Section 1 — Component Inventory

| Component | File | Original rationale | Initial classification |
|---|---|---|---|
| Round loop | `cortex_command/overnight/runner.py` | Safety loop that spawns fresh orchestrators per round; enforces wall-clock limits and circuit breakers. (Replaced the retired `runner.sh` — see `docs/overnight-operations.md`.) | load-bearing |
| Orchestrator prompt | `cortex_command/overnight/prompts/orchestrator-round.md` | Thin orchestrator prevents context saturation across rounds — reads state only, does not accumulate implementation details | load-bearing |
| Batch runner | `cortex_command/overnight/batch_runner.py` | Parallel feature dispatch; manages retry budget per task; auto-merges to main | load-bearing |
| Batch plan hand-off | `cortex_command/overnight/batch_plan.py` + orchestrator prompt §6 | Orchestrator writes batch plan as markdown file; batch runner parses it; adds a parse boundary between planning and execution | pruning-candidate |
| Retry loop | `cortex_command/pipeline/retry.py` | Fresh agent per attempt prevents context degradation across retries; routes failure types to different strategies | pruning-candidate |
| Brain agent | `cortex_command/overnight/brain.py` | Post-retry triage when retry budget exhausted; produces SKIP/DEFER/PAUSE verdict | pruning-candidate |
| Throttle manager | `cortex_command/overnight/throttle.py` | Subscription-aware concurrency; detects rate limits at runtime and reduces concurrency dynamically | load-bearing |
| Deferral system | `cortex_command/overnight/deferral.py` | Routes questions needing human input to deferred/ directory; prevents questions from blocking the session | load-bearing |
| State management | `cortex_command/overnight/state.py` | OvernightState dataclass with atomic save/load; phase transition tracking | load-bearing |
| Backlog selection | `cortex_command/overnight/backlog.py` | Scores and filters backlog items for session; dependency graph, priority, and tag scoring | load-bearing |
| Morning report | `cortex_command/overnight/report.py` | Collects session data and renders executive summary and per-feature sections | load-bearing |
| Integration recovery | `cortex_command/overnight/integration_recovery.py` | Flaky guard re-run + repair agent after integration branch test failure | pruning-candidate |
| Session plan | `cortex_command/overnight/plan.py` | Renders session plan from SelectionResult; validates repos; bootstraps state | load-bearing |
| Status display | `cortex_command/overnight/status.py` | Live status snapshot during session | load-bearing |
| Event logging | `cortex_command/overnight/events.py` | JSONL session event log; enforces event type validation | load-bearing |
| Interrupt handler | `cortex_command/overnight/interrupt.py` | Graceful pause on signal (SIGINT/SIGTERM) | load-bearing |
| Smoke tests | `cortex_command/overnight/smoke_test.py` | Pre-execution validation before session starts | load-bearing |

## Section 2 — Protocol

Follow these steps in order. Do not skip steps.

### Step 1: Output disclaimer

The very first line of your output must be:

> Note: this assessment reflects model judgment only — no empirical session data was consulted. Treat verdicts as structured hypotheses, not conclusions.

### Step 2: Runtime scan

List all `.py` and `.sh` files in `cortex_command/overnight/` and `cortex_command/pipeline/`. Compare against the inventory table above.

Report exactly: "Evaluated N components from inventory; found M additional unlisted files: [list]." If M = 0, state "No unlisted files detected."

### Step 3: Docstring check

For each listed component, read its module-level docstring and compare against the inventory rationale. Note any mismatches between the docstring and the rationale in the table.

### Step 4: Rubric evaluation

For each component in the inventory (no exemptions based on initial classification), apply the 4-question rubric in Section 3 and assign a verdict. Display each component's evaluation inline as you work through them.

### Step 5: Candidates for Review section

After all verdicts, output a "Candidates for Review" section that lists all components with verdict `experiment-candidate` or `likely-deprecated`, ordered lowest-blast-radius first.

If no components received those verdicts, output: "Candidates for Review: none — all components evaluated as load-bearing."

## Section 3 — The 4-Question Rubric

Apply these four questions to every component. No component is exempt.

1. What model limitation did this component compensate for? (Read the module docstring — quote the rationale.)
2. Is that limitation still real at Claude's current baseline? (This is a model judgment call — state confidence level explicitly: high / medium / low.)
3. What would fail or degrade if this component were removed from the runner?
4. Verdict: `load-bearing` (non-negotiable, removal causes real breakage), `experiment-candidate` (worth empirically testing removal), or `likely-deprecated` (limitation appears solved; removal is low-risk).

## Section 4 — Edge Case Handling

Handle these conditions explicitly:

- **`cortex_command/overnight/` not found**: Print "No overnight runner found at expected path" and stop. Do not attempt to evaluate any components.
- **Listed component file missing from disk**: Note "not found at path" next to that component, skip rubric evaluation for it, and flag it as a maintenance item in a separate "Maintenance Items" section at the end.
- **Module has no docstring**: Fall back to reading inline comments in the first 30 lines of the file. If no rationale is found there either, note "rationale undocumented" and flag the component.

From charleshall888/cortex-command, MIT.

More from charleshall888/cortex-command