mstar-routing-eval

Files (2)

• SKILL.md
• routing-evals.json · assets/

SKILL.md (first lines)

## Maintainer scope（维护专用 · Cursor）

- **不是 runtime pack**：日常 `specify → implement → QC` **不要求** Read 本 skill；已从 `skills/mstar-*` 运行时包中拆出，避免与「每轮必读」混淆。
- **Canonical 位置**：本仓库内 **仅此目录**（`.cursor/skills/mstar-routing-eval/`）为正文 + `assets/` SSOT；**不再**维护 `skills/mstar-routing-eval/` 副本。
- **谁在读**：`@prompt-engineer` 改规则前；维护者改 `project-manager` / harness 门禁后跑回归；`repo-maintenance` 清单涉及路由时。

## Load order（必读顺序）

**在同一会话或任务中首次 Read 本 skill 时：必须先 Read `mstar-harness-core` skill（SKILL.md，以及评估路由与门禁时需要的 `mstar-harness-core/references/`）。** 本 skill 的 `routing-evals.json` 场景**校验的是对 harness 的遵守情况**；若未建立 harness 上下文，评估易与真实门禁脱节。冲突时 **以 `mstar-harness-core` 为准**。

**摘要**：`mstar-harness-core` — 期望路由与 phase gate；本 skill — 场景集、`Routing Eval Report` 模板与回归方法。

# Morning Star Routing & Evaluation Harness

## 1. PM 路由评估

本节用于验证 `@project-manager` 是否将任务路由到正确的 agent 和门禁。

### 输入

- **场景集**：本 skill 的 **`assets/routing-evals.json`**
- **路由策略来源**：`mstar-roles` skill 的 `project-manager` 角色
- **全局约束**：`mstar-harness-core` skill（含 harness 不变量与 skill 索引）

### 评估方法

对每个场景：

1. 将实际路由与 `expected_route` 进行比对。
2. 检查所需产出是否已规划并最终产出。
3. 验证没有 `hard_fail_if` 条件被违反。
4. 验证 `Context Loaded` 已声明且包含必要文件。
5. 验证 Assignment 的语言契约：字段名英文、任务正文可中文、执行产出/报告英文（除非用户明确要求其他语言）。
6. 记录路由质量和缺失的证据。

**Authority-aware fixture interpretation**：未显式标注文件路由的断言按 ACTIVE 主线判定：`integration_merge_lease` 是 workflow 级串行合并 exclusion 的逻辑名称，经公共 workflow view 读取（per-plan execution lease 已移除，fixture 不得要求 per-row lease/claim 断言）；协调写入经公共动词的事务 + execution-token CAS。仅显式 pre-activation / engine-absent 场景保留 snapshot + same-host 文件锁语义。control root 是主 checkout 的路径定位，不要求发明 ACTIVE `control_worktree_path` 列；branch anchors 与 `integration_worktree_path` 经执行域 workflow view 读取。open findings 始终为 store linked issues，`residuals.json` 只作迁移历史。措辞 SSOT → `mstar-artifacts`、`mstar-branch-worktree` L1、`mstar-dispatch-gates` 与 `mstar-iteration/references/phase-2-worktree-lease.md` §2.0 / Coordination transactions。

### 评估执行步骤（1-2-3）

1. **准备输入**：加载当前 `assets/routing-evals.json`、`project-manager.md`、`mstar-harness-core`（含 references）。
2. **逐案判定**：按 `expected_route` / `must_have_artifacts` / `hard_fail_if` 打分并记录证据。
3. **输出报告**：按下文 **`Routing Eval Report`** 模板汇总（含 `phase_gate_compliance_rate`）。

### 评分

- `Pass`：路由和产出满足预期，无硬性失败。
- `Borderline`：存在轻微偏差但质量门禁得到保留。
- `Fail`：触发硬性失败条件或跳过关键门禁。

### Phase Gate 评分细则

用于量化 `specify -> clarify -> plan -> tasks -> implement` 合规度。

- `Pass`
  - 非 hotfix：Prepare 与 Execute 门禁均完整通过；无跳步实现。
  - hotfix：允许压缩路径，但已明确事后 `clarify/RCA` 补记安排。
- `Borderline`
  - 门禁顺序总体正确，但存在可修复的轻微缺项（例如 `Phase Gate Checklist` 字段漏写但证据可追溯）。
  - 未出现"先实现后补 gate"的行为。
- `Fail`
  - 非 hotfix 跳过 `clarify` 或 `tasks`。
  - 已知 plan drift 仍继续实现，且无 plan 回写。
  - hotfix 未承诺或未记录事后 `clarify/RCA`。

### 建议量化指标

- `phase_gate_compliance_rate = pass_cases / total_cases`
- 建议门槛：
  - 日常迭代：`>= 0.9`
  - 关键发布前：`= 1.0`

## 2. 回归信号

如果以下情况在两个或以上场景中出现，则说明路由逻辑退化：

- 实现类任务省略了 **QA gate 决策**（`mandatory` 或 `pm-acceptance` 须落 Assignment；禁止无字段即 Done）
- QC 路径相对任务类型被过度或不足使用
- 跨模块工作缺少 architect/探索阶段
- 未按 `specify -> clarify -> plan` 完成准备阶段即进入实现
- `plan` 完成后未进行 `tasks` 拆解就直接实现
- 非热修复任务使用了热修复路径
- 对业务 Git 仓库的实现类分派缺少 **`Working branch`** 或明确的 **`Branch policy`**（见 **`mstar-branch-worktree`** `SKILL.md`）
- `Context Loaded` 缺失或缺少必要的 plan 路径
- 非热修缺陷在未记录根因或可证伪假设前大规模改代码
- 生产/高危运维变更缺少回滚说明或高危清单
- 声明 Report-only 的 QA 任务却改动业务逻辑，或无谓触发全量三审
- Hotfix / clean small non-UI 误派 `qa-engineer`（应为 `QA gate: pm-acceptance`）
- UI 可见变更跳过 mandatory QA 或无 observable 证据
- Assignment 语言契约不一致（字段名漂移为中文、或要求英文报告却回报为中文且无用户豁免）
- 实现类 Assignment **缺少 `Task category`** 或与 **`Execute as`**、`Why this agent` 明显矛盾（或缺少 **`Execute as`**）
- **意图门禁**缺失：Prepare 未书面收敛真实目标/成功判据/非目标即进入开发分派
- **Dev 三角失衡**：在「新 API + 多文件/用户可见 UI」或**可并行的双模块**场景下，长期**仅**派 `@fullstack-dev`、从不派 `@frontend-dev` / `@fullstack-dev-2`，且 Assignment 未写明 `Dev routing: single-stream — …`
- **误把「单轨串行」当「全程仅 fullstack-dev」**：计划写不并行 / 逐波串行，但 PM Task Board 上多行可对等 dev 单元却**无理由**全部 `fullstack-dev`、未按 `project-manager.md` Dev 三角 §6 round-robin，也未写 `Dev owner tie-break: single id — …`
- **并行技能标签缺失**：已下发 **≥2 条并行实现 Assignment**（或 `Dev routing: parallel` / tasks 并行标记）且走 Superpowers 工作流时，**Status Update 或 Assignment 的 `Superpowers` 未**出现 **`dispatching-parallel-agents`**（或同义短语）；或 **同仓 ≥2 可写并发** 却未叠 **`using-git-worktrees`** 与检出约定
- **N invoke ≠ worktree 隔离**：同条消息已发 ≥2 implement invoke，但 Assignment 无绝对 **`Worktree path`** / 未在 invoke 前 `git worktree add`（见 **`mstar-branch-worktree`** → **`references/parallel-writable-pre-dispatch.md`**）
- **Source isolation/coordination bypass:** cross-plan writable tasks lack distinct registered row metadata checkouts or safe atomic coordinator transactions/CAS; foreign workflow integration exclusion is stolen or real integration merges run concurrently. An InProgress row without a historical per-row lease is not an orphan; never require a claim/bind/seal to continue valid facts.
- **No shared safe authority:** cross-plan parallel writable dispatch without the ACTIVE control store's transaction and execution-token CAS arbitration should be **Blocked** or **`Plan parallelism: serial`** (pre-activation / engine-absent only: same-host exclusive snapshot write lock); missing same-host flock alone is not an ACTIVE serial trigger, and a file-route race is acceptable only on current-turn `Cross-host lease race: accepted` (or equivalent) + append-only `notes.jsonl` audit. A worktree waiver alone never authorizes lockless cross-host writes or bypasses this gate — integration merges stay serial.
- **invoke 角色字段缺失（静默 generic 回退）**：已发 Task/subagent invoke，但 item 漏写与 **`Execute as`** 匹配的角色绑定字段（omp **`agent`** / Cursor **`subagent_type`** / OpenCode **`subagent`** / Kimi·ZCode **`subagent_type`**）⇒ 宿主静默回退 generic worker，却因 count=N 通过而误判「派发完成」（见 **`mstar-dispatch-gates`**、**`mstar-host/references/parallel-dispatch.md`**）。**N=1 顺序 Review-&-Edit 链**最易触发——count 门恒过，字段门是唯一保护
- **Removed entry accepted:** nonempty iteration-drive arguments boot/mutate instead of refusal or silently become whole-iteration authorization (`direct-plan-removed-entry-refused`). No-argument control still drives the full selected lifecycle (`direct-plan-noargs`).
- **Another per-row PM:** launches project-manager as subagent/terminal primary or requires per-row identity/claim/bind/transfer (`direct-plan-two-rows-one-coordinator`).
- **Configuration ceremony:** valid source/defaults cannot progress without prepared Assignment bytes, or active correction requires reseal/rebind/reset (`direct-plan-config-revisable`, `direct-plan-missing-assignment-no-ceremony`). Actual foreign identity/source conflicts still refuse.
- **Completion shortcut/transfer:** recreates handoff/accept/integration-start instead of direct complete, skips QC/QA/actual merge proof, repeats Git on replay or closes parent at row Done (`direct-plan-iteration-completion`).
- **Outer obligations lost:** development loses compound/PR/verified-merge/close; report-only invents Git/PR or substitutes QA pass for policy fulfilment (`direct-plan-development-completion`, `direct-plan-report-only-completion`).
- **Leaf promotion/disclosure:** consumes coordinator context, becomes PM or dispatches children (`direct-plan-leaf-not-promoted`).
- **Phase 2 线性等待**：存在 running child 即进入 wait，未先按 `Rescheduling checkpoint`（`before-wait`）评估独立 ready task / 已 prepare plan；或把 plan / task 编号当串行理由（case `phase2-ready-before-wait` · home 指针 → **`mstar-iteration`** → `references/phase-2-worktree-lease.md` §2.4）
- **review 返回不推动调度**：task review 返回后仍等无关 plan 的整个 lifecycle 结束才派发无关 ready 实现；或让并行轨共用同一可写 session / resumed reviewer（case `phase2-review-overlap` · home 指针 → **`mstar-iteration`** → `references/phase-2-worktree-lease.md` §2.4）
- **依赖提前释放或活动 base 被改**：review 通过即释放 dependent task；或对已派发活动 task re-split / 改 `BASE_SHA`（case `phase2-base-dependency` · home 指针 → **`mstar-iteration`** → `references/phase-2-worktree-lease.md` §2.4）
- **Owned work duplicated/capacity invented:** duplicates native running tasks or depends on deleted extra-primary slots; native capacity and actual ownership remain (`phase2-owned-and-capacity`).
- **空 ready 循环**：无 ready work 时反复自证同一事实、timer / 轮询、tick 计数或「still waiting」notice；为保持忙碌造工作（case `phase2-quiet-wait` · home 指针 → **`mstar-iteration`** → `references/phase-2-worktree-lease.md` §2.4）
- **Observer becomes transport:** reads deleted settings, launches/journals terminal PMs, polls or invents ready/completion authority (`phase2-observer-native-tasks-only`). Preserve bounded bind/checkpoint/history and native leaf dispatch.
- **`recovery-policy-product-proof`**：以 live provider receipt 作 development gate，或静默转为 mandatory QA/E2E；开发验收只要求隔离层可自证的证据。
- **`recovery-policy-isolated-proof`**：不接受 fixture / 本地构建层证据，或因未验证真实环境而阻塞开发完成；未验证行为如实记录。
- **`recovery-policy-attribution-unknown`**：猜测 model、倒填编辑时间、将真实 provenance 写入 tracked plan，或新增机器完整性门禁。
- **`recovery-policy-sparse-intent`**：强制 rebind / token-copy preflight、丢弃 verb 已应用的部分结果，或向 leaf 传 session credentials；实际输入仍按 CLI verb 显式提供。
- **`recovery-policy-host-noop`**：要求重启刻意关闭的 preference、把 `preference-off` 当工具失败，或声称已做 installed-host 验证；phase-1 no-op 与 binding 状态无关；仅 preference 开启时要求 pending binding。
- **Hash-as-record-only cases**: `hash-record-only-post-prepare`, `hash-record-only-submitted-evidence`, `hash-record-only-source-output-drift`, and `hash-record-only-request-replay` distinguish ordinary prose/report/source/output edits from real holder, scope/QA/cleanup, path/state, numeric revision and Git conflicts. Do not require adoption, restoration, resealing, re-signing or deleted hash flags. Only same-operation-id/different-request `request_hash` replay remains a hash conflict; receipt replay is independent of current output bytes.
- Run these four cases before/after with their embedded negative controls and record actual completion output plus judgments. Targeted static/link checks do not establish model compliance. Use the existing session-completion evaluation route; do not run the full corpus or create a replacement runner for this change.
- **Prototype-first Phase 1 cases**: evaluate only `iteration-loop-autonomous-direction-lock`, `iteration-phase1-cursor-plan-feedback-driven`, `iteration-start-direction-hint-parse`, `iteration-start-pause-stops-at-phase1`, `iteration-phase1-parallel-review-chain-blocked`, `iteration-start-pause-override-refused`, `iteration-start-arm-after-direction-lock`, plus the three pressure cases `iteration-phase1-prototype-current-revision`, `iteration-phase1-simple-writer-only`, and `iteration-phase1-autonomous-nonhtml-complex`. Compare baseline/candidate completions under the live coordinator model using the existing completion route, preserving actual transcripts and judgments. Check multi-turn feedback/current-version approval, single carrier and permission recovery, autonomous non-HTML retention, gap-driven actual selected invokes and mandatory writer last. Distinguish proposed/tool-free actions from real tool execution; neither source-text checks nor a permanent prose-pinning unit test proves behavior. Do not run the full corpus or invent a runner.

From btspoony/mstar-harness, MIT.

More from btspoony/mstar-harness