evaluate

Files (1)

• SKILL.md

SKILL.md (first lines)

# /ouroboros:evaluate

Evaluate an execution session with the three-stage verification pipeline.

```
/ouroboros:evaluate <session_id> [artifact]
```

## Stage authority

1. **Mechanical** (lint, build, test, static analysis, coverage; no model cost): the only stage that can grant approval. At least one configured check must run and all must pass.
2. **Semantic** (AC compliance, goal alignment, drift): advisory. Can withhold approval and give feedback; cannot grant it.
3. **Consensus** (multi-model, only on uncertainty or request): advisory. A rejection withholds approval; an approval cannot grant it or lift a Stage 2 block.

Without an executed Stage 1 check the outcome is `acceptance_state: unverified` (not approved), with the model review attached as feedback.

## Instructions

### Loading MCP tools (deferred-schema guard)

Ouroboros tools may be deferred, and a schema loaded on one turn can unload by the next. Immediately before each `ouroboros_*` call, load it with the active runtime's tool-discovery capability: `"+ouroboros evaluate"` for `ouroboros_start_evaluate`; `"+ouroboros job"` for `ouroboros_job_wait`, `ouroboros_job_status`, and `ouroboros_job_result`. A call whose schema is not loaded fails with "Invalid tool parameters". If the tool is already callable, discovery is a no-op and an empty result is expected. Use the fallback only when the tool is genuinely absent.

### Steps

1. **Session**: use the given `session_id`, else the most recent execution session id in the conversation.
2. **Artifact**: a file the user names (read it with the active runtime's `inspect_code` capability), else recent execution output; ask with `ask_user` if unclear.
2.5. **Acting verification** (behavior-bearing work): Stage 1 already runs build/test, so go further when the runtime can act. Run the result and observe the real effect (command output with `run_shell`, the endpoint's response, the rendered UI) instead of reasoning over the diff, ideally in one verification child via `orchestrate_subagents`, or inline when spawning is restricted. Probe the ACs against observed behavior and the adversarial classes (`misleading_output`, `hung_command`, `stale_state`, `dirty_worktree`, ...). Put the captured evidence (commands, outputs, artifact paths) into `artifact`. If the runtime cannot act, say behavior was not observed.
3. **Start** the background evaluation so a rejected verdict can continue through the configured Ralph chain:
   ```
   Tool: ouroboros_start_evaluate
   Arguments:
     session_id: <session ID>
     artifact: <the code/output to evaluate, plus observed-behaviour evidence from 2.5>
     seed_content: <original seed YAML, if available>
     acceptance_criterion: <specific AC to check, optional>
     artifact_type: "code"  (or "docs", "config")
     working_dir: <absolute project root, recommended>
     trigger_consensus: false  (true if user requests Stage 3)
     auto_evolve: <optional override; omit to use execution.auto_evolve>
   ```
   `working_dir` sets both Stage 1 command execution and Stage 2 source visibility; pass the absolute project root whenever known.
4. **Observe the job**: the start result carries the observer contract in `response.meta.job_observer` and in a final `<!-- ouroboros-job-observer-v1 base64 ... -->` content sentinel. If the structured meta is missing, recover the contract from the sentinel and fail closed unless there is exactly one sentinel at the end of the text, its bounded payload decodes to an object whose only key is `job_observer`, that contract passes canonical v1 validation (it equals, field for field, the v1 contract the server builds from its own `job_id`, `cursor`, `session_id`, `execution_id`, and `follow_result_job_keys`), its job identity matches the visible start receipt, and it does not differ from any structured contract; never rebuild it from visible IDs. With a valid contract, hand it unchanged to exactly one read-only observer child; its instructions already follow `chained_ralph_job_id` through `follow_result_job_keys`, and the main session must not poll the same job. Without a child session, catch up with `ouroboros_job_status` / `ouroboros_job_result` on the next turn or an explicit status request, and wait live with `ouroboros_job_wait` only when the user asked to watch. Present the convergence outcome only after any chained Ralph job is terminal. `chained_ralph_skipped: seed_unavailable` means no Seed was available: keep the rejected verdict and say continuation was safely skipped. In OpenCode plugin mode with automatic evolution disabled, the plugin child is the terminal surface and `job_id` is `None`.
5. **Present** each stage's result, the final approval decision, and on rejection the reason and suggested fixes. Pick the next action from the outcome:
   - approved: accept, or `ooo evolve` to refine further
   - Stage 1 rejected with `code_changes_detected: true`: fix the build/test failures, then `ooo evaluate`, or `ooo ralph` for an automated fix loop
   - Stage 1 rejected with `code_changes_detected: false`: `ooo run` first to produce code
   - Stage 2 rejected: `ooo run` with fixes, or `ooo evolve`
   - Stage 3 rejected: `ooo interview` to re-examine requirements, or `ooo unstuck`
   - `acceptance_state: unverified`: add executable checks to `.ouroboros/mechanical.toml` (or run `ouroboros detect`), then `ooo evaluate`; the semantic review is feedback, not a verdict

### Fallback (no Ouroboros MCP server)

Adopt the `evaluator` role yourself with the prompt the MCP server uses: `$OUROBOROS_AGENTS_DIR/evaluator.md` when that override is set; else the installed package's copy, printed with `run_shell`:

```bash
uvx --isolated --python '>=3.12' --from ouroboros-ai python -c "import importlib.resources as r, sys; print(r.files('ouroboros.agents').joinpath(sys.argv[1]).read_text(encoding='utf-8'))" evaluator.md
```

else `src/ouroboros/agents/evaluator.md` in a source checkout.

Give a qualitative, advisory evaluation against the Seed (no numerical scoring and no approval authority without the Python core).

## RFC #1392 State Breadcrumb Footer

End the final response with exactly one line `◆ <current state> → next: <recommended action>`. Derive the state from live session state via `ouroboros_session_status` when available, otherwise from this skill's actual outcome. Never use a linear `Step N of M` footer; when the next step is a real choice, list 2-3 options.

From Q00/ouroboros, MIT.

More from Q00/ouroboros