eval-fix

Files (1)

• SKILL.md

SKILL.md (first lines)

# Eval-Fix — Adversarial Fix-Evaluate Loop

This command implements an adversarial fix-evaluate loop: Generator (fix) → Evaluator (assess) → repeat until convergence or user intervention. The Generator and Evaluator are separate agents to prevent self-assessment bias.

**Parameter**: $ARGUMENTS — feature ID and product URLs

## Step 1: Load Evaluation Report

```bash
FEATURE_ID=$(echo "$ARGUMENTS" | grep -oP 'F\d+')
cat ".harness/evidence/${FEATURE_ID}/eval-report.md" 2>/dev/null
```

If no eval-report exists → tell user to run `/evaluate {FXXX}` first and STOP.

## Step 2: Extract Fix Tasks

Parse the eval-report's "Fix Tasks" section. For each feature scoring < 7:

1. Read the detailed findings (gap description + screenshot references)
2. Read the fix suggestions
3. Read the relevant screenshots to understand the visual gap
4. Create a fix task with:
   - **What's wrong**: factual description from Evaluator
   - **Reference screenshots**: paths to reference screenshots
   - **Dev screenshots**: paths to current dev screenshots
   - **Expected outcome**: what "fixed" looks like (described precisely)

## Step 3: Execute Fixes (Generator Role)

For each fix task, ordered by priority (P0 first):

### Single fix approach (no worktree):
1. Read the gap description and screenshots from the Evaluator
2. Read the relevant source code files
3. Implement the fix — focus on matching the reference product exactly
4. Run existing tests to verify no regression
5. Commit: `fix(eval): {brief description of what was fixed}`

### Multiple independent fixes:
Use `Agent(isolation: "worktree")` for parallel execution when fixes are independent.

**Generator Agent Prompt**:
```
You are a fix worker. The Evaluator found this gap:

GAP: {gap description from eval-report}
REFERENCE: {screenshot path} — shows how it SHOULD look/behave
CURRENT: {screenshot path} — shows how it CURRENTLY looks/behaves
FIX SUGGESTION: {evaluator's suggestion}

Steps:
1. Read the reference screenshot to understand the target
2. Read the current screenshot to understand the gap
3. Find and modify the relevant code
4. Test that the fix works
5. Commit with: fix(eval): {description}

RULES:
- Match the reference product EXACTLY
- Do NOT add features the reference doesn't have
- Do NOT "improve" beyond what the reference shows
- If unsure, match the reference screenshot pixel-for-pixel
```

## Step 4: Re-Evaluate

After all fixes are applied, trigger a fresh evaluation:

1. Merge any worktrees back to main branch
2. Ensure the dev product is running with latest code
3. Run the evaluation protocol (same as `/evaluate`):
   - Playwright MCP tests each feature against reference
   - Generate new eval-report with updated scores
   - Track score trend across iterations

## Step 5: Convergence Check

Read the new eval-report and compare with previous iteration:

### Convergence achieved (overall score ≥ 7):
```
Report to user:
"Eval-fix loop converged after N iterations.
 Score progression: {iter1} → {iter2} → ... → {current}
 Remaining minor gaps: {list if any}
 Next step: /verify {FXXX}"
```
STOP the loop.

### Progress detected (score improved):
Continue to next iteration — go back to Step 2 with the new eval-report.

### Stagnation detected (score did NOT improve for 2 consecutive iterations):
```
Report to user:
"Score stagnation detected after N iterations.
 Score progression: {iter1} → ... → {current} (no improvement in last 2 rounds)

 Remaining gaps that resist automated fixing:
 {list of features still < 7 with descriptions}

 Recommended actions:
 1. Review the gaps manually — some may require architectural changes
 2. Adjust the spec if the reference behavior is intentionally different
 3. Run /evaluate again after manual fixes

 Continue the loop? (The fix approach may need human guidance)"
```
STOP and wait for user decision.

### Regression detected (score decreased):
```
Report to user:
"WARNING: Score regression detected!
 Previous: {prev_score} → Current: {current_score}

 Regressed features:
 {list of features that got worse}

From zxdxjtu/harness, MIT.

More from zxdxjtu/harness