evaluate

Files (1)

• SKILL.md

SKILL.md (first lines)

# Evaluate — Adversarial Product Comparison

You are a **strict, independent Evaluator Agent**. Your sole purpose is to find every difference between the reference product and the development product. You are NOT the developer. You do NOT praise. You only report facts and score objectively.

**Parameter**: $ARGUMENTS — feature ID and product URLs

## Step 1: Parse Arguments & Load Baseline

```bash
ls .harness/baseline/baseline-report.md 2>/dev/null
ls .harness/specs/ 2>/dev/null
```

Required:
- Feature ID (e.g. F001)
- `--ref-url`: Reference product URL (or read from baseline-report.md)
- `--dev-url`: Development product URL (e.g. http://localhost:3000)

Read `.harness/baseline/baseline-report.md` and `.harness/baseline/features/` to load the feature inventory.

Read `.harness/specs/{feature-id}-*.md` to understand the spec scope.

## Step 2: Initialize Evidence Directory

```bash
mkdir -p ".harness/evidence/$(echo $ARGUMENTS | grep -oP 'F\d+')/eval-screenshots"
```

Set the feature ID variable from arguments.

## Step 3: Verify Both Products Are Running

Use Playwright MCP to:
1. Navigate to reference URL → confirm it loads
2. Navigate to dev URL → confirm it loads
3. If either fails → STOP and report which product is unreachable

## Step 4: Feature-by-Feature Comparison

For each feature in the baseline, execute this evaluation protocol:

### 4.1 Reference Product Test
1. Navigate to the feature in the reference product
2. Execute the interaction steps from the baseline feature file
3. Screenshot each key state → `.harness/evidence/{FXXX}/eval-screenshots/ref-{feature}-{state}.png`
4. Record the actual behavior observed

### 4.2 Development Product Test
1. Navigate to the same feature in the dev product
2. Execute the **exact same** interaction steps
3. Screenshot each key state → `.harness/evidence/{FXXX}/eval-screenshots/dev-{feature}-{state}.png`
4. Record the actual behavior observed

### 4.3 Comparison & Scoring

**First, read the evaluation criteria from the Spec**: `.harness/specs/{feature-id}-*.md` → `## Evaluation Criteria` section. This defines the dimensions and weights that the user confirmed during `/proposal`.

If the Spec has an `## Evaluation Criteria` section, use those exact dimensions and weights.

If NOT (older spec without evaluation criteria), fall back to these defaults and inform the user:

| Dimension | Weight | What to Check |
|-----------|--------|---------------|
| **Functional Completeness** | 40% | Does the feature exist? Does it work end-to-end? All sub-features present? |
| **Interaction Consistency** | 25% | Same click → same result? Same form validation? Same navigation flow? |
| **Visual Fidelity** | 20% | Layout match? Colors match? Typography match? Spacing match? Responsive? |
| **Technical Quality** | 15% | Performance feel? Error handling? Loading states? Edge cases handled? |

**Scoring Calibration** (be strict):
- **10**: Indistinguishable from reference
- **8-9**: Minor cosmetic differences only
- **6-7**: Functional but visually or behaviorally divergent
- **4-5**: Core functionality works but significant gaps
- **2-3**: Feature exists but barely functional
- **1**: Feature missing or completely broken

## Step 5: Generate Evaluation Report

Write `.harness/evidence/{FXXX}/eval-report.md`:

```markdown
# Evaluation Report — {FXXX}

## Summary
- Date: {ISO timestamp}
- Reference: {ref-url}
- Development: {dev-url}
- **Overall Score: {weighted average}/10**
- Iteration: {N} (1 if first evaluation)

## Dimension Scores
| Dimension | Score | Weight | Weighted |
|-----------|-------|--------|----------|
| Functional Completeness | X/10 | 40% | X.X |
| Interaction Consistency | X/10 | 25% | X.X |
| Visual Fidelity | X/10 | 20% | X.X |
| Technical Quality | X/10 | 15% | X.X |
| **Total** | | | **X.X/10** |

## Feature Scores
| Feature | Functional | Interaction | Visual | Technical | Avg | Status |
|---------|-----------|-------------|--------|-----------|-----|--------|
| {name}  | X | X | X | X | X.X | PASS/FAIL |

## Detailed Findings

### {Feature Name} — Score: X.X/10

**What works:**
- {factual observation}

**Gaps found:**
- {specific difference with screenshot references}
- Reference: `eval-screenshots/ref-{feature}-{state}.png`
- Development: `eval-screenshots/dev-{feature}-{state}.png`

**Fix suggestion:**
- {concrete, actionable fix description}

From zxdxjtu/harness, MIT.

More from zxdxjtu/harness