design-evaluation

Files (51)

• README.md
• README_EN.md
• SKILL.md
• openai.yaml · agents/
• evaluation-input.example.json · examples/
• batch-evaluation.md · references/
• classification-policy.md · references/
• design-evaluations-schema.sql · references/
• evaluation-framework.md · references/
• evidence-policy.md · references/
• idea-benchmark-methodology.md · references/
• if-benchmark-methodology.md · references/
• if-student-benchmark-methodology.md · references/
• output-template.md · references/
• if-design.json · references/profiles/award-lenses/
• if-student.json · references/profiles/award-lenses/
• red-dot-concept.json · references/profiles/award-lenses/
• red-dot-product.json · references/profiles/award-lenses/
• category-crosswalk.json · references/profiles/benchmarks/idea/
• category-profiles.json · references/profiles/benchmarks/idea/
• discipline-profiles.json · references/profiles/benchmarks/idea/
• manifest.json · references/profiles/benchmarks/idea/
• program-profile.json · references/profiles/benchmarks/idea/
• category-crosswalk.json · references/profiles/benchmarks/if-student/
• category-profiles.json · references/profiles/benchmarks/if-student/
• competition-profile.json · references/profiles/benchmarks/if-student/
• manifest.json · references/profiles/benchmarks/if-student/
• category-crosswalk.json · references/profiles/benchmarks/if/
• category-profiles.json · references/profiles/benchmarks/if/
• discipline-profiles.json · references/profiles/benchmarks/if/
• manifest.json · references/profiles/benchmarks/if/
• category-crosswalk.json · references/profiles/benchmarks/red-dot/
• category-profiles.json · references/profiles/benchmarks/red-dot/
• competition-profiles.json · references/profiles/benchmarks/red-dot/
• discipline-profiles.json · references/profiles/benchmarks/red-dot/
• manifest.json · references/profiles/benchmarks/red-dot/
• classification.json · references/profiles/
• core.json · references/profiles/
• mature-work.json · references/profiles/maturity/
• student-concept.json · references/profiles/maturity/
• sector-overlays.json · references/profiles/
• red-dot-benchmark-methodology.md · references/
• batch_evaluation.py · scripts/
• benchmark_profiles.py · scripts/
• evaluation_profiles.py · scripts/
• score_evaluation.py · scripts/
• validate_profiles.py · scripts/
• test_batch_evaluation.py · tests/
• test_benchmark_profiles.py · tests/
• test_profiles.py · tests/
• test_score_evaluation.py · tests/

SKILL.md (first lines)

# Design Evaluation

## Purpose

Evaluate design quality consistently without pretending that a score is an award outcome. Keep design quality, presentation quality, and evidence confidence separate. Require the user to choose the maturity track.

## Scope Boundary

- Evaluate the supplied design and supplied presentation materials.
- Classify one primary discipline, one primary sector, and optional secondary labels and focus tags.
- Build an evidence ledger before scoring.
- Score the general rubric and report Critical findings separately.
- Apply an optional award-aligned lens only when the user already names a target award.
- Batch-evaluate a fixed corpus only after the user approves the maturity mapping for every included record.
- Produce score-based shortlists within each maturity track; never cross-rank Student Concept and Mature Work.

Do not:

- infer or change the work maturity;
- retrieve award winners inside this skill;
- recommend which award to enter;
- turn findings into a full redesign proposal;
- audit upload limits, filenames, declarations, licences, or portal compliance;
- call the result an official iF, Red Dot, or other jury decision;
- estimate an exact probability of winning.
- label a rank percentile, top-decile membership, or score as a winning probability.

Route winner retrieval to `$design-award-search`, award selection to `$design-award-match`, concrete redesign work to `$design-optimization` when available, and final package compliance to `$design-submission-check`.

## Required User Input

Accept images, a PDF, project text, a portfolio page, video frames, prototype evidence, test records, or a structured brief.

Maturity is mandatory and must come from the user. Accept exactly:

- `Student Concept` / `学生概念`
- `Mature Work` / `成熟作品`

If maturity is absent, ask exactly one question and stop scoring:

`请选择作品成熟度：“学生概念”或“成熟作品”。`

Never infer maturity from the author's identity, image finish, prototype appearance, commercial branding, or supplied metadata. If evidence conflicts with the selected maturity, preserve the user's selection and record `Maturity evidence mismatch`.

For a batch, an explicit user-approved mapping rule counts as user selection for every record matched by that rule. Reject unmatched values rather than inferring them. Record the mapping rule and `maturity_source: user` in the batch manifest.

Offer this template when the user asks how to use the skill:

```text
Project: {name}
Maturity: Student Concept | Mature Work  # selected by the user
Primary function: {what it does}
Target user: {who uses it}
Use context: {where and when}
Materials: {attachments or links}
Evaluation mode: General | optional named award-aligned lens
```

## Evaluation Workflow

For batch work, first read [references/batch-evaluation.md](references/batch-evaluation.md). Use `scripts/batch_evaluation.py` for deterministic scoring, failure isolation, and separate-track shortlisting. Use a project adapter for private database access; never bundle database rows, images, signed URLs, or credentials in the public Skill.

### 1. Confirm the user-selected maturity

Record:

```yaml
maturity: student_concept | mature_work
maturity_source: user
```

Do not proceed with a numeric score when `maturity_source` is missing or is not `user`.

### 2. Build the evaluation profile

Read [references/classification-policy.md](references/classification-policy.md) and `references/profiles/classification.json`.

Extract:

- primary function, target user, use context, and claimed outcome;
- one primary design discipline and up to two secondary disciplines;
- one primary application sector and up to one secondary sector;
- zero or more focus tags;
- supplied material types and obvious material limitations.

The classification confidence is separate from evaluation confidence. Ask no additional question when a reasonable classification can be stated as an assumption.

### 3. Build the evidence ledger

Read [references/evidence-policy.md](references/evidence-policy.md). For every scored dimension, assign exactly one evidence state:

- `Verified`
- `Supported`
- `Claimed`
- `Missing`

Attach concise evidence references and distinguish observable facts from author claims and evaluator inference.

### 4. Load the rubric

Read [references/evaluation-framework.md](references/evaluation-framework.md).

Load:

1. `references/profiles/core.json`;
2. the user-selected maturity profile;
3. the relevant classification overlay in `references/profiles/sector-overlays.json`;
4. an optional aggregate benchmark context resolved by `scripts/benchmark_profiles.py`;
5. an optional file from `references/profiles/award-lenses/` when the user names that target.

Award lenses produce a separate alignment section. Never replace or mathematically blend the general score with an award-aligned result.

For the main iF context, read [references/if-benchmark-methodology.md](references/if-benchmark-methodology.md). Resolve an exact normalized category profile first, then its mapped discipline profile, then the core fallback.

For iF Student context, read [references/if-student-benchmark-methodology.md](references/if-student-benchmark-methodology.md). Load it only after the user has selected `student_concept`. Reject it for `mature_work`. Treat its 15 SDG categories as issue themes, never as evidence of product, communication, interface, spatial, or other design discipline. Resolve a high-sample SDG theme first and otherwise use the competition-wide student profile.

For Red Dot context, read [references/red-dot-benchmark-methodology.md](references/red-dot-benchmark-methodology.md). Keep Product Design, Brands & Communication Design, and Design Concept separate. Resolve an exact high-sample category first, then an explicitly supplied competition line, then the mapped evaluation discipline, then the core fallback. Never infer the Red Dot competition line from maturity.

For IDEA context, read [references/idea-benchmark-methodology.md](references/idea-benchmark-methodology.md). Resolve an exact high-sample category first, then a supplied or provisionally mapped discipline, then the program-wide context. Treat its discipline profiles as multi-label and non-additive. Load `Student Designs` only for `student_concept`, and never infer discipline from that category.

From SeanJ1ang/design-judge-skills, Apache-2.0.

More from SeanJ1ang/design-judge-skills