hf-l2-skill-synthesis

Files (19)

• SKILL.md
• evals.json · evals/
• trigger-queries.json · evals/
• rich-gate-scorecard.md · metrics/
• 01-github-ingestion.md · references/
• 02-pattern-classifier.md · references/
• 03-eval-framework.md · references/
• 04-quality-matrix.md · references/
• 05-template-library.md · references/
• check-overlaps.sh · scripts/
• classify-pattern.sh · scripts/
• grade-outputs.sh · scripts/
• ingest-repo.sh · scripts/
• run-trigger-evals.sh · scripts/
• validate-gate.sh · scripts/
• validate-skill.sh · scripts/
• task_plan.md
• eval-scaffold.json · templates/
• skill-scaffold.md · templates/

SKILL.md (first lines)

## Overview

Synthesize new skills from GitHub repositories, existing codebases, or design patterns. Use when creating skills from repos, classifying skill patterns, building template libraries, generating eval frameworks, or creating eval-driven skill scaffolds. Produces complete skill packages with evals and trigger queries.

## The Iron Law

```
NO SKILL WITHOUT EVALS
```

Every skill produced by this pipeline must include evaluation test cases (≥3) and trigger queries (≥20). A scaffold without evaluation coverage is incomplete. A skill without executed validation is untested.

# skill-synthesis
## On Load

1. **Fetch the canonical spec** — always fresh, never cached:
   ```bash
   webfetch https://agentskills.io/llms.txt
   ```
   Use this as the ground truth for frontmatter format, pattern definitions, and eval structure.

2. **Check for planning files** — if `task_plan.md` exists, read the Goal field. If missing, run:
   ```bash
   bash scripts/validate-gate.sh synthesize "<user-request>" <output-dir>
   ```
    This creates the planning scaffold and validates intent.

## Pipeline Overview

```
INGEST → CLASSIFY → SCAFFOLD → VALIDATE
```

### Phase 1: INGEST

**Input:** GitHub repo URL or owner/repo string
**Tools:** `repomix --remote`, `webfetch`, `websearch`

```bash
# Pack remote repo (branch-targetable, no local clone)
repomix --remote <owner/repo> \
  --include "**/SKILL.md,**/skills/**/*.md" \
  --output /tmp/skill-ingest-<timestamp>.xml

# Fetch canonical spec (always fresh)
# Saves to /tmp/spec-<timestamp>.md
webfetch https://agentskills.io/llms.txt

# Search for real-world skill implementations across GitHub
# Results form a pattern corpus for classification
websearch "site:github.com SKILL.md agent skill frontmatter"
```

**Output:** Packed XML with all SKILL.md files, fresh spec, pattern corpus

### Phase 2: CLASSIFY

For each SKILL.md found, classify along these axes:

| Axis | Categories | Detection |
|------|------------|-----------|
| **Pattern** | P1 (<200L), P2 (200-400L), P3 (400L+) | Line count + reference count |
| **Routing** | Thin router, Context router, Not-a-router | Decision tree present? Ref count 0-2? |
| **Efficiency** | Token-efficient, Context-heavy, Script-bundled, Pure-instructions | Token/line ratio, scripts/ presence |
| **Testing** | Has evals (≥3 cases), Has triggers (≥20 queries), Has scripts, Has matrix, Complete | Evaluation presence, trigger coverage |
| **Quality** | Excellent (4.5+), Good (4.0+), Acceptable (3.5+), Needs Work (<3.5) | 5-dimension weighted score |

**Output:** Classification report per skill, ranked by quality score

### Phase 3: SCAFFOLD

1. Select top 3 examples matching target category
2. Extract common structural patterns (frontmatter shape, reference topics, script types)
3. Generate scaffold:
   - SKILL.md with proper frontmatter (trigger phrases from decision tree analysis)
   - Numbered reference files (150-300 lines each)
   - Validation scripts (co-located with the skill's scripts/ directory)
   - Evals: `evals.json` (3-8 test cases) + `trigger-queries.json` (20 queries, 60/40 train/val split)
4. Write to target directory

**Output:** Complete skill directory, ready for validation

### Phase 4: VALIDATE

```bash
# Gate 0: Intent validation
bash scripts/validate-gate.sh synthesize "<user-request>" <output-dir>

# Structure validation
bash scripts/validate-skill.sh <output-dir>

# Overlap detection
bash scripts/check-overlaps.sh <output-dir>

# Trigger eval execution
bash scripts/run-trigger-evals.sh <output-dir>

# Quality scoring
bash scripts/grade-outputs.sh <output-dir>
```

**If any step fails:** Fix → re-run. Max 5 iterations.
**Output:** Validated skill with passing evals and quality score ≥ 3.5

## Decision Tree — Pick Your Path

```
User says...                                    → Load
─────────────────────────────────────────────────────────────────────
"create skills from GitHub" / "synthesize"      → references/01-github-ingestion.md
"find skill patterns" / "classify skills"       → references/02-pattern-classifier.md
"generate evals" / "test this skill"            → references/03-eval-framework.md
"score this skill" / "quality check"            → references/04-quality-matrix.md
"build a template" / "template library"         → references/05-template-library.md
```

**Rule:** Load only ONE reference file. Do not load all references.

## Reference Map

From shynlee04/hivemind-plugin, MIT.

More from shynlee04/hivemind-plugin