autoresearch

Files (2)

• SKILL.md
• eval-guide.md · references/

SKILL.md (first lines)

# Autoresearch for AI in SharePoint Skills

This skill adapts autonomous experimentation loops (Karpathy-style autoresearch) to AI in SharePoint. It doesn't rewrite the skill from scratch, but instwead runs it repeatedly, scoring every output, to improve it. Instead of spawning sub-agents on a local machine, we work within AI in SharePoint's multi-turn conversation using `load_skill` and `create_skill` as our read/write primitives.

---

## The Core Job

Take any existing AI in SharePoint skill, define what "good output" looks like as binary yes/no checks, then run an iterative loop that:

1. Applies the skill's instructions to test inputs and generates outputs
2. Scores every output against eval criteria (binary pass/fail)
3. Analyzes failure patterns and forms a mutation hypothesis
4. Mutates the skill prompt with ONE targeted change
5. Re-runs the skill, re-scores, and decides: keep or discard
6. Repeats until the score ceiling is hit or the user stops

**Output:** An improved skill (saved via `create_skill`) + a structured results log in conversation + a changelog of every mutation attempted.

---

## Before Starting: Gather Context

**STOP. Do not run any experiments until all fields below are confirmed with the user. Ask for any missing fields before proceeding.**

1. **Target skill** — Which skill do you want to optimize? (need the exact skill name as used in `load_skill`)
2. **Test inputs** — What 3–5 different prompts/scenarios should we test the skill with? These must include:
   - 1–2 **typical/happy path** inputs (the skill's bread and butter)
   - 1–2 **edge cases** (very short, very complex, ambiguous, or unusual inputs)
   - 1 **adversarial input** (something that commonly trips the skill up)

   If the user only provides happy-path inputs, push back: "What's the hardest input this skill has to handle?"

3. **Eval criteria** — What 3–6 binary yes/no checks define a good output? (see [references/eval-guide.md](references/eval-guide.md) for how to write good evals)
4. **Runs per experiment** — How many times should we apply the skill per mutation? Default: 3. (In AI in SharePoint, each run is a conversation turn, so 3 balances signal vs speed.)
5. **Budget cap** — Optional. Max number of experiment cycles. Default: no cap (runs until user stops or score ceiling hit).

---

## Step 1: Load and Understand the Target Skill

Before changing anything, load and understand the target skill completely.

1. Use `load_skill(skillName="<target>")` to read the full SKILL.md
2. If the skill references files in `references/`, load those too with `load_skill(skillName="<target>", filePath="references/<file>")`
3. Identify the skill's core job, process steps, and expected output format
4. Note any existing quality checks, anti-patterns, or constraints already in the skill

Do NOT skip this. You need to understand what the skill does before you can improve it.

**Save the original skill text in your working memory.** You'll need it as the baseline to revert to if mutations fail.

---

## Step 2: Build the Eval Suite

Convert the user's eval criteria into structured tests. Every check must be binary — pass or fail, no scales.

**Format each eval as:**

```
EVAL [N]: [Short name]
Question: [Yes/no question about the output]
Pass: [What "yes" looks like — be specific]
Fail: [What triggers "no"]
```

**Rules for good evals:**
- Binary only. Yes or no. No "rate 1–7" scales. Scales compound variability.
- Specific enough to be consistent. "Is the output good?" is too vague. "Does the output contain a direct answer in the first two sentences?" is testable.
- Not so narrow that the skill games the eval. "Contains fewer than 200 words" will make the skill optimize for brevity at the expense of everything else.
- 3–6 evals is the sweet spot. More than that and the skill starts parroting eval criteria instead of actually improving.

Load [references/eval-guide.md](references/eval-guide.md) for detailed examples and the 3-question test.

**Max score calculation:**
```
max_score = [number of evals] × [runs per experiment]
```

Example: 4 evals × 3 runs = max score of 12.

**Present the eval suite to the user for confirmation before proceeding.**

---

## Step 3: Establish Baseline

Run the skill AS-IS before changing anything. This is Experiment #0.

1. Ask the user what to name the optimized version (e.g., `summarize-email-v2`, `search-optimized`)
2. Apply the skill's instructions to each test input. For each:
   - State the test input clearly
   - Generate the output as the skill instructs
   - Present the output to the user
3. Score every output using the eval suite (see "How to Score" below)
4. Record the baseline results in the structured format (see "Results Format")

**CRITICAL: Score honestly.** You are both the generator and the scorer. Default to FAIL on ambiguity. Require explicit evidence for every PASS. See "Mitigating Self-Scoring Bias" below.

**After establishing baseline, confirm the score with the user.** If baseline is already 90%+, the skill may not need optimization — ask whether to continue.

---

## How to Score

Scoring is the most critical step. Because AI in SharePoint doesn't have separate scoring agents, you must be rigorous about self-scoring.

**For each output, score every eval:**

```
TEST INPUT: [the input used]

EVAL 1: [name] → PASS | FAIL
Evidence: [specific quote or observation from the output]
Reason: [one sentence explaining the verdict]

EVAL 2: [name] → PASS | FAIL
Evidence: [specific quote or observation]

From zrosenfield/sharepoint-ai-skills, MIT.

More from zrosenfield/sharepoint-ai-skills