prompt-optimizer

Files (7)

• SKILL.md
• SOURCES.md
• SPEC.md
• core-patterns.md · references/
• meta-optimization-loop.md · references/
• model-family-notes.md · references/
• transformed-examples.md · references/

SKILL.md (first lines)

# Prompt Optimizer

Optimize prompts with evals. Keep every instruction, example, and external context reference causal.

## Load Only What You Need

| Need | Read |
|------|------|
| New prompt | `references/core-patterns.md`, `references/model-family-notes.md`, `references/transformed-examples.md` |
| Existing prompt | `references/meta-optimization-loop.md`, `references/core-patterns.md`, `references/model-family-notes.md` |
| Model-family port | `references/model-family-notes.md`, `references/core-patterns.md` |
| Repeated failures | `references/meta-optimization-loop.md`, `references/core-patterns.md` |
| Weak or ambiguous draft | `references/transformed-examples.md` |
| Provenance | `SOURCES.md` |

## Step 1: Capture Contract

Record before editing:

- task type: new, refine, port, or debug
- target model family and snapshot, if known
- prompt surface: `system`, `developer`, `user`, tool descriptions, examples, schemas
- layer owners: platform, deployer/persona, retrieved context, user payload
- objective and non-goals
- inputs, tools, and external files available
- required output shape
- success criteria and failure cases
- hard constraints: latency, verbosity, safety, budget, tool use, style

If success criteria or examples are missing, create a small eval set first.
If the bottleneck is model choice, retrieval, tool schema, or missing evals, say so before rewriting.

## Step 2: Inventory External Context

For repo or agent prompts, list stable context by exact path:

| Context type | Examples |
|--------------|----------|
| Agent rules | `AGENTS.md`, `CLAUDE.md` |
| Specs | `specs/*.md`, `docs/api.md` |
| Policies | `SECURITY.md`, `docs/releasing.md` |
| Examples | `examples/`, `tests/fixtures/` |

Rules:

- Reference stable files by repo-relative path instead of copying them.
- Paste only excerpts needed for the prompt or eval case.
- Mark whether a file is `loaded`, `referenced`, or `out of scope`.
- Avoid vague context pointers such as "read the docs".

## Step 3: Choose Model Strategy

Read `references/model-family-notes.md`.

- Known family: optimize for that family.
- Unknown family: write a portable base plus short adapter notes.
- Snapshot changes: rerun evals.
- Cross-family divergence: specialize only the failing layer.

## Step 4: Shape Prompt

Read `references/core-patterns.md`.

- Put stable policy in `system` or `developer`.
- Put task-local facts, retrieved context, and variables in user-facing sections.
- Keep one owner per behavior rule.
- Use headings or tags only to separate content types.
- Put tool policy in prompt text; keep schemas in provider-native tools.
- Keep persona light unless it changes behavior.
- Use the shortest wording that preserves the constraint.
- Cut filler, repeated reminders, dead examples, and rationale that does not affect evals.

## Step 5: Optimize

Read `references/meta-optimization-loop.md` for refinements.

1. Baseline the current prompt on the same eval slice.
2. Cluster failures by root cause.
3. Write concrete edit criticisms.
4. Generate two to four candidates:
   - minimal-diff repair
   - structure-first rewrite
   - examples-first or tool-rule variant
   - provider adapter when needed
5. Compare candidates on the same cases.
6. Keep a short optimization log.
7. Validate the winner on holdout cases.
8. Stop on plateau, oscillation, overfit, excessive cost, or non-prompt bottleneck.

## Step 6: Return Package

Return:

1. `Target`
2. `Success Criteria`
3. `External Context`
4. `Optimized Prompt`
5. `Adapter Notes`
6. `Eval Set`
7. `Optimization Log`
8. `Residual Risks`

For existing prompts, include a concise diff-style note of the main behavioral changes.

## Failure Modes

- editing before defining the eval target
- mixing policy, examples, and raw context without boundaries
- duplicating rules across layers
- putting durable policy in user payloads
- asking for chain-of-thought
- keeping contradictory legacy instructions
- overfitting to one or two examples
- retaining examples that no longer improve evals
- fixing tool-use failures only in prompt text when tool descriptions or schemas are weak
- adding markup that does not reduce ambiguity
- using persona as a substitute for behavior rules

From getsentry/skills, Apache-2.0.

More from getsentry/skills