speedup-proof-hvn

Files (8)

• SKILL.md
• openai.yaml · agents/
• measurement-playbook.md · references/
• optimization-patterns.md · references/
• report-schema.md · references/
• benchmark_command.py · scripts/
• compare_results.py · scripts/
• generate_report.py · scripts/

SKILL.md (first lines)

# Speedup Proof

Treat every optimization as a falsifiable experiment: inspect first, propose one exact change, obtain explicit approval, preserve behavior, measure a baseline, implement only the approved change, verify correctness, measure again, and keep only a result supported by evidence.

## Require approval before edits

Treat invocation of `$speedup-proof-hvn` as consent to inspect the project, never as consent to modify it.

- Always begin with read-only analysis, even when the initial request says optimize, apply, fix, implement, or use `prove` mode.
- Do not create or edit project files, benchmark harnesses, tests, reports, dependency files, configuration, worktrees, or generated artifacts before the approval checkpoint.
- Do not install dependencies, run formatters, execute migrations, or run commands that can mutate project state before approval.
- Never interpret broad phrases such as `optimize this codebase`, `do it`, or a bare skill invocation as approval for an unseen patch.
- Ask for approval only after showing a concrete bounded proposal. Do not ask for blank-cheque approval before analysis.

Present this checkpoint and stop:

```markdown
## Proposed speedup

- Hotspot: file and line
- Evidence: why this path is likely costly
- Exact files to change or create
- Proposed transformation
- Behavior that must remain identical
- Benchmark and test plan
- Main risk
- Scope: one logical optimization

No project files have been modified. Do you want me to apply this exact change?
```

Continue only after the user explicitly approves that displayed proposal. Approval applies only to the listed files and transformation. If the scope, file list, behavior, dependencies, or optimization changes, stop and request fresh approval. Never apply a second unrelated optimization under the first approval.

## Select a mode

- Use `prove` when asked to optimize, fix, improve, or speed up code. Inspect, propose, wait for approval, then measure, edit, verify, remeasure, and report.
- Use `audit` when asked to inspect, analyze, scan, or report. Rank candidates without editing source files.
- Use `regression` when asked what became slower in a branch, commit, PR, or recent change. Compare equivalent workloads without disturbing uncommitted work.

Never edit files in `audit` mode. In every other mode, require the approval checkpoint before modifying project state.

## Run the proof workflow

### 1. Establish the contract

- Identify the exact function, endpoint, render path, query, build step, or command in scope.
- Record outputs, ordering, errors, side effects, public APIs, permissions, persistence, and other behavior that must remain stable.
- Detect the relevant tests, typecheck, lint, and build commands.
- Choose realistic input sizes and distributions. Do not prove a production claim with a toy fixture.

### 2. Find and rank candidates

- Prefer profiler output, telemetry, traces, slow tests, query logs, or existing benchmarks.
- Otherwise inspect likely hot paths for repeated scans, nested lookups, unnecessary allocations, repeated parsing or serialization, N+1 I/O, render churn, duplicate filesystem work, and avoidable recomputation.
- Rank candidates by execution frequency × input size × unit cost × confidence.
- Read `references/optimization-patterns.md` when selecting transformations.
- Pick one high-confidence candidate. Do not bundle unrelated changes into one measurement.

### 3. Request approval

- Show the required approval checkpoint with the exact hotspot, evidence, files, transformation, preserved behavior, benchmark, tests, and risk.
- State clearly that no project files have been modified.
- Stop and wait for explicit approval.
- If approval is denied, offer the read-only findings and make no changes.

### 4. Measure the baseline

- Reuse an existing benchmark only when it exercises the relevant behavior.
- Otherwise create the smallest harness that imports real production code and uses deterministic, representative fixtures.
- Exclude fixture construction from the timed region unless setup is part of the user-visible workload.
- Run at least 2 warmups and 7 measured iterations. Use more for short or variable work.
- Keep runtime, dependencies, machine conditions, fixtures, caches, and command equivalent before and after.
- Use median as the primary metric; retain p95, range, and coefficient of variation.
- Read `references/measurement-playbook.md` before creating a new benchmark.
- Use the bundled runner when command-level timing is appropriate:

```bash
python3 scripts/benchmark_command.py --label before --warmups 2 --runs 9 --output before.json -- python3 benchmark.py
```

If the workload depends on uncontrolled network latency, shared remote services, changing data, or interactive timing, isolate the variable or return `INCONCLUSIVE`.

### 5. Make the approved safe change

- Modify only the files and behavior described in the approved proposal.
- Add or strengthen a focused correctness test first when behavior is ambiguous.
- Preserve output equality, ordering, side effects, authorization, error handling, and public interfaces.
- Keep the patch small enough to attribute the measured effect.
- Do not add caching without an invalidation strategy.
- Do not remove validation, security, logging, accessibility, or error handling to manufacture a speedup.

### 6. Verify and remeasure

- Run the narrowest correctness test, then the broadest relevant test/build/typecheck.
- Rerun the exact same benchmark under equivalent conditions.
- Compare the two evidence files:

```bash
python3 scripts/compare_results.py before.json after.json comparison.json --correctness passed
```

- Reject or revert the optimization if it breaks behavior or causes a regression. Never retain a slower patch only because its Big-O estimate looks cleaner.

### 7. Report the verdict

Use one evidence label:

- `PROVEN` — Correctness checks pass and stable measurements beat the noise threshold.
- `INCONCLUSIVE` — The difference is too small, noisy, unrepresentative, or correctness is unverified.
- `REGRESSION` — The after measurement is materially slower or correctness fails.
- `UNMEASURED` — No credible benchmark could be constructed within scope.

Never claim a speedup from Big-O reasoning, code appearance, one run, or an estimated percentage. Complexity analysis may explain an observed result; it cannot replace measurement.

## Create the native report

For a completed `prove` or `regression` run, create:

- `outputs/speedup-proof-results.json` — machine-readable benchmark and comparison evidence.

From henryvn27/agent-skills, MIT.

More from henryvn27/agent-skills