axp-build-loop

Files (8)

• README.md
• SKILL.md
• README.md · eval/
• experiment.yaml · eval/
• axp · eval/fixtures/
• greet.js · eval/fixtures/greet/
• package.json · eval/fixtures/greet/
• metadata.json

SKILL.md (first lines)

# AXP Build Loop

Drive the experiment build loop end-to-end from a coding-agent chat session:
take a goal or insight, generate N candidate variants of the artifact under
test, implement them in parallel worktrees, evaluate them in isolated axp
sandboxes against a naive agent, rank the results, and either promote the
winner to a PR or run another lap mutating it.

Six stages, run sequentially: **INTAKE → GENERATE → IMPLEMENT → RUN → CHOOSE → LAP or PROMOTE**.
Complete each stage before the next. Conversation context is working state; the
`LOOP.md` journal (Stage 3) is durable state — keep it current so a fresh
session can resume the loop cold.

## Requirements

- `axp` CLI with `files:` staging support (`axp run --help` must list `--file`); Docker running. File staging is **local Docker runs only** — `--remote` and `--runner e2b` reject it.
- `git` ≥ 2.20 (worktrees), `jq`, and `gh` (only needed for the final promote stage).
- Run from the repository of the artifact under test.

## Command safety

**Run freely:** `axp validate`, `axp run --dry-run`, `axp schema`, `git worktree list`, `git status`, `git log`, `git diff`, reading any file under `.axp/`.

**Gate behind explicit user approval (use AskUserQuestion):**

- `axp run` (real run — spends tokens and time; show variant count × repeat and the estimated agent invocations)
- `git worktree add` / branch creation
- `git push`, `gh pr create` (Stage 6 — outward-facing)

Never write to the user's working tree outside `.axp/loops/` and the variant worktrees.

## Hard conventions (apply to every experiment this skill authors)

1. **Naive-user prompt.** `flow.prompt` is a realistic end-user task ("set up logging with this CLI"), never test-aware ("call the foo tool"). A test-aware prompt silently invalidates the experiment.
2. **Y₀ control.** Every matrix includes an unmodified-baseline variant. All claims are relative to Y₀.
3. **Repeat by default.** Agent behavior is stochastic. Default `--repeat 5` (ask the user to lower it only for cost reasons) and report rates ("tool fired 4/5"), not booleans.
4. **Identical mocks below the varied layer.** The surface the test agent touches (help text, errors, tool descriptions, docs, schema messages) must be real; anything beneath it (network, databases) may be stubbed — but with the *same* stub in every variant, with realistic exit codes and error text.
5. **One axis per lap.** Each lap varies one mutation axis from the playbook; the hypothesis is written down before the run.

## Per-artifact playbook

Branch on the artifact type at intake. Each variant's `setup` installs its build; mutation axes and metrics guide GENERATE and CHOOSE.

| Artifact | Mutation axes (pick ONE per lap) | Key introspection metrics |
|---|---|---|
| CLI | `--help` text; error-message wording; flag names; JSON vs text output | wrong-flag attempts, retries after error, help re-reads |
| MCP server | tool names; tool descriptions; granularity (one fat tool vs several thin) | activation rate (did the agent pick the tool at all), schema-validation failures, fallback to bash |
| Agent hooks | trigger condition; injected message wording; hook event choice | behavior delta post-hook, false-trigger rate |
| Install script | error handling; preflight checks; idempotency; verbosity | exit codes on fresh sandboxes, agent's recovery path on failure |
| Docs | structure (task-oriented vs reference); placement (README / llms.txt / agent config); length | task success with docs only, resort-to-reading-source, time to first correct call |
| Schema | field names; defaults; error message quality | validate-failure count before first valid instance, hallucinated fields |

---

## Stage 1 — INTAKE

Goal: pin down what is under test and anchor the baseline.

1. Identify the artifact type (table above) from the user's goal; ask if ambiguous.
2. Fingerprint the cwd against it: binary name in `Cargo.toml` / `package.json` `bin`, MCP server entrypoint, docs dir. If the repo doesn't contain the stated artifact, stop and ask.
3. Capture the baseline: `git rev-parse --short HEAD`, dirty-tree status. A dirty tree needs an explicit answer: is the dirty state or the committed state the baseline?
4. Pick a loop name (kebab-case, e.g. `mycli-help-text`) and create the loop home: `.axp/loops/<name>/` (add it to `.gitignore` if not covered).
5. Confirm in one gate (AskUserQuestion): artifact, baseline commit, loop name, run budget (variants × repeat).

## Stage 2 — GENERATE

Goal: a falsifiable variant plan, approved before any implementation.

1. Pick ONE mutation axis from the playbook for this lap. State the hypothesis in one sentence ("more example-driven --help reduces wrong-flag attempts").
2. Propose Y₁–Y₄ concrete mutations along that axis, plus Y₀ control. 3–5 candidates total is the sweet spot; more dilutes the repeat budget.
3. Design the evaluation:
   - `flow.prompt`: the naive-user task (convention 1).
   - Application tests: did the task outcome land (files exist, command exits 0, endpoint answers).
   - Introspection tests: read `$AXP_TRACE_PATH` with `jq` for the playbook metrics — e.g. `[ "$(jq '.tool_calls | length' "$AXP_TRACE_PATH")" -lt 30 ]`. Inspect the trace from a previous run (or lap 1's Y₀) to learn its exact fields before writing fine-grained assertions; do not invent fields.
4. Present the plan as a table (variant id, mutation, predicted effect) and gate on user OK.

## Stage 3 — IMPLEMENT

Goal: each variant's artifact built and staged; experiment YAML validated.

1. For each Yᵢ (i ≥ 1): `git worktree add .axp/loops/<name>/worktrees/y<i> <baseline>` (one approval covers all worktrees of the lap), implement the mutation there, and build the artifact. Independent mutations may be implemented by parallel subagents — one per worktree.
   - **The staged build must be usable inside the sandbox (Linux container).** Interpreted artifacts (Node/Python scripts, docs, YAML/JSON) are typically portable as-is. Compiled binaries must target Linux on the sandbox's architecture — a host macOS binary will fail at `setup` with exec-format/not-found errors. Either cross-compile (e.g. `cargo build --target x86_64-unknown-linux-gnu`, `GOOS=linux go build`) or skip host builds entirely: stage the worktree *source* and compile in that variant's `setup` (slower per variant, but always matches the sandbox). Verify the toolchain assumption early — Y₀'s `setup` failing on the baseline build means every variant will fail the same way.
2. Author `.axp/loops/<name>/lap<N>.yaml`. If an experiment-authoring skill is available in the host environment (e.g. `axp-create-experiment`), use it; otherwise follow `axp schema` and the docs at https://docs.514.ai. Per variant:
   - `overrides.files` staging the variant's build, with `source` **relative to the YAML's directory** (e.g. `./worktrees/y1/target/release/mycli` → `dest: tools/mycli`); Y₀ stages the baseline build the same way.
   - `overrides.setup` for any in-sandbox install steps. Files land **before** `setup`, so setup may reference them.
   - Add an `.axpignore` (gitignore syntax) inside any staged *directory* to keep junk out; `.git/` is excluded automatically.
3. `axp validate lap<N>.yaml`, then `axp run --dry-run` and check the staging table for `MISSING`/`UNBOUND` rows.
4. Write or update `LOOP.md` (format below) with the lap's hypothesis and variant table.

### LOOP.md format

```markdown
# Loop: <name>
Goal: <the user's goal / originating insight>
Baseline: <commit> (<clean|dirty: which>)

## Lap <N> — axis: <mutation axis>
Hypothesis: <one sentence>
| Variant | Mutation | Result (filled after CHOOSE) |
|---|---|---|
| y0 | baseline control | |
| y1 | <mutation> | |
Run: <run id> · Decision: <winner y_k / another lap / stopped> — <user's words>
```

## Stage 4 — RUN

Goal: the lap's evaluations executed without babysitting.

1. Gate: show the final command and the cost shape (`N variants × R repeats = N×R agent runs`), get approval.
2. Run `axp run lap<N>.yaml --repeat <R> --jobs <J>` **in the background**; variants already parallelize inside one run, so `--jobs` ≥ variant count.
3. While it runs, monitor the newest `.axp/runs/<run_id>/` — per-variant `run.json` appears as variants finish. Report progress; on `setup_check_failed` or `staging_failed`, stop the lap and diagnose rather than letting the matrix burn budget.

## Stage 5 — CHOOSE

Goal: a ranked, evidence-backed comparison the user decides on.

1. Collect per-variant evidence from `.axp/runs/<run_id>/variants/*/`: `run.json` (status, exit reason, cost/usage fields), `tests/application/*.json` and `tests/introspection/*.json` (pass/fail + tails). With platform auth, `axp experiment run results get` aggregates the same; without it, the artifacts suffice.
2. Rank: (a) application-test pass **rate** across repeats — an unreliable winner is not a winner; (b) playbook introspection metrics; (c) cost/wall-clock as tiebreak. Everything is reported **relative to Y₀**.

From 514-labs/agent-skills, Apache-2.0.

More from 514-labs/agent-skills