agent-benchmark

Files (5)

• SKILL.md
• artifacts-post.mjs · scripts/
• artifacts.mjs · scripts/
• check-fixture.mjs · scripts/
• test-artifacts.mjs · scripts/

SKILL.md (first lines)

# Agent benchmark

Read `<selected-repo>/docs/AGENT-BENCHMARK.md` and
`docs/AGENT-BENCHMARK-FOUNDATIONS.md` completely, not relative to an installed skill
root. STAGE A is model-free preparation, not a completed four-arm comparison.
The final binding fixture is `fixtures/benchmarks/binding-discovery/v1/`; `solver/`
is the only committed solver input. Never export manifest/private evidence/old
candidate, known repair, hidden oracle or solved Git history. Task/runtime sources
are separate; no automatic memory injection or instruction trimming.
The confirmed V2 target is Node-free Rust/Wasmtime-hosted QuickJS-WASI. V1 Node
hosting is comparison only, not a production backend/fallback. Prove V2 execution
with Node unavailable before claiming implementation; the decision is not readiness.


## Explicit all-four forced-code-mode parallel scope

For this newly authorized adapted experiment, read the selected repository's
`docs/AGENT-BENCHMARK-FORCED.md`. Use only `--pilot-forced-qualify`,
`--pilot-forced-negative`, `--pilot-forced-prepare` and the coordinator-owned
`--pilot-forced-run-approved` path with a fresh concurrent scope/manifest. The old
serial and Pi-only defaults do not become parallel or forced. Pi/historical are
explicitly Node-VM-adapted treatments, not stock controls; V1 is Node and V2 stays
native Rust/Wasmtime without a Node VM bridge/fallback. Verify trusted host/nested
execution journals and source-bound sealed archives, not model/backend labels.
Zero-use, bypass or incomplete evidence cannot rank as verified treatment.
All actors and writers settle/drain before capture/grading/sealing. Unknown tool
outcome fences every arm's new nested admission; unknown usage remains arm-local.
Timed parallel results are CPU/provider-contention affected, not clean latency
rankings. Preserve all old attempts/unknowns and Pi's lossless raw capture. Never
repair the frozen task/oracle, probe credentials or publish UI as a task worker.

## Pi-only capture repair (current narrow continuation)

For “fix output limit, rerun pi”, read the Pi-only section of the selected repo's
`docs/AGENT-BENCHMARK-PILOT.md`. The consumed four-arm attempt is immutable evidence;
never reuse its reservation or infer new authority from its receipt. The new
`--pilot-pi-qualify/--pilot-pi-prepare/--pilot-pi-run-approved` path reserves one fresh
Pi-only attempt with the same gpt-6.1-sol LOW/per-arm controls. Only the coordinator
executes it after recording the actual new human message reference. Model-free
qualification must match current execution-source/artifact hashes. Private raw Pi
journals are lossless and fsynced; bounded console metadata is not the event stream.
Keep partial/failing evidence, unknown usage, full messages/sessions and native
settlement. No production caps, task/instruction bytes or oracle change.

## Admission and coordination

The legacy paid/UI entry of `scripts/agent-benchmark.mjs` remains refused. The new
explicit pilot entry requires verified current-source qualification and exact human
limits approval; read `<repo>/docs/AGENT-BENCHMARK-PILOT.md` completely before using it.
Do not work around this with external solver scripts, alternate homes, insecure
routes or live node execution. A human must authorize the later paid experiment
and concrete pins/caps must be recorded first. Preparation assistant inference is
real spend; 'no inference' must mean no **benchmark solver inference**, not no model
work by the task owner. Missing/unpriced preparation dollars remain unknown.

Direct preparation uses the coordinator's native-owned operation, not coding
workers. Ordinary delegation still requires explicit orchestration enablement;
the later four contestant processes need their own narrow experiment authorization.
No recursion, hidden coding agents, alternate homes or failed-admission workaround.
Native task owners self-review, run focused checks, commit with provenance and use
ordinary integrate/remote readback; only coordinator owns promotion/installation/UI.
Full release gates require an explicit request. When orchestration is off the active
direct-workflow rule forbids subagents/external inference agents; benchmark
preparation does not silently lift it. Parallelize independent hermetic preparation
and tests only. Timed arms remain isolated, serial/counterbalanced by default:
provider/CPU sharing confounds a concurrent comparison.

Before any later paid launch, separately pin task source, runtime source, package,
binary, image, tools, instructions, model, reasoning, provider route and oracle.
Prove a credential-isolated shared route and complete provider-attempt/prepared-wire
capture for all arms, including summaries/retries/failures, not just final messages.
The proposed first pilot is one attempt per Pi/current wasm/Node QuickJS-WASI/Rust
Wasmtime QuickJS-WASI arm,45min each, gpt-6.1-sol LOW only if the same isolated route
is proved. The exact proposal is120dispatches/20M reported input/120k output per
arm, with one-response token overshoot and unknown subscription dollars/quota.
Approval is still missing; never convert a proposal or metadata flag into launch.
These are experiment controls, never production coding-loop caps. Canonical
architecture decision496de546 confirms V2 must be Node-free Rust/Wasmtime: no Node
process, bridge or fallback for code-mode execution. V1 Node hosting is comparison
only; benchmark/test tooling and Pi are separate. See `<repo>/docs/CODEMODE.md`
for the implemented experimental ABI and Node-unavailable source/embedded proof.
The direct isolated pilot now connects those components; actual provider acceptance
and a completed comparison remain unproven. Use this verified CLI sequence:

```
node <repo>/scripts/agent-benchmark.mjs --pilot-qualify <artifact-config> <fresh-runtime-proof>
node <repo>/scripts/test-agent-benchmark-pilot-grade.mjs <fresh-grade-proof> <exact-tool-image>
node <repo>/scripts/check-agent-benchmark-pilot-proof.mjs <artifact-config> <runtime-proof> <grade-proof>
node <repo>/scripts/agent-benchmark.mjs --pilot-prepare <preparation-config> <fresh-plan>
```

The `benchmark-pilot-qualification-proof` spell records the first CLI plus the
retained/current-source postcheck against an already-qualified grader. Required
worktrees use the verified direct CLI; do not retry unsupported spell replay.
See the pilot doc for source/artifact/approval pins, original task instruction
bytes, persistent volume snapshots, private artifact storage and exact risk/limits.
The cached tool image's critical embedded Lua is matched to actual pre-task Git
content, not inferred from its tag; full verification-binary build source is unknown.
No live provider call or production configuration change occurs in preparation.

For native engine qualification, the `codemode-native-proof` spell records
`node <repo>/scripts/test-codemode-native.mjs <binary> <fresh-evidence> [assets]`
then its `--post` source/binary/log hash check. Required worktrees use the verified
CLI, not unsupported spell replay. Embed pinned assets at build time; Node is
absent during native execution. Also run the upstream VM's benchmark-only
`test-codemode-reference.mjs <prepared-assets> <fresh-evidence>` and binding/envelope
regressions. The runtime safety limits and host differences must be disclosed;
these model-free proofs never grant paid launch or certify quality/cache/billing.

## Model-free fixture and hidden oracle

Use the existing runner substrate:

```
node <repo>/scripts/agent-benchmark.mjs --binding-export <fresh-private-output>
node <repo>/scripts/agent-benchmark.mjs --binding-preflight <pinned-native-binary> <fresh-private-evidence>
node <repo>/scripts/agent-benchmark.mjs --binding-grade <pinned-native-binary> <candidate-source> <fresh-private-evidence>
```

From Magaav/wasm-agent, MIT.

More from Magaav/wasm-agent