AGENTS.md · tester-army/e2e

Sections

1. Contracts
2. Layout
3. Commands
4. Definition of done
5. Non-obvious conventions
6. Testing quirks
7. Inspecting agent runs with unbox-ai
8. Cross-checking the web engine's tree
9. Measuring a change against main
10. Golden device trees
11. Gotchas

First lines (shown under the repo's Apache-2.0 license)

# AGENTS.md

`e2e` — an agentic end-to-end testing framework. pnpm monorepo, ESM only,
TypeScript 7.

## Contracts

There is no separate spec. The code is the contract, pinned in three places:

- The emitted `packages/e2e/dist/index.d.ts` (and `dist/engine/index.d.ts`,
  `dist/runner.d.ts`, `dist/oauth/*.d.ts`)
  is the public API. `packages/e2e/tests/types/sdk-types.ts` holds compile-time
  assertions (`@ts-expect-error` lines) for the parts that are easy to loosen
  by accident; it runs under the package `typecheck`, never under vitest.
- Wire formats live in `packages/e2e/schema/*.schema.json` (report-1,
  session-1, agent-judgment-2, and the deprecated agent-judgment-1) with a valid and an invalid fixture
  each. Integration tests validate every generated report and session envelope
  against them; `tests/unit/schema-fixtures.test.ts` checks the fixtures. A
  wire change edits the schema, both fixtures, and the producer in one review.
- Security invariants are the list under Gotchas below, enforced by tests in
  `tests/integration/agent-policy.test.ts` and the secret-ledger unit tests.
- Behavior changes update the matching `docs/**/*.mdx` page in the same
  change, including "not implemented yet" callouts, and `skills/e2e/` when the
  changed surface is described there. `scripts/check-error-codes.ts` (in
  `pnpm check`) fails when an error code in source is missing from
  `reference/errors.mdx` or `reference/engine.mdx`, or documented but raised
  nowhere.

There are no RFCs or design documents in the repo. The why lives in PR
descriptions and commit bodies; `git log` and `gh pr view` are the archive.
## Layout

`packages/` holds what publishes to npm; `apps/` holds the private apps and
suites that consume the built packages the way a user would.

- `packages/e2e` — the published `e2e` package: SDK surface, runner, CLI,
  `e2e/engine` contract. Core knows the contract and never an engine's
  internals: no engine package is imported from `src/` (grep for
  `from '@e2e-dev/` and `import('@e2e-dev/`; the presets below are the only
  hits). Engine words still appear as platform labels, capability names, and in
  error text that names a factory, such as `config/targets.ts` telling you to
  call `web()`; no path through them reaches engine code. The one exception is the
  `e2e init` scaffold presets in `src/cli/init/engines.ts`, which write the
  user's config and so name engine packages as text; the package build records
  the sibling packages' versions in `dist/cli/init/sibling-versions.json` for the
  ranges they write. Each preset owns its prompt label, dependencies, config,
  example, and run command; interactive choices derive from this list. These
  presets never import engine implementations.
  - `src/run/` runner core (scheduler, units, workers, retries, sessions;
    `standalone.ts` is the worker a host or MCP's one-shot attempt keeps
    alive, `standalone-attempt.ts` one attempt on it: reservation, launch,
    fixtures, deadline, close, `host-trace.ts` its `trace.md`, and
    `select-target.ts` the target a session names, shared with MCP;
    `open-session.ts` exposes `openHost` on `e2e/runner` for another test
    runner: worker-lifetime provisioning, app processes, and engine init,
    then `host.openSession({ name })` for each test's fixtures, deadline,
    steps, artifacts, and trace at close. Only one hosted attempt may be
    opening, live, or closing per process; close it before opening another),
    `src/collect/` registration+selection, `src/locator/` locator AST/engine,
    `src/agent/` the agent (the `act` executor socket plus the judgment
    methods), `src/mcp/` the `e2e mcp` server (a live session that rides
    the `act` socket with a queue executor so every MCP call is a harness
    action), `src/oauth/` subscription sign-in (the `e2e login`, `logout`,
    and `models` commands and the `e2e/oauth/*` model constructors; each
    constructor subpath is the place its `@ai-sdk/*` optional peers are
    imported, and the Copilot constructor loads `@ai-sdk/openai` lazily, only
    for the Responses models, so the CLI boots without any of them). The
    constructors and the CLI
    are the whole public surface: the flows, stores, and fetch behind them
    are module-private, not a library for other products. `tests/live/` holds hand-run
    checks that need a stored login and are never part of `pnpm test`.
- `packages/web` — the published `@e2e-dev/web` package: the
  browser engine, built with the public `defineEngine`, contributing the
  `browser` fixture and `expect(browser)`. It depends on `e2e` (peer), never the
  reverse; a target names it explicitly as `engine: web()`. There is
  no default engine and no well-known id registry in core. It imports from
  `e2e/engine` only: the semantics every engine must reproduce
  (error taxonomy, text and URL matching, assertion polling, JSON-value rules)
  are exported there, and there is no `e2e/internal` subpath.
- `packages/mobile` — the published `@e2e-dev/mobile` package: the
  iOS/Android engine on agent-device, built with the same public
  `defineEngine`, contributing the `device` fixture. Like the web engine it
  depends on `e2e` (peer), never the reverse; the engine implementation
  imports from `e2e/engine` only, and a target names it as `engine: mobile()`.
  The `@e2e-dev/mobile/tools` subpath holds the agent-side `open_app`, `swipe`,
  and `alert` tools; focused-field typing uses the engine's keyboard grammar.
  The main entry never loads the AI SDK.
- `packages/kernel` - the published `@e2e-dev/kernel` package: Kernel hosted
  browsers for the web engine. An official integration with a hosted service
  is one package per service, named after it (`@e2e-dev/<service>`), with the
  vendor SDK and the engine it plugs into as peers. It implements that
  engine's provider seam (`BrowserProvider` for web, `DeviceProvider` for
  mobile) and imports the engine's types only; the engines never know it
  exists. A service with providers for both engines ships them in that one
  package, each engine an optional peer and each provider on its own entry
  that never imports the other engine (`@e2e-dev/testmu` and
  `@e2e-dev/testmu/web`).
  Its docs page under `docs/integrations/` ships with the vendor's logo as
  the sidebar icon and on its overview card (`authoring-docs`, "Page
  icons"); `scripts/check-docs-logos.ts` fails `pnpm docs:check` without
  them.
- `packages/lightpanda` - the published `@e2e-dev/lightpanda` package: a
  `BrowserProvider` that starts a local `lightpanda serve` per worker slot
  and stops it with the run; no vendor SDK, the binary is the dependency.
- `packages/eas` - the published `@e2e-dev/eas` package: EAS Simulators
  hosted iOS simulators and Android emulators for the mobile engine
  (`DeviceProvider`). Expo publishes no SDK for the sessions API, so it calls
  Expo's GraphQL API with `fetch`, and `@e2e-dev/mobile` is its only peer.
- `packages/smol` - the published `@e2e-dev/smol` package: Chromium in
  smol machines microVMs on the runner's own computer for the web engine
  (`BrowserProvider`), through the `smolmachines` SDK's embedded engine
  (peer). Each worker slot boots one warm browser machine from the pinned
  Playwright Ubuntu image; a Node TCP relay exposes DevTools, and SDK agent
  readiness allows machine startup without waiting for published ports. In
  `attempt` scope every attempt runs in a copy-on-write branch. With `app`,
  the app under test runs in the same machine, so a branch also copies its
  running app and data on the machine. When maintaining this integration,
  update the provider's SDK dev dependency to a published version and raise
  its peer minimum only if the provider needs new SDK behavior. Keep the
  Playwright image and Chromium version in sync with `@e2e-dev/web`. Run
  `pnpm --filter @e2e-dev/smol test:unit` and `pnpm check`, then verify live
  machines in both attempt and worker scopes with an app running in the VM.
- `packages/limrun` — the Limrun hosted mobile device provider. Each worker
  leases one remote device, installs its app, and releases the device at exit.
- `packages/decision` — the published `@e2e-dev/decision` package: a
  `StepExecutor` (`decisionExecutor()`) that drives `agent.act` and
  `agent.assert` through an AI SDK *decision* model answering `choice`
  questions with probability distributions, plus an optional small language
  model that writes field values when the decision model picks `type`.
  `minProbability` and `minConfidence` gate a chosen operation, target,
  secret, or assertion verdict. It also exports `decisionFilter()`, a
  `TestFilter` that asks a decision model whether the pull request's diff
  can affect each test, packing test files into calls under the model's
  size limit, 8 calls at a time, and leaves out the unlikely ones. It runs
  every test on an `e2e.config` or lockfile change, and asks nothing about a
  test whose test file, setup file, or stored screenshots changed. A failed
  call leaves its tests in and the rest are still judged; after 3 failed
  calls it stops asking. When every call fails, or git or a file read
  fails, it throws, and the run warns and runs every test.
- `packages/github` — the published `@e2e-dev/github` package: the reporter
  that posts the run as one pull request comment from GitHub Actions and
  keeps it current on reruns, writing the same text to the job summary. It
  reads the token, event, and repository when the run finishes, never at
  config load, and renders the page with `renderMarkdownReport` from `e2e`.
- `packages/testmu` — the published `@e2e-dev/testmu` package: TestMu AI
  (formerly LambdaTest) hosted Android emulators, iOS simulators, and real
  devices for the mobile engine (`DeviceProvider`), through agent-device's
  `@agent-device/testmu` plugin, and hosted Chrome and Edge for the web
  engine (`BrowserProvider`) from `@e2e-dev/testmu/web`.
- `apps/testbed` (`@e2e-dev/testbed`, private) — dogfood project that
  consumes the **built** packages like a real user would: the playground app
  where every runner feature (sessions, routes, downloads, frames, uploads,
  serial groups, the executor seams, verdict edge cases, the reporter under
  stress, `explore`) has a deterministic test. Hard UI surfaces belong to the
  benchmarks, not here.
- `apps/web-benchmark` (`@e2e-dev/web-benchmark`, private) — a Next.js app of
  self-contained hard-surface scenarios (shadow DOM, canvas, iframes, native
  dialogs, planted bugs) at `/e/<slug>`, copied from the tester-army web
  benchmark, plus the e2e suites written against them (`tests/` and
  `tests-agent/` both gate PRs; the agentic one runs from its committed