AGENTS.md · experientiallabs/experiential

Sections

1. Toolchain
2. Repository checks
3. CLI package ownership
4. Evidence, simulation, and routing lifecycle
5. Worker-agent execution
6. Optimization surfaces
7. Python
8. Writing
9. Rules
10. One package
11. Docs

First lines (shown under the repo's Apache-2.0 license)

# Agent guide — experiential

Experiential builds immutable task evidence from agent traces, composes and fits frozen model routers,
runs those routers on loopback, and executes bounded SFT from persisted datasets. All importable
code lives under `exp/`; benchmark data arrives as a dependency (see rule 6).

## Product and repository boundaries

Make open-source Experiential excellent for individual users. Keep enterprise operations,
organization management, billing, and deployment orchestration in other repos. Enterprise
builders can build that layer themselves, but this boundary must never compromise our
gateway's performance, security, or maintainability.

Public code, documentation, commit messages, PR descriptions, and comments must describe the
standalone open-source behavior. Keep private repository links, internal deployment details,
customer information, and private operational or experiment evidence in private repos or local
records. Limit disclosure cleanup to sensitive cases; preserve legitimate public interfaces
and ordinary technical terminology. Keep shared interfaces small, performant, and reliable.

## Toolchain

Managed with `uv`; lint/format with `ruff`; type-check with `ty`.

```bash
uv sync --extra dev
uv run ruff check .
uv run ruff format --check .
uv run ty check
uv run pytest -q
```

## Repository checks

- Every new or rewritten hand-authored source, configuration, and documentation file with a
  covered suffix stays below 1,000 physical lines. The executable limit is 999 lines and counts
  comments and blank lines. Test modules named `*_test.py` are exempt so cohesive tests are not
  split solely to satisfy a line count. Generated lock files are excluded. Generated code belongs
  in an explicitly named `generated/` directory and is never edited by hand.
- Full-repository Ruff check, Ruff format check, and ty check are required on every change. No
  pre-existing lint or type failures are grandfathered.
- Production imports follow the approved dependency direction: common may not import runtime,
  simulation, optimize, or cli; runtime may not import simulation, optimize, or cli; simulation
  may not import optimize or cli; optimize may not import cli. Optimize owns application
  orchestration and may depend inward on common, runtime, and simulation. The AST gate rejects
  every current forbidden edge directly and proves that the package graph is acyclic.
- The root CLI command set is exact: `build`, `capture`, `config`, `eval`, `login`, `optimize`, and `run`.
  An invocation without a subcommand opens the default gateway home screen.
  `exp/cli/app_test.py` and the release tests
  enforce the current command and distribution shape.

## CLI package ownership

- `exp/cli/app.py` owns root command composition only. Command implementations live in the
  `build/`, `capture/`, `config/`, `evaluation/`, `judge/`, `optimize/`, and `gateway/` packages. Gateway serving and the
  default home screen live under `gateway/`.
- `exp/cli/providers/` owns provider discovery, model selection, and catalog setup shared by
  commands. Command-specific orchestration stays with its command package. In particular,
  router-candidate collection belongs to `exp/cli/optimize/`.
- `exp/cli/shared/` owns reusable terminal, Typer, consent, picker, and progress primitives. It
  must not import command packages. `exp/cli/providers/` may import `shared/`, but it must not
  import command packages.
- Keep the `exp/cli/` root closed to new command implementation modules. Package-wide CLI tests
  may live in `exp/cli/tests/`; module tests stay beside the module they cover.

## Evidence, simulation, and routing lifecycle

- `exp/common/traces/ingest/` owns reusable source parsing, trace normalization, provenance,
  streaming and import persistence. Gateway-specific capture/protocol adapters live under
  `exp/runtime/gateway/ingest/` and supply canonical evidence to the shared ingestion layer.
- `exp/simulation/` owns representative-task mining, typed simulation specs, current engines,
  orchestration, artifact construction, and comparisons. New modules for those responsibilities
  go inside `exp/simulation/`, never at the flat `exp/` root.
- `exp build PROJECT --traces TRACE_FILE --source SOURCE --root ROOT` imports canonical traces
  into shared `<root>/gateway/traffic.db` SQLite storage, then mines scenarios from that exact
  saved evidence. `--source gateway --identity ID` selects retained captures for one explicit
  identity. Imports preserve source provenance, prompts, tools, results, normalization exclusions
  and model identity; immutable import IDs and project associations are transactional and
  idempotent. Ingestion remains a common Python operation, not a separate CLI command.
  Build writes manifest-bound fit and held-out tasks plus
  `proposals_pending` review state, builds both RAG indexes with cost-aware spend consent,
  and binds the grounded world model without a completion or judge call. Route each corpus
  through an explicit canonical source loader. The interactive wizard defaults to provider
  configuration and build preparation; router optimization requires an explicit selection.
- New file trace sources belong in `exp/common/traces/ingest/`, normalize into the `Trace` and
  `TraceSpan` contracts in `exp/common/traces/`, and register with the explicit shared source table.
  Runtime-specific source adapters stay with their runtime owner. Common ingestion never imports
  runtime or simulation code; source selection is composed by the consuming command.
- Python applications use `exp.compose_router` to complete review, plan-bound simulation,
  judgment, fitting, held-out verification, reporting, and runtime loading. Callers inject the
  approved review and setup suppliers, simulator factory, judge, runtime catalog, and finite
  simulation-dollar and judgment-call ceilings. Preserve its phase boundary: held-out evidence
  opens only after fit evidence, policy locking, and remaining-budget checks pass. Router fitting
  never runs or consumes world-model fidelity evaluation.
- `exp optimize router PROJECT --root ROOT` discovers the completed build, fit-only RAG, grounded
  world model, judge syllabus and provenance, and confirmed router candidates from the project.
  Human calibration is recommended but optional. The command freezes one shared provider ceiling
  before calls, simulates and judges missing fit evidence, locks the fit policy before held-out
  execution, and exactly replays completed work.
  World-model fidelity testing is a separately invoked common-evaluation mode with no authority
  over router fitting or runtime activation. Its reports contain measurements only and never carry
  an approval, denial, gate, threshold, or decision.
- `exp` opens the branded home screen. Its `Run Gateway` choice starts the initialized
  authenticated multi-alias gateway on loopback, running first-time setup when no gateway exists,
  while `exp --project PROJECT --root ROOT --port
  PORT [--ghost]` retains the single-project compatibility server. Both expose OpenAI Chat
  Completions, Responses, and Models routes. Public request and
  response types come
  from the official OpenAI SDK. Chat retries use the standard `Idempotency-Key`; Responses
  continuations use `previous_response_id`. Experiential never joins unrelated Chat callers by transcript
  prefix and requires no proprietary request fields or headers. Durable journaling is the default;
  ghost mode performs routed calls without saving traffic or replay state. Request-time embedding
  failure uses the frozen conservative baseline, and neither path mutates policy or evidence.

## Worker-agent execution

- Agent execution code lives under `exp/runtime/`: whole-episode customer agents, executable
  environments, model clients, and frozen router execution. Optimization may depend on runtime;
  runtime code must not depend on simulation or optimization algorithms.
- No-subcommand `exp` serves only explicit active gateway aliases. The `--project` form serves one
  frozen router policy. Simulation callers choose an `AgentRuntime` and `EnvironmentRuntime`
  directly, in process.
- The multi-alias worker's catalog build is per-alias fail-safe: a granted alias that cannot build
  (bad rung, missing revision, malformed capabilities, or a rung with no native wire dialect) is
  excluded and marked UNAVAILABLE (surfaced in the startup receipt and the `catalog.aliases_excluded`
  log; a direct call to it 503s), and the worker binds and serves every other alias. No single
  catalog row may abort the whole build or crash the worker; only an entirely empty servable set
  fails startup loud. This is EXCLUSION, never a fallback — the RUST-only serving contract holds, so
  an unservable alias is dropped, never served through some other path. The single-alias owned
  project gateway (`--project`) keeps the all-or-nothing `native_serving_blockers` gate, since its
  one alias has nothing else to serve.
- Local Pi and process-environment adapters execute external code on the user's machine only when
  a caller explicitly selects them. Preserve bounded processes, the explicit working directory,
  and fail-closed support checks.
- Customer agents implement the whole-episode `AgentRuntime` contract and receive only an injected
  candidate model plus an execute-only `EnvironmentSession`. The built-in Pi adapter invokes an
  externally installed executable.
- Executable environments implement the lifecycle-owning `EnvironmentRuntime` contract. Local and
  injected remote backends preserve exact resource identity, bounded execution, usage metering,
  and fail-closed cleanup evidence. A remote adapter declares and implements its own finite close
  primitive before use, and all of its cleanup runs inside that primitive. The sandbox ledger
  releases an exact ID only after the adapter positively proves cleanup.

## Optimization surfaces

- Customer agent execution lives only in `exp/runtime/agents/`, executable environments live only
  in `exp/runtime/environments/`, and sandbox simulation lives only in
  `exp/simulation/engines/sandbox.py`. Agent-search and benchmark-scoring work belongs to the
  private `agent-optimization` repo; send it there.
- `exp/optimize/router/` owns provider-free offline fit, policy locking, held-out reporting,
  application composition, and their immutable artifacts. Online selection and provider execution
  belong to `exp/runtime/router/`. Keep those two boundaries explicit.
- Router application entrypoints stay in `composition.py` and `activation.py`. Automatic evidence
  orchestration lives in `automatic/`, manual judge calibration in `judging/`, offline policy work
  in `fit/`, and evaluation preparation in `evaluation/`. The durable judgment ledger remains at
  `judgment_budget.py`.
- The root CLI is locked to `build`, `capture`, `config`, `eval`, `login`, `optimize`, and `run`. Capture runs
  in the foreground with no management subcommands. The optimize group is locked
  to `router` and `model`; the config group is locked to `budget`, `gateway`, `judge`, `providers`,
  and `telemetry`. Widening any of those three sets, whether with a command, an alias, or a flag, is a
  deliberate change to the locked surface and needs the same scrutiny as a public API change.