smoke-test

Files (2)

• SKILL.md
• openai.yaml · agents/

SKILL.md (first lines)

1) Resolve the smoke command.
   - Read `codex.toml` and use `[commands].smoke` when present.
   - If absent, use `uv run python scripts/smoke.py` when `scripts/smoke.py` exists.
   - If neither exists, fail as `config_error` and report the missing command/script.
2) Scope smoke to the requested work item when provided.
   - Prefer `--work-item <work-id>` for `scripts/smoke.py`.
   - Treat `--feature` as a deprecated compatibility alias only.
3) Run smoke using the real entrypoint path.
   - Smoke must execute a subprocess command path (CLI/module/script), not only import-level calls.
   - Use tiny fixture inputs under `tests/fixtures/smoke/` when available.
   - Pass the selected mode to the child as `SMOKE_MODE`; offline mode must not make provider network calls.
   - Select live mode only when credentials match a selected scenario provider, unless explicitly forced.
4) Enforce required artifact contract under `artifacts/smoke/<run-id>/` (or equivalent configured directory):
   - `summary.json`
   - `cases/<case_id>.json`
   - `stdout.txt`
   - `stderr.txt`
   - `timing.json`
5) Validate outputs.
   - Always validate the configured JSON Schema, not merely JSON shape.
   - Always evaluate configured declarative invariants (required paths, equality, membership, and numeric ranges).
   - Golden comparison is optional per scenario and must support explicit `--record` updates.
   - Require child usage telemetry when token or cost ceilings are configured, and fail closed when a configured budget cannot be verified.
6) Summarize every run:
   - command run
   - work item ID (`work_item`) when provided
   - fixture(s)/scenario(s)
   - exit code
   - artifact locations
   - wall-clock time
   - budget telemetry (`requests_used`, `time_budget_exceeded`, max request/time settings)
   - source identity (Git commit/dirty diff hash, registry hash, fixture hashes, generated time)
   - failure bucket counts
7) Classify failures into one primary bucket per case/run summary:
   - `config_error`, `fixture_error`, `auth`, `permission_error`, `network_error`, `timeout`, `rate_limit`, `quota_exceeded`, `provider_error`, `invalid_response`, `parse_error`, `schema_error`, `invariant_error`, `golden_mismatch`, `budget_exceeded`, `artifact_error`, `subprocess_error`, `internal_error`
8) Use this exit-code mapping:
   - `0` pass
   - `10` config/fixture/auth/permission failure
   - `12` artifact failure
   - `20` subprocess failure
   - `22` network failure
   - `23` timeout
   - `24` rate limit
   - `25` quota exceeded
   - `26` provider error
   - `30` invalid/parse error
   - `40` schema validation failure
   - `41` invariants failure
   - `50` golden mismatch
   - `55` budget exceeded
   - `60` internal/unclassified
9) If multiple failures occur, prioritize exit code in this order:
   - artifact -> config/fixture/auth/permission -> quota -> rate_limit -> network -> timeout -> provider -> subprocess -> invalid/parse -> schema -> invariant -> golden -> budget -> internal
10) Recommend next action:
   - deterministic smoke/runtime/packaging issues -> `debug-loop`
   - fixture/golden updates only when behavior change is intentional and documented

From rafaie/codex_home, Apache-2.0.

More from rafaie/codex_home