write-system-test

Files (1)

• SKILL.md

SKILL.md (first lines)

# Write a system test

Each scenario has exactly one system test. The test starts the built module as a separate process and talks to
it only through its public surface.

## File and name

- Location: `modules/<name>/system-tests/`.
- One file per scenario, named after the ID: `SCN-PROOF-004.test.ts`.

## Example

```ts
import { scenario } from "@opensoftware/testkit";

scenario("SCN-PROOF-004", async ({ sut, fakes, given, when, then }) => {
  const forge = await fakes.forge();                 // Forgejo container, real API
  const pr = await forge.openPullRequest("demo");

  await given("a pull request with a Playwright trace", async () => {
    await sut.files.put("trace.zip", fakes.fixture("playwright-trace.zip"));
  });

  let run;
  await when('the developer runs "proof attach" for the pull request', async () => {
    run = await sut.cli(["attach", "--pr", pr.url, "--trace", "trace.zip"]);
  });

  await then("the pull request has a comment that links the evidence bundle", async () => {
    expect(run.exitCode).toBe(0);
    expect(await forge.comments(pr)).toContainEqual(expect.stringMatching(/evidence bundle/));
  });

  await then('the pull request has a check named "proof" with the result "success"', async () => {
    expect(await forge.check(pr, "proof")).toMatchObject({ conclusion: "success" });
  });
});
```

Every module is TypeScript (ADR 0009). A plugin written in another language, such as the Harbor agent plugin,
is tested with this same testkit through its CLI or API.

## Rules

- `sut` comes from the testkit. It runs the module's built artifact (CLI, server or container), never the source.
- Allowed: `sut.cli`, `sut.http` (generated client from `contracts/`), `sut.mcp`, `sut.events`, `sut.files`
  (only the folders the module documents as input or output), and `fakes.*`.
- Forbidden: imports from `src/`, database drivers, reading the module's data folder, environment variables that
  are not documented in the module's README, `sleep`, real time, real network, real models.
- Outside systems are fakes from the testkit: a scripted model gateway, a scripted ACP harness, fake Slack,
  a real Forgejo container as forge, PGlite in the test's process for Postgres (ADR 0049), containers for NATS
  and S3.
- Keep all state inside the scenario. Scenario files share a Vitest worker (ADR 0051), so a test never changes
  `process.env`, the working directory or a global of the test process.
- Keep the scenario cheap: each mutant of a line it reaches reruns it (ADR 0061). Set state up through `$INPUT`
  or a fake, start module processes only through `sut`, inside the scenario, and never wait for time to pass. The
  rules are under "Test speed" in `docs/quality/testing-strategy.md`.
- Every `given`, `when` and `then` label repeats the scenario's step text exactly. `And` steps use the keyword
  of the step before them.
- Each `then` must assert on something the module produced. An assertion on a value the test created itself
  does not count.
- Run the test before you implement: it must fail, and the failure must be about the missing behavior, not a
  setup error. Write the failure line into the pull request description.

From sesigl/open-software-factory, Apache-2.0.

More from sesigl/open-software-factory