cli-e2e-testcase-writer

Files (1)

• SKILL.md

SKILL.md (first lines)

# CLI E2E Testcase Writer

Work on one domain per run. Produce exactly two artifacts for that domain:
- workflow testcase files under `tests/cli_e2e/{domain}/`
- `tests/cli_e2e/{domain}/coverage.md`

Focus on domain testcase files. Do not change shared E2E support code such as `tests/cli_e2e/core.go` unless the user explicitly asks. Treat `tests/cli_e2e/demo/` as reference only.

## Core standard

- Make the testcase scenario-based and self-contained.
- Prove one workflow end to end: create plus follow-up read, or mutate plus teardown.
- Prefer one file per workflow or one closely related feature.
- For mutable flows, prove persisted state with read-after-write assertions, not just exit code.
- Leave prerequisite-heavy paths uncovered when they cannot be proven, and explain why in `coverage.md`.

## Workflow

### 1. Explore the live CLI before writing code

```bash
lark-cli --help
lark-cli <domain> --help
lark-cli <domain> +<shortcut> -h
lark-cli <domain> <group> --help
lark-cli <domain> <group> <method> -h
lark-cli schema <domain>.<group>.<method>
```

### 2. Count leaf commands for the denominator

- A leaf command is one that executes an action — it has no further subcommands.
- If `lark-cli <domain> <group> --help` lists no subcommands, `<group>` itself is the leaf.
- Count `task +create` as one leaf and `task tasks get` as one leaf.
- Do not count parameter combinations.
- Reuse coverage already present under `tests/cli_e2e/{domain}/`. Do not count `tests/cli_e2e/demo/`.

### 3. Choose the proof surface before editing

Identify the provable risks for the touched workflow: invalid input, missing prerequisite, identity or permission, state transition, output shape, cleanup safety. If only the happy path is testable, document the blocked risk areas in `coverage.md`.

### 4. Add or update the workflow testcase

- Use `clie2e.RunCmd(ctx, clie2e.Request{...})`.
- Put command path and plain flags in `Args`; put JSON in `Params` (URL/path parameters) and `Data` (request body).
- Prefer one top-level test per workflow with `t.Run` substeps.
- Register teardown on `parentT.Cleanup` so it survives subtest failures.
- When touching an existing command, verify the JSON response shape is stable: assert status type, field paths, and identifiers consumed by later steps before changing assertions.

### 5. Run and iterate

Run `go test ./tests/cli_e2e/{domain} -count=1` while iterating and before finishing. If command shape or behavior is unclear, re-check help or schema (step 1) before changing assertions.

### 6. Refresh the domain outputs

- Update the workflow testcase files.
- Update `coverage.md`: recompute the denominator from live help output, mark each command as `shortcut` or `api`, and keep one command table for the whole domain.

## Testcase rules

- Override `BinaryPath`, `DefaultAs`, or `Format` on `clie2e.Request` only when the testcase truly needs it.
- Use `require.NoError`, `result.AssertExitCode`, `result.AssertStdoutStatus`, `assert`, and `gjson`.
- Shortcut responses (`{ok: bool}`) assert `true`; API responses (`{code: int}`) assert `0`.
- Use `t.Helper()` only for setup or assertion helpers that are called from multiple tests.
- Use table-driven tests only when the scenario shape repeats across inputs.
- For expected failures, assert stderr content and exit code when the environment makes them deterministic.
- If identity or external fixtures cannot be proven, leave the command uncovered and document the prerequisite rather than faking confidence.

## coverage.md

Keep `coverage.md` brief and mechanical. Include:
- a domain-specific H1 title
- a metrics section with denominator, covered count, and coverage rate
- a summary section restating each `Test...` workflow, key `t.Run(...)` proof points, and main blockers
- one command table for all commands

Recommended structure:

```markdown
# <Domain> CLI E2E Coverage

## Metrics
- Denominator: N leaf commands
- Covered: N
- Coverage: N%

## Summary
- TestXxx: ... key `t.Run(...)` proof points ...
- Blocked area: ...

## Command Table
| Status | Cmd | Type | Testcase | Key parameter shapes | Notes / uncovered reason |
| --- | --- | --- | --- | --- | --- |
| ✓ | task +create | shortcut | task_status_workflow_test.go::TestTask_StatusWorkflow | basic create; create with due | |
| ✕ | task +assign | shortcut |  | none | requires real user open_id |
```

- Mark each command `shortcut` or `api`.
- Write testcase entries in `go test -run` friendly form.
- Commands only exercised in `parentT.Cleanup` teardown are not counted as covered.
- Do not split covered and uncovered commands into separate sections.

## Guardrails

- Run as bot identity only; do not assume `--as user` works.
- Do not place new real coverage under `tests/cli_e2e/demo/`.
- Do not depend on preexisting remote data.
- Do not fabricate open_ids, chats, docs, or other remote fixtures.
- Prefer deterministic negative cases over tenant-dependent assertions.
- Do not guess `Params` or `Data` fields when help or schema can tell you the exact shape.
- Do not hardcode obvious defaults unless the command truly requires explicit flags.
- Do not put agent, model, or vendor brand names in visible remote test data; use neutral prefixes such as `lark-cli-e2e-` or `<domain>-e2e-`.
- A command is covered only when the testcase asserts returned fields or persisted state, not just exit code.
- Cleanup-only execution is not primary coverage, except `delete` in the same workflow that created the resource.

From larksuite/cli, MIT.

More from larksuite/cli