test-doctor

Files (19)

• .gitignore
• .npmrc
• .vally.yaml
• LICENSE
• README.md
• SKILL.md
• README.md · evals/
• eval.yaml · evals/test-doctor/
• README.md · evals/test-doctor/fixtures/shipping/
• package.json · evals/test-doctor/fixtures/shipping/
• shipping.mjs · evals/test-doctor/fixtures/shipping/
• shipping.test.mjs · evals/test-doctor/fixtures/shipping/test/
• package-lock.json
• package.json
• test-quality.md · references/
• fixtures.test.mjs · test/
• registration.test.mjs · test/
• skill.test.mjs · test/
• thumbnail.png

SKILL.md (first lines)

# Test Doctor

Find gaps, weak assertions, and flaky tests. Fix them without gaming the suite.
Work directly with the repository's tools; no DevX runtime, model panel, shared
protocol directory, or particular test framework is required.

## Modes

| Request | Behavior |
| --- | --- |
| `test-doctor [path]` | Audit, report findings, stop. Do not edit project files. |
| `test-doctor fix [path]` | Audit, repair material test gaps, verify, report. |

An explicit natural-language request to write or repair tests also selects fix
mode. A question about test quality selects audit mode. Ask only when the
requested action or target is materially ambiguous.

Default to the whole repository. An explicit path limits the audit, not just its
report. Discover all test projects in a monorepo. Do not silently substitute a
diff-only review or a sample for a full audit. If the work cannot be completed,
name the inspected scope and the remaining work.

## Workflow

### 1. Discover the setup and intended behavior

Read repository instructions, manifests, lockfiles, test scripts, CI, runner
configuration, coverage gates, and representative source/test pairs. Identify:

- Each test project, language, package manager, runner, and test directory.
- Unit, integration, component, browser, and other relevant test layers.
- Documented contracts and critical flows, especially authorization, validation,
  error handling, CLI/tool protocols, concurrency, and persistence.
- Existing conventions for fixtures, assertions, test placement, and isolation.

Use checked-in commands and project-local tools. Read scripts before execution;
do not assume npm, append generic runner flags, or download tools implicitly.
Prefer existing orchestration that already covers all projects. Where commands
overlap, run a suite once rather than executing it through every wrapper.

Derive expected behavior from specifications, public contracts, requirements,
and established sibling behavior, not just today's implementation. Record
ambiguities rather than cementing a possible bug into a new expected value.
For equivalent entry points, test one independently established intended contract, not
separate expectations copied from each implementation. Identify affected surfaces and
the canonical behavior owner. Impacted sibling paths outside the authorized audit scope
remain unverified gaps: report them without expanding edits or claiming parity.
Open [test quality checks](references/test-quality.md) before reviewing tests.

### 2. Collect a baseline

Run each in-scope suite in a finite, non-watch mode. Record the exact command,
working directory, exit status, test counts, failures, skips, retries, and
duration when available. Distinguish assertion failures from environment,
configuration, missing dependency, and unavailable service failures.

Use the existing coverage command when available; a coverage run that executes
the suite can supply the baseline too. Collect line, branch, and function
coverage, per-file gaps, configured exclusions, and policy results. Check that
reports describe the current source/configuration, not stale artifacts.
No coverage tool is a measurement gap, not evidence of zero or full coverage.

Audit mode must not install dependencies, change configuration, update
snapshots, or save reports inside the project. Test-runner artifacts are allowed;
disclose unexpected tracked-file changes and do not silently revert them.
If the documented test command itself changes tracked files, request permission
before running it. Report missing tooling rather than automatically setting it up.

Failures take priority but do not erase other inspectable findings. Qualify
coverage conclusions when execution failed. Never claim a partial run was green.

### 3. Review the evidence

Check all of these dimensions; mark each verified, findings, blocked, or not
applicable. Keep observed defects separate from suspected risks.

| Dimension | What to examine |
| --- | --- |
| Correctness | Do assertions prove the intended behavior and fail for a credible regression? |
| Coverage | Material branches, boundaries, failure paths, and cross-module contracts; configured gates and exclusions. |
| Reliability | Async completion, state restoration, cleanup, clocks, randomness, external services, and order dependence. |
| Maintenance | Redundant tests, brittle snapshots, implementation coupling, misleading names, focused/skipped tests, slow setup. |
| Compatibility | Installed test tools, actual deprecated APIs, incompatible configuration, and relevant support status. |
| Runtime | Existing E2E results and observable behavior of important browser-facing journeys, when applicable. |

Prioritize security properties and reachable tool/CLI handlers, not filename
matches alone. Look beyond test-file correspondence: integration tests may cover
several modules, while a matching test file may assert almost nothing.

Exercise real adapter integration seams and meaningful negative cases against that
contract: approval ordering and information, denial/cancel, retries/idempotency/recovery,
and durable or partial results where applicable. Do not mock the behavior under test.
Allow justified capability, native UI, and authentication-adapter differences;
do not require identical pixels or duplicate every E2E.

Treat ratios, test counts, snapshots, sleeps, mocks, and file/network access as
investigation signals, not automatic defects. Do not require a fixed number of
tests per function, test every constant, or impose an invented coverage target.
Honor repository gates; if none exists, report gaps without inventing a pass.

Use current official documentation or registry metadata before claiming a tool
is outdated, unsupported, or deprecated. Record the date/source or say currency
was not checked. Not using the newest API is not itself a health defect.
Identify migration needs, but do not upgrade dependencies as part of a test fix.

For suspected flakes, repeat the smallest relevant test or group a bounded
number of times using the runner's supported repeat/order/seed controls.
Preserve first-attempt failures and seeds; a retry-pass is still a finding.
Retries of failures are not repetitions of every test. State the number of
attempts; successful repeats do not prove the suite is flake-free.

For browser-facing projects, use the existing E2E framework, whether Playwright,
Cypress, Selenium, or another supported tool. Missing coverage of a critical
journey matters; absence of Playwright does not. With an available local app and
browser tools, inspect rendering, console errors, and unexpected failed requests.
Distinguish expected negative responses from broken application requests.
If the app, browser, credentials, or services are unavailable, mark runtime
verification blocked. Never call an unobserved frontend healthy.

From jongio/skills, MIT.

More from jongio/skills