audit-functional-suitability

Files (1)

• SKILL.md

SKILL.md (first lines)

TODAY=!`date +%Y-%m-%d`

# Functional Suitability Audit (ISO/IEC 25010)

Audits the codebase for **functional suitability**: does the software provide the functions needed to meet stated requirements, completely and correctly?
It is the bottom-up check that what was supposed to be built is actually built and working.

This is the **Functional suitability** characteristic of the [ISO/IEC 25010](https://en.wikipedia.org/wiki/ISO/IEC_25010) quality model.
Distinct from `check-issue-status` (which checks whether a *single issue* is addressed), this audit scans the whole implementation against its requirements corpus and the code's own incompleteness markers (stubs, TODOs, disabled tests).

## Prerequisites

- Working directory is the root of the repository
- `.sdlc/features/*/requirements.md` present (improves completeness scoring; if absent, the audit falls back to code-honesty signals only)
- Read `.sdlc/context/project-overview.md` if present for scope
- `gh` CLI for open bug-issue signals (optional)

## What This Checks

| Sub-characteristic | What it means | Signals scanned |
|---|---|---|
| Functional completeness | All required functions are implemented | requirements with no matching code; `TODO`/`FIXME`/`NotImplemented`/`stub`/`501`/`raise NotImplementedError`; feature flags wired but never enabled |
| Functional correctness | Functions produce correct results | open bug issues; skipped/disabled/`xfail` tests; assertion-free tests; logic that silently no-ops (empty `except`, `return None` on happy paths) |
| Functional appropriateness | Functions are suitable for the intended use | requirements whose implementation exists but diverges from the stated acceptance criteria; over-broad or surprising behavior |

## Steps

### 1. Inventory requirements (if .sdlc exists)

For each `.sdlc/features/*/requirements.md`, extract functional requirements (`FR-N`) and their acceptance criteria.
Record each as a completeness target.

```
find .sdlc/features -name requirements.md 2>/dev/null
```

If no `.sdlc/`, record "no requirements corpus" and rely on code-honesty signals (step 2) plus bug issues (step 3).

### 2. Scan code-honesty markers

Find places where the code admits it is incomplete:

```
rg -n "TODO|FIXME|XXX|HACK|NotImplemented|not implemented|raise NotImplemented|501 Not Implemented|stub|placeholder|coming soon" \
  -g '*.{py,ts,js,go,rs,java}' . \
  | rg -v "test|_test|spec|vendor|node_modules|\.venv"
```

Each marker is a completeness finding (the code states it is not done).

### 3. Check requirements coverage

For each functional requirement, search the code for evidence of implementation (keywords from the requirement).
Requirements with no evidence are completeness gaps.

```
rg -n "<requirement keyword>" -g '*.py' .
```

### 4. Correctness signals

- Open bug issues (group by area to spot correctness hotspots):
  ```
  gh search issues --repo <repo> --state open --label bug --limit 100 --json number,title,labels
  ```
- Skipped / disabled / expected-fail tests:
  ```
  rg -n "pytest.mark.skip|@skip|@Ignore|xfail|\.skip\(|test\.todo|it.skip|describe.skip" -g '*.{py,ts,js}' .
  ```
- Assertion-free tests (tests with no assert/expect):
  ```
  rg --files-without-match "assert|expect|should" -g 'test_*.py' -g '*_test.*' .
  ```
- Silent no-ops: empty exception handlers and functions that return without acting:
  ```
  rg -n -U -B1 "except[^:]*:\s*\n\s*(pass|continue|\.\.\.)" -g '*.py' .
  ```

### 5. Appropriateness spot-check

For a sample of requirements where code exists, compare behavior to the acceptance criteria.
Flag clear divergences (e.g., requirement says "soft delete" but code hard-deletes).
This is judgment-based; keep it to a sample and flag for human review.

### 6. Confirm the decisive findings

Pick the critical or high findings that decide the report.
Run the one or two you can: write a scratch script or test under `/tmp` that calls the code, run it, and paste the output, reaching `L3 - Executed` (see [`../sdlc/references/evidence.md`](../sdlc/references/evidence.md)).
Never write scratch files into the repository; this audit is read-only and leaves no artifacts behind.
Label every other finding with its level and pointer: `L1 - Cited` for a `file:line`, or `L2 - Ruled out` for a walked failure path.
When a decisive finding cannot be executed, mark it `unproven` and state what runtime evidence it needed and why that was infeasible.

### 7. Report

Classify each finding by severity (below), then print the report in the Output Format.
Do not modify any files.

## Severity

| Severity | Criteria |
|---|---|
| Critical | A documented requirement has zero implementation; a correctness bug on a critical path with an open issue |
| High | Completeness gap on a core function; cluster of skipped tests in a critical module |
| Medium | TODO/stub in a non-critical path; isolated skipped test |
| Low | Cosmetic TODO; minor divergence from acceptance criteria |

## Output Format

```
# Functional Suitability Audit — {TODAY}

## Summary
- Requirements corpus: <present (N FRs) | absent>
- Completeness gaps: N critical, N high
- Correctness signals: N bug issues, N skipped tests, N assertion-free tests
- Code-honesty markers: N

## Completeness (requirements with no implementation)
| FR | Feature | Requirement | Evidence found | Severity |

From tomzx/agents, MIT.

More from tomzx/agents