ios-evidence-reporting

Files (1)

• SKILL.md

SKILL.md (first lines)

# iOS Evidence-Based Reporting

AI confidence is not evidence. AI reasoning is not runtime evidence. A code
change is not automatically a verified fix. Every claim in a report is
backed by evidence at a specific tier — and a claim never gets reported at
a higher tier than the evidence actually reached.

## The seven evidence tiers

From weakest to strongest. A claim's tier is the *weakest* link that
actually backs it — gathering strong evidence for half a claim doesn't
license reporting the whole claim at that strength.

1. **ASSUMPTION** — a belief based on reasoning, not evidence. Never
   reported as verified. Phrase it as a hypothesis: "likely a decoding
   cost" not "is a decoding cost."
2. **STATIC_ANALYSIS** — evidence from reading source, a compiler
   diagnostic, AST/dependency inspection, or a static-analysis tool
   (including an MCP analyzer such as `ios-agent-mcp`) run over source
   without building or running the app. Reading the output of `grep`,
   `wc`, or `git diff` also lands here — running a text command over
   source is still static analysis, not runtime evidence, regardless of
   whether the agent doing it is otherwise read-only. Example: "no
   obvious retain cycle found during static inspection" — this does
   **not** mean "no memory leak exists."
3. **BUILD_VERIFIED** — the project actually compiled. Example:
   "`xcodebuild` completed successfully."
4. **TEST_VERIFIED** — an automated test suite ran and its result is
   shown. Example: "42 tests passed."
5. **RUNTIME_VERIFIED** — the app was actually launched, installed, or
   interacted with (simulator or device) and its behavior observed — a
   screenshot, a deep link reaching the right screen, a manual
   walkthrough. Confirms *behavior*, not a number — see the next tier
   for that.
6. **RUNTIME_MEASURED** — an actual number came from actually running
   the app: Instruments (Allocations, Leaks, Time Profiler), MetricKit,
   XCTMetric, `os_signpost`, a launch-time or memory measurement.
   Required for any performance or memory *improvement* claim — no
   lower tier can support one.
7. **HUMAN_VERIFICATION** — the agent cannot perform the verification
   this claim needs (driving the Instruments GUI, for instance) and
   says so explicitly, rather than guessing or silently skipping it.

Tiers 3-6 all require something to have actually *run*; only 5 and 6
require the app itself to have run. Static analysis — including an MCP
tool's structured findings — never outranks build/test/runtime tiers,
no matter how sophisticated the analysis.

### MCP/tool-assisted analysis is still whatever tier it actually is

Canonical rule, referenced rather than re-derived: an MCP server's
output is graded by what it actually did, not by the fact that a tool
(rather than a human) did it.

- `ios-agent`-style servers are static analyzers — they read source and
  never build or run the app. Their findings are `STATIC_ANALYSIS`,
  identical tier to a manual code read, never higher, no matter how
  structured or confident-looking their output is.
- `ios-simulator`-style servers actually build, install, and launch the
  app — their `build_project`/`run_tests`/`launch_app`/`screenshot`
  output can genuinely earn `BUILD_VERIFIED`, `TEST_VERIFIED`, or
  `RUNTIME_VERIFIED`, because the app actually ran. It still cannot
  produce `RUNTIME_MEASURED` on its own unless it reports an actual
  measured number (a screenshot confirms behavior, not a quantity).

### A pattern match is a lead, not a finding

Canonical rule, referenced rather than re-derived: a `grep` hit, an MCP
tool flag, or any other pattern match is a *candidate* to verify, never
a confirmed issue on its own — regardless of which agent produced it or
how narrow the pattern looks. "`UserDefaults` called near the word
`token`" is a lead; "this stores the session token insecurely" requires
having actually traced that it's a real token, on that path, with no
Keychain wrapper already handling it. Report a match as a lead — cite
the evidence, state what would confirm it — until the trace is actually
done. This applies most often in `ios-security-review`,
`ios-legacy-mapping`, and `ios-feature-implementation`'s Security
Review, but the rule itself lives here rather than being restated in
each.

### Independent review requirement

Any report reaching `BUILD_VERIFIED` or higher — not just
`STATIC_ANALYSIS` — is handed to `ios-evidence-reviewer` before being
presented as final. The agent that ran the build/test/measurement is
not the sole authority on whether its own report is honest about it;
see `ios-evidence-reviewer` for what it checks. Skip this only for a
report making no claim above `STATIC_ANALYSIS`/`ASSUMPTION` (a pure
read-only consult has nothing for the reviewer to add).

## Claim → minimum evidence matrix

| Claim | Minimum evidence |
|---|---|
| Code compiles | `BUILD_VERIFIED` |
| Unit/UI tests pass | `TEST_VERIFIED` |
| A feature's logic/behavior is correct | `TEST_VERIFIED` (the tests that exercise it) |
| The UI renders/looks/behaves correctly | `RUNTIME_VERIFIED` — a passing unit test cannot confirm a visual or layout claim; something has to actually be seen running |
| An API is available at the stated minimum | `STATIC_ANALYSIS` (availability check) |
| No obvious retain cycle | `STATIC_ANALYSIS` only — never "no leak exists" |
| No runtime leak observed | `RUNTIME_MEASURED` (Instruments/Memory Graph) |
| Memory usage improved | `RUNTIME_MEASURED`, before/after |
| Performance improved | `RUNTIME_MEASURED`, before/after |
| Thread-safe | `STATIC_ANALYSIS` + concurrency-relevant tests — never inspection alone |
| Architecture preserved / consistent | `STATIC_ANALYSIS` (architecture review) |
| Security issue addressed | `STATIC_ANALYSIS`/security review, plus a regression test where practical |
| App Store ready | `STATIC_ANALYSIS` (code-visible checks) — metadata/screenshots still need `HUMAN_VERIFICATION` |

If an agent can't reach the tier a claim needs, it reports the tier it
actually reached and marks the gap — it never rounds up.

## Never say, unless the evidence supports it

Do not write these words in a final report unless the matrix above is
satisfied for that exact claim:

- **"fixed" / "working"** — unless `RUNTIME_VERIFIED` or `TEST_VERIFIED`
  backs it.
- **"optimized" / "faster" / "uses less memory"** — unless

From apexbymanish/claude-ai-agents-ios, MIT.

More from apexbymanish/claude-ai-agents-ios