skeptical-triage

Files (1)

• SKILL.md

SKILL.md (first lines)

# Skeptical Triage

Filter false positives from multi-angle review, security audit, QA regression flags, or any high-stakes judgment before it turns into a blocker.

Three rounds of skeptical self-review + an impartial arbiter, with a confidence score from the vote.

## When to invoke

| Caller | Finding type | Apply triage? |
|--------|--------------|---------------|
| `/review` | Angle 2/4/7/9 P0/P1 (security, SQL, privacy, concurrency) | Yes |
| `/review --deep` | Any angle P0/P1 | Yes |
| `security-officer` | CSO audit P0/P1 | Yes |
| `security-officer` | Secret in source/git, confirmed CVE | **No** — hard finding |
| `qa-engineer` | Flaky-test verdict (is this a regression or flake?) | Yes |
| `architect` | ADR trade-off dispute (option A vs. B when both look reasonable) | Yes |
| Any | P2/advisory | No |

## The 4-step pattern

Run these sequentially. Each round sees prior reasoning. Arbiter sees all rounds.

### Round 1 — Reachability / Premise

Question: **is the premise true?**
- For security/reliability: can an external attacker reach this code path with untrusted input? Trace input flow backward from the bug site to its origin. If only trusted internal callers → lean INVALID.
- For regressions: does the failing behavior reproduce from a clean state on the target branch?
- For ADR trade-offs: is the constraint that forces the choice actually binding? (e.g. "we need <10ms p99" — is that real or aspirational?)

Output: `{round: 1, verdict: VALID|INVALID|UNCERTAIN, reasoning: "...", crux: "single key fact"}`

### Round 2 — Verify cited defenses / counter-evidence

Question: **are claimed defenses real and sufficient?**
- Every cited defense → use `Grep` to find its actual implementation line.
- Resolve constant names to numeric values. `MAX_BUF_SIZE` is not a verified bound — `#define MAX_BUF_SIZE 64` is.
- For regressions: is the cited "test covers this" actually asserting the right invariant?
- For ADR: is the cited benchmark/precedent real (grep for it, read it), or rumored?

If you cannot point to the line that enforces the defense, **it does not exist.**

Output: same JSON shape, with `grep_used: true/false`.

### Round 3 — Missed angles

Question: **what did Rounds 1-2 not consider?**
- Error paths, integer overflow, race windows, different callers, platform differences
- Do NOT rehash prior rounds — add new evidence or concede
- For QA: retry logic masking the failure? Test pollution from another test?
- For ADR: option C that neither reviewer raised?

Output: same JSON shape.

### Arbiter

Input: all 3 rounds + original finding/question + source code.

Question: **final call — which side has the stronger evidence?**
- Deliver single `verdict: VALID|INVALID` (no UNCERTAIN — make the call).
- Deliver one-sentence `crux` — the key fact the verdict turns on.
- If 3 prior rounds all said the same thing, only override with overwhelming new evidence and explain why.

Output:
```json
{
  "verdict": "VALID",
  "crux": "memcpy at auth.c:142 copies network-controlled len bytes into 64-byte stack buffer with no bound check",
  "reasoning": "Rounds 1 and 3 verified attacker reach; Round 2 found no size check in 50 LOC radius; arbiter confirms no caller clamps len."
}
```

## Hard rules

Burn these into every round's prompt:

1. **Absence of defense → VALID, not UNCERTAIN.** If you searched for a defense and did not find one, that is the answer. "Other code probably handles this" is not a valid defense.
2. **A constant name is not a verified bound — only its resolved value is.** Grep for the `#define` / `const` declaration.
3. **Name the line or it does not exist.** Vague references to "assumptions in this codebase" do not count.
4. **Do not contradict your own conclusion in the same response.** If you verified a defense is insufficient, that is the verdict. Stop searching for reasons to flip.
5. **Code quality issue ≠ security vulnerability.** Data race on diagnostic state, NULL check on internal-only API, UB only in debug builds → INVALID.
6. **Trust your own reasoning.** If you see the crux on first read, don't manufacture a counter-argument.

## Confidence scoring

```
confidence = valid_rounds_before_arbiter / 3
```

- `100%` (VVV) — 3/3 rounds VALID. Arbiter rubber-stamps unless it finds something brand-new.
- `67%` (VVI or VIV or IVV) — majority VALID. Arbiter breaks tie with new evidence.
- `33%` (IIV or IVI or VII) — majority INVALID. Arbiter usually confirms INVALID.
- `0%` (III) — 3/3 INVALID. Arbiter rarely overrides.

Arbiter **overrides** the final verdict; confidence reflects the round vote for transparency. Record both in the output so humans can see where the arbiter diverged.

## Applying triage results to severity

Once the arbiter returns:

| Arbiter verdict | Confidence | Severity action |
|-----------------|------------|-----------------|
| `VALID` | ≥ 50% | Keep original severity |
| `VALID` | < 50% | Demote: P0→P1, P1→P2 |
| `INVALID` | any | Remove from gate tally, record as `[FILTERED]` in report for audit |
| `UNCERTAIN` (only if arbiter could not decide) | n/a | Keep original severity, flag for manual CTO review |

## Output schema

Every caller logs triage results to `.great_cto/triage-log.jsonl` (append-only, one JSON per line):

```json
{
  "timestamp": "2026-04-19T12:34:56Z",
  "caller": "review|security-officer|qa-engineer|architect",
  "finding_id": "SEC-042",
  "file": "src/auth.c:142",
  "original_severity": "P0",
  "rounds": [
    {"round": 1, "verdict": "VALID",   "crux": "..."},

From avelikiy/great_cto, MIT.

More from avelikiy/great_cto