post-patch-validation

Files (5)

• SKILL.md
• openai.yaml · agents/
• evidence-model.md · references/
• post_patch_validation.py · scripts/
• pyproject.toml · scripts/

SKILL.md (first lines)

# Post-Patch Validation

Validate a security patch against the reported bug and the surrounding code it affects. Give the
patch author reproducible failures to fix and identify what still needs testing. Apply the same
checks to human and agent patches. The diff, author, upstream implementation, and original proof
of concept alone cannot establish correctness.

## When to Use

- A security fix, remediation commit, patch file, or pull request already exists.
- An AI-generated patch needs validation before human review or merge.
- A fix may cover one exploit path while missing variants of the same root cause.
- A security fix may alter legitimate behavior or introduce a new vulnerability.
- A patch author needs concrete failures and coverage gaps before another revision.

## When NOT to Use

- No patch exists yet; use vulnerability discovery or fix implementation first.
- The task is to review an audit finding against a report without executing patch evidence.
- The task is only to convert a finding into a permanent project test.
- The target is remote or production. This skill executes local code and tests only.
- The user has not authorized execution of the repository's code or test suite.

## Quick Start

1. Pin the vulnerable base and patched input. Prefer immutable commits. For uncommitted work,
   create a binary patch file first; do not validate in the user's working tree.
2. Scaffold a pinned plan:

   ```bash
   uv run {baseDir}/scripts/post_patch_validation.py scaffold \
     --repo . \
     --base-ref <vulnerable-ref> \
     --patched-ref <patched-ref> \
     --finding-id <stable-id> \
     --finding-summary "<root cause and impact>" \
     --evidence-level runtime \
     --output post-patch-validation/plan.json
   ```

   Use `--patch-file <path>` instead of `--patched-ref` for a patch artifact. Choose the highest
   honest evidence level: `source` for source/patch invariants only, `build` when target code is
   compiled or analyzed but the reported behavior is not executed, or `runtime` when the checks
   execute the reported behavior and its safety assertions.
3. Inspect the finding, diff, callers, sibling paths, cleanup/error paths, and existing tests.
   Populate `checks` in the generated plan. Run `print-schema` for the structural schema:

   ```bash
   uv run {baseDir}/scripts/post_patch_validation.py print-schema
   ```

4. Run `validate-plan` for the complete validation, including coverage, command restrictions,
   and pinned inputs, before executing code:

   ```bash
   uv run {baseDir}/scripts/post_patch_validation.py validate-plan \
     --plan post-patch-validation/plan.json
   ```

5. Execute the evidence plan:

   ```bash
   uv run {baseDir}/scripts/post_patch_validation.py run \
     --plan post-patch-validation/plan.json \
     --output post-patch-validation/results
   ```

6. Report `result.json`, `report.md`, the evidence level, and the complete `assessment`.
   Return each finding to the patch author with its check ID, assertion, and saved logs. Identify
   each validation gap separately, including gaps that coexist with supported findings. After
   the author revises the patch, pin the new inputs and save a fresh validation run. Preserve the
   prior evidence. Passing supplied checks still requires human review before acceptance.

## Evidence Contract

The runner rejects incomplete plans. Supply at least one check of every kind:

| Kind | Required observation |
|---|---|
| `control` | Benign harness succeeds on both base and patch |
| `exploit` | Original safety assertion fails on base and succeeds on patch |
| `variant` | A distinct root-cause variant fails on base and succeeds on patch |
| `behavior` | Unaffected behavior succeeds with byte-identical selected output |
| `regression` | Targeted non-security regression check succeeds on both revisions |
| `security` | Adjacent/new-vulnerability check succeeds on base and patch |
| `suite` | Existing project suite, sanitizer, or deterministic fuzz campaign succeeds on patch |

Commands are argv arrays, never shell strings. Put complex setup in a checked-in or plan artifact
script and invoke it with `{plan_dir}`. The runner fixes locale/timezone/hash-seed inputs, executes
checks in lexical ID order, records raw stdout/stderr, and never edits the original worktree.
Each check's `timeout_seconds` defaults to 300 and accepts integers from 1 through 3600.
Exceeding the timeout leaves a validation gap. A timeout alone does not establish a regression.
Every plan also contains a sorted `submodules` array (`[]` when none). Scaffolding infers affected
Gitlinks from the changed-file inventory. The runner initializes those pinned commits from the
source repository's existing Git module objects, never from `.gitmodules` network URLs; initialize
or fetch them in the source repository before validation.

### Exploit and variant checks must prove they ran

A nonzero exit does not mean the vulnerability reproduced. An import error, a failed build, a
missing dependency, and a failed safety assertion all exit nonzero and are indistinguishable to the
runner. Every `exploit` and `variant` check must print and flush `PPV_REACHED` immediately
before it evaluates its assertion, on both revisions:

```json
"argv": ["python3", "-c", "import app; value = app.render('<'); print('PPV_REACHED', flush=True); assert value == '&lt;'"]
```

The token is also in the environment as `PPV_REACHED_MARKER`. It must land on **stdout, as a line
of its own**. Stderr is not scanned, because a Python `SyntaxError` traceback echoes the offending
source and would otherwise satisfy the check for a harness that executed nothing. A run without it
leaves a `marker_missing` gap. Independent findings from other checks remain in the result.
Flush explicitly: a harness whose payload segfaults or calls `_exit` loses buffered output and forfeits its own evidence.

These checks also run **side-blind**. `{side}` is not expanded for them, `PPV_SIDE` is absent from
their environment, the checkout directory is randomly named, and the plan validator rejects any
`exploit` or `variant` check whose `argv` or `env` mentions either. An assertion that can see which
revision it is on can assert on *that* instead of on the code, which is the cheapest possible way
to fake a reproduction followed by a fix.

From trailofbits/skills, CC-BY-SA-4.0.

More from trailofbits/skills