code-audit-jev

Files (49)

• README.md
• SKILL.md
• action.yml
• ci.yml · mock-data/sample-repo/.github/workflows/
• pr.yml · mock-data/sample-repo/.github/workflows/
• Dockerfile · mock-data/sample-repo/
• documents.py · mock-data/sample-repo/api/handlers/
• extras.py · mock-data/sample-repo/api/handlers/
• auth.py · mock-data/sample-repo/api/services/
• cleanup.py · mock-data/sample-repo/api/services/
• app.md · mock-data/sample-repo/
• storage.tf · mock-data/sample-repo/infra/
• labels.csv · mock-data/sample-repo/
• negatives.csv · mock-data/sample-repo/
• test_documents.py · mock-data/sample-repo/tests/
• Comment.tsx · mock-data/sample-repo/web/src/
• methodology.md · references/
• README.md · rules/
• _core.toml · rules/
• csrf.toml · rules/
• idor.toml · rules/
• information_disclosure.toml · rules/
• injection.toml · rules/
• insecure_configuration.toml · rules/
• mass_assignment.toml · rules/
• missing_authorization.toml · rules/
• nosql_ldap_injection.toml · rules/
• open_redirect.toml · rules/
• path_traversal.toml · rules/
• resource_exhaustion.toml · rules/
• secret_exposure.toml · rules/
• session_and_token_handling.toml · rules/
• ssrf.toml · rules/
• supply_chain.toml · rules/
• tenant_isolation.toml · rules/
• unsafe_deletion.toml · rules/
• unsafe_deserialization.toml · rules/
• upload_validation.toml · rules/
• weak_cryptography.toml · rules/
• xml_xxe.toml · rules/
• xss.toml · rules/
• compare.py · scripts/
• diff.py · scripts/
• extract.py · scripts/
• inventory.py · scripts/
• jev.py · scripts/
• output.py · scripts/
• rules.py · scripts/
• run.py · scripts/

SKILL.md (first lines)

# Code Audit Eval

> **Research proof of concept.** Jev reads one unit of code at a time. It
> can see what is in the unit, the file's imports and guard names, and a
> paragraph about the system; it cannot follow a call into another file,
> read a route table elsewhere, or reason about a lifecycle across
> services. So every row it produces is a **candidate** for a person or a
> reasoning agent to trace, and a clean unit means "nothing visible here",
> not "safe". Cross-file and architectural defects are outside its reach by
> design; the full `/audit-security` skill exists for those. Repeat the
> short form of this at the end of every run.

## What it does

1. **Inventory.** `scripts/inventory.py` walks the target, skips vendored,
   generated, lock and binary files, classifies each file's language and
   role (http_handler, event_worker, shared_library, frontend,
   infra_terraform, ci_pipeline, container, config, script, test, docs).
   A folder whose children are git repositories is a fleet: every
   repository is run separately and summarised in `fleet.csv`.
2. **Extract.** `scripts/extract.py` splits files into units: Python by
   `ast` (functions, methods, class bodies, the module top), TypeScript /
   JavaScript / Go / Java / Terraform by brace depth (declarations, route
   registrations, resource blocks), YAML / Dockerfile / shell as one unit
   per file. Units over ~4.4k tokens are chunked. Every unit carries the
   file's imports and the guard names found anywhere in the file.
3. **Rules.** `rules/*.toml`, one file per category (22 of them: injection, missing
   authorization, tenant isolation, IDOR, secrets, SSRF, open redirect, CSRF,
   traversal, upload validation, deserialization, XXE, NoSQL/LDAP injection,
   mass assignment, weak crypto, session and token handling, disclosure,
   XSS, resource exhaustion, insecure configuration, supply chain, unsafe
   deletion). Each declares the class text Jev chooses
   between, its yes/no questions, its regex signals (computed in code and
   handed to Jev as facts with line numbers) and how the answers compose
   into a vector. `_core.toml` holds the model, the scoring, the severity
   scale and the shared questions. `scripts/rules.py` loads the folder; the
   runner knows nothing about any category. `rules/README.md` is the
   author's guide; a repo adds its own rules with `--rules DIR`.
4. **Judge.** `scripts/run.py` sends each unit with the file context, the
   signals and the `app` paragraph to Jev in one request (one Choice, one
   Noul per declared question, 29 questions in all, one Score) and composes the answers: vectors
   ("missing authorization" is handles-request AND privileged AND
   no-auth-check, multiplied in code), a 0-100 score, a band, drops test and
   non-production units, and applies the rules' floors. About 7-9k tokens
   and 450 ms per unit (the 22 rules' questions are ~7k of that, sent once
   per unit and evaluated in parallel: a rule adds tokens, not calls).
5. **Report.** The same `run.py` prints the band counts and the attention
   rows, and writes `results.json` / `results.csv`, `findings.md` (one entry
   per file and category with the snippet, for the tracing agent), the PR
   summary and SARIF when asked, and `results.xlsx` if openpyxl is installed.
   `--diff BASE` judges only the units a change touched, on both sides of
   the merge base, and reports band rises. `--explain FILE[:UNIT]` prints
   the exact state Jev saw.
6. **Compare.** `scripts/compare.py` matches a run against a reference
   finding list (an `/audit-security` report, or a CSV) finding by finding.
7. **Share.** `action.yml` packages it as a composite GitHub Action:
   `uses: .../code-audit-jev@<sha>` with the key as a secret, SARIF to the
   Security tab.

Every question, class, weight, threshold and regex lives in `rules/`.

## Workflow

`SKILL_DIR` means the folder containing this SKILL.md.

### 1. Understand what the user wants

- **"Scan this repo"**: one target, default scope, then you read
  `findings.md` and explain the top candidates.
- **"Which of these N repos need a real audit"**: a fleet folder; the
  deliverable is `fleet.csv` plus the top candidates per repo.
- **"Is Jev good enough to replace the full audit"**: run both and use
  `compare.py`; the honest answer is in `references/methodology.md`.

### 2. Write the `app` paragraph

One paragraph about the system, written by you from the repo's docs
(CLAUDE.md, README, architecture docs). It must say: how requests are
authenticated and whether a framework gate covers every handler or each
handler must guard itself; how tenants or customers are isolated and where
the tenant id is supposed to come from; which inputs are trusted
(environment, CI substitutions); which resources are intentionally public;
which sanitizers and pinned clients exist. This is the difference between
Jev flagging every handler without a decorator and Jev knowing the
framework gate exists. Write it to the staging folder (your scratchpad or
`$TMPDIR/code-audit-jev/<timestamp>/app.md`), never into the target repo.
`mock-data/sample-repo/app.md` is a model.

### 3. Check prerequisites

```bash
python3 --version   # 3.10 or newer
test -n "$TYPESAFE_API_KEY" || test -f ~/.config/typesafe/env || echo "need a TypeSafe key"
```

Nothing to install: stdlib only (openpyxl, if present, adds an xlsx). Keys come from
https://console.typesafe.ai/keys; if the user has none, stop and ask.

### 4. Dry-run, then run

```bash
python3 "$SKILL_DIR/scripts/run.py" --target /path/to/repo --app "$STAGING/app.md" \
  --exclude archive --out "$STAGING/out" --dry-run
```

prints the file and unit counts, the role mix and the token estimate with
no API calls, and writes `units.json`. Exclude frozen or vendored trees the
inventory cannot recognise (`--exclude archive`). Then:

```bash
python3 "$SKILL_DIR/scripts/run.py" --target /path/to/repo --app "$STAGING/app.md" \
  --exclude archive --out "$STAGING/out" --workers 8
```

`--scope signals` judges only units with a regex signal or a
security-relevant role (about 40% cheaper, misses defects with no
greppable shape). `--include PREFIX` narrows to a directory. `--limit N`
is for a trial. `--labels labels.csv` (columns `path,category[,id]`)
scores agreement with a reference list. To debug one unit:

From SecurityMindedSolutions/ai-skills, MIT.

More from SecurityMindedSolutions/ai-skills