prompt-injection-eval

Files (19)

• README.md
• SKILL.md
• README.md · mock-data/
• app-context.md · mock-data/
• results-detail.csv · mock-data/example-output/
• results.csv · mock-data/example-output/
• results.json · mock-data/example-output/
• results.xlsx · mock-data/example-output/
• prompts.csv · mock-data/
• methodology.md · references/
• baseline.py · scripts/
• bootstrap.py · scripts/
• code_signals.py · scripts/
• evaluate.py · scripts/
• gate.py · scripts/
• jev.py · scripts/
• llm_judge.py · scripts/
• questions.py · scripts/
• report.py · scripts/

SKILL.md (first lines)

# Prompt Injection Eval

> **Research proof of concept.** This skill asks TypeSafe's Jev a fixed set of
> questions about a prompt and turns the answers into a risk score and an
> allow / review / block decision. It is a signal for a security gate, not a
> guarantee: no detector catches every injection, and this one belongs
> alongside least-privilege tools, output checks and logging, not in place of
> them. Before wiring it into a real system, run the evaluator on a labelled
> sample of your own traffic, tune the thresholds in `scripts/questions.py`,
> and run it in **alert mode** (log and review, never block) until the false
> alarm rate on real users is known. Only then consider block mode. Repeat the
> short form of this to the user at the end of every run.

## What it does

1. **Gate.** `scripts/gate.py` exposes `check_prompt(user_input, app,
   recent_turns=None)`. First, deterministic code checks
   (`code_signals.py`) for invisible Unicode, chat-template tokens, hidden
   HTML, homoglyph words, gibberish suffixes and fake turn markers; each
   forces a minimum decision. Then one request to Jev carrying a paragraph
   describing the app, the prompt, and the recent user turns if given, with
   ten typed questions answered together: instruction override, persona
   hijack, secret or prompt extraction, action misuse beyond the app's
   purpose, false authority, obfuscation, instructions embedded in pasted
   content, format injection (fake system or assistant turns), multi-turn
   assembly, and an overall manipulation severity, plus a labelled attack
   type. Code weights the answers into a 0-100 risk score and applies the
   rules in `questions.py`: block on high risk, on high severity with
   confidence, or on any single near-certain signal; review on moderate risk,
   any strong signal, or an unsure allow; else allow. About 2,100 tokens, 9
   cents per thousand prompts, 300 ms median.
2. **Evaluate.** `scripts/evaluate.py` runs a CSV of prompts through the gate,
   a built-in regex phrase list, and optionally an LLM judge via any
   OpenAI-compatible endpoint. With `label` (benign|injection) in the CSV it
   reports caught / missed / false alarms, precision, recall and F1 per
   approach, with measured latency and cost per thousand prompts, and prices
   the LLM judge's measured tokens at Claude list rates. Output is
   `results.xlsx` (Results, Details, Comparison, Summary, Read me) plus CSV and
   JSON.
3. **Explain.** You, the agent, read the results and tell the user what was
   caught, what was missed, what the false alarms were and why, and what the
   numbers mean for their traffic.

Every question, weight, threshold and rule lives in `scripts/questions.py`.

## Workflow

`SKILL_DIR` means the folder containing this SKILL.md.

### 1. Understand what the user wants

Three shapes of request:

- **"Evaluate my prompts"**: they have prompts (logs, a CSV, a folder of text
  files, tickets, an export from their LLM gateway) and want scores and a
  comparison. Go to step 2.
- **"Gate my app"**: they want the check in their backend. Run the evaluator
  on their sample first (never wire in an untested threshold), then go to
  step 5.
- **"Just test one"**: `python3 "$SKILL_DIR/scripts/gate.py" "<prompt>"
  "<app description>"` prints the decision and every signal as JSON.

### 2. Stage the prompts

The evaluator reads one CSV with a `prompt` column and optional `label`,
`category`, `id`, and `history` (earlier user turns, oldest first, separated
by ` || `, for split-across-turns cases). Wherever the prompts live (local files, a log export, a
database query, an LLM gateway's request log, a ticketing system via a
connector, pasted text), fetch them and write that CSV into a temp staging
folder (your scratchpad, or `$TMPDIR/prompt-injection-eval/<timestamp>/`),
never into the user's working tree. Prompts are user data; the copies exist
for the run. If the user has ground truth (a list of known attacks, or a
labelled set), put it in `label`. Without labels the run still produces
scores and decisions, just no precision/recall.

Also write `app.md`: one paragraph on what the assistant is for, what it can
do, what tools it has, and what it must not do. Ask the user if you cannot
infer it. This is the `app` field Jev judges against; a vague description
makes "action outside the app's purpose" vague too.

### 3. Check prerequisites

```bash
python3 --version   # 3.10 or newer
test -n "$TYPESAFE_API_KEY" || test -f ~/.config/typesafe/env || echo "need a TypeSafe key"
```

Nothing to install. First run creates a private venv under the system temp
dir for the one dependency (openpyxl). `uv run` works too. Key resolution:
`TYPESAFE_API_KEY` in the environment, else `~/.config/typesafe/env`. Keys
come from https://console.typesafe.ai/keys; if the user has none, stop and
ask.

### 4. Run the evaluator

```bash
python3 "$SKILL_DIR/scripts/evaluate.py" \
  --prompts "$STAGING/prompts.csv" \
  --app "$STAGING/app.md" \
  --out "$STAGING/out"
```

Add `--llm-judge <model id>` to also run an LLM judge for the comparison,
with `OPENAI_API_KEY` and optionally `OPENAI_BASE_URL` set (OpenAI, OpenRouter,
a gateway in front of Claude, Ollama). Without it, the Comparison sheet
carries an estimated LLM-judge row from token counts, labelled as not run.
`--workers N` (default 4) sets parallel requests; `--model` picks the
TypeSafe model.

The run prints every prompt with its decision, risk, the regex verdict and
the label, then the counts, Jev tokens, cost and latency, then the comparison
table. Read `results.json` for anything the table does not show.

### 5. Wire it in (only when asked, and only after step 4)

`gate.py` is written to be copied. `check_prompt(user_input, app,
recent_turns=None)` returns:

```json

From SecurityMindedSolutions/ai-skills, MIT.

More from SecurityMindedSolutions/ai-skills