traffic-triage-eval

Files (36)

• README.md
• SKILL.md
• app.md · mock-data/
• events.jsonl · mock-data/
• 103.99.1.7.jsonl · mock-data/example-output/investigate/
• 141.98.11.5.jsonl · mock-data/example-output/investigate/
• 176.65.144.71.jsonl · mock-data/example-output/investigate/
• 194.26.29.4.jsonl · mock-data/example-output/investigate/
• 198.18.0.1.jsonl · mock-data/example-output/investigate/
• 198.51.100.50.jsonl · mock-data/example-output/investigate/
• 203.0.113.12.jsonl · mock-data/example-output/investigate/
• 203.0.113.14.jsonl · mock-data/example-output/investigate/
• 23.129.64.1.jsonl · mock-data/example-output/investigate/
• 2a02_4780_1_1__1.jsonl · mock-data/example-output/investigate/
• 5.188.86.2.jsonl · mock-data/example-output/investigate/
• 89.248.165.2.jsonl · mock-data/example-output/investigate/
• README.md · mock-data/example-output/investigate/
• results-detail.csv · mock-data/example-output/
• results.csv · mock-data/example-output/
• results.json · mock-data/example-output/
• results.xlsx · mock-data/example-output/
• generate.py · mock-data/
• labels.csv · mock-data/
• methodology.md · references/
• schema.md · references/
• bootstrap.py · scripts/
• classify.py · scripts/
• evaluate.py · scripts/
• jev.py · scripts/
• limits.py · scripts/
• profile.py · scripts/
• questions.py · scripts/
• report.py · scripts/
• schema.py · scripts/
• signals.py · scripts/
• validate.py · scripts/

SKILL.md (first lines)

# Traffic Triage Eval

> **Research proof of concept.** This skill reduces each IP's requests to a
> profile in code and asks TypeSafe's Jev what the pattern means. It is a
> triage signal for a person, not an incident finding: `malicious` means
> "read these requests", not "block this address", and `benign_user` means
> "nothing stood out", not "verified human". Verify against the raw log
> before acting. Repeat the short form of this at the end of every run.

## What it does

1. **Validate.** `scripts/validate.py` reads the events you staged (JSONL,
   JSON array or CSV in the schema in `references/schema.md`), drops rows
   that fail a required field with a reason you can act on, applies the IP
   or time-window filter, and prints which optional fields are populated so
   you know which signals the run has.
2. **Profile.** `scripts/profile.py` groups by IP and computes everything
   numeric and temporal in code: counts, span, peak per minute, status mix,
   WAF verdicts, hosts, User-Agent classes, probe-path families, payload
   families, enumeration and route templates, directory and extension
   rollups, a coverage sentence, and short samples spent on the paths no
   rule described. Jev does not count and reads dates as text, so nothing
   numeric is left for it to do.
3. **Classify.** `scripts/classify.py` sends each profile with a paragraph
   about the site (`app`) to Jev in one request: a category choice, ten
   yes/no signals and a four-level threat rating. Code turns the answers
   into a 0-100 threat score with a band (Benign / Nuisance / Concerning /
   Attack), and raises the category where it is certain (payloads on real
   routes, credential volume with WAF evidence, scanner evidence with no
   app knowledge), never lowering it. About 3,500 tokens, 15 cents per
   thousand IPs, 340 ms median.
4. **Report and carve out.** `scripts/evaluate.py` runs all of it, prints
   two tables (category counts; the attention and malicious rows with
   their evidence), writes `results.xlsx` (Results, Details, Summary, Read
   me) plus CSV and JSON, one row per IP, attention rows first, and writes
   the **raw rows of every IP that is malicious, unclear or flagged** to
   `out/investigate/<ip>.jsonl` with an index `README.md`, so the
   investigation starts from the run directory and never re-queries the
   log source. With `--labels` it also scores agreement per category.
5. **Explain.** You, the agent, read the attention rows and tell the user
   what each flagged IP did, in plain words, with the paths.

Every question, category, threshold and rule lives in `scripts/questions.py`.
Every regex lives in `scripts/signals.py`.

## Workflow

`SKILL_DIR` means the folder containing this SKILL.md.

### 1. Understand what the user wants

- **"What was this IP doing?"** or **"Who was hitting us between X and Y?"**:
  they will name IPs or a window. Retrieval scope follows from that.
- **"Triage our WAF / LB logs"**: a window, usually the last day or week.
- **"Is this scanning or an attack?"**: same run; the answer is in the
  category, score and evidence columns.

Ask for the window or IPs if neither is given. Ask which hosts are theirs if
the site has several.

### 2. Get the events into the schema

**This skill does not retrieve logs and does not dictate how.** Work out
with the user how to pull them: a cloud CLI, an MCP server, Athena, a SIEM
export, a log sink, a file they already have. Whatever the source, map each
request to one JSON line in the schema in `references/schema.md` (four
required fields: `ts`, `ip`, `method`, `path`; `status` and the rest add
signals, and a WAF log without statuses is fine) and write the file into a staging folder (your scratchpad, or
`$TMPDIR/traffic-triage-eval/<timestamp>/`), never into the user's working
tree. The schema page has a field-source table for GCLB, ALB and AWS WAF to
speed up the mapping. If you pull only a window, pull the whole window for
every IP; a profile built from a slice of an IP's traffic is a slice.

If the source fronts the origin with a CDN, the client IP is the CDN's; drop
those rows or expect one profile per CDN node.

Also write `app.md`: one paragraph on what the site is, which hostnames are
its own, what its real route families are (API prefixes, page paths, the
query parameters that matter), what health checks and schedulers it runs,
and what it does **not** run (WordPress, PHP, Java, a CMS, `/admin`). Say
if static buckets return 200 with the SPA shell for unknown paths. This is
the `app` field Jev judges against; the difference between "background
noise" and "malicious" is whether the traffic knows this application, and
Jev only knows the application from this paragraph.

### 3. Check prerequisites

```bash
python3 --version   # 3.10 or newer
test -n "$TYPESAFE_API_KEY" || test -f ~/.config/typesafe/env || echo "need a TypeSafe key"
```

Nothing to install. First run creates a private venv under the system temp
dir for the one dependency (openpyxl). `uv run` works too. Key resolution:
`TYPESAFE_API_KEY` in the environment, else `~/.config/typesafe/env`. Keys
come from https://console.typesafe.ai/keys; if the user has none, stop and
ask.

### 4. Validate, then dry-run, then run

```bash
python3 "$SKILL_DIR/scripts/validate.py" --events "$STAGING/events.jsonl"
```

Read the drop reasons and the coverage table. If `ua`, `host` or `query` is
near 0%, fix the mapping before spending Jev calls; those carry most of the
signal. Then:

```bash
python3 "$SKILL_DIR/scripts/evaluate.py" \
  --events "$STAGING/events.jsonl" --app "$STAGING/app.md" \
  --out "$STAGING/out" --dry-run
```

prints every IP with its code signals and token estimate and writes
`profiles.json`, with no API calls. Then the real run, with `--ip A --ip B`
to limit to named IPs, `--from` / `--to` (ISO 8601) for a window inside the
file, `--min-requests N` to skip one-hit IPs, `--labels labels.csv` (columns
`ip,label`) for agreement scoring, `--workers 8` for speed:

From SecurityMindedSolutions/ai-skills, MIT.

More from SecurityMindedSolutions/ai-skills