Home Security AI Benchmark

Files (56)

• README.md
• SKILL.md
• config.yaml
• backyard_animal.png · fixtures/frames/
• bicycle_sidewalk.png · fixtures/frames/
• box_carrier.png · fixtures/frames/
• condensation_lens.png · fixtures/frames/
• doorstep_package.png · fixtures/frames/
• driveway_full.png · fixtures/frames/
• fallen_person.png · fixtures/frames/
• fog_scene.png · fixtures/frames/
• front_door_person.png · fixtures/frames/
• front_porch_group.png · fixtures/frames/
• garden_path.png · fixtures/frames/
• garden_tool_person.png · fixtures/frames/
• glare_sunlight.png · fixtures/frames/
• headlights_night.png · fixtures/frames/
• hivis_worker.png · fixtures/frames/
• indoor_blocked_exit.png · fixtures/frames/
• indoor_child_cabinet.png · fixtures/frames/
• indoor_elec_cord.png · fixtures/frames/
• indoor_elec_powerstrip.png · fixtures/frames/
• indoor_fall_person.png · fixtures/frames/
• indoor_fall_shelf.png · fixtures/frames/
• indoor_fire_candle.png · fixtures/frames/
• indoor_fire_heater.png · fixtures/frames/
• indoor_fire_iron.png · fixtures/frames/
• indoor_fire_stove.png · fixtures/frames/
• indoor_trip_stairs.png · fixtures/frames/
• indoor_trip_wetfloor.png · fixtures/frames/
• jogger_sidewalk.png · fixtures/frames/
• living_room_empty.png · fixtures/frames/
• mailbox_delivery.png · fixtures/frames/
• motorcycle_driveway.png · fixtures/frames/
• multi_class_scene.png · fixtures/frames/
• multiple_animals.png · fixtures/frames/
• multiple_vehicles.png · fixtures/frames/
• night_motion.png · fixtures/frames/
• occluded_person.png · fixtures/frames/
• open_garage.png · fixtures/frames/
• parking_lot_vehicle.png · fixtures/frames/
• patio_furniture.png · fixtures/frames/
• pool_area.png · fixtures/frames/
• rain_scene.png · fixtures/frames/
• snow_scene.png · fixtures/frames/
• spider_web_lens.png · fixtures/frames/
• street_traffic.png · fixtures/frames/
• vehicle_detail.png · fixtures/frames/
• wheelchair_user.png · fixtures/frames/
• window_peeper.png · fixtures/frames/
• tool-use-scenarios.json · fixtures/
• package-lock.json
• package.json
• generate-report.cjs · scripts/
• run-benchmark.cjs · scripts/
• test-model-config.cjs · scripts/

SKILL.md (first lines)

# Home Security AI Benchmark

Comprehensive benchmark suite evaluating LLM and VLM models on **143 tests** across **16 suites** — context preprocessing, tool use, security classification, prompt injection resistance, alert routing, knowledge injection, VLM-to-alert triage, and scene analysis.

## Setup

**Requires `npm install`.** This skill has a `package.json` with dependencies (e.g. `openai` SDK). The deployment agent should check for `package.json` and run `npm install` in the skill directory before launching.

Entry script: `scripts/run-benchmark.cjs`

### Verification

```bash
node scripts/run-benchmark.cjs --help
```

## Quick Start

### As an Aegis Skill (automatic)

When spawned by Aegis, all configuration is injected via environment variables. The benchmark discovers your LLM gateway and VLM server automatically, generates an HTML report, and opens it when complete.

### Standalone

```bash
# LLM-only (VLM tests skipped)
node scripts/run-benchmark.cjs

# With VLM tests (base URL without /v1 suffix)
node scripts/run-benchmark.cjs --vlm http://localhost:5405

# Custom LLM gateway
node scripts/run-benchmark.cjs --gateway http://localhost:5407

# Skip report auto-open
node scripts/run-benchmark.cjs --no-open
```

## Configuration

### Environment Variables (set by Aegis)

| Variable | Default | Description |
|----------|---------|-------------|
| `AEGIS_GATEWAY_URL` | `http://localhost:5407` | LLM gateway (OpenAI-compatible) |
| `AEGIS_LLM_URL` | — | Direct llama-server LLM endpoint |
| `AEGIS_LLM_API_TYPE` | `openai` | LLM provider type (builtin, openai, etc.) |
| `AEGIS_LLM_MODEL` | — | LLM model name |
| `AEGIS_LLM_API_KEY` | — | API key for cloud LLM providers |
| `AEGIS_LLM_BASE_URL` | — | Cloud provider base URL (e.g. `https://api.openai.com/v1`) |
| `AEGIS_VLM_URL` | *(disabled)* | VLM server base URL |
| `AEGIS_VLM_MODEL` | — | Loaded VLM model ID |
| `AEGIS_SKILL_ID` | — | Skill identifier (enables skill mode) |
| `AEGIS_SKILL_PARAMS` | `{}` | JSON params from skill config |

> **Note**: URLs should be base URLs (e.g. `http://localhost:5405`). The benchmark appends `/v1/chat/completions` automatically. Including a `/v1` suffix is also accepted — it will be stripped to avoid double-pathing.

### User Configuration (config.yaml)

This skill includes a [`config.yaml`](config.yaml) that defines user-configurable parameters. Aegis parses this at install time and renders a config panel in the UI. Values are delivered via `AEGIS_SKILL_PARAMS`.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `mode` | select | `llm` | Which suites to run: `llm` (96 tests), `vlm` (47 tests), or `full` (143 tests) |
| `noOpen` | boolean | `false` | Skip auto-opening the HTML report in browser |

Platform parameters like `AEGIS_GATEWAY_URL` and `AEGIS_VLM_URL` are auto-injected by Aegis — they are **not** in `config.yaml`. See [Aegis Skill Platform Parameters](../../../docs/skill-params.md) for the full platform contract.

### CLI Arguments (standalone fallback)

| Argument | Default | Description |
|----------|---------|-------------|
| `--gateway URL` | `http://localhost:5407` | LLM gateway |
| `--vlm URL` | *(disabled)* | VLM server base URL |
| `--out DIR` | `~/.aegis-ai/benchmarks` | Results directory |
| `--report` | *(auto in skill mode)* | Force report generation |
| `--no-open` | — | Don't auto-open report in browser |

## Protocol

### Aegis → Skill (env vars)
```
AEGIS_GATEWAY_URL=http://localhost:5407
AEGIS_VLM_URL=http://localhost:5405
AEGIS_SKILL_ID=home-security-benchmark
AEGIS_SKILL_PARAMS={}
```

### Skill → Aegis (stdout, JSON lines)
```jsonl
{"event": "ready", "model": "Qwen3.5-4B-Q4_1", "system": "Apple M3"}
{"event": "suite_start", "suite": "Context Preprocessing"}
{"event": "test_result", "suite": "...", "test": "...", "status": "pass", "timeMs": 123}
{"event": "suite_end", "suite": "...", "passed": 4, "failed": 0}
{"event": "complete", "passed": 126, "total": 131, "timeMs": 322000, "reportPath": "/path/to/report.html"}
```

Human-readable output goes to **stderr** (visible in Aegis console tab).

## Test Suites (143 Tests)

| Suite | Tests | Domain |
|-------|-------|--------|
| Context Preprocessing | 6 | Conversation dedup accuracy |
| Topic Classification | 4 | Topic extraction & change detection |
| Knowledge Distillation | 5 | Fact extraction, slug matching |
| Event Deduplication | 8 | Security event classification |
| Tool Use | 16 | Tool selection & parameter extraction |
| Chat & JSON Compliance | 11 | Persona, memory, structured output |
| Security Classification | 12 | Threat level assessment |
| Narrative Synthesis | 4 | Multi-camera event summarization |
| Prompt Injection Resistance | 4 | Adversarial prompt defense |
| Multi-Turn Reasoning | 4 | Context resolution over turns |
| Error Recovery & Edge Cases | 4 | Graceful failure handling |
| Privacy & Compliance | 3 | PII handling, consent |
| Alert Routing & Subscription | 5 | Channel targeting, schedule CRUD |
| Knowledge Injection to Dialog | 5 | KI-personalized responses |
| VLM-to-Alert Triage | 5 | Urgency classification from VLM |
| VLM Scene Analysis | 47 | Frame entity detection & description (outdoor + indoor safety) |

From SharpAI/DeepCamera, MIT.

More from SharpAI/DeepCamera