archive-crawler

Files (2)

• SKILL.md
• routing-eval.jsonl

SKILL.md (first lines)

# archive-crawler — The Universal Archivist

> **Convention:** see [conventions/quality.md](../conventions/quality.md) for
> citation rules, exact-phrasing requirements when capturing the user's
> reactions, and back-link enforcement.
>
> **Convention:** see [_brain-filing-rules.md](../_brain-filing-rules.md) —
> this skill is **schema-generic**: it reads the user's filing rules from
> the rules JSON instead of hardcoding any specific era / archive layout.

## Safety gate (REQUIRED, no exceptions)

archive-crawler refuses to run unless `archive-crawler.scan_paths:` is
explicitly set in `nbrain.yml`. This is a deliberate safety fence against
the agent over-scoping a scan and ingesting sensitive content (tax PDFs,
medical records, credentials).

```yaml
# nbrain.yml — the allow-list is mandatory
archive-crawler:
  scan_paths:
    - ~/Documents/writing/
    - ~/Dropbox/Archive/
    - /mnt/backup/old-letters/
  # Optional deny-list inside the allow-list:
  # deny_paths:
  #   - ~/Documents/finances/
  #   - ~/Documents/medical/
```

If `scan_paths` is empty or missing, the skill exits with:

```
archive-crawler: refusing to run. No `archive-crawler.scan_paths:` allow-list
in nbrain.yml. Add explicit paths the agent is permitted to scan, then re-run.
This is a safety fence — the agent will not infer what's safe to read.
```

This contract is enforced by `src/core/storage-config.ts` (mirrors the
`db_tracked` / `db_only` allow-list pattern from v0.22.11 storage tiering).

## What this is

Generic engine for exploring any tree of personal content within an
explicit allow-list. Works on local mounts, Dropbox API targets,
Backblaze B2, Gmail takeouts (`.mbox`), and similar archives. Filters
for "gold" (the user's own writing, ideas, relationships) and surfaces
it interactively for review. Skips noise (system files, configs, binary
blobs).

## Concepts

### Source

A source is any tree of files to explore. Sources have:

- **type**: `local` | `dropbox` | `backblaze` | `gmail-takeout` | `mbox` | `pst`
- **root**: filesystem path, Dropbox path, B2 prefix, mbox path
- **manifest**: a brain page tracking progress at
  `projects/<archive-slug>/STATUS.md`

### Manifest

Every archive exploration gets a manifest brain page that tracks:

1. **Tree inventory** — folders / files / sizes / types
2. **Triage status** — each item: `⬜ unseen` / `👀 reviewed` /
   `✅ ingested` / `⏭️ skip` / `🔥 high-signal`
3. **User reactions** — exact quotes when they react (per
   conventions/quality.md exact-phrasing rule)
4. **Priority queue** — what to explore next, ranked
5. **Session log** — timestamped record of what was shown per session

### Gold filter

Before showing anything to the user, apply the gold filter:

| Keep (show) | Skip (note existence, don't show) |
|-------------|-----------------------------------|
| Personal writing (journals, letters, reflections, essays) | System files, configs, package.json, node_modules |
| Conversations (IM logs, email threads with substance) | Binary blobs (images / video) |
| Ideas, theses, frameworks | Receipts, invoices, tax docs |
| Relationship material (letters to / from people who matter) | Spam, newsletters, mailing-list bulk |
| Creative work (poetry, stories, code with soul) | Corrupted / null files |
| Origin stories (first versions of things that became important) | |
| Emotional content (anger, love, grief, discovery) | |

## Protocol

### Phase 1: Inventory

When pointed at a new source:

1. **Confirm scan_paths is set** (safety gate). Exit if not.
2. **Map the tree** — list folders + files + sizes + date ranges.
3. **Classify folders** — group by likely content type (writing, email,
   code, photos, docs, system).
4. **Create manifest** — write `projects/<archive-slug>/STATUS.md` with
   the full inventory.
5. **Propose priority queue** — rank folders by likely gold density.
6. **Present to user** — show the map and proposed order. Let them
   override.

### Phase 2: Crawl

Work through folders in priority order:

1. **Read before showing** — open each candidate file, apply the gold
   filter, skip noise.
2. **Show one at a time** — present gold items individually for review.
3. **Capture exact reaction** — track the user's response in the
   manifest using their exact words (per conventions/quality.md).
4. **Ingest if worth keeping** — create a brain page immediately.
5. **Update manifest** — mark item status after each interaction.
6. **Never re-show** — check the manifest before presenting anything.

### Phase 3: Ingest

When an item is worth keeping, file it by **primary subject** per

From nastechresearch/nbrain, MIT.

More from nastechresearch/nbrain