obsidian-ingest

Files (7)

• SKILL.md
• extract_conversations.py
• fetch_export.py
• fetch_youtube.py
• retire_sources.py
• test_retire_sources.py
• test_vault_config.py

SKILL.md (first lines)

# obsidian-ingest

Universal entry point for ingesting source files from the **vault-root `_sources/`** folder into derived notes. Auto-detects file types and routes accordingly. Never requires a second command.

**One mode, one source directory.** Everything is dropped in `_sources/` at the vault root; routing spans the whole vault and unmatched items are routed interactively. There is no project-scoped variant and no project path argument.

> **`_sources/` at the vault root is the single source of truth.** Per-project `<project>/_sources/` directories are **not** a thing: never scanned, never created, never ingested from. If one appears, it is an anomaly, and `obsidian-lint-light` will surface its contents as orphan sources.
>
> **Two kinds of anomaly. Tell them apart before you touch anything.** Check whether the notes that cite these files record them as `retired: true`.
>
> - **Retired — a restore or a sync brought back deleted files.** Do not move the files, and do not ingest them. Retirement means the content already reached the notes and the hashes are kept as lineage. A move makes the inventory read them as `verdict: new` and re-ingest what the notes already hold. Report the directory to the user and stop.
> - **Not retired — genuinely uningested files.** Move them to the vault-root `_sources/` and ingest from there. Delete the directory afterward, retiring the entries first with `retire_sources.py --scope <project>`.
>
> If some files in the directory are retired and some are not, treat each file on its own terms. Do not apply one verdict to the whole directory.

## Inputs

- **`--status` flag** *(optional)*: report-only mode. When present, the command prints the timestamp of the last ingest run and does nothing else — no source reads, no inventory scan, no writes.

- **`--files-backlog` flag** *(optional)*: backfill mode. Lists already-ingested conversations whose files never reached a note, lets the human pick, and writes only those files. It runs `obsidian-claude-export-ingest` §3b and nothing else — no inventory scan, no normal ingest.

There are no other arguments. Any path supplied is ignored; the source directory is always the vault-root `_sources/`.

## Entry Point Workflow

### Status short-circuit (`--status`)

If invoked with `--status`, do **only** this and then stop — before any preflight or source access:

1. Read `_meta/last-ingest.md`.
2. If it exists, print its timestamp (local time):

   ```
   Last ingest: <YYYY-MM-DD HH:MM ZZZ>
   ```

   Print the human-readable body line verbatim (it is already local time). Do not print the rest of the file.
3. If it does not exist, print:

   ```
   No ingest recorded yet — run /obsidian:ingest to ingest sources.
   ```

4. If any other argument was supplied alongside `--status`, `--status` wins: print the status as above, then note that the other arguments were ignored.

Perform no inventory scan, no `_sources/` reads, and no writes in this mode.

### Files backlog short-circuit (`--files-backlog`)

If invoked with `--files-backlog`, run the preflight (§0), then follow `obsidian-claude-export-ingest` §3b. Then append a design-log line and regenerate indexes (§6, §7) if a note changed. Do not update `_meta/last-ingest.md` — a backfill is not an ingest of new sources.

### 0. Preflight

**Obsidian CLI check (non-fatal):** Run once at session start:
```bash
python3 .agents/skills/obsidian-ingest/extract_conversations.py check-cli
```
If it exits non-zero, note that topic matching will fall back to grep and continue — do not abort the ingest run.

There is no mode to detect. The source directory is always the vault-root `_sources/`, routing always spans the whole vault, and unmatched conversations are always routed interactively by the sub-skill (§2a). Proceed with Steps 1–8.

### 1. Verify _sources/ exists

If the vault-root `_sources/` directory does not exist, stop:

```
No _sources/ directory found at _sources/
Nothing to ingest.
```

### 2–3. Build inventory, discover and classify source files — one command

Run the `inventory` subcommand. It performs **all** of Steps 2 and 3 — note scanning, UUID inventory, SHA256 inventory, ignored-UUID set, source discovery, type classification, hashing, and the change verdict — and emits one JSON blob:

```bash
python3 .agents/skills/obsidian-ingest/extract_conversations.py inventory
```

Take no arguments. (The script accepts a `--scope` flag, but it is a diagnostic left for `retire_sources.py` and stray-directory checks — ingest never passes it.)

> **Dependency:** `inventory` needs PyYAML (`pip install pyyaml`), the same dependency `obsidian-lint-light` already has. It is imported lazily, so every other subcommand — including the Step 0 `check-cli` — still works without it. If `inventory` exits 2 with an install hint, that is the cause.

> **Do not hand-write a scan script, and do not reimplement any of this inline.** Earlier runs wrote throwaway Python to walk notes and parse frontmatter every single time — that is pure token waste and it drifts from the rules below. One call replaces it. Equally: **never use an `ls`/`find` listing already visible in the conversation.** The user may have listed `_sources/` before invoking the skill; that snapshot is stale the moment a file is added. The `inventory` call *is* the discovery step, so running it satisfies this requirement.

**Output fields:**

| Field | Meaning |
|---|---|
| `mode` / `scope` | Always `vault` / `null` for an ingest run |
| `source_dir` / `source_dir_exists` | The vault-root `_sources/` (use for the Step 1 check) |
| `notes_scanned` | Count of notes contributing to the inventory |
| `uuid_inventory` | `{uuid: {note, message_count, last_ingested}}` — pass to the sub-skill |
| `ignored_uuids` | `[uuid]` from `_meta/ignored-uuids.md` |
| `sources[]` | One entry per discovered file: `filename`, `path`, `type`, `sha256`, `verdict`, `notes`, `recorded_sha256`, `missing_uuids` |
| `missing_uuids` | Claude exports only. Conversations present in the file that no note has ingested and that are not in `ignored_uuids`. Non-empty means `verdict` is `incomplete`. Empty for every other type and verdict. |
| `owners` | `{note_path: human\|agent\|shared}` — use for the Step 5d owner check without re-reading frontmatter |

**`type`** is `claude-export` (a `.json` or `.zip` holding a valid conversations export), `claude-projects` (a projects export zip — see §4b), `chatgpt-export` (an OpenAI data export zip — see §4c), or `document` (everything else, including a `.zip` with neither — report those unprocessable).

**`verdict`** is the Step 5b decision, precomputed:

| `verdict` | Meaning | Action |
|---|---|---|
| `unchanged` | Hash matches the value recorded in `notes` | Skip. Count as unchanged. Run nothing further against this file. |
| `incomplete` | Claude export only. Hash matches, but conversations inside it were never ingested | **Do not skip.** Process exactly the uuids in `missing_uuids` and no others. |
| `changed` | Hash differs from the recorded value | Update — append an `## Update` section (§5e) |
| `untracked` | Cited by a note as a flat `sources:` string with no sha256 | Treat as an update; target note is already known |
| `new` | Not cited by any note | New ingest; discover a target via §5c |

> **Why `incomplete` exists.** A multi-conversation export is not one unit of content. The first note that records the export's sha256 used to mark the whole file `unchanged`, so the next run skipped it and every conversation that had not yet been written was dropped — permanently, with nothing in the report saying so. A partial run is ordinary: a crash, a lost network, a context compaction, or the user stopping the run. This was measured on 2026-08-09, when an export read `unchanged` with 5 of 16 conversations ingested. The `inventory` command now checks conversation coverage, not just the hash.

**`notes`** lists *every* note citing that source, not just one — a single export routes to many notes, and a document whose decision touches several notes is cited by each. When `verdict` is not `new`, these are the known targets and §5c discovery is skipped.

> **Why the inventory is keyed on (project folder, filename), not the bare `_sources/<name>` path:** recorded paths are project-relative by convention, so a bare basename key collides across projects. A collision does not merely cause a wrong skip — it can append an `## Update` section to a note in an entirely different project. The script resolves each recorded path project-relative first, then vault-root, matching `lint.py`.

> **Underscore directories are excluded zones** (see AGENTS.md): never scanned for inventory, never used as routing/topic-match targets (§2a / §5c), and never listed as note content in indexes. The `inventory` command already enforces this. Read-only ones — `_sources/` (read only as ingest *input*) and pure-reference dirs like `Reference/_reference/` — are never written to. `_meta/` is the exception: it is the writable operational store (indexes, `last-ingest.md`, design-log, ignored-uuids) and is still excluded from scanning/routing.

> A dropped `.zip` needs no manual extraction — `detect`/`list`/`classify`/`extract` read the export JSON from inside the archive (preferring a `conversations.json` member). Hashing uses the **inner** export content, not the zip packaging — so an unchanged re-download hashes the same (the unchanged-skip actually fires) even though the zip's outer bytes vary, and a zip and a hand-extracted `conversations.json` with identical content share one hash.

From DoxaLogosGit/obsidian-agent-vault, MIT.

More from DoxaLogosGit/obsidian-agent-vault