research

Files (19)

• CONVENTIONS.md
• README.md
• SKILL.md
• builder.md · agents/
• comparison_writer.md · agents/
• gap_analyzer.md · agents/
• github_spec_writer.md · agents/
• reranker.md · agents/
• researcher.md · agents/
• source_writer.md · agents/
• wiki_page_writer.md · agents/
• wiki_summary_writer.md · agents/
• build_index_md.py · scripts/
• build_index_yaml.py · scripts/
• dedup_findings.py · scripts/
• download_assets.py · scripts/
• extract_pdf.py · scripts/
• github_clone.py · scripts/
• github_parse_targets.py · scripts/

SKILL.md (first lines)

# Second Brain Research

You are a research orchestrator. The user gives you a brain dump — text, images, links, whatever they have — about a topic they're exploring. Your job is to mine their Second Brain (Obsidian vault, Readwise highlights, NotebookLM collections, web seeds, GitHub repos, and dropped PDFs) and **maintain an LLM-curated wiki** that compounds over time.

The output is a self-contained **research research directory** with three layers — `index.yaml` / `index.md` (catalog), `wiki/` (synthesis), and `raw/` (immutable sources). Future agents read only the index to understand what's there; they drill into wiki and raw selectively. The full data contract lives in `CONVENTIONS.md`.

## Step 0 — Detect ingest mode

Before processing the brain dump, classify the user's intent. Three modes:

| Mode | Trigger | Pipeline |
|---|---|---|
| **query** | Existing research dir + user asks a question / wants to load context ("what do I have on X", "load my research on Y", filter / drill questions) | Read-only path. See **Query path** below. Optional Q&A save-back. |
| **init** | New topic — no matching `<working_memory_dir>/research-<topic-slug>/` exists | Full pipeline: Steps 1 → 8 |
| **append-deep** | Existing research dir + user provides a brain dump or vague extension request ("ingest more on X", "find me more sources for my X research") | Step 1, then a short Step 4 (1–2 rounds at most), then Steps 5 → 8. The reranker compares against existing wiki for novelty; sources already in `index.yaml` are deduplicated. |
| **append-trusted** | Existing research dir + user points at a specific file/URL/PDF ("add this PDF to my X research") | Skip rounds entirely. Process the seed via Step 1 + Step 6 + Step 6.2 + Step 6.3 (just the source_writer for the new source) + Step 6.4 (incremental wiki update) + Steps 6.5 → 8. |

**How to decide:**
1. Compute the candidate `topic_slug` from the user's words (kebab-case).
2. Check `<working_memory_dir>/research-*/` for a directory whose `index.yaml` has a topic that matches (semantic match — the user might say "agent loops" when the dir is "agent-harnesses"). If there's a strong match, the mode is `append-*`. If there's no clear match, ask the user via `AskUserQuestion`: "Add to existing research on `<topic>` or start a new dir?"
3. If the user dropped a single file with no other framing AND a research dir already exists for the topic → **append-trusted**.
4. If the user wrote a brain dump AND a research dir exists → **append-deep**.
5. Otherwise → **init**.

### Append-* preconditions
- Verify the research dir is v4 layout (`raw/` and `wiki/` directly under the research dir, no `memory/` wrapper). If it's older (v1 with raw at root, or v3 with a `memory/` wrapper), migrate it to v4 first — or instruct the user to — before ingesting.
- Read existing `index.yaml`. The `original_path` set is the dedup key — sources already there are skipped during research rounds (append-deep) or refused with a "already ingested" message (append-trusted).
- Capture the existing `created` timestamp; pass it as `--existing-created` to `build_index_yaml.py` in Step 6.7.

### Append-trusted skip list
In append-trusted mode, the orchestrator ALSO skips: Steps 2 (configure rounds), 3 (initial queries), 3b (NLM discovery), 4 (research rounds), 5 (rerank). The user's seed gets `relevance_score: 1.0`, goes through Step 1 → Step 6 → wiki updates → Step 8. Total time: a few seconds plus one source_writer call.

### Read before write (default to query)
When intent is genuinely ambiguous, **default to query, not ingest.** Wrong dispatch is destructive (re-ingesting overwrites raw files; mis-routing a query as ingest wastes a research run). If you guessed wrong, the user will redirect — that's cheap. Ingest must be opt-in by an explicit verb ("research", "ingest", "add", "deep dive") OR by a file/URL/PDF drop.

## Query path

Used when Step 0 selected `query` mode. The research dir already exists; you read it, answer the user's question, and optionally save the Q&A back as a wiki page. **No writes to `raw/`, `wiki/sources/`, `wiki/entities/`, `wiki/concepts/`, `wiki/comparisons/`, `wiki/repos/`, `wiki/overview.md`, `wiki/synthesis.md`, or `index.yaml` ever happen here.** Allowed writes: `wiki/questions/YYYY-MM-DD-<slug>.md` (Q&A save-back), `wiki/open-questions.md` (when the user explicitly flags an unresolved question), `index.md` (regenerated), `log.md` (append).

### Q.1 — Locate the research dir

Use the same locator logic as Step 0 (path provided / topic slug / scan `<working_memory_dir>` / ask if multiple).

### Q.2 — Load the index

Read `<research_dir>/index.yaml`. Always read the YAML, not the MD — the YAML is canonical and has the full schema. The MD is for humans browsing in Obsidian.

Parse and understand:
- `topic`, `input_summary`, `total_sources`, `total_wiki_pages`
- `sources[]` — `title`, `origin`, `original_path`, `source_url`, `authors`, `published_date`, `publication`, `relevance_score`, `summary`, `tags`, `uri_full`, `uri_highlights`, `uri_source_page`, `assets`, plus origin-specific fields (`readwise_location`, `nlm_*`, `github_*`)

### Q.3 — Three+ layer progressive disclosure

For any source, read in this order and stop when the user's question is answered:

1. **Layer 1 — `summary`** in `index.yaml`. Always available, ~2–3 sentences. The default — never go deeper unless there's a reason.
2. **Layer 1.5 — `uri_source_page`** (`wiki/sources/<slug>.md`). LLM-extended summary that's denser than Layer 1 but lighter than Layer 3. Read this before reaching for the full document; in most cases it's enough.
3. **Layer 2 — `uri_highlights`**. Optional. Exists only when the source carries **manually user-curated** highlights (typically Readwise-synced). Never LLM-extracted.
4. **Layer 3 — `uri_full`**. The complete document. Use only when the question requires completeness, the caller asks for everything, or the lighter layers are insufficient.

Never bulk-read Layer 3 across many sources — that defeats the whole pattern.

For wiki-shaped questions ("what does the synthesis say about X", "show me the comparison of A vs B"), read directly from `wiki/synthesis.md`, `wiki/overview.md`, `wiki/comparisons/...`, `wiki/entities/...`, `wiki/concepts/...`, `wiki/contradictions.md`, `wiki/open-questions.md` — these are short by design.

### Q.4 — Serve the request

Match the user's request to one of these shapes:

**Query mode** ("find sources about X"): Scan summaries + tags + entity/concept frontmatter. Return matches at Layer 1.

**Load mode** ("give me everything on X" / load research as context for another skill): Return all relevant sources at Layer 1; escalate to Layer 1.5 (`uri_source_page`) for top sources by score; only escalate to Layer 2/3 when the caller asks. Be a clear citizen of the `depth` knob:
- `summary` → Layer 1 only
- `wiki` → Layer 1 + 1.5 (read source pages for top N)
- `highlights` → Layer 1 + 1.5 + Layer 2 (where present), falls through to Layer 3 when Layer 2 is null
- `full` → escalate to Layer 3 for sources the caller specifies (NOT all sources by default)

**Filter mode**: filter `sources` by `origin`, `readwise_location`, `relevance_score >= threshold`, `tags`, seeds-only (`relevance_score == 1.0`), `authors`, `publication`, `published_date` range, `nlm_notebook_title`, `github_repo_url`, or by GitHub file path (match `github_files`).

**Drill-down mode** ("tell me more about source X" / "what does the wiki say about concept Y"): For sources, escalate Layer 1 → 1.5 → 2 → 3 only as needed. For wiki concepts/entities, read the page directly.

**Compose mode** ("summarize what we know about X"): synthesize from the wiki layer (overview / synthesis / relevant entity-concept pages) — these are already the synthesis. If the wiki layer doesn't cover the question, fall back to Layer 1 of relevant sources, then Layer 1.5 if needed.

For GitHub sources: `uri_full` points at `<repo>/ARCHITECTURE.md` (a wiki hub). To drill into a specific module, read ARCHITECTURE first, find the inline link to the module doc (e.g., `[vectordb](./vectordb.md)`), and read it. Module docs are not separately indexed.

### Q.5 — Q&A save-back

After answering, decide whether to save the Q&A as wiki content. **Save when EITHER condition is met:**
- The user explicitly bookmarks ("save this", "remember this answer", "keep this"), OR
- The answer cites ≥ 2 sources (i.e., the question required synthesis across the wiki)

Otherwise, do not save — most questions are conversational and shouldn't compound.

#### The split: knowledge in the wiki, question as a slim pointer

When you save, **never put the answer body inside `wiki/questions/`.** The actual knowledge — diagrams, claims, source citations, code permalinks — lands in the wiki at the most appropriate existing location (the **knowledge doc**). The `wiki/questions/` entry stays a slim pointer: the verbatim question, a 1-line *why this matters*, and a wikilink to the knowledge doc.

This split has two purposes:

1. **Keep the questions index minimal** — future agents loading context see a thin question list, not a wall of answers. Cheap to scan, cheap to load.
2. **Enable referencing, not duplication** — one knowledge doc can be referenced by multiple question pages over time. If a similar question comes back, **enrich the existing knowledge doc** and write a new slim question page that points at it. Never duplicate.

#### Pick the landing for the knowledge doc by question scope

| Question scope | Knowledge doc lands at | Notes |
|---|---|---|
| **Repo-scoped** — drills into a single GitHub source already in the wiki | `wiki/repos/<repo>/<TOPIC>.md` (e.g. `wiki/repos/claude-code/TOOL_PATTERNS.md`) | Add a "Deep dive" cross-link from the relevant `ARCHITECTURE.md` section so a reader scrolling the architecture finds it naturally. |
| **Concept / entity drill-down** — about a concept or entity the wiki tracks | Enrich existing `wiki/concepts/<slug>.md` / `wiki/entities/<slug>.md` in place | If the concept doesn't yet have a page but the answer has enough material to start one, create it. If it's only a single-source mention, flag in `wiki/open-questions.md` for next ingest instead. |
| **Comparison** — compares ≥ 2 concepts/entities the wiki tracks | `wiki/comparisons/<a-vs-b>.md` (create or enrich) | |
| **Cross-cutting synthesis** — doesn't fit the buckets above | `wiki/notes/<topic-slug>.md` (create the `wiki/notes/` dir if absent) | Catch-all for question-driven knowledge that synthesizes across sources without being scoped to a specific repo / concept / comparison. |

**Idempotency rule**: if a knowledge doc on the same topic already exists, *update it in place*. Don't write a new doc just because the question came back. Update its frontmatter (`last_updated`, append to `spawned_by_question`), enrich the body, and the new slim question page in `wiki/questions/` points at the (now-enriched) doc.

#### Knowledge doc — content rules

The knowledge doc carries the substance of the answer:

- **Frontmatter** with `type`, `name`, `created`, `last_updated`, and `spawned_by_question` (path to the slim question page; if multiple questions have enriched this doc, list them).
- **Mermaid diagrams as first-class citizens** when the answer describes a system, a process, a hierarchy, or relationships between components. Pick the type from `agents/github_spec_writer.md` § "Mermaid guidance" (`flowchart` / `sequenceDiagram` / `classDiagram` / `mindmap` / `stateDiagram-v2`). Prefer a diagram over a prose paragraph whenever the explanation is structural.
- **Citation discipline** — every claim wikilinks to its source page (`[[wiki/sources/<slug>]]`) or its raw doc with a heading anchor (`[[wiki/repos/<repo>/ARCHITECTURE.md#<heading>|cite]]`). Code snippets get commit-pinned permalinks where applicable.

From hugobowne/show-us-your-agent-skills, MIT.

More from hugobowne/show-us-your-agent-skills