wiki-ingest

Files (1)

• SKILL.md

SKILL.md (first lines)

# Wiki Ingest Skill

Two workflows, depending on the input.

---

## Branch A: Literature & Document Ingestion
*A raw file or a URL to a paper/article (PDF, HTML, Markdown, text), or a request to ingest a specific reference.*

> [!TIP]
> For automating ingestion (especially of PDFs), you can run the library script locally in a two-step process:
> 1. Run the script with `--extract-only` to download the file (if it's a URL) and extract its text:
>    `python Scripts/library.py <file_path_or_url> --extract-only`
> 2. The script outputs the local path of the file (under `LOCAL_PATH: ...`) and the extracted text. Read the text, perform metadata extraction manually as the LLM (following the schema in A3), and save the structured metadata to a temporary JSON file inside the conversation's scratch directory: `<appDataDir>\brain\<conversation-id>\scratch/metadata.json`.
> 3. Complete the ingestion by running the script again with `--metadata`, `--no-confirm`, and `--delete-original`:
>    `python Scripts/library.py <local_path_returned> --metadata <path_to_metadata_json> --no-confirm --delete-original`
> This automatically handles file renaming/copying, markdown note creation, and index updating. Proceed to Branch B if needed.

### A1: Locate Source
- URL (starts with `http`): download to `Wiki/Library/tmp.<ext>`. Use `.pdf` for arbitrary/binary URLs; preserve `.html`/`.md`/`.txt` if present in the path.
- Local path: locate it.

### A2: Extract Text
Read Markdown/text directly. For PDF/HTML, parse text cleanly, ignoring binary/markup wrapping. You may use the command line tool "markitdown" for this.

### A3: Extract Metadata
1. **title:** APA-style Title Case.
2. **filename:** from title + year, descriptive, under 40 chars (e.g. `Attention_Is_All_You_Need_2017`). Replace spaces, hyphens, slashes, backslashes, dots, question marks, curly braces, and `'s` with single underscores; strip double/leading/trailing underscores.
3. **tag:** one category from `default_tags` in the plugin's `tags.json`.
4. **tag_list:** up to 5 PascalCase tags for indexing, excluding any in `negative_tags` (`tags.json`).
5. **abstract:** exact abstract from the paper; replace unicode with ASCII/LaTeX equivalents; inline math as `$...$`.
6. **topic1-4:** one short sentence each, starting with an action verb - main method, second method, discussion, results (e.g. "Presents...").

### A4: Confirm with User
- Check whether the document is relevant to existing projects, collaborators, or pages - i.e. whether Branch B synthesis applies.
- Present the metadata (Title, Filename, Tag, Tag list, Abstract, Topics) then ask: **"Is this correct?"**

### A5: Save, Index, Synthesize (once confirmed)
1. **Copy source** to `Wiki/Library/<filename>.<ext>`; delete the temp download if used.
2. **Branch B (if confirmed):** run steps B2 and B3 using the paper's contents.
3. **Markdown note (PDFs only)** at `Wiki/Library/<filename>.md`:
   ```markdown
   >[!abstract]-
   > <abstract>
   - :luc_file_text: [[<filename>.pdf]]
   - <topic1>
   - <topic2>
   - <topic3>
   - <topic4>
   #<tag> #<tag_list_item1> #<tag_list_item2> ...
   -------------
   ```
4. **Index** — prepend to the top of `Wiki/_Index.md`:
   ```markdown
   ### <title>
   >[!abstract]-
   > <abstract>
   - :luc_file_text: [[<filename>.pdf]] ❌
   - :luc_file: [[<filename>.md|Notes]]
   - <topic1>
   - <topic2>
   - <topic3>
   - <topic4>
   ```
   *(Non-PDF source: omit the `- :luc_file:` line.)*

---

## Branch B: Research Notes & Project Ingestion
*Raw research notes, discussion questions, project updates, or meeting minutes.*

### B1: Synthesize
Read the input to identify the core concept, project updates, methods, or open questions. Clarify anything ambiguous with the user.

### B2: Create or Update Pages
Draft or modify the relevant page in `Wiki/Projects/` or `Wiki/`. Merge rather than overwrite — note whether new input refines, expands, or contradicts prior claims. Add Obsidian links (`[[Link]]`) connecting the concept to existing projects, collaborators, and methods.

### B3: Update Navigation
- Prepend to `Wiki/log.md`: `## [YYYY-MM-DD HH:MM] ingest | <Title>`.
- If new pages were created, add their links to `Wiki/wiki.md` under the right category.

## Reminders
- Only touch `Wiki/_Index.md` and `Library` via the steps above; do not otherwise hand-edit them.
- Preserve existing cross-references when editing a page.
- Tone: dry, objective, scientific, concise.

From not-a-feature/wiki-skill, LGPL-2.1.

More from not-a-feature/wiki-skill