insight-extractor

Files (10)

• SKILL.md
• output-format.md · references/
• pdf.md · references/
• web-article.md · references/
• youtube.md · references/
• fetch_article.py · scripts/
• fetch_youtube.py · scripts/
• process_pdf.py · scripts/
• extract_pdf.py · scripts/utils/
• parse_vtt.py · scripts/utils/

SKILL.md (first lines)

# Insight Extractor

Extract crisp, actionable insights from YouTube videos, web articles, and research papers.
Outputs go to `knowledge/insights/`; raw fetched content is preserved in `knowledge/raw/`.

**Always read `references/output-format.md`** for the required output structure, insight
categories, file naming, and quality bar.

---

## Workflow

```
1. Detect resource type  (table below)
2. Multiple resources? → ask standalone vs. consolidated first, then proceed
3. Ensure workspace folders exist
4. Extract content  (follow the reference file for the detected type)
5. Generate insights  (per references/output-format.md)
6. Save to knowledge/insights/<slug>.md and confirm path to user
```

---

## Step 1 — Detect Resource Type

| Input pattern | Type | Reference |
|---|---|---|
| `youtube.com/watch`, `youtu.be/` | YouTube video | `references/youtube.md` |
| `arxiv.org`, `semanticscholar.org`, `openreview.net`, `dl.acm.org`, `ieeexplore.ieee.org`, or any URL ending in `.pdf` | PDF (remote) | `references/pdf.md` |
| Local file path ending in `.pdf` | PDF (local) | `references/pdf.md` |
| Any other `http/https` URL | Web article / blog | `references/web-article.md` |

---

## Step 3 — Ensure Workspace Folders Exist

```bash
mkdir -p knowledge/insights knowledge/raw/transcripts knowledge/raw/articles knowledge/raw/pdfs
```

---

## Step 4 — Extract Content

Read the reference file for the detected type and follow it exactly. Each reference file
specifies where to save intermediate raw content within the `knowledge/raw/` folder.

---

## Step 5 — Generate Insights

Read `references/output-format.md`. Key principles:
- Every bullet = something worth remembering or acting on — no padding.
- Extract the *implication*, not a restatement of what the resource says.
- Skip categories with nothing meaningful.
- Add diagrams (Mermaid / ASCII) only where they compress a concept better than bullets.

---

## Multiple Resources

When the user provides 2+ resources in one request:
1. List detected types (don't extract yet).
2. Ask: **"Separate insight files for each, or one consolidated file grouped by theme?"**
   - Standalone: different topics, or resources will be referenced independently.
   - Consolidated: shared theme, cross-referencing adds value.
3. Extract all (parallelize where possible).
4. Consolidated output: group by insight category, tag each insight's source in brackets.

---

## Extensibility

To add a new resource type:
1. Add a row to the detection table above, pointing to a new reference file.
2. Create `references/<type>.md` — follow the same pattern as existing reference files:
   acquire content → save raw to `knowledge/raw/<type>/` → produce clean plain text.
3. Steps 5–6 require no changes.

Future candidates: podcasts (audio transcript API), GitHub repos, Notion exports, newsletter issues, Twitter/X threads.

From tezansahu/ai-garage, GPL-3.0.

More from tezansahu/ai-garage