wiki-fetch

Files (1)

• SKILL.md

SKILL.md (first lines)

# wiki-fetch

Orchestrate fetching a URL or research query into the vault using the best available adapter, then clean, tag, and deduplicate the result.

Use when asked to:
- "fetch this URL into the vault"
- "ingest this page"
- "research this topic"
- "add this to raw/"
- "scrape this site"
- "pull this into the wiki"

**User-visible progress** — Before any long command, state **which adapter** you will try first and **why** (from the priority table below). After **`llm-wiki ingest …`** or equivalent, **summarize stdout/stderr** (paths under **`raw/`**, errors, retries). When handing off to **wiki-ingest**, follow that skill’s **Visibility** section so the merge is not silent.

---

## Adapter Priority

**Priority 0 — Read the config first:**

Before checking what's installed or which keys are set, open `llm-wiki/config.json` → `integrations` and build the enabled list. Only consider adapters where **`enabled: true`**; skip any adapter disabled in config even if the key exists.

```bash
llm-wiki integrations status   # shows enabled/disabled + key check for each adapter
```

Then use the **first** ready and enabled adapter from this order:

| Priority | Adapter | Config key | Condition | Best for |
|----------|---------|------------|-----------|----------|
| 0 | **config check** | — | Read `llm-wiki/config.json`; build enabled list | — |
| 1 | **Brave Search** | `integrations.brave` | `enabled: true` + `BRAVE_SEARCH_API_KEY` set | LLM-ready extracted content, news, web, AI answers |
| 2 | **Firecrawl CLI** | `integrations.firecrawl` | `enabled: true` + `which firecrawl` succeeds | Cleanest markdown, JS-rendered pages |
| 3 | **Firecrawl REST** | `integrations.firecrawl` | `enabled: true` + `FIRECRAWL_API_KEY` set | Same quality, no CLI install |
| 4 | **Perplexity** | `integrations.perplexity` | `enabled: true` + `PERPLEXITY_API_KEY` set | Research questions / synthesis |
| 5 | **Twitter** | `integrations.twitter` | `enabled: true` + any `x.com`/`twitter.com` URL | Tweets (zero-config via FxTwitter); threads/search with bird CLI |
| 6 | **HackerNews** | — | URL matches `news.ycombinator.com` | HN threads + comments |
| 7 | **Playwright** | `integrations.playwright` | `enabled: true` + `pip install playwright` + `playwright install chromium` | Same **`raw/`** markdown shape as **`url`** — use when the page is JS-heavy or **`url`** returned empty |
| 8 | **stdlib url** | — | Always available | Fallback — HTML stripped to text |

**If `url` output is empty or tiny:** Offer **`playwright`** (CLI adapter above) **or** Firecrawl **or** install **Playwright MCP** in Cursor/Claude for interactive browsing — then still save via **`llm-wiki ingest playwright …`** or paste into **`raw/`** using the same title/URL/body pattern as other clips.

---

## Step 1 — Determine what is being fetched

Ask the user (or infer from context):

- **Is it a URL?** → Use Firecrawl CLI/REST or HackerNews adapter
- **Is it a research question / topic?** → Use Perplexity adapter
- **Is it a Twitter/X thread URL?** → Use Twitter adapter (if configured)
- **Is it a Firestore collection path?** → Use Firebase adapter (if configured)

---

## Step 2 — Check adapter availability

Run this command and read the output:
```bash
llm-wiki integrations status
```

Parse each line for `checks=ok` (ready) vs `checks=Set ...` (needs setup).

If the preferred adapter is NOT ready, either:
- **Offer to configure it** (run `llm-wiki integrations wizard` or set the key — see Step 2a)
- **Fall back** to the next available adapter and inform the user

### Step 2a — Configure a missing adapter (nested)

If the user wants to set up an integration:

```bash
# Firecrawl CLI (preferred — free tier available)
npm install -g firecrawl-cli
firecrawl login --browser
# OR with API key:
export FIRECRAWL_API_KEY=fc-YOUR-KEY
# Add to ~/.zshrc for persistence

# Firecrawl REST only
export FIRECRAWL_API_KEY=fc-YOUR-KEY

# Perplexity
export PERPLEXITY_API_KEY=pplx-YOUR-KEY

# Twitter
export TWITTER_BEARER_TOKEN=YOUR-TOKEN

# Firebase
export FIREBASE_API_KEY=YOUR-KEY
# OR: export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json

# Playwright (CLI ingest adapter — no API key)
pip install playwright
playwright install chromium
# Optional: enable Playwright MCP in Cursor/Claude for interactive browser tools (separate from vault MCP)
```

Then re-run `llm-wiki integrations status` to confirm.

To persist keys in Claude Code's environment (available to all tools):
Edit `~/.claude/settings.json` and add/update the `env` block:
```json
{
  "env": {
    "FIRECRAWL_API_KEY": "fc-...",
    "PERPLEXITY_API_KEY": "pplx-..."
  }
}
```

---

## Step 3 — Run the ingest

Use the chosen adapter. Supply `--out` to control the destination path inside `raw/`.

```bash

From SkinnnyJay/wiki-llm, MIT.

More from SkinnnyJay/wiki-llm