keyword-extractor

Files (13)

• SKILL.md
• 1_extract_graph_chunks.py · scripts/
• 2_merge_and_build_graph.py · scripts/
• 3_escalate_to_concepts.py · scripts/
• 4_extract_action_phrases.py · scripts/
• chunk_source.py · scripts/
• export_keywords.py · scripts/
• filter_by_relevance.py · scripts/
• gen_statistical_candidates.py · scripts/
• llm_call.py · scripts/
• llm_escalate_concepts.py · scripts/
• llm_gen_candidates.py · scripts/
• llm_verify_and_dedup.py · scripts/

SKILL.md (first lines)

# Keyword Extractor (ATE Pipeline)

> **Goal:** Extract all terms related to `target_context` from source documents (PDF/MD/TXT). Optimized for **recall** — not missing any terms, even those appearing only once — followed by dedup and omission checking to ensure completeness.

---

## Why a Multi-Stage Pipeline

A single pass (RAKE/YAKE/LLM once) is insufficient because there are two distinct sources of error:

- **Statistical (YAKE):** underestimates rare terms (frequency 1).
- **Single-pass LLM over long documents:** attention degrades in the middle of the document → misses the middle section.

This pipeline combines both tracks, filters for domain relevance, then performs dedup + omission-check.

---

## Quality Rules (Mandatory)

1. **Recall-First Filtering** — The cosine threshold in `filter_by_relevance.py` MUST be lenient (default `≥ 0.25`). Precision is handled in `llm_verify_and_dedup.py`. Early filtering cannot be recovered in later steps.
2. **No Self-Curating Prompts** — The LLM candidate-gen prompt MUST be explicit: _"list ALL terms, including those appearing only once, DO NOT filter by importance."_
3. **Omission-Check In-Context** — Do not split omission-check into a separate call. The model MUST see the entire existing list + the original chunk in THE SAME single call.
4. **Source Traceability** — Every term in the output MUST carry `source_chunks` (source chunk_id) and `first_extraction_method` (statistical / llm / omission_check).
5. **Omission Loop Limit** — Maximum 2 omission-check loops. Stop early if any loop has Δ = 0 (no new terms).

---

## Inputs

- `target_context` — String describing the target topic (e.g. `"Arduino programming for ESP32-S3 as an IoT product"`)
- Source files: PDF, MD, or TXT in `projects/<project>/context/` or an arbitrary path
- `OPENAI_API_KEY` in `.env` (for LLM + embedding scripts)

## Outputs

| File                                   | Description                                                          |
| -------------------------------------- | -------------------------------------------------------------------- |
| `.work/kw/chunks.json`                 | Chunks with heading_trail                                            |
| `.work/kw/candidates_statistical.json` | YAKE candidates                                                      |
| `.work/kw/candidates_llm.json`         | LLM candidates                                                       |
| `.work/kw/candidates_filtered.json`    | Union filtered by embedding cosine                                   |
| `.work/kw/candidates_filtered.md`      | Preview markdown for quick agent review                              |
| `.work/kw/keywords_verified.json`      | After dedup + omission-check                                         |
| `.work/kw/verify-report.md`            | **Human review checkpoint**: before/after, delta from omission-check |
| `output/keywords.tsv`                  | Final output (6 columns: term, aliases, relevance_score, source_chunks, first_extraction_method, category) |
| `output/keywords.json`                 | Final output as JSON                                                 |

---

## Scripts

```
skills/keyword-extractor/scripts/
├── chunk_source.py              # ❌ LLM  — cut source by heading, PDF/MD/TXT
├── gen_statistical_candidates.py # ❌ LLM  — YAKE on each chunk
├── llm_gen_candidates.py        # ✅ LLM  — exhaustive listing per-chunk
├── filter_by_relevance.py       # ⚠️ embed — cosine vs target-context (lenient)
├── llm_verify_and_dedup.py      # ✅ LLM  — dedup variants + omission loop
├── export_keywords.py           # ❌ LLM  — write final TSV/JSON output
└── llm_escalate_concepts.py     # ✅ LLM  — keyword → neutral concept + match Master Tree
```

---

## Workflow Commands

| Command                                       | Owner                | LLM? | Result                                                              |
| --------------------------------------------- | -------------------- | ---- | ------------------------------------------------------------------- |
| `/scaffold-keywords <target> --source <path>` | scaffolder           | ❌   | Create `.work/kw/`, write config                                    |
| `/extract-terms`                              | `@keyword-extractor` | ✅   | `candidates_filtered.md`                                            |
| `/verify-terms`                               | `@keyword-extractor` | ✅   | `verify-report.md` ← **review checkpoint**                          |
| `/finalize-keywords`                          | `@keyword-extractor` | ❌   | `output/keywords.tsv` + inject into context-audit                   |
| `/escalate-concepts`                          | `@keyword-extractor` | ✅   | `concept_candidates.tsv` + `concept_escalation.md` ← **review checkpoint** |

---

## Integration with `/context-audit`

After `/finalize-keywords`, the agent will:

1. Copy `output/keywords.tsv` into `projects/<project>/context/keywords.tsv`
2. Update `projects/<project>/.work/context-audit.md` by adding a **`## ATE Keywords`** section containing the exhaustive term list
3. `@context-analyzer`, when running `/context-audit`, will check for the existence of `keywords.tsv` and integrate it into the domain breakdown

---

## Run Sequence

```bash
# Step 1: Scaffold (no API key needed)
# /scaffold-keywords "Arduino ESP32-S3 IoT" --source projects/<slug>/context/

# Step 2: Extract (requires OPENAI_API_KEY)
# /extract-terms

# Step 3: Verify + review verify-report.md
# /verify-terms

# Step 4: Finalize (inject into context-audit)
# /finalize-keywords
```

## Dependencies

```bash
pip install pdfplumber yake numpy openai pydantic
```

---

## Alternative/Experimental Pipeline: GraphRAG

> ⚠️ The pipeline below is experimental; do not use it in the standard workflow (`/run-pipeline`). Retained for reference.

This skill uses an LLM-Native GraphRAG model to scan the entire project document, extract Entities (Nodes) and Relationships (Edges), then filters down to neutral Concepts.

### Output Files (GraphRAG)

1. `.work/kw/chunks_graph/*.json`: Local graphs for each chunk.

From orchable/knowledge-graph-public, MIT.

More from orchable/knowledge-graph-public