taxonomy-mapper

Files (10)

• SKILL.md
• master_tree.json · resources/
• master_tree_embeddings.json · resources/
• mlo-knowlege-tree.tsv · resources/
• detect_master_gaps.py · scripts/
• extract_document_hierarchy.py · scripts/
• parse_master_tree.py · scripts/
• query_master_tree.py · scripts/
• semantic_search_master.py · scripts/
• test_taxonomy_improvements.py · scripts/

SKILL.md (first lines)

# Taxonomy Mapper

> **Goal:** You are the `@taxonomy-mapper`. Take the raw syllabus topics from `/context-audit` and map them to the Master Knowledge Graph using N:N Multi-parent Topology. If `concept_candidates.tsv` exists (from `/escalate-concepts`), use those pre-computed `matched_master_code` hints as the **primary mapping signal** — do NOT re-derive what ATE has already resolved.

## Inputs

| File | Required | Role |
|------|----------|------|
| `.work/context-audit.md` | ✅ | Domain breakdown, knowledge scope |
| `output/concept_candidates.tsv` | 🔵 optional | **Pre-computed concept→Master mapping** from ATE pipeline (`/escalate-concepts`) |

## Outputs
- `.work/mapping-plan.md`

## Process

### Step 1 — Load ATE Concept Hints (if available)

Check whether `projects/<project>/output/concept_candidates.tsv` exists.

If it does, read it and split into 2 groups:
- **Matched** (`is_new_concept=False`, `match_confidence >= 0.80`): `matched_master_code` is already known → include directly in the plan, marked `[ATE-MATCHED]`. **No need to search the Master Tree again.**
- **Low-confidence** (`0.70 ≤ match_confidence < 0.80`): include in the plan with the `[VERIFY]` flag for reviewer confirmation.
- **Gap D** (`is_new_concept=True`): no code yet → propose `[NEW NODE PROPOSAL]`.

Mapping from ATE **replaces** the need to search/infer from the Master Tree for concepts that already exist.
Only perform a fresh search for domains in `context-audit.md` that **do not** appear in `concept_candidates.tsv`.

### Step 2 — Taxonomy Search (for the remainder)

For domains in `context-audit.md` not covered by ATE:
1. Read the master tree from `skills/taxonomy-mapper/resources/mlo-knowlege-tree.tsv` (or `master_tree.json`)
2. Find suitable `fields`, `subjects`, `categories`, `topics`, `concepts`
3. **N:N Graph Reuse Principle:** Prefer REUSING existing Category/Topic nodes rather than creating new ones

### Step 3 — Assemble `mapping-plan.md`

Organize according to a clear structure:

```markdown
## ATE-Matched Concepts (from /escalate-concepts)
- CONCEPT_CODE_1  ← matched_master_code, confidence=0.92
- CONCEPT_CODE_2  ← matched_master_code, confidence=0.85

## [VERIFY] Low-Confidence Matches
- CONCEPT_CODE_3  ← matched_master_code, confidence=0.74 — needs confirmation

## Gap D — New Node Proposals (from /escalate-concepts)
### [NEW NODE PROPOSAL] PROPOSED_CODE
- Name: ...
- Description: ...
- Parent codes: TOPIC_CODE_1, TOPIC_CODE_2

## Additional Taxonomy (from context-audit search)
<!-- Codes found via Master Tree search -->
```

For existing codes: list them only.
For `[NEW NODE PROPOSAL]`: provide the full `UPPER_SNAKE_CASE` code (noun phrase), Name, Description, and parent codes.

### Step 4 — STOP for Approval

Do not write any project TSV files. Present `mapping-plan.md` for user approval, then run `/build-tree`.

## Scripts

- `extract_document_hierarchy.py` — Extract hierarchical structure from source documents (PDF/MD/TXT/DOCX), LLM classifies each node into a Master Tree layer. Output: `structured_hints.json`.
- `semantic_search_master.py` — Semantic search of the Master Tree using embeddings. Uses `master_tree_embeddings.json` to find the top-k nearest concept/topic/category for each syllabus item.
- `detect_master_gaps.py` — Detect syllabus domains not present in the Master Tree. Output: `master_gap_report.md`.
- `parse_master_tree.py` — Parse Master TSV → JSON (uses shared `master_tree_parser.py`).

## Rules

- `match_confidence >= 0.80` → trust & use directly (no re-derivation)
- `0.70 ≤ match_confidence < 0.80` → flag `[VERIFY]`
- Gap D concepts → always `[NEW NODE PROPOSAL]`
- *Note: `apply_plan_to_staging.py` is only used in the `/crawl-roadmap` workflow.*

From orchable/knowledge-graph-public, MIT.

More from orchable/knowledge-graph-public