learning-objective-generator

Files (24)

• SKILL.md
• approaches.tsv · resources/
• bloom_verbs.tsv · resources/
• master_learning_objectives.tsv · resources/
• 0_classify_action_phrases.py · scripts/
• enrich_lo_keywords.py · scripts/
• fix_concept_columns.py · scripts/
• fix_coverage.py · scripts/
• fix_coverage_v2.py · scripts/
• fix_coverage_v3.py · scripts/
• fix_coverage_v4.py · scripts/
• fix_sio_parents.py · scripts/
• fix_validation_errors.py · scripts/
• generate_all_los.py · scripts/
• generate_ulocio_new.py · scripts/
• global_map_prerequisites.py · scripts/
• llm_extract_lo.py · scripts/
• llm_generate_hierarchical_lo.py · scripts/
• llm_map_prerequisites.py · scripts/
• regenerate_swift_los.py · scripts/
• regenerate_ulocio_by_topic.py · scripts/
• test_lo_improvements.py · scripts/
• translate_los_english.py · scripts/
• translate_taxonomy_descriptions.py · scripts/

SKILL.md (first lines)

# Learning Objective Generator

> **Goal:** Generate `learning-objectives.tsv` by extracting LOs from the project's syllabus material. Must run AFTER `/build-tree` so that valid concept codes are available for grounding.

## Pedagogical Model — Abstraction Axis

| Layer   | Nature (Tri-Level Hypothesis [T6])                             | Constraints & Tests                                                                                                   | Identifier Code                    |
| ------- | -------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | ---------------------------------- |
| **ULO** | **Computational Level**: Core competency, WHAT/WHY             | Prioritize high-level Bloom verbs (**Evaluate/Create**); `parent_lo_code = ""`                                        | `ULO-<FEATURE_SLUG>`               |
| **CIO** | **Algorithmic Level**: Procedural representation, **language-neutral** | **MANDATORY Marr 2-Language Test** (test map to $\ge 2$ tools, not tied to token order); each CIO has $\ge 2$ child SIOs | `CIO-<FEATURE_SLUG>`               |
| **SIO** | **Implementational Level**: Technology-specific                | Technology name MUST appear in `name`/`description`                                                                    | `SIO-<TECH_PREFIX>-<FEATURE_SLUG>` |

> ⚠️ Specific technologies (Swift, Python, SQL...) **are determined by context**, not hardcoded. The script automatically detects them from `context/`, `context-audit.md`, `raw_pdf.txt`.

## SIO Cross-Referencing & Naming Conventions

1. **SIO Code Structure:** `SIO-<TECH_PREFIX>-<FEATURE_SLUG>` (in `UPPER_SNAKE_CASE` format).
   - Example: `SIO-JS-OBJECT-LITERAL-METHODS`, `SIO-SWIFT-DICTIONARY-LITERAL-METHODS`, `SIO-PY-DICT-KEYS-VALUES-GET`.
2. **Cross-Technology Pattern Reference:**
   - The **Concept / ULO / CIO** layer is **100% Neutral** and shared across all projects.
   - When building SIOs for a new technology (e.g., Swift), the Agent **MUST REFERENCE & CROSS-CHECK** existing SIOs from other projects (JS, Python...) connected to the same CIO/Concept codes to inherit the structure and convert names/equivalent keywords for the new technology.

## Required Format

The `learning-objectives.tsv` file consists of **9 columns** (tab-separated):

| Column                | Description                                                                 |
| --------------------- | --------------------------------------------------------------------------- |
| `code`                | Identifier code (e.g., `ULO-DATA-TYPES`, `CIO-ARRAY-TRAVERSAL`, `SIO-JS-ARRAY-MAP`) |
| `name`                | Concise name                                                                |
| `description`         | `"The learner will be able to [verb] [object]..."` (in English, 1-3 sentences)      |
| `lo_type`             | ULO → `UNIVERSAL`; CIO → `CONCEPTUAL_IMPL`; SIO → `SPECIFIC_IMPL`           |
| `parent_lo_code`      | ULO → `""`; CIO → ULO code(s); SIO → CIO code(s). Separated by `,` for N:N relationships |
| `concept_codes`       | List of linked concept codes (separated by `,`)                             |
| `bloom_level`         | Bloom level (e.g., `REMEMBER`, `UNDERSTAND`, `APPLY`, `EVALUATE`, `CREATE`) |
| `knowledge_dimension` | `FACTUAL`, `CONCEPTUAL`, `PROCEDURAL`, or `METACOGNITIVE` [T1]              |
| `assessment_approach` | Direct assessment method (e.g., `project`, `quiz`, `code-review`)           |

## Prerequisite

1. `/build-tree` has been run → `concepts.tsv` exists (the LLM needs the list of valid concept codes)
2. `context/` contains a syllabus file (PDF, DOCX...) or `.work/context-audit.md` has content

## Inputs

- `projects/<project>/context/` — original syllabus file (PDF, DOCX...)
- `projects/<project>/.work/raw_pdf.txt` or `.work/context-audit.md` — syllabus text
- `projects/<project>/output/concepts.tsv` — valid concept codes (grounding)

## Outputs

- `projects/<project>/output/learning-objectives.tsv`

## Academic Improvements (Current Version)

### P0 — `assessment_approach` is directly generated by the LLM
- Pydantic models (ULO, CIO, SIO) all have a required `assessment_approach` field
- The LLM is prompted to select an assessment method appropriate to the Bloom level + Knowledge Dimension
- ULO prioritizes `project`, `essay`, `presentation` (relational understanding — Skemp T4)
- CIO prioritizes `code-review`, `quiz`, `project` (algorithmic understanding)
- SIO prioritizes `code-review`, `debugging-exercise`, `quiz` (instrumental understanding)
- Satisfies §7 (Skemp T4, Perkins T8): each ULO/CIO has ≥1 direct assessment method

### P1 — Bloom Verb Validation Matrix
- Resource file: `resources/bloom_verbs.tsv` (6 Bloom levels × allowed verbs)
- The Merge phase automatically checks whether the first verb in the description belongs to the declared Bloom level
- Warnings are written to `.work/hlo/bloom_verb_warnings.json` — does not block, only warns
- Detects misalignment: "explain" declared as EVALUATE, "design" declared as REMEMBER, etc.

### P2 — Inter-rater Reliability for CIO Generation
- Each CIO batch runs **2 independent LLM passes** (temperatures 0.2 and 0.3)
- Union + dedup by code — reduces hallucination, increases coverage
- Pattern similar to ADR-0005 (Concept DAG majority vote)

### P3 — Backward Design (Evidence-First)
- The ULO_SYSTEM prompt adds a step: "Before writing the ULO, ask yourself: How will the learner demonstrate this competency?"
- Evidence determines the `assessment_approach` and Bloom level
- Reversed direction: Concept → Desired Evidence → ULO (instead of Concept → ULO → assessment afterwards)

## Scripts

### Primary: `llm_generate_hierarchical_lo.py` (Hierarchical 4-Phase)

Primary script: generates LOs across 3 separate layers, then merges. Each phase corresponds to a workflow command:

| Phase        | Command                   | Description                                                                           |
| ------------ | ------------------------- | ------------------------------------------------------------------------------------- |
| **A** — ULOs | `/generate-ulos`          | Generate Universal LOs from concepts. Bloom prioritizes Evaluate/Create. **Human review checkpoint.** |
| **B** — CIOs | `/generate-cios`          | Generate Conceptual Impl LOs with **Marr 2-Language Test** per-CIO. **Human review checkpoint.** |
| **C** — SIOs | `/generate-sios`          | Generate Specific Impl LOs (tech-specific)                                             |
| **merge**    | `--phase merge` (separate)| Merge ULO + CIO + SIO into the final TSV file                                          |

```bash
# Phase A: Generate ULOs
python3 skills/learning-objective-generator/scripts/llm_generate_hierarchical_lo.py --project <slug> --phase ulos

# Phase B: Generate CIOs (with Marr Test)
python3 skills/learning-objective-generator/scripts/llm_generate_hierarchical_lo.py --project <slug> --phase cios

# Phase C: Generate SIOs
python3 skills/learning-objective-generator/scripts/llm_generate_hierarchical_lo.py --project <slug> --phase sios

# Merge: Merge ULO + CIO + SIO into the final TSV file
python3 skills/learning-objective-generator/scripts/llm_generate_hierarchical_lo.py --project <slug> --phase merge
```

### `llm_map_prerequisites.py` (Phase E — ADR-0005)

Generates `lo_prerequisites.tsv` and updates `concepts.tsv` (column `prerequisite_concept_codes`) following a 4-step process: Domain Partitioning → Concept DAG (LLM) → ULO Derivation → LLM Verify.

```bash
python3 skills/learning-objective-generator/scripts/llm_map_prerequisites.py --project <slug>
```

### Legacy 1-shot: `llm_extract_lo.py`

> ⚠️ Legacy/fast path. Generates all LOs in a single LLM call, without phase separation. Not recommended for new projects — use `llm_generate_hierarchical_lo.py` instead.

```bash

From orchable/knowledge-graph-public, MIT.

More from orchable/knowledge-graph-public